Files
pytorch/caffe2/core
Mark Richardson 8bce88d9de [caffe2] dont call cudnnDestroy on thread exit (crashes on windows with cuda 11/12) (#95382)
Summary:
My team has been hitting a mysterious crash for a few months on a windows binary that uses Caffe2 inside a worker thread.

When this thread gets destroyed, there is an error at this line in context_gpu.h where the state of this operation gives CUDNN_STATUS_INTERNAL_ERROR instead of CUDNN_STATUS_SUCCESS.

When enabling cudnn debug logs (via the env variables nvidia specifies), I can see that the context is destroyed twice, even though this code only destroys it once, so something mysterious is causing a double free.

This seems very very similar to the issue/fix described here for pytorch:
https://github.com/pytorch/pytorch/issues/17658
https://github.com/apache/tvm/pull/8267

And pytorch handles this in the same way, by just not calling cudnnDestroy

This seems to have become an issue with cuda11, but I tested cuda12 as well and found that the issue persists so this needs to be somehow fixed.

Test Plan:
CI

I checked that the specific windows binary I am using is able to create and drestroy caffe2-invoking threads without causing the application to crash.

buck run arvr/mode/win/cuda11/opt //arvr/projects/nimble/prod/tools/MonoHandTrackingVis

Differential Revision: D43538017

Pull Request resolved: https://github.com/pytorch/pytorch/pull/95382
Approved by: https://github.com/malfet
2023-03-10 06:42:51 +00:00
..
2022-10-28 04:56:40 +00:00
…
2022-09-08 01:49:55 +00:00
2021-06-14 13:46:11 -07:00
2021-12-10 04:26:23 -08:00
2021-12-10 04:26:23 -08:00
…
…
…
…
2021-08-11 14:07:10 -07:00
…
…
…
2021-12-10 04:26:23 -08:00
…
2021-10-07 08:06:09 -07:00
2021-10-07 08:06:09 -07:00