CPP20 introduced an rvalue overload to stringstream.str() . Without it, the string is copied out for the str() call every time. Now, we can steal the internal string buffer and return it directly through the rvalue overload. Mechanical find and replace assisted by codex.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/186552
Approved by: https://github.com/malfet
Summary:
Fix deadlock caused by https://github.com/pytorch/pytorch/pull/173568, which stashes a contextvars.Context as a SafePyObject in C++ thread_local storage during every backward() call. When a C++ thread exits, __call_tls_dtors destroys the thread_local and the SafePyObject destructor tries to acquire the GIL — deadlocking if the GIL holder is blocked on that thread (as in UP's GpuThreadGroup).
The old cleanup (_stash_obj_in_tls("context", None)) replaced the entry with a SafePyObject(None) but still left a SafePyObject in the map. This adds _remove_obj_from_tls which erases the key entirely, so no SafePyObject survives to thread exit.
Test Plan:
Green signals
test_correctness_fsdp_semi_frozen and test_correctness_one_pp_clip_mixed under UP tests passing locally
Reviewed By: malfet
Differential Revision: D103314406
Pull Request resolved: https://github.com/pytorch/pytorch/pull/182083
Approved by: https://github.com/rtimpe, https://github.com/malfet
C++20 provides a `contains` member function for all associative containers.
Prior to C++20, the common pattern to check if an element is a member of a
container was to either `container.find(key) != container.end()` or
`container.count(key) > 0`. This PR replaces those instances with `contains`.
All the changes were done by running clang-tidy with the
`readability-container-contains` check enabled.
PyTorch now supports C++20 #176662
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179436
Approved by: https://github.com/Skylion007
Adds `torch._tensor_iterator`: a pybind11 binding of `at::TensorIterator`
and `at::TensorIteratorConfig`. The Python entry point is
`TensorIterator(**kwargs)`, which takes operand lists and the same flag
knobs that drive `at::TensorIteratorConfig`, then exposes the post-build
shape, dtype, device, and per-operand strides for inspection. There is
no `for_each` -- this is build-pipeline only. Factory shortcuts mirror
the C++ named constructors (`binary_op`, `unary_op`, `comparison_op`,
`nullary_op`, `reduce_op`, `binary_float_op`, `unary_float_op`).
Equivalent of `at::TensorIteratorConfig` in Python:
```
it = TensorIterator(
outputs=[None],
const_inputs=[a, b],
promote_inputs_to_common_dtype=True,
cast_common_dtype_to_outputs=True,
enforce_safe_casting_to_output=True,
)
```
Operands take the owning `MaybeOwned<TensorBase>` path so storage
outlives the Python tensor handles. `outputs` accepts `None` placeholders
for outputs the iterator should allocate itself. The binding rejects
`is_reduction` with an undefined output (the C++ `reduce_op` named
constructor gates this, but manual flag composition would otherwise
bypass it).
A developer note at `docs/source/notes/tensor_iterator.md` documents
the C++ fluent -> Python kwargs mapping and the canonical-recipe caveats
(operand order is fixed at `outputs -> inputs -> const_inputs`; setters
are applied as final state, not as a sequence).
This is an alternative API design to the fluent-builder version; the
two stacks are intentionally being evaluated in parallel.
Authored by Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/184603
Approved by: https://github.com/ngimel, https://github.com/drisspg
Closes#55093
This changes `Tensor.__torch_function__` to skip only one level of
`__torch_function__` calls. This is achieved by adding a new state variable
`skip_next_torch_function` which is reset to false every time
`has_torch_function` is checked, which works both in C++ and Python.
In the issue we discussed making `DisableTorchFunction` public, but with this
you can just write `super().__torch_function__(func, types, args, kwargs)`
to achieve the intended effect whilst being clear that this is
intended for use in `__torch_function__` implementations only.
This revives https://github.com/pytorch/pytorch/pull/74728
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/177530
Approved by: https://github.com/albanD, https://github.com/anijain2305
Closes#55093
This changes `Tensor.__torch_function__` to skip only one level of
`__torch_function__` calls. This is achieved by adding a new state variable
`skip_next_torch_function` which is reset to false every time
`has_torch_function` is checked, which works both in C++ and Python.
In the issue we discussed making `DisableTorchFunction` public, but with this
you can just write `super().__torch_function__(func, types, args, kwargs)`
to achieve the intended effect whilst being clear that this is
intended for use in `__torch_function__` implementations only.
This revives https://github.com/pytorch/pytorch/pull/74728
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/177530
Approved by: https://github.com/albanD
Replace the raw PyObject* on Node with an atomic PyObjectSlot and use the same strategy that we already use for Tensor and Storage PyObject wrappers.
Use the same ownership pattern as THPCppFunction: THPFunction holds an owning intrusive_ptr<Node> and reads back via pyobj_slot(), with get_or_init managing the Python wrapper lifecycle. This replaces the previous scheme where PyNode owned THPFunction via a raw PyObject* and THPFunction held a weak_intrusive_ptr back.
Pass-by-value + move for intrusive_ptr parameters, remove an unnecessary temporary, and use try_emplace in NodeCalls::lookup.
Authored with Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/181390
Approved by: https://github.com/Skylion007
Expose dtype abbreviations (e.g., `torch.float32.abbr` → `"f32"`) as a
C++ property on `torch.dtype`, making them discoverable without reaching
into internal utilities. The abbreviation mapping is implemented in
`c10::getScalarTypeAbbr()` and `torch.utils._dtype_abbrs` is rewritten
to derive from the new property.
Also adds coverage for all Bits types (Bits2x4, Bits4x2, Bits8) and
uses a consistent `bNxM` naming scheme for Bits abbreviations to avoid
collision with Bool's `b8`. Adds tests and docs for the new property.
Co-authored-by: Claude <noreply@anthropic.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/177296
Approved by: https://github.com/albanD
### Summary
This PR replaces the hard-coded copy-on-write (COW) materialization logic in StorageImpl with a general-purpose, pluggable function pointer hook. Instead of StorageImpl knowing about COW internals directly (is_cow(), maybe_materialize_cow(), the friend declaration into `cow::materialize_cow_storage()`), it now exposes a single `MaterializeFn` slot (a plain `void(*)(StorageImpl&)`) that any backend can register to intercept write-path data pointer access.
COW becomes one consumer of this interface via the free function `c10::impl::cow::materialize_cow`. The existing COW semantics (lazy clone, refcounted shared data, copy-on-write materialization) are fully preserved; only the entry point changes.
### Motivation
Today, StorageImpl has COW-specific knowledge baked into its hot paths: it checks a deleter function pointer to detect COW storages, calls a friend free function to materialize them, and couples StorageImpl.h to COW.h COWDeleter.h via transitive includes. This makes it difficult for other backends to hook into the same materialization point without further special-casing StorageImpl.
The immediate benefit is COW cleanup, but the broader motivation is extensibility. Today, any backend that needs to intercept the moment a storage's data pointer is demanded for mutation, to commit deferred allocations, trigger device-to-host transfers, synchronize on async compute, or resolve placeholder buffers; would have to add another one-off check to StorageImpl's hot paths. A single pluggable hook at the write-path materialization point lets accelerator backends register that behavior with a plain function pointer, at zero additional fast-path cost. This is also the natural interception point for graph-based compilers operating in eager mode that need to materialize symbolic tensors into concrete buffers on first mutable access.
### Testing
All existing COW tests pass. New tests cover the materializer lifecycle, one-shot enforcement, and interaction with set_data_ptr_no_materialize.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179063
Approved by: https://github.com/ezyang, https://github.com/kurtamohler
## Summary
During CUDA graph capture, autograd nodes may hold a stale reference to a non-capturing stream from a prior warmup iteration. When the autograd engine encounters such a node, it issues `cudaStreamWaitEvent` on that stream, which pulls a non-capturing stream into the capture graph and invalidates it with an opaque `cudaErrorStreamCaptureInvalidated`.
[PR #166136](https://github.com/pytorch/pytorch/pull/166136) added a `UserWarning` in `input_buffer.cpp` that detects when AccumulateGrad's stream doesn't match the producer node's stream. This warning correctly identifies the problem, but does not prevent the resulting crash.
- **Default behavior (flag OFF)**: When the stale stream is the **default stream** (stream 0), raises a clear `RuntimeError` with the offending node name, stream ID, and actionable guidance — replacing the opaque CUDA error. Non-default stale streams are left alone (existing behavior).
- **Opt-in override (flag ON)**: `torch.autograd.graph.set_override_stale_capture_stream(True)` automatically redirects **any** stale non-capturing stream to the producer's capturing stream, allowing the capture to proceed.
## Reproducer
Minimal script — default-stream warmup followed by side-stream capture:
```python
import torch
model = torch.nn.Linear(32, 32).cuda()
x = torch.ones(4, 32, device="cuda")
# Warmup on default stream — AccumulateGrad nodes snapshot stream 0
loss = model(x).sum()
loss.backward()
model.zero_grad()
# Capture on a side stream
s = torch.cuda.Stream()
with torch.cuda.stream(s):
static_x = torch.ones(4, 32, device="cuda")
static_loss = model(static_x).sum()
static_loss.backward()
model.zero_grad()
torch.cuda.synchronize()
g = torch.cuda.CUDAGraph()
g.capture_begin()
static_loss = model(static_x).sum()
static_loss.backward() # crashes here
g.capture_end()
```
**Before this PR:** opaque `cudaErrorStreamCaptureInvalidated` or `cudaErrorStreamCaptureImplicit` crash.
**After this PR (flag OFF, default):**
```
RuntimeError: During CUDA graph capture, autograd node
'torch::autograd::AccumulateGrad' has a stale reference to the default
stream (stream 0) from warmup. This will invalidate the capture because
cudaStreamWaitEvent on the default stream pulls a non-capturing stream
into the graph.
To fix, either:
(a) Run warmup on a non-default stream, or
(b) Call torch.autograd.graph.set_override_stale_capture_stream(True)
to automatically redirect stale nodes to the capturing stream.
```
**After this PR (flag ON):** capture succeeds, correct gradients.
## Root Cause
Autograd nodes snapshot `current_stream` into `InputMetadata::stream_` at construction time.
https://github.com/pytorch/pytorch/blob/40fd3f7499057949fb4192a742e36ed624b7eec2/torch/csrc/autograd/input_metadata.cpp#L41
If a node (e.g., `AccumulateGrad`) is created during warmup when the default stream is current, it permanently references stream 0. During CUDA graph capture on a side stream, `evaluate_function` switches to that stale stream and calls `cudaStreamWaitEvent`, pulling stream 0 into the capture.
## Behavior by Scenario
| Scenario | Warmup stream | Capture stream | Before this PR | After this PR (flag OFF) | After this PR (flag ON) |
|----------|--------------|----------------|---------------|--------------------------|-------------------------|
| Default-stream warmup | default (0) | side stream (capturing) | Opaque `cudaErrorStreamCaptureInvalidated` crash | Clear `RuntimeError` with node name + fix instructions | Override stale stream |
| Side-stream warmup | side stream A | side stream B (capturing) | May work (depends on lazy init) | No change — non-default stale streams are not errored | **Override stale stream (see discussion)** |
### Discussion: Performance implications of the Scenario 2 override
When the flag is enabled and the stale stream is a non-default side stream (Scenario 2), the override places the node on the capturing stream. One concern is whether this could alter the captured CUDA graph topology or unintentionally serialize work that was meant to run in parallel across multiple streams.
We have not formally proven that the CUDA graph topology is preserved for all possible graph structures. However, in our current testing, we have not observed any topology changes. We would welcome feedback on whether a more conservative policy for Scenario 2 would be preferable.
## Detection Logic
In `engine.cpp` `evaluate_function`, at three sites (node processing, fan-out new buffer, fan-out existing buffer):
```
if node's stream is not capturing:
if any producer's ready_stream is capturing:
if override flag ON:
adopt producer's capturing stream + TORCH_WARN_ONCE
else if node's stream == default stream:
TORCH_CHECK(false, "actionable error message")
```
## Dependency
This PR depends on [PR #171443](https://github.com/pytorch/pytorch/pull/171443) (`c10::Stream::is_capturing()` by @guangyey), which introduces the device-generic `is_capturing()` method on `c10::Stream` and the corresponding `isStreamCapturing()` virtual on `DeviceGuardImplInterface`. The current patch cherry-picks that API; we will rebase onto #171443 once it merges.
## Reference
- **[Issue #164302](https://github.com/pytorch/pytorch/issues/164302)** — @ngimel identified the AccumulateGrad stream origin, @soulitzer pointed to the warning-only fix.
- **[PR #166136](https://github.com/pytorch/pytorch/pull/166136)** — Added the `UserWarning` in `input_buffer.cpp`. This PR builds on that — the warning identifies the problem, this PR adds a flag to override the stream it.
- **[CUDA graph docs](https://pytorch.org/docs/stable/notes/cuda.html#constraints)** — "Capture must occur on a non-default stream." The docs acknowledge the constraint but autograd violates it internally when nodes snapshot stream 0 during warmup.
- **[PR #171443](https://github.com/pytorch/pytorch/pull/171443)** — `c10::Stream::is_capturing()` API used by this PR.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/180090
Approved by: https://github.com/eqy, https://github.com/soulitzer, https://github.com/ngimel
### Summary
Replaces the `OpaqueGenerator` Python wrapper from #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.
C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen
Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.
With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.
Authored with Claude.
### Test plan
- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
Replace std::shared_ptr<Node> with c10::intrusive_ptr<Node> throughout the autograd system. Node now inherits from c10::intrusive_ptr_target instead of std::enable_shared_from_this<Node>.
**BC Breaking**:
This changes the signature of `Tensor::grad_fn`. Accesses to `Tensor.grad_fn()` should change from `std::shared_ptr<Node>` to `c10::intrusive_ptr<Node>`. Similarly, construction of a C++ autograd function should change from:
```c++
std::shared_ptr<CustomCppNode> node(new CustomCppNode(), torch::autograd::deleteNode);
```
to
```c++
auto node = c10::make_intrusive<CustomCppNode>();
```
Authored with Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/181139
Approved by: https://github.com/albanD
### Summary
Replaces the `OpaqueGenerator` Python wrapper from #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.
C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen
Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.
With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.
Authored with Claude.
### Test plan
- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
### Summary
Replaces the `OpaqueGenerator` Python wrapper from #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.
C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen
Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.
With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.
Authored with Claude.
### Test plan
- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
### Summary
Replaces the `OpaqueGenerator` Python wrapper from #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.
C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen
Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.
With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.
Authored with Claude.
### Test plan
- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
### Summary
Replaces the `OpaqueGenerator` Python wrapper from #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.
C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen
Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.
With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.
Authored with Claude.
### Test plan
- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
This change was separated from #170609, and adds a context flag to select the backend kernel for depthwise convolution. This allows easy benchmarking of cuDNN performance relative to fallback performance for heuristic tuning (and could also be used for debugging), while maintaining default behavior for standard use.
There are three options: `["auto", "cudnn", "native"]`. The `"auto"` option chooses a kernel based on the existing heuristic function. The `"cudnn"` and `"native"` flags skip the heuristic, instead always dispatching to a cuDNN kernel or a fallback kernel, respectively.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/176500
Approved by: https://github.com/malfet, https://github.com/eqy
This adds Python functions to get, set, and reset cuBLAS or cuBLASLt workspace sizes. Workspaces are updated lazily the next time they are accessed if the new workspace size is greater than the current size.
@ngimel, is this what you were hoping for? I believe the current BLAS backend can already be queried with `torch.backends.cuda.preferred_blas_library`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/177912
Approved by: https://github.com/ngimel, https://github.com/malfet, https://github.com/eqy
### Summary
Replaces the `OpaqueGenerator` Python wrapper from #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.
C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen
Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.
With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.
Authored with Claude.
### Test plan
- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
ghstack dependencies: #179657
TorchVitals was an undocumented feature gated behind the TORCH_VITAL
environment variable that allowed setting and reading key-value "vital signs" (e.g. whether CUDA or DataLoader was used). It was never documented and the test itself had a "FIXME: document or deprecate" comment.
Originally added in PR #51047 (Jan 2021), with a Python API in #53238 and a CUDA vital in #58059. There have been no feature changes since June 2021 — all subsequent commits were mechanical cleanups (clang-tidy,
thread-safe getenv, etc.).
Update: `torch.set_vital` was used from TorchData (see https://github.com/meta-pytorch/data/pull/1537 ) so will keep the dummy API around
Authored with Claude.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/178479
Approved by: https://github.com/msaroufim, https://github.com/albanD, https://github.com/mlazos
This PR adds [hipDNN](https://github.com/ROCm/rocm-libraries/tree/develop/projects/hipdnn) convolution support to PyTorch on ROCm, following the newer dispatch mechanism which no longer requires per-backend frontend op registration.
New ROCm kernel providers will be able to register plugins and ship faster kernels through hipDNN without touching PyTorch code. The graph-based API of hipDNN also supports fused kernels. The eventual goal is to replace all MIOpen-specific PyTorch code with hipDNN, wherein MIOpen will be just one of multiple first-class kernel providers.
# Note
This PR is being opened parallel to #177534 in anticipation that batch norm's dispatch migration issues might stall the review process for hipdnn as a whole.
# Summary
- Integrates the lightweight hipDNN frontend graph API as an alternative convolution backend to MIOpen on ROCm, covering forward, backward-data, and backward-weight for 2D/3D convolutions including transposed variants.
- Adds cmake `USE_HIPDNN` detection, `torch.backends.hipdnn` Python module with `enabled`/`flags()` context manager, and `compiledWithHipDNN()` runtime hook.
- Implements a generic thread-local `ParamsLRUCache` to amortize hipDNN `graph->build()` cost across repeated convolution calls with identical parameters.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/178515
Approved by: https://github.com/jeffdaily
Co-authored-by: Dmitry Nikolaev <dmitry.nikolaev@amd.com>
Summary:
CONTEXT: D97799113 was reverted by D98681402. This relands the original change.
Port GitHub PR https://github.com/pytorch/pytorch/issues/172246 to internal. This adds composable_kernel (CK) backend
support for scaled-dot-product-attention varlen path on ROCm:
- Fix philox seed/offset access in varlen backward to guard on is_dropout
- Fix softmax_lse shape and stride for CK varlen forward path
- Add `is_ck_sdpa_available()` API to `torch.backends.cuda`
- Update varlen_attn fake to use correct logsumexp shape per backend
- Add CK backend parametrization to varlen attention tests
Co-authored-by: Chinmay Kuchinad ChinmayDattanand.Kuchinad@amd.com
Co-authored-by: Jeff Daily jeff.daily@amd.com
Test Plan: buck test on relevant test targets
Differential Revision: D98693429
Pulled By: haoyuz
Pull Request resolved: https://github.com/pytorch/pytorch/pull/178729
Approved by: https://github.com/echen4096
The old implementation used a static std::unordered_set protected only by the GIL. Under free-threaded Python (3.14t) this is a data race.
Replace it with a Python dict stored in per-module state. The hot path (already-seen event) calls PyDict_Contains which is lock-free in 3.14t. The cold path (first occurrence) uses PyDict_SetDefaultRef which uses CPython's per-dict fine-grained locking to atomically check-and-insert, avoiding TOCTOU races between threads.
Authored with Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/178867
Approved by: https://github.com/Skylion007
TorchVitals was an undocumented feature gated behind the TORCH_VITAL
environment variable that allowed setting and reading key-value "vital
signs" (e.g. whether CUDA or DataLoader was used). It was never
documented and the test itself had a "FIXME: document or deprecate"
comment.
Originally added in PR #51047 (Jan 2021), with a Python API in #53238
and a CUDA vital in #58059. There have been no feature changes since
June 2021 — all subsequent commits were mechanical cleanups (clang-tidy,
thread-safe getenv, etc.).
A search on cs.github.com and general web search confirmed no external
projects depend on torch.set_vital, torch.read_vitals, or
torch.vitals_enabled — usage was limited to pytorch/pytorch itself.
Authored with Claude.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/178479
Approved by: https://github.com/msaroufim, https://github.com/albanD
## Summary
Enables variable-length (varlen) attention support for the Composable Kernel (CK) SDPA backend on ROCm.
## Changes
### Forward pass (`mha_varlen_fwd_ck.hip`)
- Fixed LSE tensor allocation: changed from 3D `{batch_size, num_heads, max_seqlen_q}` to 2D `{num_heads, total_q}` to match CK group mode expectation
- Fixed `nhead_stride_lse`: use `stride(0)` instead of `stride(1)` for 2D layout
- Fixed `batch_stride_lse`: set to `0` (no batch dimension in group mode LSE)
- Fixed `min_seqlen_q`: changed from `-1` to `1` (valid minimum for kernel dispatch)
### Backward pass (`mha_varlen_bwd_ck.hip`)
- Fixed philox seed/offset access: guarded with `if (is_dropout)` to avoid dtype mismatch when dropout is disabled
### Test infrastructure (`test/test_varlen_attention.py`)
- Added `sdpa_backend` parametrization to test both `aotriton` and `ck` backends on ROCm
- Backend selection: `["aotriton", "ck"]` when CK is available, `["aotriton"]` otherwise
- Uses `preferred_rocm_fa_library()` to switch backends per test
### Platform detection
- Added `PLATFORM_SUPPORTS_CK_SDPA` in `torch/testing/_internal/common_cuda.py`
- Added `_is_ck_sdpa_available()` in `torch/csrc/Module.cpp` for runtime CK availability check
### Fake tensor implementation (`torch/nn/attention/varlen.py`)
- Updated `_varlen_attn_fake` to use standard `[num_heads, total_q]` logsumexp format (matches aotriton/CUDA)
## Dependencies: CK submodule bump
- **CK PR**: (https://github.com/ROCm/rocm-libraries/pull/4292)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/172246
Approved by: https://github.com/jeffdaily