Commit Graph

777 Commits

Author SHA1 Message Date
Z Liu 4c5b5ab032 [ROCm] Enable hipFile (#191069)
This PR enables hipFile in PyTorch by adding mappings to hipify and updating cmake files, adds more tests for GDS, and also adds a helper variable and a function to facilitate tests' skip condition.
hipFile only ships with ROCm on ROCm 7.14 and later, therefore hipFile is only required after that version.

Authored with Claude.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/191069
Approved by: https://github.com/jeffdaily
2026-08-08 18:55:04 +00:00
Nikita Shulga aa7b0ba9cb Expose c10::utils::{get,set}_env to Python (#191015)
Adds torch._utils.getenv/setenv, thin wrappers over torch._C._getenv/_setenv bindings for c10::utils::[get|set]_env
These share c10's environment mutex, so reads and writes are serialized and stay consistent with other native code

Callers that add torch/nccl environment should prefer these over os.environ to avoid potential crash/race condition
Note that setenv updates at the C level and does not update Python's cached os.environ dict

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/191015
Approved by: https://github.com/cyyever, https://github.com/albanD
2026-07-27 16:27:41 +00:00
Kurt Mohler 01fb937d25 [MPS] Add MPS stream pool and Python bindings (#190375)
This PR adds a pool of 32 MPS streams that remain alive through the whole process, similar to CUDA's stream pool, to avoid the overhead of repeatedly creating and destroying streams.

Python bindings are added to obtain a stream from the pool, set the current stream for kernels to be executed on, and synchronize streams.

MPSAllocator's buffer reuse code is modified so that buffers can only be reused by the stream that initially allocated them, to avoid data races between streams. The CUDA allocator does this as well. If a buffer could be reused by a different stream, an expensive sync would be required to ensure that the previous stream has actually finished executing any kernels that used that buffer.

This PR does not add the python frontend APIs like `torch.mps.Stream`. That will be done in a follow-up PR.

Co-authored with Claude
Pull Request resolved: https://github.com/pytorch/pytorch/pull/190375
Approved by: https://github.com/malfet
2026-07-22 19:56:32 +00:00
Edward Z. Yang bcb1aa33b7 Add non-throwing query for whether Storage.data_ptr() would raise (#189268)
Some storages are set up so that accessing data_ptr() raises a
RuntimeError: FakeTensor and FunctionalTensor arm throw_on_mutable_data_ptr_
via _set_throw_on_mutable_data_ptr, and cudagraph-invalidated storages arm
throw_on_immutable_data_ptr_ via _set_storage_data_ptr_access_error_msg.
Previously the only way to detect this state was to call data_ptr() inside a
try/except RuntimeError, which is awkward and forces an exception through the
hot path.

Both flags are private on StorageImpl with no getters. This exposes them as
two orthogonal getters (throw_on_immutable_data_ptr / throw_on_mutable_data_ptr)
rather than a single combined predicate, so callers can distinguish the
immutable vs mutable throw reason if needed. They are bound into torch._C as
_storage_throw_on_immutable_data_ptr / _storage_throw_on_mutable_data_ptr
(modeled on _storage_Use_Count, taking storage._cdata), and torch.storage
grows a module-level _throws_on_data_ptr_access helper that ORs the two to
mirror exactly what Python's storage.data_ptr() checks. The getters return the
raw flags; the OR lives only in the Python helper.

Materializer hooks are intentionally excluded: they materialize rather than
throw, so data_ptr() would not raise.

Test Plan:

```
python test/test_torch.py -k test_storage_throws_on_data_ptr_access -v
```

The test asserts the helper agrees with a try/except RuntimeError baseline
across a normal storage, a FakeTensor with unsafe data_ptr access disallowed,
and a storage armed and then cleared via
_set_storage_data_ptr_access_error_msg. It is device-generic and passes on
both CPU and CUDA.

Authored-with: Claude Opus 4.8 (Claude Code)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/189268
Approved by: https://github.com/soulitzer
2026-07-19 01:47:18 +00:00
rraminen_amdeng 1a3b927f62 Enable hipsparselt version, initCusparseltBindings and related UTs (#178737)
This PR enables the following on ROCm >= 7.12

- `torch.backends.cusparselt.version()`
- Enable `initCusparseltBindings` to avoid `AttributeError: module 'torch._C' has no attribute '_cusparselt'`
- Enable the below 31 UTs
```
# hipSPARSELt allows INT8 sparse × INT8 dense matmul
TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape0_cuda_int8
TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape1_cuda_int8
TestSparseSemiStructuredCUDA::test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape2_cuda_int8
TestSparseSemiStructuredCUDA::test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape0_cuda_int8

# hipsparseLtDenseDescriptorInit() UTs are passing on ROCm
TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_compile_autotune_bfloat16_cuda
TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_compile_autotune_float16_cuda
TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_compile_autotune_int32_cuda
TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_mixed_dtype_bfloat16_cuda
TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_mixed_dtype_float16_cuda
TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_alpha_mixed_dtype_int32_cuda

# torch.backends.cusparselt.version() not implemented error is fixed
TestSparseSemiStructuredCUSPARSELTCUDA::test_cusparselt_backend_cuda

# Other enabled UTs
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_addmm_padded_output_shape_sparse_second_dense_input_shape0_cuda_bfloat16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_addmm_padded_output_shape_sparse_second_dense_input_shape0_cuda_float16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_addmm_padded_output_shape_sparse_second_dense_input_shape1_cuda_bfloat16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_addmm_padded_output_shape_sparse_second_dense_input_shape1_cuda_float16

test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_search_cuda_bfloat16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_search_cuda_float16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_mm_search_cuda_int8
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_cslt_sparse_tensor_with_alg_id_cuda
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_csrc_cslt_sparse_mm_search_cuda_bfloat16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_csrc_cslt_sparse_mm_search_cuda_float16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_csrc_cslt_sparse_mm_search_cuda_int8

test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_mm_padded_output_shape_dense_input_shape0_cuda_bfloat16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_mm_padded_output_shape_dense_input_shape0_cuda_float16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_mm_padded_output_shape_dense_input_shape1_cuda_bfloat16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_mm_padded_output_shape_dense_input_shape1_cuda_float16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_mm_padded_output_shape_sparse_second_dense_input_shape0_cuda_bfloat16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_mm_padded_output_shape_sparse_second_dense_input_shape0_cuda_float16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_mm_padded_output_shape_sparse_second_dense_input_shape1_cuda_bfloat16
test/test_sparse_semi_structured.py::TestSparseSemiStructuredCUSPARSELTCUDA::test_mm_padded_output_shape_sparse_second_dense_input_shape1_cuda_float16
```

Pull Request resolved: https://github.com/pytorch/pytorch/pull/178737
Approved by: https://github.com/jeffdaily, https://github.com/pragupta
2026-07-17 19:32:14 +00:00
Jason Ansel 2736ea01d7 Avoid native eager router graph breaks under Dynamo (#186388)
Fixes #186354

The native override eager router is installed as the real backend kernel for
aten overrides. Normally that path should evaluate eager-only predicates and
fall back through the captured boxed kernel, which avoids recursively
redispatching to the same override.

A torch.device context can make Dynamo inline DeviceContext.__torch_function__
and trace into the native bmm eager router. In that trace-time path, the router
was evaluating eager-only code such as torch._C._is_cow_tensor and the boxed
fallback, which produced graph breaks before Inductor could use the normal aten
path.

Use torch.compiler.is_dynamo_compiling() to detect the narrow case where Dynamo
is actively tracing the Python router and emit the resolved aten overload
instead. This is deliberately narrower than torch.compiler.is_compiling(): the
broader compile-session flag may be true while eager code is actually running,
and redispatching to aten in that case can recurse back into the same backend
router. Ordinary eager execution still uses the existing predicate and captured
boxed fallback path.

The alternative of using torch.compiler.is_compiling() was rejected because it
is true for the whole compile session and can recurse in real eager execution.
Changing the bmm predicate to avoid _is_cow_tensor would only fix one native
predicate and would leave boxed fallback graph breaks for other overrides.

Test Plan:
- python test/python_native/test_registry.py -k test_compile_session_flag_falls_through_without_recursion
- python test/dynamo/test_repros.py -k test_device_context_matmul_avoids_native_bmm_router_graph_break
- python test/python_native/test_registry.py
- python test/python_native/test_decomp.py
- lintrunner -a
- git diff --check

Benchmark Results:
- python CUDA bmm eager-router smoke benchmark, shape (64, 8, 1) x (64, 1, 128), 1000 iterations: 0.025610 s total, 25.610 us/call after the fix.

Generated by my agent

Pull Request resolved: https://github.com/pytorch/pytorch/pull/186388
Approved by: https://github.com/drisspg, https://github.com/slayton58
2026-07-16 05:34:59 +00:00
PyTorch MergeBot d540d310dd Revert "Avoid native eager router graph breaks under Dynamo (#186388)"
This reverts commit 3a6383d72a.

Reverted https://github.com/pytorch/pytorch/pull/186388 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable ([comment](https://github.com/pytorch/pytorch/pull/186388#issuecomment-4975032039))
2026-07-14 23:33:31 +00:00
Jason Ansel 3a6383d72a Avoid native eager router graph breaks under Dynamo (#186388)
Fixes #186354

The native override eager router is installed as the real backend kernel for
aten overrides. Normally that path should evaluate eager-only predicates and
fall back through the captured boxed kernel, which avoids recursively
redispatching to the same override.

A torch.device context can make Dynamo inline DeviceContext.__torch_function__
and trace into the native bmm eager router. In that trace-time path, the router
was evaluating eager-only code such as torch._C._is_cow_tensor and the boxed
fallback, which produced graph breaks before Inductor could use the normal aten
path.

Use torch.compiler.is_dynamo_compiling() to detect the narrow case where Dynamo
is actively tracing the Python router and emit the resolved aten overload
instead. This is deliberately narrower than torch.compiler.is_compiling(): the
broader compile-session flag may be true while eager code is actually running,
and redispatching to aten in that case can recurse back into the same backend
router. Ordinary eager execution still uses the existing predicate and captured
boxed fallback path.

The alternative of using torch.compiler.is_compiling() was rejected because it
is true for the whole compile session and can recurse in real eager execution.
Changing the bmm predicate to avoid _is_cow_tensor would only fix one native
predicate and would leave boxed fallback graph breaks for other overrides.

Test Plan:
- python test/python_native/test_registry.py -k test_compile_session_flag_falls_through_without_recursion
- python test/dynamo/test_repros.py -k test_device_context_matmul_avoids_native_bmm_router_graph_break
- python test/python_native/test_registry.py
- python test/python_native/test_decomp.py
- lintrunner -a
- git diff --check

Benchmark Results:
- python CUDA bmm eager-router smoke benchmark, shape (64, 8, 1) x (64, 1, 128), 1000 iterations: 0.025610 s total, 25.610 us/call after the fix.

Generated by my agent

Pull Request resolved: https://github.com/pytorch/pytorch/pull/186388
Approved by: https://github.com/drisspg, https://github.com/slayton58
2026-07-14 23:03:04 +00:00
Angel Li 9975f8bb38 swap isinstance(t, FakeTensor) to is_fake (#188978)
closed #188694 bc need to stack smth on top copying pr summary over...

in preparation for C++ FakeTensor migration, i am landing this change first to change isinstance(t, FakeTensor) to is_fake_tensor and adding a check for C++ FakeTensor to the is_fake_tensor function. but since C++ faketensor is not being used yet this PR should be a no-op

most loc should be converting isinstance(t, FakeTensor) to is_fake_tensor(t)
is_fake_tensor is defined in fake_tensor.py and is now the base case for the is_fake() recursive function
for attributes like fake_device, real_tensor, constant, C++ will access them differently via a pybind function to expose, so i made helper functions to wrap that logic. for now, C++ path returns None

also added a lint rule ISINSTANCE_FAKE_TENSOR to prevent future uses of isinstance(t, FakeTensor) unless you actually want to check for Python FakeTensor specifically (ie in fake_tensor.py) but this should be removed after C++ FakeTensor fully takes over

Pull Request resolved: https://github.com/pytorch/pytorch/pull/188978
Approved by: https://github.com/zou3519
2026-07-14 04:49:37 +00:00
PyTorch MergeBot fe2c46f8ba Revert "swap isinstance(t, FakeTensor) to is_fake (#188978)"
This reverts commit 49e1e7c651.

Reverted https://github.com/pytorch/pytorch/pull/188978 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable ([comment](https://github.com/pytorch/pytorch/pull/188978#issuecomment-4962918514))
2026-07-13 21:38:31 +00:00
Angel Li 49e1e7c651 swap isinstance(t, FakeTensor) to is_fake (#188978)
closed #188694 bc need to stack smth on top copying pr summary over...

in preparation for C++ FakeTensor migration, i am landing this change first to change isinstance(t, FakeTensor) to is_fake_tensor and adding a check for C++ FakeTensor to the is_fake_tensor function. but since C++ faketensor is not being used yet this PR should be a no-op

most loc should be converting isinstance(t, FakeTensor) to is_fake_tensor(t)
is_fake_tensor is defined in fake_tensor.py and is now the base case for the is_fake() recursive function
for attributes like fake_device, real_tensor, constant, C++ will access them differently via a pybind function to expose, so i made helper functions to wrap that logic. for now, C++ path returns None

also added a lint rule ISINSTANCE_FAKE_TENSOR to prevent future uses of isinstance(t, FakeTensor) unless you actually want to check for Python FakeTensor specifically (ie in fake_tensor.py) but this should be removed after C++ FakeTensor fully takes over

Pull Request resolved: https://github.com/pytorch/pytorch/pull/188978
Approved by: https://github.com/zou3519
2026-07-13 21:03:21 +00:00
Aaron Gokaslan 28a6c68c85 [BE][Ez]: Apply missing std::move calls via clang-tidy (#189583)
Missing std::move calls suggested by the new clang-tidy rule in nightly: [performance-use-std-move](https://clang.llvm.org/extra/clang-tidy/checks/performance/use-std-move.html) . Double checked with Claude and Codex . Only one false positive in the entire codebase. The check is very conservative by default.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/189583
Approved by: https://github.com/malfet
2026-07-12 00:50:51 +00:00
PyTorch MergeBot f79d8f22b1 Revert "swap isinstance(t, FakeTensor) to is_fake (#188978)"
This reverts commit b32b94e853.

Reverted https://github.com/pytorch/pytorch/pull/188978 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable ([comment](https://github.com/pytorch/pytorch/pull/188978#issuecomment-4937609151))
2026-07-10 16:59:18 +00:00
Angel Li b32b94e853 swap isinstance(t, FakeTensor) to is_fake (#188978)
closed #188694 bc need to stack smth on top copying pr summary over...

in preparation for C++ FakeTensor migration, i am landing this change first to change isinstance(t, FakeTensor) to is_fake_tensor and adding a check for C++ FakeTensor to the is_fake_tensor function. but since C++ faketensor is not being used yet this PR should be a no-op

most loc should be converting isinstance(t, FakeTensor) to is_fake_tensor(t)
is_fake_tensor is defined in fake_tensor.py and is now the base case for the is_fake() recursive function
for attributes like fake_device, real_tensor, constant, C++ will access them differently via a pybind function to expose, so i made helper functions to wrap that logic. for now, C++ path returns None

also added a lint rule ISINSTANCE_FAKE_TENSOR to prevent future uses of isinstance(t, FakeTensor) unless you actually want to check for Python FakeTensor specifically (ie in fake_tensor.py) but this should be removed after C++ FakeTensor fully takes over

Pull Request resolved: https://github.com/pytorch/pytorch/pull/188978
Approved by: https://github.com/zou3519
2026-07-10 16:29:25 +00:00
PyTorch MergeBot 4dd9de427f Revert "Add non-throwing query for whether Storage.data_ptr() would raise (#189268)"
This reverts commit 8c0048857b.

Reverted https://github.com/pytorch/pytorch/pull/189268 on behalf of https://github.com/atalman due to see internal diff: D111171964 for revert reason ([comment](https://github.com/pytorch/pytorch/pull/189268#issuecomment-4936271807))
2026-07-10 14:21:42 +00:00
Edward Z. Yang 8c0048857b Add non-throwing query for whether Storage.data_ptr() would raise (#189268)
Some storages are set up so that accessing data_ptr() raises a
RuntimeError: FakeTensor and FunctionalTensor arm throw_on_mutable_data_ptr_
via _set_throw_on_mutable_data_ptr, and cudagraph-invalidated storages arm
throw_on_immutable_data_ptr_ via _set_storage_data_ptr_access_error_msg.
Previously the only way to detect this state was to call data_ptr() inside a
try/except RuntimeError, which is awkward and forces an exception through the
hot path.

Both flags are private on StorageImpl with no getters. This exposes them as
two orthogonal getters (throw_on_immutable_data_ptr / throw_on_mutable_data_ptr)
rather than a single combined predicate, so callers can distinguish the
immutable vs mutable throw reason if needed. They are bound into torch._C as
_storage_throw_on_immutable_data_ptr / _storage_throw_on_mutable_data_ptr
(modeled on _storage_Use_Count, taking storage._cdata), and torch.storage
grows a module-level _throws_on_data_ptr_access helper that ORs the two to
mirror exactly what Python's storage.data_ptr() checks. The getters return the
raw flags; the OR lives only in the Python helper.

Materializer hooks are intentionally excluded: they materialize rather than
throw, so data_ptr() would not raise.

Test Plan:

```
python test/test_torch.py -k test_storage_throws_on_data_ptr_access -v
```

The test asserts the helper agrees with a try/except RuntimeError baseline
across a normal storage, a FakeTensor with unsafe data_ptr access disallowed,
and a storage armed and then cleared via
_set_storage_data_ptr_access_error_msg. It is device-generic and passes on
both CPU and CUDA.

Authored-with: Claude Opus 4.8 (Claude Code)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/189268
Approved by: https://github.com/soulitzer
2026-07-09 01:01:50 +00:00
Grayson Derossi 86f2fb2c1e Add cublasLt as a backend for grouped GEMM (bf16/fp16 support) (#177037)
This PR adds a cublasLt backend to grouped GEMM. The cublasLt backend is default for fp16 on Blackwell for CUDA 13.2+ and Hopper for CUDA 13.3+, and can also be enabled for bf16 on those device/version combos with `torch.backends.cuda.matmul.prefer_cublaslt_grouped_gemm = True`. The cublasLt grouped GEMM path is fully compatible with `torch.compile` and CUDA Graphs.

There are some constraints (see [docs](https://docs.nvidia.com/cuda/cublas/index.html#id106)) but these are mostly handed internally- the only one that users need to be aware of is that matrices and leading dimensions must be 16-byte aligned, so matrices with some dimensions not divisible by 16 may require padding + slicing.

Here's an initial [benchmarking results spreadsheet](https://docs.google.com/spreadsheets/d/1Av3RwwmLERLKlZ2826rPAduAribjF8KQLKTHxn5Wb2Y/edit?usp=sharing). Note the two tabs. For uniform groups, the cublasLt backend is faster than the fallback (fp16) but slower than the CUTLASS kernel (bf16) except for the largest size tested. For MOE-like cases where groups are not uniform, cublasLt appears to be on average faster across the board.
[Benchmarking script](https://gist.github.com/gderossi/515581a165b9b8eb031ba7a1a5a01c55)

Scaled grouped GEMM for FP8 types is also working with all currently supported scaling types (tensorwise, per-group tensorwise, MXFP8) at #177038, to be stacked on top of this.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/177037
Approved by: https://github.com/eqy, https://github.com/drisspg, https://github.com/ngimel
2026-07-08 18:14:34 +00:00
Angel Li fcf3f244af [C++ Fake] fake dispatch tables for decomps/op_impl/prim (#188699)
creating tables on C++ side for decomps/op_impls/prims so that C++ faketensor will have all necessary info on C++ side before deciding whether or not to cross into Python to run decomps/op_impls/etc

adding the additional registrations to `torch/_prims/__init__.py`, `torch/_decomp/__init__.py`, `torch/_subclasses/fake_impls.py`

rn there will be 2 tables (one in python one in c++) but can revisit in a followup pr if its worth removing the python one after migration is complete
Pull Request resolved: https://github.com/pytorch/pytorch/pull/188699
Approved by: https://github.com/ezyang
2026-07-07 22:23:19 +00:00
Simon Layton 6dc52141b1 [dlpack] Add read-only DLPack export and ReadOnlyTensorWrapper (#188554)
Adds read-only DLPack export, motivated by passing read-only tensors
into CuteDSL over the fast tvm-ffi exchange path without materializing
copy-on-write storage.

`ReadOnlyTensorWrapper` (`torch.utils.dlpack`) is the entry point: a
`Tensor` subclass that exports read-only and is export-only (every op
other than the DLPack protocol methods raises). It drives two surfaces:

- `__dlpack__` gains a `read_only` kwarg.
- a new const C exchange API (`torch._C._const_dlpack_exchange_api`)
  mirrors the fast exchange vtable but exports read-only, which is the
  path tvm-ffi / CuteDSL take.

Underneath, the read-only flag only exists on the versioned struct, so
`read_only` is threaded through `toDLPackImpl`/`fillVersion`:

- data is taken from `const_data_ptr()`, so exporting a COW tensor does
  not materialize it, and the flag is set.
- requesting it on the unversioned path is an error.

Test Plan:

```
python test/test_dlpack.py
python test/python_native/test_cutedsl_smoketest.py -k ReadOnly
```

The first runs the full DLPack suite, including new CPU coverage for the
export semantics: a COW tensor isn't materialized on read-only export
(while a writable control still is), read-only requires the versioned
protocol, the wrapper rejects non-DLPack ops, and a NumPy consumer ends
up with a non-writeable array. The second runs on a B200 and pushes a
read-only-wrapped COW tensor through a real CuteDSL kernel over the
tvm-ffi path, checking the result is correct and the tensor is still COW
afterward.

Authored with assistance from an AI coding assistant.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/188554
Approved by: https://github.com/ngimel
2026-07-01 18:31:26 +00:00
Matthias Jouanneaux ca983a7651 [CUDA] torch.cuda: move green contexts to cuda python bindings (#185527)
This PR implements the current `torch.cuda.GreenContext` interface using cuda-python bindings, to align with advanced usage of CUDA graphs, and similar features.

There are small changes in behavior:

1. cuda-bindings is now required for anything related to green contexts
2. GreenContext.set_context cannot be called twice in a row anymore (previously undefined behavior)
3. GreenContext.pop_context raises if set_context wasn't called before (previously undefined behavior)
4. custom streams are destroyed on green context deleter (previously leaked, but this was a minor concern)

Edit: additional note about using strings for API calls in python. Avoiding strings means that the green_contexts.py gets littered with pyrefly comments, which I thought was ugly. I don't think there is an easy way to avoid this, but if the comments are OK, I'm fine with avoiding strings and accessing the objects directly.
I'm also happy to submit a subsequent PR to update this in graphs.py as well to align things.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/185527
Approved by: https://github.com/eqy, https://github.com/ngimel
2026-06-23 19:09:09 +00:00
Jagadish Krishnamoorthy 793fe7f6ac [ROCm] Split CK SDPA vs CK GEMM arch gating (#187267)
Gate preferred_blas_library CK on ckGemmSupported (includes gfx90a) and keep SDPA/ROCm FA on ckSPDASupported. Update _is_ck_sdpa_available accordingly.

Skip test_ck_blas_library_mm bfloat16 on MI200 (gfx90a).

Pull Request resolved: https://github.com/pytorch/pytorch/pull/187267
Approved by: https://github.com/alugorey, https://github.com/jeffdaily

Co-authored-by: Jeff Daily <jeff.daily@amd.com>
2026-06-18 18:51:47 +00:00
Aaron Gokaslan 75eef9d885 [BE][Ez]: Use rvalue overload for stringstream str (#186552)
CPP20 introduced an rvalue overload to stringstream.str() . Without it, the string is copied out for the str() call every time. Now, we can steal the internal string buffer and return it directly through the rvalue overload. Mechanical find and replace assisted by codex.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/186552
Approved by: https://github.com/malfet
2026-06-08 00:09:08 +00:00
William Wen 4fc800d274 _remove_obj_from_tls to fix deadlock involving _stash_obj_in_tls (#182083) (#182083)
Summary:

Fix deadlock caused by https://github.com/pytorch/pytorch/pull/173568, which stashes a contextvars.Context as a SafePyObject in C++ thread_local storage during every backward() call. When a C++ thread exits, __call_tls_dtors destroys the thread_local and the SafePyObject destructor tries to acquire the GIL — deadlocking if the GIL holder is blocked on that thread (as in UP's GpuThreadGroup).

The old cleanup (_stash_obj_in_tls("context", None)) replaced the entry with a SafePyObject(None) but still left a SafePyObject in the map. This adds _remove_obj_from_tls which erases the key entirely, so no SafePyObject survives to thread exit.

Test Plan:
Green signals

test_correctness_fsdp_semi_frozen and test_correctness_one_pp_clip_mixed under UP tests passing locally

Reviewed By: malfet

Differential Revision: D103314406

Pull Request resolved: https://github.com/pytorch/pytorch/pull/182083
Approved by: https://github.com/rtimpe, https://github.com/malfet
2026-06-05 13:50:24 +00:00
Frederik Gossen 896205f4a2 Fix typo in SwizzleType enum docstring (#185582)
Fix "Blackwell-stype" → "Blackwell-style" in the SWIZZLE_32_4_4
Python binding docstring.

Authored with Claude (typo_terminator2).
Pull Request resolved: https://github.com/pytorch/pytorch/pull/185582
Approved by: https://github.com/zou3519, https://github.com/cyyever
2026-05-30 11:14:30 +00:00
PyTorch MergeBot 7d0f597644 Revert "Run clang-tidy with readability-container-contains (#179436)"
This reverts commit 7029dbea3c.

Reverted https://github.com/pytorch/pytorch/pull/179436 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable ([comment](https://github.com/pytorch/pytorch/pull/179436#issuecomment-4571115368))
2026-05-29 05:48:07 +00:00
Lakshay Garg 7029dbea3c Run clang-tidy with readability-container-contains (#179436)
C++20 provides a `contains` member function for all associative containers.
Prior to C++20, the common pattern to check if an element is a member of a
container was to either `container.find(key) != container.end()` or
`container.count(key) > 0`. This PR replaces those instances with `contains`.

All the changes were done by running clang-tidy with the
`readability-container-contains` check enabled.

PyTorch now supports C++20 #176662

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179436
Approved by: https://github.com/Skylion007
2026-05-29 04:55:28 +00:00
Artur Kłoniecki a6adf6cf41 Generalize CurrentWorkStream to get correct out_stream on XPU. (#179140)
Previously only CUDA path was supported correctly, and XPU fell on the same path as CPU, which is incorrect in this context.

This PR introduces XPU-specific path for this.

Fixes: https://github.com/intel/torch-xpu-ops/issues/3074
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179140
Approved by: https://github.com/guangyey, https://github.com/albanD

Co-authored-by: Yu, Guangye <guangye.yu@intel.com>
2026-05-25 01:58:10 +00:00
Simon Layton d86174d498 [TensorIterator] Expose the build pipeline to Python (kwargs API) (#184603)
Adds `torch._tensor_iterator`: a pybind11 binding of `at::TensorIterator`
and `at::TensorIteratorConfig`. The Python entry point is
`TensorIterator(**kwargs)`, which takes operand lists and the same flag
knobs that drive `at::TensorIteratorConfig`, then exposes the post-build
shape, dtype, device, and per-operand strides for inspection. There is
no `for_each` -- this is build-pipeline only. Factory shortcuts mirror
the C++ named constructors (`binary_op`, `unary_op`, `comparison_op`,
`nullary_op`, `reduce_op`, `binary_float_op`, `unary_float_op`).

Equivalent of `at::TensorIteratorConfig` in Python:

```
it = TensorIterator(
    outputs=[None],
    const_inputs=[a, b],
    promote_inputs_to_common_dtype=True,
    cast_common_dtype_to_outputs=True,
    enforce_safe_casting_to_output=True,
)
```

Operands take the owning `MaybeOwned<TensorBase>` path so storage
outlives the Python tensor handles. `outputs` accepts `None` placeholders
for outputs the iterator should allocate itself. The binding rejects
`is_reduction` with an undefined output (the C++ `reduce_op` named
constructor gates this, but manual flag composition would otherwise
bypass it).

A developer note at `docs/source/notes/tensor_iterator.md` documents
the C++ fluent -> Python kwargs mapping and the canonical-recipe caveats
(operand order is fixed at `outputs -> inputs -> const_inputs`; setters
are applied as final state, not as a sequence).

This is an alternative API design to the fluent-builder version; the
two stacks are intentionally being evaluated in parallel.

Authored by Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/184603
Approved by: https://github.com/ngimel, https://github.com/drisspg
2026-05-22 11:30:48 +00:00
cyy da550360fc Simplify Python wrappers (Py_NewRef/Py_IsNone) and fix ref leaks (#184174)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/184174
Approved by: https://github.com/jansel
2026-05-19 07:29:29 +00:00
Edward Z. Yang 4480f61c86 Skip only hop of __torch_function__ at a time (#177530)
Closes #55093

This changes `Tensor.__torch_function__` to skip only one level of
`__torch_function__` calls. This is achieved by adding a new state variable
`skip_next_torch_function` which is reset to false every time
`has_torch_function` is checked, which works both in C++ and Python.

In the issue we discussed making `DisableTorchFunction` public, but with this
you can just write `super().__torch_function__(func, types, args, kwargs)`
to achieve the intended effect whilst being clear that this is
intended for use in `__torch_function__` implementations only.

This revives https://github.com/pytorch/pytorch/pull/74728

Signed-off-by: Edward Z. Yang <ezyang@meta.com>

Pull Request resolved: https://github.com/pytorch/pytorch/pull/177530
Approved by: https://github.com/albanD, https://github.com/anijain2305
2026-05-12 03:13:24 +00:00
Yuanyuan Chen 496ea8261b Use Py_NewRef to collapse Py_INCREF followed by return (#180761)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/180761
Approved by: https://github.com/jansel
2026-05-09 09:08:44 +00:00
PyTorch MergeBot e3bbce4e05 Revert "Skip only hop of __torch_function__ at a time (#177530)"
This reverts commit fcffcab942.

Reverted https://github.com/pytorch/pytorch/pull/177530 on behalf of https://github.com/malfet due to This indeed broke all dynamo wrapped tests, both blocking and non-blocking ones, but hightlighting only latter https://hud.pytorch.org/hud/pytorch/pytorch/53b18aee8241dfbbfdb4ea100ab25e857e5a160f/1?per_page=50&name_filter=pull%20%2F%20linux-jammy-py3.10-clang18%20%2F%20test%20(dy&mergeEphemeralLF=true ([comment](https://github.com/pytorch/pytorch/pull/177530#issuecomment-4397611822))
2026-05-07 13:41:04 +00:00
Edward Z. Yang fcffcab942 Skip only hop of __torch_function__ at a time (#177530)
Closes #55093

This changes `Tensor.__torch_function__` to skip only one level of
`__torch_function__` calls. This is achieved by adding a new state variable
`skip_next_torch_function` which is reset to false every time
`has_torch_function` is checked, which works both in C++ and Python.

In the issue we discussed making `DisableTorchFunction` public, but with this
you can just write `super().__torch_function__(func, types, args, kwargs)`
to achieve the intended effect whilst being clear that this is
intended for use in `__torch_function__` implementations only.

This revives https://github.com/pytorch/pytorch/pull/74728

Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/177530
Approved by: https://github.com/albanD
2026-05-07 00:32:09 +00:00
Sam Gross 1d551526ec [autograd] Thread-safe Python wrapping for Node (#181390)
Replace the raw PyObject* on Node with an atomic PyObjectSlot and use the same strategy that we already use for Tensor and Storage PyObject wrappers.

Use the same ownership pattern as THPCppFunction: THPFunction holds an owning intrusive_ptr<Node> and reads back via pyobj_slot(), with get_or_init managing the Python wrapper lifecycle. This replaces the previous scheme where PyNode owned THPFunction via a raw PyObject* and THPFunction held a weak_intrusive_ptr back.

Pass-by-value + move for intrusive_ptr parameters, remove an unnecessary temporary, and use try_emplace in NodeCalls::lookup.

Authored with Claude.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/181390
Approved by: https://github.com/Skylion007
2026-05-01 22:12:05 +00:00
Sam Gross 968b088e49 [autograd] Convert shared_ptr<Node> to intrusive_ptr<Node> (v2) (#181782) (#181782)
Summary:

Reland "https://github.com/pytorch/pytorch/pull/181139" with a fix to CppNode<T>:

CppNode<T> needed to override `release_resources()` to reset the `AutogradContext`. WIthout it, the weak reference cycle from `CppNode<T>.ctx_.grad_fn_` was enough to keep the entire structure alive.

Reviewed By: albanD

bifferential Revision: D102653085

Pull Request resolved: https://github.com/pytorch/pytorch/pull/181782
Approved by: https://github.com/albanD
2026-04-30 16:26:35 +00:00
Edward Z. Yang 2c7cecb5b7 Add abbr property to torch.dtype (#177296)
Expose dtype abbreviations (e.g., `torch.float32.abbr` → `"f32"`) as a
C++ property on `torch.dtype`, making them discoverable without reaching
into internal utilities. The abbreviation mapping is implemented in
`c10::getScalarTypeAbbr()` and `torch.utils._dtype_abbrs` is rewritten
to derive from the new property.

Also adds coverage for all Bits types (Bits2x4, Bits4x2, Bits8) and
uses a consistent `bNxM` naming scheme for Bits abbreviations to avoid
collision with Bool's `b8`. Adds tests and docs for the new property.

Co-authored-by: Claude <noreply@anthropic.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/177296
Approved by: https://github.com/albanD
2026-04-30 02:57:04 +00:00
Rui Silva d776d9d659 Add pluggable MaterializeFn hook to StorageImpl (#179063)
### Summary

This PR replaces the hard-coded copy-on-write (COW) materialization logic in StorageImpl with a general-purpose, pluggable function pointer hook. Instead of StorageImpl knowing about COW internals directly (is_cow(), maybe_materialize_cow(), the friend declaration into `cow::materialize_cow_storage()`), it now exposes a single `MaterializeFn` slot (a plain `void(*)(StorageImpl&)`) that any backend can register to intercept write-path data pointer access.

COW becomes one consumer of this interface via the free function `c10::impl::cow::materialize_cow`. The existing COW semantics (lazy clone, refcounted shared data, copy-on-write materialization) are fully preserved; only the entry point changes.

### Motivation

Today, StorageImpl has COW-specific knowledge baked into its hot paths: it checks a deleter function pointer to detect COW storages, calls a friend free function to materialize them, and couples StorageImpl.h to COW.h COWDeleter.h via transitive includes. This makes it difficult for other backends to hook into the same materialization point without further special-casing StorageImpl.

The immediate benefit is COW cleanup, but the broader motivation is extensibility. Today, any backend that needs to intercept the moment a storage's data pointer is demanded for mutation, to commit deferred allocations, trigger device-to-host transfers, synchronize on async compute, or resolve placeholder buffers; would have to add another one-off check to StorageImpl's hot paths. A single pluggable hook at the write-path materialization point lets accelerator backends register that behavior with a plain function pointer, at zero additional fast-path cost. This is also the natural interception point for graph-based compilers operating in eager mode that need to materialize symbolic tensors into concrete buffers on first mutable access.

### Testing

All existing COW tests pass. New tests cover the materializer lifecycle, one-shot enforcement, and interaction with set_data_ptr_no_materialize.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179063
Approved by: https://github.com/ezyang, https://github.com/kurtamohler
2026-04-29 00:09:22 +00:00
Frank Lin 1b2c1d8689 Detect and fix stale stream references in autograd during CUDA graph capture (#180090)
## Summary

During CUDA graph capture, autograd nodes may hold a stale reference to a non-capturing stream from a prior warmup iteration. When the autograd engine encounters such a node, it issues `cudaStreamWaitEvent` on that stream, which pulls a non-capturing stream into the capture graph and invalidates it with an opaque `cudaErrorStreamCaptureInvalidated`.

[PR #166136](https://github.com/pytorch/pytorch/pull/166136) added a `UserWarning` in `input_buffer.cpp` that detects when AccumulateGrad's stream doesn't match the producer node's stream. This warning correctly identifies the problem, but does not prevent the resulting crash.

- **Default behavior (flag OFF)**: When the stale stream is the **default stream** (stream 0), raises a clear `RuntimeError` with the offending node name, stream ID, and actionable guidance — replacing the opaque CUDA error. Non-default stale streams are left alone (existing behavior).
- **Opt-in override (flag ON)**: `torch.autograd.graph.set_override_stale_capture_stream(True)` automatically redirects **any** stale non-capturing stream to the producer's capturing stream, allowing the capture to proceed.

## Reproducer

Minimal script — default-stream warmup followed by side-stream capture:

```python
import torch

model = torch.nn.Linear(32, 32).cuda()
x = torch.ones(4, 32, device="cuda")

# Warmup on default stream — AccumulateGrad nodes snapshot stream 0
loss = model(x).sum()
loss.backward()
model.zero_grad()

# Capture on a side stream
s = torch.cuda.Stream()
with torch.cuda.stream(s):
    static_x = torch.ones(4, 32, device="cuda")
    static_loss = model(static_x).sum()
    static_loss.backward()
    model.zero_grad()
    torch.cuda.synchronize()

    g = torch.cuda.CUDAGraph()
    g.capture_begin()
    static_loss = model(static_x).sum()
    static_loss.backward()  # crashes here
    g.capture_end()
```

**Before this PR:** opaque `cudaErrorStreamCaptureInvalidated` or `cudaErrorStreamCaptureImplicit` crash.

**After this PR (flag OFF, default):**
```
RuntimeError: During CUDA graph capture, autograd node
'torch::autograd::AccumulateGrad' has a stale reference to the default
stream (stream 0) from warmup. This will invalidate the capture because
cudaStreamWaitEvent on the default stream pulls a non-capturing stream
into the graph.

To fix, either:
  (a) Run warmup on a non-default stream, or
  (b) Call torch.autograd.graph.set_override_stale_capture_stream(True)
      to automatically redirect stale nodes to the capturing stream.
```

**After this PR (flag ON):** capture succeeds, correct gradients.

## Root Cause

Autograd nodes snapshot `current_stream` into `InputMetadata::stream_` at construction time.
https://github.com/pytorch/pytorch/blob/40fd3f7499057949fb4192a742e36ed624b7eec2/torch/csrc/autograd/input_metadata.cpp#L41

If a node (e.g., `AccumulateGrad`) is created during warmup when the default stream is current, it permanently references stream 0. During CUDA graph capture on a side stream, `evaluate_function` switches to that stale stream and calls `cudaStreamWaitEvent`, pulling stream 0 into the capture.

## Behavior by Scenario

| Scenario | Warmup stream | Capture stream | Before this PR | After this PR (flag OFF) | After this PR (flag ON) |
|----------|--------------|----------------|---------------|--------------------------|-------------------------|
| Default-stream warmup | default (0) | side stream (capturing) | Opaque `cudaErrorStreamCaptureInvalidated` crash | Clear `RuntimeError` with node name + fix instructions | Override stale stream |
| Side-stream warmup | side stream A | side stream B (capturing) | May work (depends on lazy init) | No change — non-default stale streams are not errored | **Override stale stream (see discussion)**  |

### Discussion: Performance implications of the Scenario 2 override

When the flag is enabled and the stale stream is a non-default side stream (Scenario 2), the override places the node on the capturing stream. One concern is whether this could alter the captured CUDA graph topology or unintentionally serialize work that was meant to run in parallel across multiple streams.

We have not formally proven that the CUDA graph topology is preserved for all possible graph structures. However, in our current testing, we have not observed any topology changes. We would welcome feedback on whether a more conservative policy for Scenario 2 would be preferable.

## Detection Logic

In `engine.cpp` `evaluate_function`, at three sites (node processing, fan-out new buffer, fan-out existing buffer):

```
if node's stream is not capturing:
    if any producer's ready_stream is capturing:
        if override flag ON:
            adopt producer's capturing stream + TORCH_WARN_ONCE
        else if node's stream == default stream:
            TORCH_CHECK(false, "actionable error message")
```

## Dependency

This PR depends on [PR #171443](https://github.com/pytorch/pytorch/pull/171443) (`c10::Stream::is_capturing()` by @guangyey), which introduces the device-generic `is_capturing()` method on `c10::Stream` and the corresponding `isStreamCapturing()` virtual on `DeviceGuardImplInterface`. The current patch cherry-picks that API; we will rebase onto #171443 once it merges.

## Reference

- **[Issue #164302](https://github.com/pytorch/pytorch/issues/164302)** — @ngimel identified the AccumulateGrad stream origin, @soulitzer pointed to the warning-only fix.
- **[PR #166136](https://github.com/pytorch/pytorch/pull/166136)** — Added the `UserWarning` in `input_buffer.cpp`. This PR builds on that — the warning identifies the problem, this PR adds a flag to override the stream it.
- **[CUDA graph docs](https://pytorch.org/docs/stable/notes/cuda.html#constraints)** — "Capture must occur on a non-default stream." The docs acknowledge the constraint but autograd violates it internally when nodes snapshot stream 0 during warmup.
- **[PR #171443](https://github.com/pytorch/pytorch/pull/171443)** — `c10::Stream::is_capturing()` API used by this PR.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/180090
Approved by: https://github.com/eqy, https://github.com/soulitzer, https://github.com/ngimel
2026-04-28 02:12:00 +00:00
Sam Gross 5910df01a5 Revert "[autograd] Convert shared_ptr<Node> to intrusive_ptr<Node>" (#181432)
Summary: The PR may cause increased memory usage.

Test Plan: contbuild & CI

Differential Revision: D102382560

Pull Request resolved: https://github.com/pytorch/pytorch/pull/181432
Approved by: https://github.com/atalman
2026-04-25 14:59:21 +00:00
Shangdi Yu 441bf01d46 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-24 18:13:34 +00:00
PyTorch MergeBot 07841b8fe8 Revert "Make Generator an Opaque Object (#179661)"
This reverts commit 7e0ab7a126.

Reverted https://github.com/pytorch/pytorch/pull/179661 on behalf of https://github.com/atalman due to sorry need to revert again, not sure why windows builds where not executed ([comment](https://github.com/pytorch/pytorch/pull/179661#issuecomment-4314245101))
2026-04-24 15:18:51 +00:00
Sam Gross 2d92e585fd [autograd] Convert shared_ptr<Node> to intrusive_ptr<Node> (#181139)
Replace std::shared_ptr<Node> with c10::intrusive_ptr<Node> throughout the autograd system. Node now inherits from c10::intrusive_ptr_target instead of std::enable_shared_from_this<Node>.

**BC Breaking**:

This changes the signature of `Tensor::grad_fn`.  Accesses to `Tensor.grad_fn()` should change from `std::shared_ptr<Node>` to `c10::intrusive_ptr<Node>`. Similarly, construction of a C++ autograd function should change from:

```c++
std::shared_ptr<CustomCppNode> node(new CustomCppNode(), torch::autograd::deleteNode);
```

to

```c++
auto node = c10::make_intrusive<CustomCppNode>();
```

Authored with Claude.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/181139
Approved by: https://github.com/albanD
2026-04-23 22:01:46 +00:00
Shangdi Yu 7e0ab7a126 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-23 17:15:28 +00:00
Zhijing Li 4918ae2275 [inductor] Add _FastCudaLauncher: vectorcall C extension for pre-bound kernel launch (#180507) (#180507)
Summary:

## Motivation

D100887242 added a fast path to CachingAutotuner.run() that bypasses the
Python preamble (~1.4µs savings). However, the actual kernel launch still goes
through StaticallyLaunchedCudaKernel.run() → _StaticCudaLauncher._launch_kernel(),
which carries ~2.2µs of overhead above bare cuLaunchKernel:

  - PyArg_ParseTuple("KiiiiisOK") parsing 9 args per call      ~0.3µs
  - cuCtxGetCurrent validation                                  ~0.1µs
  - Python data_ptr() + cuPointerGetAttribute per tensor arg    ~0.3µs × N
  - parseKernelArgs type-switching loop                         ~0.2µs

All of these are redundant on steady-state launches: kernel metadata is fixed,
CUDA context is guaranteed valid, and tensor pointers are always valid device
pointers.

## Approach

Add `_FastCudaLauncher`, a C Python type that pre-binds kernel metadata at
construction time and uses PEP 590 vectorcall for the fastest Python→C
dispatch:

1. **Construction** (once per kernel): pre-binds CUfunction, block dims,
   shared mem, arg types, scratch slots. Pre-computes
   `kernelArgs[i] = &argStorage[i]`.

2. **__call__** (every launch, via vectorcall — no tuple allocation):
   - Extracts grid + stream directly from the vectorcall args array
   - For tensor args: `THPVariable_Unpack(obj).data_ptr()` — zero-cost
     reinterpret_cast to C++ `at::Tensor`, no Python method call, no
     cuPointerGetAttribute validation
   - For scalar args: same `convertType` as existing path
   - Calls `cuLaunchKernel` directly — no cuCtxGetCurrent check

Integrated into CachingAutotuner: after D100887242 selects the fast launcher,
`_build_fast_launcher()` upgrades it to use `_FastCudaLauncher` by cloning
the launcher function with the runner replaced. Falls back silently to the
original launcher if conditions aren't met (non-static path, missing kernel
info, import failure, etc.).

Gated by `torch._inductor.config.use_fast_triton_launcher` (default ON),
controllable via `TORCHINDUCTOR_USE_FAST_TRITON_LAUNCHER=0`.

## Results

### Microbenchmark (H100, mode/opt, tritonbench launch_latency)

Per-kernel overhead (`nop_inductor_per_kernel`):
- Before (D100887242): 5.64µs (0-arg), 7.05µs (19-arg)
- After:               3.12µs (0-arg), 3.28µs (19-arg)
- Δ:                  -2.52µs (-45%), -3.77µs (-53%)

Raw cuLaunchKernel baseline: 2.09µs (0-arg)
Per-kernel gap to raw CUDA: 3.12µs - 2.09µs = 1.03µs (only ~1µs above driver cost)

### Microbenchmark (GB200, mode/opt, tritonbench launch_latency)

Per-kernel overhead (`nop_inductor_per_kernel`):
- Before (D100881626): 5.21µs (0-arg), 6.93µs (19-arg)
- After:               3.07µs (0-arg), 3.06µs (19-arg)
- Δ:                  -2.14µs (-41%), -3.87µs (-56%)

Raw cuLaunchKernel baseline: 1.32µs (0-arg)
Per-kernel gap to raw CUDA: 3.07µs - 1.32µs = 1.75µs

### E2E model benchmark (H100, torchbench, inductor, no cudagraphs, inference, bfloat16)

Combined effect of D100887242 + this diff vs baseline (D100881626):

| Model        | Calls | Baseline | After  | Δ latency           |
|--------------|------|----------|--------|---------------------|
| hf_Bert      |   285 | 2.513ms  | 1.907ms| -0.606ms (-24.1%)   |
| resnet18     |    69 | 0.894ms  | 0.601ms| -0.293ms (-32.8%)   |
| resnet50     |   175 | 2.242ms  | 2.227ms| ~0% (compute-bound) |
| mobilenet_v2 |   153 | 2.537ms  | 1.147ms| -1.390ms (-54.8%)   |

### E2E model benchmark (GB200, torchbench, inductor, no cudagraphs, inference, bfloat16)

Combined effect of D100887242 + this diff vs baseline (D100881626):

| Model        | Calls | Baseline | After   | Δ latency           |
|--------------|------|----------|---------|---------------------|
| hf_Bert      |   285 | 5.205ms  | 4.371ms | -0.834ms (-16.0%)   |
| resnet18     |    69 | 4.668ms  | 4.667ms | ~0% (noise)         |
| resnet50     |   175 | 8.734ms  | 8.690ms | ~0% (compute-bound) |
| mobilenet_v2 |   153 | 3.929ms  | 2.777ms | -1.152ms (-29.3%)   |

Models with many small kernels (mobilenet_v2, hf_Bert) see the
largest improvements. Compute-bound models (resnet50) are unaffected — the
optimization has zero cost when kernel execution dominates.

With cudagraphs ON, the optimization has ~zero effect (expected — cudagraphs
bypass the Python dispatch path):

| Model (GB200, cudagraphs ON) | Baseline | After   | Δ        |
|------------------------------|----------|---------|----------|
| hf_Bert                      | 1.407ms  | 1.418ms | ~0%      |
| resnet50                     | 8.414ms  | 8.467ms | ~0%      |
| mobilenet_v2                 | 0.892ms  | 0.875ms | ~0%      |

### Inductor Perf Nightly

H100: https://fburl.com/xa99g949
A100: https://fburl.com/491o10qu
(default & inductor deterministic mode are correct mode to check: disable cudagraph, no AOTI)
A100 shows less perf improve might because each kernel has longer execution time, so relative improvement is less.

Test Plan:
## Tritonbench (H100)

```
buck2 run fbcode//mode/opt fbcode//pytorch/tritonbench:run_lite -- \
  --op launch_latency \
  --only nop_inductor_kernel,nop_inductor_per_kernel,nop_triton_compiled_kernel_run,nop_triton_direct_culaunch \
  --metrics walltime --simple-output
```

```
  x_val  nop_inductor_kernel  nop_inductor_per_kernel  nop_triton_compiled_kernel_run  nop_triton_direct_culaunch
      0            0.0248156              0.00312076                     0.00358178                   0.00208895
     19            0.0259619              0.00328259                     0.00501875                   0.00352997
```

## Tritonbench (GB200)

```
buck2 run fbcode//mode/opt \
  -c fbcode.enable_gpu_sections=true \
  -c fbcode.platform010_cuda_version=12.8 \
  -c fbcode.nvcc_arch=b200a \
  fbcode//pytorch/tritonbench:run_lite -- \
  --op launch_latency \
  --only nop_inductor_per_kernel,nop_triton_compiled_kernel_run,nop_triton_direct_culaunch \
  --metrics walltime --simple-output
```

```
  x_val  nop_inductor_per_kernel  nop_triton_compiled_kernel_run  nop_triton_direct_culaunch
      0              0.00307118                      0.0025503                   0.00132063
     19              0.00306199                      0.00360913                  0.00229927
```

## E2E torchbench A/B (no cudagraphs)

```
# Fast ON (default):
buck2 run fbcode//mode/opt fbcode//caffe2/benchmarks/dynamo:torchbench -- \
  --performance --inference --bfloat16 --inductor --disable-cudagraphs \
  --device cuda --only mobilenet_v2

# Fast OFF (fallback):
TORCHINDUCTOR_USE_FAST_TRITON_LAUNCHER=0 buck2 run fbcode//mode/opt \
  fbcode//caffe2/benchmarks/dynamo:torchbench -- \
  --performance --inference --bfloat16 --inductor --disable-cudagraphs \
  --device cuda --only mobilenet_v2
```

## Build

```
buck2 build fbcode//mode/opt fbcode//caffe2:torch_python_cuda_without_torch
```

## Correctness

```
buck2 test fbcode//mode/opt fbcode//caffe2/test:test_torchinductor
```

## Unit test

```
buck2 test fbcode//mode/opt fbcode//caffe2/test/inductor:static_triton_launcher -- 'TestFastCudaLauncherCompileResult'
```

Reviewed By: PaulZhang12

Differential Revision: D101027739

Pull Request resolved: https://github.com/pytorch/pytorch/pull/180507
Approved by: https://github.com/PaulZhang12
2026-04-23 05:26:48 +00:00
PyTorch MergeBot 1aa8e27453 Revert "Make Generator an Opaque Object (#179661)"
This reverts commit 4aef62c058.

Reverted https://github.com/pytorch/pytorch/pull/179661 on behalf of https://github.com/yushangdi due to breaks internal buck dependency ([comment](https://github.com/pytorch/pytorch/pull/179661#issuecomment-4300265707))
2026-04-22 22:10:14 +00:00
PyTorch MergeBot 9372b02db2 Revert "[autograd] Convert shared_ptr<Node> to intrusive_ptr<Node> (#179766)"
This reverts commit 0309973bc7.

Reverted https://github.com/pytorch/pytorch/pull/179766 on behalf of https://github.com/ZainRizvi due to Sorry but this is breaking internally. See diff D101855555 for details. To validate your fixes internally, you can follow the instructions here: https://fburl.com/fixing-ghfirst-reverts ([comment](https://github.com/pytorch/pytorch/pull/179766#issuecomment-4297869022))
2026-04-22 16:06:55 +00:00
Shangdi Yu 4aef62c058 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-21 16:21:24 +00:00
Sam Gross 0309973bc7 [autograd] Convert shared_ptr<Node> to intrusive_ptr<Node> (#179766)
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.14.0) (oldest at bottom):

* #179767
* -> #179766
* #179765
* #179764

Replace std::shared_ptr<Node> with c10::intrusive_ptr<Node> throughout
the autograd system. Node now inherits from c10::intrusive_ptr_target
instead of std::enable_shared_from_this<Node>.

Authored with Claude.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179766
Approved by: https://github.com/albanD
ghstack dependencies: #179764, #179765
2026-04-21 03:13:54 +00:00
PyTorch MergeBot cb242afe1b Revert "Make Generator an Opaque Object (#179661)"
This reverts commit 18357d9483.

Reverted https://github.com/pytorch/pytorch/pull/179661 on behalf of https://github.com/yushangdi due to breaks pr_time_benchmarks ([comment](https://github.com/pytorch/pytorch/pull/179661#issuecomment-4285042651))
2026-04-21 00:09:01 +00:00
Shangdi Yu 18357d9483 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-20 19:55:21 +00:00