Commit Graph

746 Commits

Author SHA1 Message Date
PyTorch MergeBot e3bbce4e05 Revert "Skip only hop of __torch_function__ at a time (#177530)"
This reverts commit fcffcab942.

Reverted https://github.com/pytorch/pytorch/pull/177530 on behalf of https://github.com/malfet due to This indeed broke all dynamo wrapped tests, both blocking and non-blocking ones, but hightlighting only latter https://hud.pytorch.org/hud/pytorch/pytorch/53b18aee8241dfbbfdb4ea100ab25e857e5a160f/1?per_page=50&name_filter=pull%20%2F%20linux-jammy-py3.10-clang18%20%2F%20test%20(dy&mergeEphemeralLF=true ([comment](https://github.com/pytorch/pytorch/pull/177530#issuecomment-4397611822))
2026-05-07 13:41:04 +00:00
Edward Z. Yang fcffcab942 Skip only hop of __torch_function__ at a time (#177530)
Closes #55093

This changes `Tensor.__torch_function__` to skip only one level of
`__torch_function__` calls. This is achieved by adding a new state variable
`skip_next_torch_function` which is reset to false every time
`has_torch_function` is checked, which works both in C++ and Python.

In the issue we discussed making `DisableTorchFunction` public, but with this
you can just write `super().__torch_function__(func, types, args, kwargs)`
to achieve the intended effect whilst being clear that this is
intended for use in `__torch_function__` implementations only.

This revives https://github.com/pytorch/pytorch/pull/74728

Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/177530
Approved by: https://github.com/albanD
2026-05-07 00:32:09 +00:00
Sam Gross 1d551526ec [autograd] Thread-safe Python wrapping for Node (#181390)
Replace the raw PyObject* on Node with an atomic PyObjectSlot and use the same strategy that we already use for Tensor and Storage PyObject wrappers.

Use the same ownership pattern as THPCppFunction: THPFunction holds an owning intrusive_ptr<Node> and reads back via pyobj_slot(), with get_or_init managing the Python wrapper lifecycle. This replaces the previous scheme where PyNode owned THPFunction via a raw PyObject* and THPFunction held a weak_intrusive_ptr back.

Pass-by-value + move for intrusive_ptr parameters, remove an unnecessary temporary, and use try_emplace in NodeCalls::lookup.

Authored with Claude.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/181390
Approved by: https://github.com/Skylion007
2026-05-01 22:12:05 +00:00
Sam Gross 968b088e49 [autograd] Convert shared_ptr<Node> to intrusive_ptr<Node> (v2) (#181782) (#181782)
Summary:

Reland "https://github.com/pytorch/pytorch/pull/181139" with a fix to CppNode<T>:

CppNode<T> needed to override `release_resources()` to reset the `AutogradContext`. WIthout it, the weak reference cycle from `CppNode<T>.ctx_.grad_fn_` was enough to keep the entire structure alive.

Reviewed By: albanD

bifferential Revision: D102653085

Pull Request resolved: https://github.com/pytorch/pytorch/pull/181782
Approved by: https://github.com/albanD
2026-04-30 16:26:35 +00:00
Edward Z. Yang 2c7cecb5b7 Add abbr property to torch.dtype (#177296)
Expose dtype abbreviations (e.g., `torch.float32.abbr` → `"f32"`) as a
C++ property on `torch.dtype`, making them discoverable without reaching
into internal utilities. The abbreviation mapping is implemented in
`c10::getScalarTypeAbbr()` and `torch.utils._dtype_abbrs` is rewritten
to derive from the new property.

Also adds coverage for all Bits types (Bits2x4, Bits4x2, Bits8) and
uses a consistent `bNxM` naming scheme for Bits abbreviations to avoid
collision with Bool's `b8`. Adds tests and docs for the new property.

Co-authored-by: Claude <noreply@anthropic.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/177296
Approved by: https://github.com/albanD
2026-04-30 02:57:04 +00:00
Rui Silva d776d9d659 Add pluggable MaterializeFn hook to StorageImpl (#179063)
### Summary

This PR replaces the hard-coded copy-on-write (COW) materialization logic in StorageImpl with a general-purpose, pluggable function pointer hook. Instead of StorageImpl knowing about COW internals directly (is_cow(), maybe_materialize_cow(), the friend declaration into `cow::materialize_cow_storage()`), it now exposes a single `MaterializeFn` slot (a plain `void(*)(StorageImpl&)`) that any backend can register to intercept write-path data pointer access.

COW becomes one consumer of this interface via the free function `c10::impl::cow::materialize_cow`. The existing COW semantics (lazy clone, refcounted shared data, copy-on-write materialization) are fully preserved; only the entry point changes.

### Motivation

Today, StorageImpl has COW-specific knowledge baked into its hot paths: it checks a deleter function pointer to detect COW storages, calls a friend free function to materialize them, and couples StorageImpl.h to COW.h COWDeleter.h via transitive includes. This makes it difficult for other backends to hook into the same materialization point without further special-casing StorageImpl.

The immediate benefit is COW cleanup, but the broader motivation is extensibility. Today, any backend that needs to intercept the moment a storage's data pointer is demanded for mutation, to commit deferred allocations, trigger device-to-host transfers, synchronize on async compute, or resolve placeholder buffers; would have to add another one-off check to StorageImpl's hot paths. A single pluggable hook at the write-path materialization point lets accelerator backends register that behavior with a plain function pointer, at zero additional fast-path cost. This is also the natural interception point for graph-based compilers operating in eager mode that need to materialize symbolic tensors into concrete buffers on first mutable access.

### Testing

All existing COW tests pass. New tests cover the materializer lifecycle, one-shot enforcement, and interaction with set_data_ptr_no_materialize.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/179063
Approved by: https://github.com/ezyang, https://github.com/kurtamohler
2026-04-29 00:09:22 +00:00
Frank Lin 1b2c1d8689 Detect and fix stale stream references in autograd during CUDA graph capture (#180090)
## Summary

During CUDA graph capture, autograd nodes may hold a stale reference to a non-capturing stream from a prior warmup iteration. When the autograd engine encounters such a node, it issues `cudaStreamWaitEvent` on that stream, which pulls a non-capturing stream into the capture graph and invalidates it with an opaque `cudaErrorStreamCaptureInvalidated`.

[PR #166136](https://github.com/pytorch/pytorch/pull/166136) added a `UserWarning` in `input_buffer.cpp` that detects when AccumulateGrad's stream doesn't match the producer node's stream. This warning correctly identifies the problem, but does not prevent the resulting crash.

- **Default behavior (flag OFF)**: When the stale stream is the **default stream** (stream 0), raises a clear `RuntimeError` with the offending node name, stream ID, and actionable guidance — replacing the opaque CUDA error. Non-default stale streams are left alone (existing behavior).
- **Opt-in override (flag ON)**: `torch.autograd.graph.set_override_stale_capture_stream(True)` automatically redirects **any** stale non-capturing stream to the producer's capturing stream, allowing the capture to proceed.

## Reproducer

Minimal script — default-stream warmup followed by side-stream capture:

```python
import torch

model = torch.nn.Linear(32, 32).cuda()
x = torch.ones(4, 32, device="cuda")

# Warmup on default stream — AccumulateGrad nodes snapshot stream 0
loss = model(x).sum()
loss.backward()
model.zero_grad()

# Capture on a side stream
s = torch.cuda.Stream()
with torch.cuda.stream(s):
    static_x = torch.ones(4, 32, device="cuda")
    static_loss = model(static_x).sum()
    static_loss.backward()
    model.zero_grad()
    torch.cuda.synchronize()

    g = torch.cuda.CUDAGraph()
    g.capture_begin()
    static_loss = model(static_x).sum()
    static_loss.backward()  # crashes here
    g.capture_end()
```

**Before this PR:** opaque `cudaErrorStreamCaptureInvalidated` or `cudaErrorStreamCaptureImplicit` crash.

**After this PR (flag OFF, default):**
```
RuntimeError: During CUDA graph capture, autograd node
'torch::autograd::AccumulateGrad' has a stale reference to the default
stream (stream 0) from warmup. This will invalidate the capture because
cudaStreamWaitEvent on the default stream pulls a non-capturing stream
into the graph.

To fix, either:
  (a) Run warmup on a non-default stream, or
  (b) Call torch.autograd.graph.set_override_stale_capture_stream(True)
      to automatically redirect stale nodes to the capturing stream.
```

**After this PR (flag ON):** capture succeeds, correct gradients.

## Root Cause

Autograd nodes snapshot `current_stream` into `InputMetadata::stream_` at construction time.
https://github.com/pytorch/pytorch/blob/40fd3f7499057949fb4192a742e36ed624b7eec2/torch/csrc/autograd/input_metadata.cpp#L41

If a node (e.g., `AccumulateGrad`) is created during warmup when the default stream is current, it permanently references stream 0. During CUDA graph capture on a side stream, `evaluate_function` switches to that stale stream and calls `cudaStreamWaitEvent`, pulling stream 0 into the capture.

## Behavior by Scenario

| Scenario | Warmup stream | Capture stream | Before this PR | After this PR (flag OFF) | After this PR (flag ON) |
|----------|--------------|----------------|---------------|--------------------------|-------------------------|
| Default-stream warmup | default (0) | side stream (capturing) | Opaque `cudaErrorStreamCaptureInvalidated` crash | Clear `RuntimeError` with node name + fix instructions | Override stale stream |
| Side-stream warmup | side stream A | side stream B (capturing) | May work (depends on lazy init) | No change — non-default stale streams are not errored | **Override stale stream (see discussion)**  |

### Discussion: Performance implications of the Scenario 2 override

When the flag is enabled and the stale stream is a non-default side stream (Scenario 2), the override places the node on the capturing stream. One concern is whether this could alter the captured CUDA graph topology or unintentionally serialize work that was meant to run in parallel across multiple streams.

We have not formally proven that the CUDA graph topology is preserved for all possible graph structures. However, in our current testing, we have not observed any topology changes. We would welcome feedback on whether a more conservative policy for Scenario 2 would be preferable.

## Detection Logic

In `engine.cpp` `evaluate_function`, at three sites (node processing, fan-out new buffer, fan-out existing buffer):

```
if node's stream is not capturing:
    if any producer's ready_stream is capturing:
        if override flag ON:
            adopt producer's capturing stream + TORCH_WARN_ONCE
        else if node's stream == default stream:
            TORCH_CHECK(false, "actionable error message")
```

## Dependency

This PR depends on [PR #171443](https://github.com/pytorch/pytorch/pull/171443) (`c10::Stream::is_capturing()` by @guangyey), which introduces the device-generic `is_capturing()` method on `c10::Stream` and the corresponding `isStreamCapturing()` virtual on `DeviceGuardImplInterface`. The current patch cherry-picks that API; we will rebase onto #171443 once it merges.

## Reference

- **[Issue #164302](https://github.com/pytorch/pytorch/issues/164302)** — @ngimel identified the AccumulateGrad stream origin, @soulitzer pointed to the warning-only fix.
- **[PR #166136](https://github.com/pytorch/pytorch/pull/166136)** — Added the `UserWarning` in `input_buffer.cpp`. This PR builds on that — the warning identifies the problem, this PR adds a flag to override the stream it.
- **[CUDA graph docs](https://pytorch.org/docs/stable/notes/cuda.html#constraints)** — "Capture must occur on a non-default stream." The docs acknowledge the constraint but autograd violates it internally when nodes snapshot stream 0 during warmup.
- **[PR #171443](https://github.com/pytorch/pytorch/pull/171443)** — `c10::Stream::is_capturing()` API used by this PR.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/180090
Approved by: https://github.com/eqy, https://github.com/soulitzer, https://github.com/ngimel
2026-04-28 02:12:00 +00:00
Sam Gross 5910df01a5 Revert "[autograd] Convert shared_ptr<Node> to intrusive_ptr<Node>" (#181432)
Summary: The PR may cause increased memory usage.

Test Plan: contbuild & CI

Differential Revision: D102382560

Pull Request resolved: https://github.com/pytorch/pytorch/pull/181432
Approved by: https://github.com/atalman
2026-04-25 14:59:21 +00:00
Shangdi Yu 441bf01d46 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-24 18:13:34 +00:00
PyTorch MergeBot 07841b8fe8 Revert "Make Generator an Opaque Object (#179661)"
This reverts commit 7e0ab7a126.

Reverted https://github.com/pytorch/pytorch/pull/179661 on behalf of https://github.com/atalman due to sorry need to revert again, not sure why windows builds where not executed ([comment](https://github.com/pytorch/pytorch/pull/179661#issuecomment-4314245101))
2026-04-24 15:18:51 +00:00
Sam Gross 2d92e585fd [autograd] Convert shared_ptr<Node> to intrusive_ptr<Node> (#181139)
Replace std::shared_ptr<Node> with c10::intrusive_ptr<Node> throughout the autograd system. Node now inherits from c10::intrusive_ptr_target instead of std::enable_shared_from_this<Node>.

**BC Breaking**:

This changes the signature of `Tensor::grad_fn`.  Accesses to `Tensor.grad_fn()` should change from `std::shared_ptr<Node>` to `c10::intrusive_ptr<Node>`. Similarly, construction of a C++ autograd function should change from:

```c++
std::shared_ptr<CustomCppNode> node(new CustomCppNode(), torch::autograd::deleteNode);
```

to

```c++
auto node = c10::make_intrusive<CustomCppNode>();
```

Authored with Claude.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/181139
Approved by: https://github.com/albanD
2026-04-23 22:01:46 +00:00
Shangdi Yu 7e0ab7a126 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-23 17:15:28 +00:00
Zhijing Li 4918ae2275 [inductor] Add _FastCudaLauncher: vectorcall C extension for pre-bound kernel launch (#180507) (#180507)
Summary:

## Motivation

D100887242 added a fast path to CachingAutotuner.run() that bypasses the
Python preamble (~1.4µs savings). However, the actual kernel launch still goes
through StaticallyLaunchedCudaKernel.run() → _StaticCudaLauncher._launch_kernel(),
which carries ~2.2µs of overhead above bare cuLaunchKernel:

  - PyArg_ParseTuple("KiiiiisOK") parsing 9 args per call      ~0.3µs
  - cuCtxGetCurrent validation                                  ~0.1µs
  - Python data_ptr() + cuPointerGetAttribute per tensor arg    ~0.3µs × N
  - parseKernelArgs type-switching loop                         ~0.2µs

All of these are redundant on steady-state launches: kernel metadata is fixed,
CUDA context is guaranteed valid, and tensor pointers are always valid device
pointers.

## Approach

Add `_FastCudaLauncher`, a C Python type that pre-binds kernel metadata at
construction time and uses PEP 590 vectorcall for the fastest Python→C
dispatch:

1. **Construction** (once per kernel): pre-binds CUfunction, block dims,
   shared mem, arg types, scratch slots. Pre-computes
   `kernelArgs[i] = &argStorage[i]`.

2. **__call__** (every launch, via vectorcall — no tuple allocation):
   - Extracts grid + stream directly from the vectorcall args array
   - For tensor args: `THPVariable_Unpack(obj).data_ptr()` — zero-cost
     reinterpret_cast to C++ `at::Tensor`, no Python method call, no
     cuPointerGetAttribute validation
   - For scalar args: same `convertType` as existing path
   - Calls `cuLaunchKernel` directly — no cuCtxGetCurrent check

Integrated into CachingAutotuner: after D100887242 selects the fast launcher,
`_build_fast_launcher()` upgrades it to use `_FastCudaLauncher` by cloning
the launcher function with the runner replaced. Falls back silently to the
original launcher if conditions aren't met (non-static path, missing kernel
info, import failure, etc.).

Gated by `torch._inductor.config.use_fast_triton_launcher` (default ON),
controllable via `TORCHINDUCTOR_USE_FAST_TRITON_LAUNCHER=0`.

## Results

### Microbenchmark (H100, mode/opt, tritonbench launch_latency)

Per-kernel overhead (`nop_inductor_per_kernel`):
- Before (D100887242): 5.64µs (0-arg), 7.05µs (19-arg)
- After:               3.12µs (0-arg), 3.28µs (19-arg)
- Δ:                  -2.52µs (-45%), -3.77µs (-53%)

Raw cuLaunchKernel baseline: 2.09µs (0-arg)
Per-kernel gap to raw CUDA: 3.12µs - 2.09µs = 1.03µs (only ~1µs above driver cost)

### Microbenchmark (GB200, mode/opt, tritonbench launch_latency)

Per-kernel overhead (`nop_inductor_per_kernel`):
- Before (D100881626): 5.21µs (0-arg), 6.93µs (19-arg)
- After:               3.07µs (0-arg), 3.06µs (19-arg)
- Δ:                  -2.14µs (-41%), -3.87µs (-56%)

Raw cuLaunchKernel baseline: 1.32µs (0-arg)
Per-kernel gap to raw CUDA: 3.07µs - 1.32µs = 1.75µs

### E2E model benchmark (H100, torchbench, inductor, no cudagraphs, inference, bfloat16)

Combined effect of D100887242 + this diff vs baseline (D100881626):

| Model        | Calls | Baseline | After  | Δ latency           |
|--------------|------|----------|--------|---------------------|
| hf_Bert      |   285 | 2.513ms  | 1.907ms| -0.606ms (-24.1%)   |
| resnet18     |    69 | 0.894ms  | 0.601ms| -0.293ms (-32.8%)   |
| resnet50     |   175 | 2.242ms  | 2.227ms| ~0% (compute-bound) |
| mobilenet_v2 |   153 | 2.537ms  | 1.147ms| -1.390ms (-54.8%)   |

### E2E model benchmark (GB200, torchbench, inductor, no cudagraphs, inference, bfloat16)

Combined effect of D100887242 + this diff vs baseline (D100881626):

| Model        | Calls | Baseline | After   | Δ latency           |
|--------------|------|----------|---------|---------------------|
| hf_Bert      |   285 | 5.205ms  | 4.371ms | -0.834ms (-16.0%)   |
| resnet18     |    69 | 4.668ms  | 4.667ms | ~0% (noise)         |
| resnet50     |   175 | 8.734ms  | 8.690ms | ~0% (compute-bound) |
| mobilenet_v2 |   153 | 3.929ms  | 2.777ms | -1.152ms (-29.3%)   |

Models with many small kernels (mobilenet_v2, hf_Bert) see the
largest improvements. Compute-bound models (resnet50) are unaffected — the
optimization has zero cost when kernel execution dominates.

With cudagraphs ON, the optimization has ~zero effect (expected — cudagraphs
bypass the Python dispatch path):

| Model (GB200, cudagraphs ON) | Baseline | After   | Δ        |
|------------------------------|----------|---------|----------|
| hf_Bert                      | 1.407ms  | 1.418ms | ~0%      |
| resnet50                     | 8.414ms  | 8.467ms | ~0%      |
| mobilenet_v2                 | 0.892ms  | 0.875ms | ~0%      |

### Inductor Perf Nightly

H100: https://fburl.com/xa99g949
A100: https://fburl.com/491o10qu
(default & inductor deterministic mode are correct mode to check: disable cudagraph, no AOTI)
A100 shows less perf improve might because each kernel has longer execution time, so relative improvement is less.

Test Plan:
## Tritonbench (H100)

```
buck2 run fbcode//mode/opt fbcode//pytorch/tritonbench:run_lite -- \
  --op launch_latency \
  --only nop_inductor_kernel,nop_inductor_per_kernel,nop_triton_compiled_kernel_run,nop_triton_direct_culaunch \
  --metrics walltime --simple-output
```

```
  x_val  nop_inductor_kernel  nop_inductor_per_kernel  nop_triton_compiled_kernel_run  nop_triton_direct_culaunch
      0            0.0248156              0.00312076                     0.00358178                   0.00208895
     19            0.0259619              0.00328259                     0.00501875                   0.00352997
```

## Tritonbench (GB200)

```
buck2 run fbcode//mode/opt \
  -c fbcode.enable_gpu_sections=true \
  -c fbcode.platform010_cuda_version=12.8 \
  -c fbcode.nvcc_arch=b200a \
  fbcode//pytorch/tritonbench:run_lite -- \
  --op launch_latency \
  --only nop_inductor_per_kernel,nop_triton_compiled_kernel_run,nop_triton_direct_culaunch \
  --metrics walltime --simple-output
```

```
  x_val  nop_inductor_per_kernel  nop_triton_compiled_kernel_run  nop_triton_direct_culaunch
      0              0.00307118                      0.0025503                   0.00132063
     19              0.00306199                      0.00360913                  0.00229927
```

## E2E torchbench A/B (no cudagraphs)

```
# Fast ON (default):
buck2 run fbcode//mode/opt fbcode//caffe2/benchmarks/dynamo:torchbench -- \
  --performance --inference --bfloat16 --inductor --disable-cudagraphs \
  --device cuda --only mobilenet_v2

# Fast OFF (fallback):
TORCHINDUCTOR_USE_FAST_TRITON_LAUNCHER=0 buck2 run fbcode//mode/opt \
  fbcode//caffe2/benchmarks/dynamo:torchbench -- \
  --performance --inference --bfloat16 --inductor --disable-cudagraphs \
  --device cuda --only mobilenet_v2
```

## Build

```
buck2 build fbcode//mode/opt fbcode//caffe2:torch_python_cuda_without_torch
```

## Correctness

```
buck2 test fbcode//mode/opt fbcode//caffe2/test:test_torchinductor
```

## Unit test

```
buck2 test fbcode//mode/opt fbcode//caffe2/test/inductor:static_triton_launcher -- 'TestFastCudaLauncherCompileResult'
```

Reviewed By: PaulZhang12

Differential Revision: D101027739

Pull Request resolved: https://github.com/pytorch/pytorch/pull/180507
Approved by: https://github.com/PaulZhang12
2026-04-23 05:26:48 +00:00
PyTorch MergeBot 1aa8e27453 Revert "Make Generator an Opaque Object (#179661)"
This reverts commit 4aef62c058.

Reverted https://github.com/pytorch/pytorch/pull/179661 on behalf of https://github.com/yushangdi due to breaks internal buck dependency ([comment](https://github.com/pytorch/pytorch/pull/179661#issuecomment-4300265707))
2026-04-22 22:10:14 +00:00
PyTorch MergeBot 9372b02db2 Revert "[autograd] Convert shared_ptr<Node> to intrusive_ptr<Node> (#179766)"
This reverts commit 0309973bc7.

Reverted https://github.com/pytorch/pytorch/pull/179766 on behalf of https://github.com/ZainRizvi due to Sorry but this is breaking internally. See diff D101855555 for details. To validate your fixes internally, you can follow the instructions here: https://fburl.com/fixing-ghfirst-reverts ([comment](https://github.com/pytorch/pytorch/pull/179766#issuecomment-4297869022))
2026-04-22 16:06:55 +00:00
Shangdi Yu 4aef62c058 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-21 16:21:24 +00:00
Sam Gross 0309973bc7 [autograd] Convert shared_ptr<Node> to intrusive_ptr<Node> (#179766)
Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.14.0) (oldest at bottom):

* #179767
* -> #179766
* #179765
* #179764

Replace std::shared_ptr<Node> with c10::intrusive_ptr<Node> throughout
the autograd system. Node now inherits from c10::intrusive_ptr_target
instead of std::enable_shared_from_this<Node>.

Authored with Claude.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179766
Approved by: https://github.com/albanD
ghstack dependencies: #179764, #179765
2026-04-21 03:13:54 +00:00
PyTorch MergeBot cb242afe1b Revert "Make Generator an Opaque Object (#179661)"
This reverts commit 18357d9483.

Reverted https://github.com/pytorch/pytorch/pull/179661 on behalf of https://github.com/yushangdi due to breaks pr_time_benchmarks ([comment](https://github.com/pytorch/pytorch/pull/179661#issuecomment-4285042651))
2026-04-21 00:09:01 +00:00
Shangdi Yu 18357d9483 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-20 19:55:21 +00:00
PyTorch MergeBot 5c61b13b23 Revert "Make Generator an Opaque Object (#179661)"
This reverts commit 3fa6c02521.

Reverted https://github.com/pytorch/pytorch/pull/179661 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable ([comment](https://github.com/pytorch/pytorch/pull/179661#issuecomment-4283639775))
2026-04-20 19:22:36 +00:00
Shangdi Yu 3fa6c02521 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)
- `python test/dynamo/test_misc.py -k test_torch_generator_initial_seed`

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
2026-04-20 18:52:45 +00:00
cyy cf14e0ab47 Use Py_IsNone/Py_IsTrue/Py_IsFalse for singleton identity checks (#180753)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/180753
Approved by: https://github.com/Skylion007
2026-04-19 05:54:28 +00:00
Grayson Derossi 1dba3349d7 Add cuDNN depthwise convolution backend flag (#176500)
This change was separated from #170609, and adds a context flag to select the backend kernel for depthwise convolution. This allows easy benchmarking of cuDNN performance relative to fallback performance for heuristic tuning (and could also be used for debugging), while maintaining default behavior for standard use.

There are three options: `["auto", "cudnn", "native"]`. The `"auto"` option chooses a kernel based on the existing heuristic function. The `"cudnn"` and `"native"` flags skip the heuristic, instead always dispatching to a cuDNN kernel or a fallback kernel, respectively.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/176500
Approved by: https://github.com/malfet, https://github.com/eqy
2026-04-17 08:41:04 +00:00
Grayson Derossi d61bcb2575 [CUDA][cuBLAS] Add get/set/reset functions for BLAS workspaces (#177912)
This adds Python functions to get, set, and reset cuBLAS or cuBLASLt workspace sizes. Workspaces are updated lazily the next time they are accessed if the new workspace size is greater than the current size.

@ngimel, is this what you were hoping for? I believe the current BLAS backend can already be queried with `torch.backends.cuda.preferred_blas_library`.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/177912
Approved by: https://github.com/ngimel, https://github.com/malfet, https://github.com/eqy
2026-04-16 00:11:46 +00:00
PyTorch MergeBot cc8d1bdb03 Revert "Make Generator an Opaque Object (#179661)"
This reverts commit 4510a721a2.

Reverted https://github.com/pytorch/pytorch/pull/179661 on behalf of https://github.com/pytorch-auto-revert due to Reverted automatically by pytorch's autorevert, to avoid this behaviour add the tag autorevert: disable ([comment](https://github.com/pytorch/pytorch/pull/179661#issuecomment-4217993074))
2026-04-09 22:18:36 +00:00
Shangdi Yu 4510a721a2 Make Generator an Opaque Object (#179661)
### Summary

Replaces the `OpaqueGenerator` Python wrapper from  #179658 by making
`torch._C.Generator` itself a proper opaque reference type. This eliminates the
wrapper class entirely.

C++ changes to `THPGenerator`:
- Set `OpaqueBaseMeta` as the metaclass (same pattern as `ProcessGroup`)
- Add weakref support (`tp_weaklistoffset` + `weakreflist` field)
- Add `tp_richcompare` (identity-based `__eq__`/`__ne__`)
- Add `tp_hash` (identity-based)
- Add `__fx_repr__` method for FX codegen

Also fixes `THPModule_addDocStr` to use `PyType_Check(obj)` instead of
`Py_TYPE(obj) == &PyType_Type`, since Generator's metaclass is now
`OpaqueBaseMeta` rather than `type`.

With these changes, `torch._C.Generator` is registered directly via
`register_opaque_type(torch._C.Generator, typ="value")` and flows through
`make_fx`, FX codegen, and the repro infrastructure without any wrapper.

Authored with Claude.

### Test plan

- `python test/dynamo/test_after_aot.py` (all 6 tests)
- `python test/inductor/test_torchinductor.py -k test_randn_generator_cuda`
- `python test/test_opaque_obj_v2.py TestOpaqueGenerator` (make_fx + correctness)
- ` python test/test_torch.py -k test_pickle_generator` (generator can be pickled)
- `python test/dynamo/test_dynamic_shapes.py -k test_user_getattr1_dynamic_shapes`
- `python agent_space/repro.py` (end-to-end repro)
- `python test/test_fx.py -k test_impure_random` (Generator is a C type, it has __init__ without __code__)

Pull Request resolved: https://github.com/pytorch/pytorch/pull/179661
Approved by: https://github.com/angelayi
ghstack dependencies: #179657
2026-04-09 21:17:34 +00:00
Nikita Shulga 50d4815bd4 Remove TorchVitals (#178479)
TorchVitals was an undocumented feature gated behind the TORCH_VITAL
environment variable that allowed setting and reading key-value "vital signs" (e.g. whether CUDA or DataLoader was used). It was never documented and the test itself had a "FIXME: document or deprecate" comment.

Originally added in PR #51047 (Jan 2021), with a Python API in #53238 and a CUDA vital in #58059. There have been no feature changes since June 2021 — all subsequent commits were mechanical cleanups (clang-tidy,
thread-safe getenv, etc.).

Update: `torch.set_vital` was used from TorchData (see https://github.com/meta-pytorch/data/pull/1537 ) so will keep the dummy API around

Authored with Claude.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Pull Request resolved: https://github.com/pytorch/pytorch/pull/178479
Approved by: https://github.com/msaroufim, https://github.com/albanD, https://github.com/mlazos
2026-04-03 06:19:18 +00:00
PyTorch MergeBot 3f18f3f5ba Revert "[ROCm] Add hipDNN backend support for convolution (#178515)"
This reverts commit 3a52b28751.

Reverted https://github.com/pytorch/pytorch/pull/178515 on behalf of https://github.com/izaitsevfb due to breaks internal builds ([comment](https://github.com/pytorch/pytorch/pull/178515#issuecomment-4180671650))
2026-04-02 21:50:15 +00:00
Rob Timpe f6ef243ef8 Implement thread-local config with contextvars (#173568)
This is an alternative approach to https://github.com/pytorch/pytorch/issues/118387 using context vars

Pull Request resolved: https://github.com/pytorch/pytorch/pull/173568
Approved by: https://github.com/williamwen42
ghstack dependencies: #178950
2026-04-01 22:20:03 +00:00
zjgarvey 3a52b28751 [ROCm] Add hipDNN backend support for convolution (#178515)
This PR adds [hipDNN](https://github.com/ROCm/rocm-libraries/tree/develop/projects/hipdnn) convolution support to PyTorch on ROCm, following the newer dispatch mechanism which no longer requires per-backend frontend op registration.

New ROCm kernel providers will be able to register plugins and ship faster kernels through hipDNN without touching PyTorch code. The graph-based API of hipDNN also supports fused kernels. The eventual goal is to replace all MIOpen-specific PyTorch code with hipDNN, wherein MIOpen will be just one of multiple first-class kernel providers.

# Note

This PR is being opened parallel to #177534 in anticipation that batch norm's dispatch migration issues might stall the review process for hipdnn as a whole.

# Summary

- Integrates the lightweight hipDNN frontend graph API as an alternative convolution backend to MIOpen on ROCm, covering forward, backward-data, and backward-weight for 2D/3D convolutions including transposed variants.

- Adds cmake `USE_HIPDNN` detection, `torch.backends.hipdnn` Python module with `enabled`/`flags()` context manager, and `compiledWithHipDNN()` runtime hook.

- Implements a generic thread-local `ParamsLRUCache` to amortize hipDNN `graph->build()` cost across repeated convolution calls with identical parameters.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/178515
Approved by: https://github.com/jeffdaily

Co-authored-by: Dmitry Nikolaev <dmitry.nikolaev@amd.com>
2026-04-01 20:30:55 +00:00
Haoyu Zhang 4f6f0da0a9 Reland D97799113: [CK SDPA] Add CK backend support for varlen attention (#178322) (#178322) (#178729)
Summary:
CONTEXT: D97799113 was reverted by D98681402. This relands the original change.

Port GitHub PR https://github.com/pytorch/pytorch/issues/172246 to internal. This adds composable_kernel (CK) backend
support for scaled-dot-product-attention varlen path on ROCm:

- Fix philox seed/offset access in varlen backward to guard on is_dropout
- Fix softmax_lse shape and stride for CK varlen forward path
- Add `is_ck_sdpa_available()` API to `torch.backends.cuda`
- Update varlen_attn fake to use correct logsumexp shape per backend
- Add CK backend parametrization to varlen attention tests

Co-authored-by: Chinmay Kuchinad ChinmayDattanand.Kuchinad@amd.com
Co-authored-by: Jeff Daily jeff.daily@amd.com

Test Plan: buck test on relevant test targets

Differential Revision: D98693429

Pulled By: haoyuz

Pull Request resolved: https://github.com/pytorch/pytorch/pull/178729
Approved by: https://github.com/echen4096
2026-04-01 15:16:53 +00:00
Sam Gross 3eeaf56b00 Make LogAPIUsageOnceFromPython thread-safe for free-threaded Python (#178867)
The old implementation used a static std::unordered_set protected only by the GIL. Under free-threaded Python (3.14t) this is a data race.

Replace it with a Python dict stored in per-module state. The hot path (already-seen event) calls PyDict_Contains which is lock-free in 3.14t. The cold path (first occurrence) uses PyDict_SetDefaultRef which uses CPython's per-dict fine-grained locking to atomically check-and-insert, avoiding TOCTOU races between threads.

Authored with Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/178867
Approved by: https://github.com/Skylion007
2026-04-01 14:58:03 +00:00
PyTorch MergeBot 537b1955c1 Revert "Remove TorchVitals (#178479)"
This reverts commit 179e2d57a9.

Reverted https://github.com/pytorch/pytorch/pull/178479 on behalf of https://github.com/georgehong due to change is breaking internal tests: AttributeError: module 'torch' has no attribute 'set_vital' ([comment](https://github.com/pytorch/pytorch/pull/178479#issuecomment-4145885127))
2026-03-27 23:25:57 +00:00
Nikita Shulga 179e2d57a9 Remove TorchVitals (#178479)
TorchVitals was an undocumented feature gated behind the TORCH_VITAL
environment variable that allowed setting and reading key-value "vital
signs" (e.g. whether CUDA or DataLoader was used). It was never
documented and the test itself had a "FIXME: document or deprecate"
comment.

Originally added in PR #51047 (Jan 2021), with a Python API in #53238
and a CUDA vital in #58059. There have been no feature changes since
June 2021 — all subsequent commits were mechanical cleanups (clang-tidy,
thread-safe getenv, etc.).

A search on cs.github.com and general web search confirmed no external
projects depend on torch.set_vital, torch.read_vitals, or
torch.vitals_enabled — usage was limited to pytorch/pytorch itself.

Authored with Claude.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Pull Request resolved: https://github.com/pytorch/pytorch/pull/178479
Approved by: https://github.com/msaroufim, https://github.com/albanD
2026-03-26 22:00:52 +00:00
PyTorch MergeBot 554573d861 Revert "Implement thread-local config with contextvars (#173568)"
This reverts commit c904bdcf76.

Reverted https://github.com/pytorch/pytorch/pull/173568 on behalf of https://github.com/georgehong due to tests failing with torch._dynamo hit config.recompile_limit (8) ([comment](https://github.com/pytorch/pytorch/pull/173568#issuecomment-4129118162))
2026-03-25 19:17:48 +00:00
rraminen 120a341a01 [ROCm] Enable hipSPARSELt UTs (#178285)
This PR:

- Enables torch.backends.cusparselt.is_available() on ROCm
- Enables hiSPARSELt UTs on ROCm
- Skip the following UTs on ROCm
```
test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape0_cuda_int8
test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape1_cuda_int8
test_mm_sparse_first_NN_backend_cusparselt_dense_input_shape2_cuda_int8
test_mm_sparse_first_NT_backend_cusparselt_dense_input_shape0_cuda_int8
```

Pull Request resolved: https://github.com/pytorch/pytorch/pull/178285
Approved by: https://github.com/jeffdaily
2026-03-24 22:47:05 +00:00
Rob Timpe c904bdcf76 Implement thread-local config with contextvars (#173568)
This is an alternative approach to https://github.com/pytorch/pytorch/issues/118387 using context vars

Pull Request resolved: https://github.com/pytorch/pytorch/pull/173568
Approved by: https://github.com/williamwen42
2026-03-24 17:09:50 +00:00
PyTorch MergeBot 939efd00d8 Revert "[ROCm][CK] Enable variable-length attention support for CK SDPA backend (#172246)"
This reverts commit e1b7388686.

Reverted https://github.com/pytorch/pytorch/pull/172246 on behalf of https://github.com/huydhn due to sorry, failing builds with older HIP versions, see below ([comment](https://github.com/pytorch/pytorch/pull/172246#issuecomment-4033763552))
2026-03-10 19:13:24 +00:00
PyTorch MergeBot 2ee3377bb7 Revert "[fx] Move _Namespace to C++ (#170962)"
This reverts commit 5c68844a2a.

Reverted https://github.com/pytorch/pytorch/pull/170962 on behalf of https://github.com/wdvr due to reverted in fbcode ([comment](https://github.com/pytorch/pytorch/pull/170962#issuecomment-4027702285))
2026-03-10 00:00:56 +00:00
Chinmay Kuchinad e1b7388686 [ROCm][CK] Enable variable-length attention support for CK SDPA backend (#172246)
## Summary

Enables variable-length (varlen) attention support for the Composable Kernel (CK) SDPA backend on ROCm.

## Changes

### Forward pass (`mha_varlen_fwd_ck.hip`)

- Fixed LSE tensor allocation: changed from 3D `{batch_size, num_heads, max_seqlen_q}` to 2D `{num_heads, total_q}` to match CK group mode expectation
- Fixed `nhead_stride_lse`: use `stride(0)` instead of `stride(1)` for 2D layout
- Fixed `batch_stride_lse`: set to `0` (no batch dimension in group mode LSE)
- Fixed `min_seqlen_q`: changed from `-1` to `1` (valid minimum for kernel dispatch)

### Backward pass (`mha_varlen_bwd_ck.hip`)

- Fixed philox seed/offset access: guarded with `if (is_dropout)` to avoid dtype mismatch when dropout is disabled

### Test infrastructure (`test/test_varlen_attention.py`)

- Added `sdpa_backend` parametrization to test both `aotriton` and `ck` backends on ROCm
- Backend selection: `["aotriton", "ck"]` when CK is available, `["aotriton"]` otherwise
- Uses `preferred_rocm_fa_library()` to switch backends per test

### Platform detection

- Added `PLATFORM_SUPPORTS_CK_SDPA` in `torch/testing/_internal/common_cuda.py`
- Added `_is_ck_sdpa_available()` in `torch/csrc/Module.cpp` for runtime CK availability check

### Fake tensor implementation (`torch/nn/attention/varlen.py`)

- Updated `_varlen_attn_fake` to use standard `[num_heads, total_q]` logsumexp format (matches aotriton/CUDA)

## Dependencies: CK submodule bump

- **CK PR**: (https://github.com/ROCm/rocm-libraries/pull/4292)

Pull Request resolved: https://github.com/pytorch/pytorch/pull/172246
Approved by: https://github.com/jeffdaily
2026-03-04 00:28:02 +00:00
PyTorch MergeBot 0a06050b14 Revert "[fx] Move _FindNodesLookupTable and _node_list to C++ (#170973)"
This reverts commit 6a78d3dff7.

Reverted https://github.com/pytorch/pytorch/pull/170973 on behalf of https://github.com/facebook-github-bot due to Diff reverted internally ([comment](https://github.com/pytorch/pytorch/pull/170973#issuecomment-3980822092))
2026-03-01 19:20:43 +00:00
Jason Ansel 6a78d3dff7 [fx] Move _FindNodesLookupTable and _node_list to C++ (#170973)
Cumulative Changes (PR 170946 → PR 170973)

  | Benchmark                             | PR 170946 | PR 170973 | Delta |
  |---------------------------------------|-----------|-----------|-------|
  | add_loop_eager                        | 3,308M    | 3,206M    | -3.1% |
  | aotdispatcher_partitioner_cpu2        | 1,944M    | 1,865M    | -4.1% |
  | aotdispatcher_partitioner_cpu         | 8,340M    | 8,062M    | -3.3% |
  | sum_floordiv_regression               | 3,700M    | 3,591M    | -3.0% |
  | aotdispatcher_training_nosubclass_cpu | 3,242M    | 3,168M    | -2.3% |
  | update_hint_regression                | 1,674M    | 1,641M    | -2.0% |
  | symint_sum_loop                       | 4,682M    | 4,590M    | -2.0% |
  | add_loop_inductor_gpu                 | 26,418M   | 26,031M   | -1.5% |
  | basic_NestedModule_eager              | 6,171M    | 6,084M    | -1.4% |
  | add_loop_inductor                     | 29,736M   | 29,342M   | -1.3% |

Pull Request resolved: https://github.com/pytorch/pytorch/pull/170973
Approved by: https://github.com/oulgen
ghstack dependencies: #170946, #170962
2026-02-26 17:48:17 +00:00
Jason Ansel 5c68844a2a [fx] Move _Namespace to C++ (#170962)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/170962
Approved by: https://github.com/oulgen
ghstack dependencies: #170946
2026-02-26 17:48:17 +00:00
Kushagra Rastogi 4f3d429973 Raise Warning Once when invariants check is disabled (#171695)
Fixes #170626

Pull Request resolved: https://github.com/pytorch/pytorch/pull/171695
Approved by: https://github.com/pearu, https://github.com/isuruf
2026-02-11 16:21:39 +00:00
Chinmay Kuchinad 03b8651970 [ROCm] Enable StaticCudaLauncher for ROCm (#166492)
This PR enables ROCm/HIP support for PyTorch's StaticCudaLauncher, which provides static compilation and launching of Triton kernels. The implementation has been tested on AMD MI300 and MI200 hardware.

**Changes**

**Python (torch/_inductor/runtime/)**
- static_cuda_launcher.py: Added ROCm detection, .hsaco binary support, and ROCm-specific scratch parameter handling
- triton_heuristics.py: Updated device type checks to support both cuda and hip

**C++ (torch/csrc/)**
- Module.cpp: Enabled StaticCudaLauncher for ROCm builds
- inductor/static_cuda_launcher.cpp: Added HIP API equivalents for all CUDA driver calls
- inductor/static_cuda_launcher.h: Updated header guard

**Tests (test/inductor/)**
- test_static_cuda_launcher.py: Removed @skipIfRocm decorators and updated binary file handling

**Enabled Unit Tests**
All tests in test/inductor/test_static_cuda_launcher.py now pass on ROCm:
1. Fixes #168654.
2. Fixes #168655.
3. Fixes #168656.
4. Fixes #168657.
5. Fixes #168658.
6. Fixes #168659.
7. Fixes #168660.
8. Fixes #168661.
9. Fixes #168662.
10. Fixes #168663.
11. Fixes #168664.
12. Fixes #168666
13. Fixes #168667
14. Fixes #168668
15. Fixes #168669
16. Fixes #168670
17. Fixes #168671
18. Fixes #168575

Pull Request resolved: https://github.com/pytorch/pytorch/pull/166492
Approved by: https://github.com/jeffdaily
2026-02-03 17:01:30 +00:00
Ma, Jing1 b71eaf9fb0 [1/2][XPU] Implement XPUGraph capture and replay (#166285)
XPUGraph PR submission plan, following this [RFC](https://github.com/pytorch/pytorch/issues/162143).

- [x] 1, enhance XPUGenerator to cover XPUGeneratorState and philoxState
- [x] 2, the implementation of MemPool in XPU device allocator
- [ ] ->3, the implementations of capture_begin/capture_end/instantiate for XPUGraph
- [ ] 4, the implementations of replay/debug_dump/reset for XPUGraph
- [ ] 5, XPUGraph python APIs: is_current_stream_capturing/graph_pool_handle/graph
- [ ] 6, XPUGraph python APIs: make_graphed_callables

Pull Request resolved: https://github.com/pytorch/pytorch/pull/166285
Approved by: https://github.com/EikanWang, https://github.com/gujinghui, https://github.com/eellison, https://github.com/guangyey
2026-01-31 12:45:42 +00:00
xinan.lin 84b7563c37 [xpu][feature][static_launcher] 3/4: Add xpu static launcher cpp module. (#168952)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/168952
Approved by: https://github.com/EikanWang, https://github.com/desertfire
ghstack dependencies: #169121
2026-01-26 07:03:14 +00:00
Ma, Jing1 f9e35b5409 [3/3][XPU][feature] The frontend python APIs for XPU MemPool (#166843)
The implementation plan of MemPool for XPU, which is the dependance of [XPUGraph](https://github.com/pytorch/pytorch/pull/166285), following the [RFC](https://github.com/pytorch/pytorch/issues/162143).

- [x] #166831
- [x] #166833
- [ ] ->#166843

Pull Request resolved: https://github.com/pytorch/pytorch/pull/166843
Approved by: https://github.com/EikanWang, https://github.com/gujinghui, https://github.com/eellison
2026-01-24 09:32:23 +00:00
xinan.lin 4ec170a846 [xpu][static_launcher] 2/4: Refactor static cuda launcher to static triton launcher. (#169121)
This PR refactor the cuda static launcher to triton static launcher to be able to be reused by XPU triton backend as follow:
- Rename class `StaticallyLaunchedCudaKernel` -> class `StaticallyLaunchedTritonKernel`, and then keep `StaticallyLaunchedCudaKernel` inherit the  `StaticallyLaunchedTritonKernel`. Then there will be `StaticallyLaunchedXpuKernel`.
- Rename static_cuda_launcher.py -> static_triton_launcher.py
- Rename torch/csrc/inductor/static_cuda_launcher.cpp -> torch/csrc/inductor/static_launcher/cuda.cpp
- Rename test/_inductor/test_static_cuda_launcher.py -> test/_inductor/test_static_triton_launcher.py

Pull Request resolved: https://github.com/pytorch/pytorch/pull/169121
Approved by: https://github.com/EikanWang, https://github.com/desertfire
2026-01-23 17:23:19 +00:00
Howard Zhang 6407e23a77 Add FA3 to SDPA (#172040)
### Summary:
Added support for flash attention v3 to SDPA
Only supports fp8 forward pass for now
Created new `_fa3.py` mirroring `_fa4.py` registration
modified `torch/nn/attention/__init__.py` and `_registry.py` to add FA3 support
Added C++ hook to expose FA3 activation (to allow for fp8 dtype)

### Design:
- The design follows the same basic structure as the [FA4 implementation](https://github.com/pytorch/pytorch/pull/167348).
- I added a `_fa3.py` which mirrors the same registration framework as `_fa4.py` (wiring up the new fa3 impl with aten ops `_flash_attention_forward`, `_flash_attention_backward`, `_scaled_dot_product_flash_attention`, `_scaled_dot_product_flash_attention_backward`)
- I additionally added a C++ hook to expose the FA3 activation to `sdp_utils.cpp`, and added a new function to check for low precision dtypes in flash attention (since before fp8 was not allowed)
- Note that only fp8 forward pass is supported for now. We can add fp16/bf16 support later, but fp16/bf16 performance with FA4 is better anyway. The backwards pass currently will throw an error: "FA3 does not support backward pass. Either: 1. Use torch.no_grad() for inference. 2. Unregister FA3 before training: `restore_flash_attention_impl`"

### Test Plan:
Install FA3 library: https://github.com/Dao-AILab/flash-attention/tree/main, follow directions for "FlashAttention-3 beta release", tldr; clone the repo, cd into hopper (where the FA3 implementation lives), python setup.py install

Following the same steps as FA4 to keep it consistent:
```python
activate_flash_attention_impl("FA3")
with sdpa_kernel(SDPBackend.FLASH_ATTENTION):
        _ = F.scaled_dot_product_attention(q, k, v)
```
It also works with torch.compile
```python
activate_flash_attention_impl("FA3")
def sdpa_fn(q, k, v):
      with sdpa_kernel(SDPBackend.FLASH_ATTENTION):
          return F.scaled_dot_product_attention(q, k, v)

compiled_fn = torch.compile(sdpa_fn, fullgraph=True)
_ = compiled_fn(q, k, v)
```

### Some quick runtime results:
I ran some very quick tests with the pytorch profiler to test the runtime of fp8 using FA3 versus bf16 using FA4. Mostly just to catch any easy mistakes slowing it down. These are the results when running on (8, 16, 1024, 128) shape tensors:
FA3 eager:
[sdpa_profiler_fa3_eager.txt](https://github.com/user-attachments/files/24511657/sdpa_profiler_fa3_eager.txt)
FA3 compile:
[sdpa_profiler_fa3_compile.txt](https://github.com/user-attachments/files/24511664/sdpa_profiler_fa3_compile.txt)
FA4 eager:
[sdpa_profiler_fa4_eager.txt](https://github.com/user-attachments/files/24511667/sdpa_profiler_fa4_eager.txt)
FA4 compile:
[sdpa_profiler_fa4_compile.txt](https://github.com/user-attachments/files/24511669/sdpa_profiler_fa4_compile.txt)

### Some quick accuracy results:
I ran a very quick SQNR test on (8, 16, 1024, 128) shape tensors 100 times. I got an average SQNR of 25.55.

This is still a draft, so I'll be doing a quick test with the memory next. Then, I'll do some tests on the diffusers library stable diffusion models.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/172040
Approved by: https://github.com/drisspg
2026-01-22 23:43:56 +00:00