[CPP FakeTensor] Support MKLDNN fake tensors (#204906)

## Linked issue or supporting maintainer

> Stacked above #204905.

## BC-breaking?

> No.

## Summary (human written only)

> follow the exact pattern youve been doing so far don't change any fucntionality of the code itself does that make sense?

## Agent details (AI-generated)

Authored with Codex assistance. Preserve logical MKLDNN metadata on native fakes while using strided Meta storage, and support the feature's dense conversion and composite paths. Enable all 29 existing MKLDNN cases: aliasing, four symbolic/active conversion variants, and 24 composite variants. Retain their metadata, identity, dtype, symbolic-error, and functional-wrapper assertions. Both forward to_dense and to_dense_backward use the recorded logical layout to select the MKLDNN branch for native fakes, whose physical storage is Meta. The backward check reads native metadata without acquiring the GIL.

The feature already removes its obsolete C++ xfails. Preserve MKLDNN availability checks and the unsupported real-propagation exclusions; skipped shadow cases are not claimed as working coverage. The recorded dispatch keys distinguish logical MKLDNN tensors from their strided Meta storage.

### Test Plan

The independent native build passed, including the shared-header rebuild. All 29 MKLDNN cases passed in each backend. The staged CPU runner passed 813 tests and 32 subtests without skips. Ten-class Python comparison:872 passed,8 skipped,5 expected failures,40 subtests. C++ discovery:770 passed,85 failed,6 skipped,4 expected failures. Inherited dispatcher/view failures remain deferred. Cleanup lint passed.

```bash
pip install -e . -v --no-build-isolation
CPP_FAKETENSOR=1 python test/test_fake_tensor.py --use-pytest --cpp-faketensor-tests ../prs/204906/candidate-classes.json -q --tb=short --junitxml=../prs/204906/discovery.xml
CPP_FAKETENSOR=0 python test/test_fake_tensor.py --use-pytest --cpp-faketensor-tests ../prs/204906/candidate-classes.json -q --tb=short --junitxml=../prs/204906/python.xml
CPP_FAKETENSOR=1 CUDA_VISIBLE_DEVICES='' TEST_CONFIG=cpp_faketensor python test/run_test.py --include test_fake_tensor test_custom_ops test_meta test_sparse --enable-td
spin quicklint -a -- .ci/pytorch/cpp_faketensor_tests.json
```

The follow-up incremental build passed after independent review. A direct native
to_dense_backward probe now returns MKLDNN metadata, and the four existing
non-shadow backward composite cases passed; their four shadow variants remain
skipped. This does not assert that direct Python FakeTensor backward reports the
same layout.

```bash
CPP_FAKETENSOR=1 PYTHONPATH="$PWD" python test/test_fake_tensor.py -k '*test_mkldnn_composite*backward*'
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/204906
Approved by: https://github.com/ezyang
ghstack dependencies: #204813, #204814, #204902, #204903, #204904, #204905
This commit is contained in:
Angel Li
2026-11-17 13:33:36 -08:00
committed by PyTorch MergeBot
parent 0368230cc7
commit d71087e616
8 changed files with 139 additions and 47 deletions
+3
View File
@@ -135,6 +135,7 @@
"test_fake_tensor.py::TestCppFakeTensorModeBindings::test_unset_rebuilds_mode_*",
"test_fake_tensor.py::TestFakeTensor::test__adaptive_avg_pool2d_backward",
"test_fake_tensor.py::TestFakeTensor::test_alias_call",
"test_fake_tensor.py::TestFakeTensor::test_alias_fake_mkldnn",
"test_fake_tensor.py::TestFakeTensor::test_allow_meta",
"test_fake_tensor.py::TestFakeTensor::test_as_strided_negative_stride_error",
"test_fake_tensor.py::TestFakeTensor::test_assert_tensor_metadata_with_list",
@@ -174,6 +175,8 @@
"test_fake_tensor.py::TestFakeTensor::test_matmul_rank4_unbacked_batch_dim",
"test_fake_tensor.py::TestFakeTensor::test_meta_storage_trace_uses_hint_for_symbolic_size",
"test_fake_tensor.py::TestFakeTensor::test_mixed_real_and_fake_inputs_*",
"test_fake_tensor.py::TestFakeTensor::test_mkldnn_composite_*",
"test_fake_tensor.py::TestFakeTensor::test_mkldnn_to_dense_*",
"test_fake_tensor.py::TestFakeTensor::test_mm_out_dtype",
"test_fake_tensor.py::TestFakeTensor::test_mode_*",
"test_fake_tensor.py::TestFakeTensor::test_nan_to_num",
+15 -1
View File
@@ -605,7 +605,13 @@ Tensor to_dense_backward(
semantics. Masked semantics of tensors is implemented in the
framework of masked tensors.
*/
const auto input_layout = input_.layout();
auto input_layout = input_.layout();
if (input_.is_fake()) {
const auto keys = input_.unsafeGetTensorImpl()->fake_dispatch_keys();
if (keys.has_value() && keys->has_all(c10::mkldnn_ks)) {
input_layout = kMkldnn;
}
}
const bool masked_grad = masked_grad_.value_or(true);
switch (input_layout) {
case kStrided:
@@ -674,6 +680,14 @@ Tensor to_dense(
if (tensor.layout() == c10::kMkldnn) {
return tensor._to_dense(dtype, masked_grad);
}
if (tensor.is_fake()) {
// Match FakeTensor.to_dense() without exposing MKLDNN to native layout
// queries.
const auto keys = tensor.unsafeGetTensorImpl()->fake_dispatch_keys();
if (keys.has_value() && keys->has_all(c10::mkldnn_ks)) {
return tensor._to_dense(dtype, masked_grad);
}
}
TORCH_CHECK(
tensor.layout() == c10::kStrided,
"to_dense does not support layout ",
+2 -3
View File
@@ -239,9 +239,8 @@ struct C10_API BackendMeta : intrusive_ptr_target {
// A fake can record the dispatch keys of the real tensor it stands for
// (ExtraMeta::fake_dispatch_keys_, Python's FakeTensor.dispatch_keys), so that
// guards built from the fake check the real tensor's keys rather than the
// fake's own. Python also derives is_mkldnn and layout from them; here only
// guards read them, so a fake recorded with an mkldnn tensor's keys still
// reports its own layout.
// fake's own. Python's is_mkldnn and layout properties also read them, while
// native queries continue to see the fake's strided Meta backing.
// Python keeps them on the output of a top-level alias or detach (dispatched,
// through functionalization, or by calling fast_detach directly), but not of
// other views. The fake fallback (FakeFallbackKernel.cpp) does the same for a
+86 -18
View File
@@ -99,7 +99,6 @@ from torch.testing._internal.common_utils import (
TEST_WITH_SLOW,
TEST_WITH_TORCHDYNAMO,
TestCase,
xfailIf,
xfailIfTorchDynamo,
)
from torch.testing._internal.custom_op_db import custom_op_db
@@ -542,8 +541,13 @@ class TestFakeTensor(_TestFakeTensorHelpers, TestCase):
dispatch_calls = []
fake_impl_calls = []
op_name = (
"_torch_testing::fake_tensor_dispatch_parameter_subclass"
f"_{type(self).__name__}"
)
@torch.library.custom_op(
"_torch_testing::fake_tensor_dispatch_parameter_subclass",
op_name,
mutates_args=(),
)
def custom_op(weight: torch.Tensor) -> torch.Tensor:
@@ -1054,13 +1058,12 @@ class TestFakeTensor(_TestFakeTensorHelpers, TestCase):
@unittest.skipIf(not torch.backends.mkldnn.is_available(), "MKLDNN not available")
@skipIfPropagateRealTensors("Propagate real tensor not supported")
# mkldnn nyi in C++ fake tensors
@xfailIf(torch._dynamo.config.use_cpp_fake_tensor)
def test_alias_fake_mkldnn(self):
if torch._functorch.config.fake_tensor_propagate_real_tensors:
self.skipTest("Propagate real tensor not supported")
real = torch.randn(2, 3).to_mkldnn()
with FakeTensorMode() as fake_mode:
x = fake_mode.from_tensor(torch.randn(2, 3).to_mkldnn())
x = fake_mode.from_tensor(real)
y = torch.ops.aten.alias.default(x)
self.assertTrue(y.is_mkldnn)
self.assertEqual(torch._C._dispatch_key_set(y), torch._C._dispatch_key_set(x))
@@ -1082,9 +1085,17 @@ class TestFakeTensor(_TestFakeTensorHelpers, TestCase):
self.assertEqual(t.device, torch.device("cpu"))
@unittest.skipIf(not torch.backends.mkldnn.is_available(), "MKLDNN not available")
# mkldnn nyi in C++ fake tensors
@xfailIf(torch._dynamo.config.use_cpp_fake_tensor)
def test_mkldnn_to_dense(self):
@parametrize("symbolic", [False, True])
@parametrize("active", [False, True])
def test_mkldnn_to_dense(self, symbolic, active):
if not active and TEST_WITH_TORCHDYNAMO:
self.skipTest(
"Dynamo cannot guard explicit MKLDNN fakes outside their mode"
)
if torch._dynamo.config.use_cpp_fake_tensor and TEST_WITH_TORCHDYNAMO:
self.skipTest(
"Dynamo guards on MKLDNN keys differ from the C++ fake's dispatch keys"
)
from torch._subclasses.functional_tensor import (
FunctionalTensor,
FunctionalTensorMode,
@@ -1093,10 +1104,16 @@ class TestFakeTensor(_TestFakeTensorHelpers, TestCase):
if torch._functorch.config.fake_tensor_propagate_real_tensors:
self.skipTest("Propagate real tensor not supported")
real = torch.randn(2, 3).to_mkldnn()
with FakeTensorMode() as fake_mode:
fake_mode = FakeTensorMode(shape_env=ShapeEnv() if symbolic else None)
with fake_mode:
x = fake_mode.from_tensor(real)
dense = torch.randn(2, 3)
with fake_mode if active else contextlib.nullcontext():
self.assertTrue(x.is_mkldnn)
self.assertEqual(x.layout, torch._mkldnn) # type: ignore[attr-defined]
with in_kernel_invocation_manager(fake_mode):
self.assertTrue(x.is_mkldnn)
self.assertEqual(x.layout, torch._mkldnn) # type: ignore[attr-defined]
y = x.to_dense()
self.assertFalse(y.is_mkldnn)
@@ -1116,6 +1133,7 @@ class TestFakeTensor(_TestFakeTensorHelpers, TestCase):
detached = x.detach()
self.assertTrue(detached.is_mkldnn)
self.assertTrue(x.data.is_mkldnn)
y = detached.to_dense()
self.assertFalse(y.is_mkldnn)
self.assertEqual(y.layout, torch.strided)
@@ -1128,7 +1146,6 @@ class TestFakeTensor(_TestFakeTensorHelpers, TestCase):
self.assertEqual(y.layout, torch.strided)
self.assertEqual(y.stride(), (3, 1))
dense = torch.randn(2, 3)
mkldnn = torch.ops.aten.to_mkldnn.default(dense)
self.assertTrue(mkldnn.is_mkldnn)
y = torch.ops.aten.to_dense.default(mkldnn)
@@ -1153,16 +1170,67 @@ class TestFakeTensor(_TestFakeTensorHelpers, TestCase):
@unittest.skipIf(not torch.backends.mkldnn.is_available(), "MKLDNN not available")
@skipIfPropagateRealTensors("Propagate real tensor not supported")
# mkldnn nyi in C++ fake tensors
@xfailIf(torch._dynamo.config.use_cpp_fake_tensor)
def test_mkldnn_unsafe_fallback(self):
@parametrize("symbolic", [False, True])
@parametrize("active", [False, True])
@parametrize(
"op",
[
"reshape",
"linear",
"max_pool2d",
"max_pool3d",
"adaptive_avg_pool2d",
"backward",
],
)
def test_mkldnn_composite(self, symbolic, active, op):
if not active and TEST_WITH_TORCHDYNAMO:
self.skipTest(
"Dynamo cannot guard explicit MKLDNN fakes outside their mode"
)
if op == "backward" and TEST_WITH_TORCHDYNAMO:
self.skipTest("Compiled autograd does not support explicit MKLDNN fakes")
if torch._functorch.config.fake_tensor_propagate_real_tensors:
self.skipTest("Propagate real tensor not supported")
with FakeTensorMode() as fake_mode:
x = fake_mode.from_tensor(torch.randn(1, 2, 8, 8).to_mkldnn())
y = torch.nn.functional.max_pool2d(x, 3, 2, 1)
self.assertTrue(is_fake_tensor(y))
self.assertEqual(y.shape, (1, 2, 4, 4))
shape = (2, 3, 8, 8, 8) if op == "max_pool3d" else (2, 3, 8, 8)
real = torch.randn(shape).to_mkldnn().requires_grad_(op == "backward")
real_weight = torch.randn(4, 8) if op == "linear" else None
fake_mode = FakeTensorMode(shape_env=ShapeEnv() if symbolic else None)
with fake_mode:
x = fake_mode.from_tensor(real)
weight = fake_mode.from_tensor(real_weight) if op == "linear" else None
with fake_mode if active else contextlib.nullcontext():
if op == "reshape":
y = x.reshape(x.shape[0], -1)
expected_shape = (2, 192)
self.assertEqual(y.untyped_storage()._cdata, x.untyped_storage()._cdata)
elif op == "linear":
y = torch.nn.functional.linear(x, weight)
expected_shape = (2, 3, 8, 4)
elif op == "max_pool2d":
y = torch.nn.functional.max_pool2d(x, 3, 2, 1)
expected_shape = (2, 3, 4, 4)
elif op == "max_pool3d":
if symbolic:
# Python's Meta kernel reads the MKLDNN property in new_empty.
with self.assertRaisesRegex(
UnsupportedOperatorException, "aten.max_pool3d_with_indices"
):
torch.nn.functional.max_pool3d(x, 3, 2, 1)
return
y = torch.nn.functional.max_pool3d(x, 3, 2, 1)
expected_shape = (2, 3, 4, 4, 4)
elif op == "adaptive_avg_pool2d":
y = torch.nn.functional.adaptive_avg_pool2d(x, (4, 4))
expected_shape = (2, 3, 4, 4)
else:
x.relu().to_dense().sum().backward()
y = x.grad
expected_shape = shape
self.assertTrue(is_fake_tensor(y))
self.assertEqual(y.shape, expected_shape)
self.assertFalse(y.is_mkldnn)
self.assertEqual(y.layout, torch.strided)
def test_compare_tensor_meta_unbacked_numel(self):
from torch.fx.experimental.symbolic_shapes import _constrain_range_for_size
+15 -18
View File
@@ -1784,19 +1784,17 @@ def has_meta(func: OpOverload) -> bool:
def maybe_to_dense_mkldnn(
fake_mode: FakeTensorMode,
a: FakeTensor,
fake_mode: FakeTensorMode | CppFakeTensorMode,
a: torch.Tensor,
dtype: torch.dtype | None = None,
masked_grad: bool | None = None,
) -> object:
# this function invokes in_kernel_invocation_manager and creates python
# FakeTensor, revisit later for C++ behaviour
if (
not isinstance(a, FakeTensor) # noqa: ISINSTANCE_FAKE_TENSOR
or not a.is_mkldnn
):
if not is_fake_tensor(a) or not a.is_mkldnn:
return NotImplemented
fake_device = maybe_get_fake_device(a)
if fake_device is None:
raise AssertionError("expected a fake tensor device")
out_dtype = dtype if dtype is not None else a.dtype
if a.dtype not in _MKLDNN_TO_DENSE_DTYPES:
raise RuntimeError(
@@ -1821,7 +1819,9 @@ def maybe_to_dense_mkldnn(
dtype=out_dtype,
device="meta",
)
return FakeTensor(fake_mode, out, a.fake_device)
return fake_mode.fake_tensor_converter.from_meta_and_device(
fake_mode, out, fake_device
)
class _ToDenseMkldnn(torch.autograd.Function):
@@ -1973,17 +1973,12 @@ aten.to_dense.default.py_impl(torch._C.DispatchKey.CompositeImplicitAutograd)(
@register_op_impl(aten.to_mkldnn.default)
def to_mkldnn(
fake_mode: FakeTensorMode,
fake_mode: FakeTensorMode | CppFakeTensorMode,
func: OpOverload,
a: FakeTensor,
a: torch.Tensor,
dtype: torch.dtype | None = None,
) -> object:
if not isinstance(a, FakeTensor): # noqa: ISINSTANCE_FAKE_TENSOR
if is_fake_tensor(a):
raise NotImplementedError(
"The C++ fake tensor fallback does not support mkldnn tensors yet, "
f"found in {func.name()}"
)
if not is_fake_tensor(a):
return NotImplemented
fake_device = maybe_get_fake_device(a)
@@ -2019,7 +2014,9 @@ def to_mkldnn(
dtype=out_dtype,
device="meta",
)
return FakeTensor(fake_mode, out, fake_device, dispatch_keys=_MKLDNN_DISPATCH_KEYS)
return fake_mode.fake_tensor_converter.from_meta_and_device(
fake_mode, out, fake_device, dispatch_keys=_MKLDNN_DISPATCH_KEYS
)
# These are for the `torch._foreach_...` ops like `torch._foreach_add`.
+5 -4
View File
@@ -367,10 +367,9 @@ def is_fake(x: object) -> TypeGuard[Tensor]:
# maybe_get_fake_mode returns a CppFakeTensorMode for a C++ fake or a wrapper of
# one, active_fake_mode while a C++ mode is installed, and detect_fake_mode in
# either case. CppFakeTensorMode is not a FakeTensorMode subclass and has little of
# its interface yet. Its from_tensor does not support MKLDNN tensors or
# tensors with nested ints. Its converter's from_real_tensor rejects constants
# that are fakes or not plain tensors. Its from_meta_and_device raises
# NotImplementedError
# its interface yet. Its from_tensor does not support tensors with nested ints.
# Its converter's from_real_tensor rejects constants that are fakes or not plain
# tensors, and its converter's from_meta_and_device raises NotImplementedError
# for a pytype other than Tensor. So a caller that needs a Python
# FakeTensorMode narrows the result with
# torch._guards.require_python_fake_mode, which raises RuntimeError for a C++
@@ -530,6 +529,8 @@ def maybe_clear_fake_constant(x: object) -> None:
def maybe_set_fake_dispatch_keys(x: object, keys: torch.DispatchKeySet | None) -> None:
if isinstance(x, FakeTensor): # noqa: ISINSTANCE_FAKE_TENSOR
x.dispatch_keys = keys
elif isinstance(x, Tensor) and torch._C._is_fake_tensor(x):
torch._C._set_fake_dispatch_keys(x, keys)
def maybe_get_fake_dispatch_keys(x: object) -> torch.DispatchKeySet | None:
-1
View File
@@ -1725,7 +1725,6 @@ class MetaConverter(Generic[_TensorT]):
sizes, strides, dtype=t.dtype, device="meta"
)
)
require_python_fake_mode(maybe_get_fake_mode(r))
if self.copy_data:
with torch.no_grad(), no_dispatch():
if t.size is None:
+13 -2
View File
@@ -3174,13 +3174,23 @@ static PyObject* THPVariable_is_sparse_csr(THPVariable* self, void* unused) {
END_HANDLE_TH_ERRORS
}
// Match FakeTensor's Python properties without changing native layout queries.
static bool is_fake_mkldnn(const at::Tensor& tensor) {
if (!tensor.is_fake()) {
return false;
}
const auto keys = tensor.unsafeGetTensorImpl()->fake_dispatch_keys();
return keys.has_value() && keys->has_all(c10::mkldnn_ks);
}
static PyObject* THPVariable_is_mkldnn(THPVariable* self, void* unused) {
HANDLE_TH_ERRORS
if (has_torch_function((PyObject*)self)) {
return handle_torch_function_getter(self, "is_mkldnn");
}
auto& self_ = THPVariable_Unpack(self);
return torch::autograd::utils::wrap(self_.is_mkldnn());
return torch::autograd::utils::wrap(
self_.is_mkldnn() || is_fake_mkldnn(self_));
END_HANDLE_TH_ERRORS
}
@@ -3280,7 +3290,8 @@ static PyObject* THPVariable_layout(THPVariable* self, void* unused) {
return handle_torch_function_getter(self, "layout");
}
auto& self_ = THPVariable_Unpack(self);
return torch::autograd::utils::wrap(self_.layout());
return torch::autograd::utils::wrap(
is_fake_mkldnn(self_) ? at::kMkldnn : self_.layout());
END_HANDLE_TH_ERRORS
}