Apply one precedence rule to all forwarded build env vars (#201666)

Go through the following checklist
- [x] Passes lint (`spin fixlint`)
- [x] Added/updated tests
- [x] Updated documentation (if applicable)
- [ ] Included benchmark results (for PRs impacting perf)

---

## Linked issue or supporting maintainer

Part of #201664

## BC-breaking?

No. One behavior change for existing build directories: a `BLAS`, `MKL_THREADING` or `CUDNN_*` value exported in the shell now re-applies on every build and overwrites a ccmake edit of that entry, as `USE_*` and `BUILD_*` already did since #188242.

## Summary (human written only)

This PR homogenizes the treatment of configure options for the Pytorch build.
It is part of #201664, implementing most of the necessary changes that are not covered in other PRs or dependent on future changes.

## Agent details (if applicable)

Drafted with Claude Code; diff and commit message reviewed by hand. The commit message follows.

cmake/EnvVarForwarding.cmake forwarded its variables with two different semantics. Names matched by prefix (BUILD_*, USE_*, CMAKE_*) have overridden the cache on every configure since #188242, so changing USE_CUDA in the environment takes effect on the next build. The names in the passthrough and alias lists (BLAS, TORCH_CUDA_ARCH_LIST, CUDNN_*, MKL_THREADING, ...) still used the original seed-only loop, guarded by NOT DEFINED, so they were written once and then ignored: changing BLAS in the environment did nothing to an existing build directory, and -D beat the environment for these names while losing to it for the prefixed ones.

This change routes the passthrough names and all aliases through the same _envfwd_apply the prefix names use, so one rule holds for every knob: environment variables win over -D / cmake.define, those win over pyproject.toml, and all three win over what the build directory remembers. For developers this means a BLAS, MKL_THREADING or CUDNN_* value exported in the shell now re-applies on every build, overwriting a ccmake edit of that entry, exactly as USE_* and BUILD_* already did. The _ENV_ALIASES and _LOW_PRIORITY_ALIASES lists collapse into one, with a single semantics: an alias applies only when its canonical name is not itself in the environment, and where several aliases name the same variable the first listed one wins. Among the aliases, the only observable difference is that a canonical name now beats its alias when both are set (CUDNN_LIBRARY over CUDNN_LIB_DIR); no CI job sets both. As a consequence of going through _envfwd_apply, the passthrough and alias names are now also recorded in env_forwarded.txt, so the native-AOT stage-2 drift check covers them, as its docstring already claimed.

The alternative, restoring flags-over-environment for the prefixed names instead, was rejected: scikit-build-core resolves its own and every declared config-setting as environment > -C > default, and a project cannot invert that, so flags-over-environment could not have been delivered uniformly. #200651 binds torch.blas and torch.cuda-arch-list to these names and documents that the bound environment variable takes precedence over the setting; that sentence is only true under this rule, so that PR rebases on this one.

The README "Adjust Build Options" and CONTRIBUTING "subsequent builds" paragraphs stated the old seed-only behavior and now state the rule; the EnvVarForwarding header carries it too.

Test Plan

tools/test/test_env_var_forwarding.py configures a NONE-language CMake project that includes the module and asserts the four tiers for a prefix name and for BLAS (environment over -D, environment over cache, cache kept when the environment is silent, -D over cache), plus canonical-over-alias, alias-over-cache, first-listed-alias-wins and the drift-file recording. Against the unmodified module four of the eight tests fail, exactly the BLAS and alias cases.

```
python -m pytest tools/test/test_env_var_forwarding.py -q
lintrunner -a cmake/EnvVarForwarding.cmake tools/test/test_env_var_forwarding.py tools/native_aot/build_stage2.py README.md CONTRIBUTING.md
```

Every commit that touched the module since its introduction was traced to its PR and checked against the new semantics; each CMake-side write to a forwarded variable is either a non-FORCE default or a normal-variable shadow, both of which behave the same under either semantics.

Authored with an AI assistant; the diff was reviewed by hand.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Pull Request resolved: https://github.com/pytorch/pytorch/pull/201666
Approved by: https://github.com/atalman
This commit is contained in:
Klaus Zimmermann
2026-11-09 15:59:25 +00:00
committed by PyTorch MergeBot
parent 02a1dc067a
commit 8a9a9e737d
5 changed files with 181 additions and 50 deletions
+7 -4
View File
@@ -997,10 +997,13 @@ BUILD_CONFIG USE_CUDA=1 pip install --no-build-isolation -v -e .
BUILD_CONFIG USE_CUDA=1 USE_DISTRIBUTED=1 pip install --no-build-isolation -v -e .
```
For subsequent builds (i.e., when `build/CMakeCache.txt` exists), the build
options passed for the first time will persist; please run `ccmake build/`, run
`cmake-gui build/`, or directly edit `build/CMakeCache.txt` to adapt build
options.
For subsequent builds (i.e., when `build/CMakeCache.txt` exists), an option
that is no longer in the environment keeps the value the cache remembers from
the first build, while a variable that is in the environment is re-applied on
every build. Environment variables win over `-C` flags, `-C` flags win over
`pyproject.toml`, and all three win over the cache. To change a cached option
without setting its variable, run `ccmake build/`, run `cmake-gui build/`, or
directly edit `build/CMakeCache.txt`.
### Profile build time
+5 -4
View File
@@ -396,10 +396,11 @@ MACOSX_DEPLOYMENT_TARGET=11.0 spin develop
After a first `spin develop`, you can inspect or adjust the CMake cache interactively with
`ccmake build` (or `cmake-gui build`) and rebuild with `spin develop`; your edits persist, because
reconfiguration does not use `--fresh` and environment variables only seed cache entries that are
not already set. The corollary: once a variable is in the cache, changing its environment variable
no longer affects it -- edit the cache directly (or delete `build/CMakeCache.txt`) to change it.
The build type and compiler are the exception; the build always re-applies them.
reconfiguration does not use `--fresh`. Environment variables win over `-C` flags, `-C` flags win
over `pyproject.toml`, and all three win over what the build directory remembers from last time:
a forwarded environment variable is re-applied on every build, so a cache edit only survives for
variables the environment and the flags leave alone. To change one setting for a single command,
put the variable in front of the command.
### Docker Image
+29 -31
View File
@@ -3,9 +3,16 @@
# Forwarding rule: an environment variable reaches CMake (as a cache variable of
# the same name) if it (a) starts with BUILD_, USE_, or CMAKE_, (b) ends with
# EXITCODE or EXITCODE__TRYRUN_OUTPUT, or (c) appears in the _ENV_ALIASES /
# _ENV_PASSTHROUGH / _LOW_PRIORITY_ALIASES lists below. Anything else is not
# _ENV_PASSTHROUGH lists below. Anything else is not
# forwarded -- set it as a CMake option with -D / cmake.define instead.
#
# Precedence: a forwarded variable overrides the cache on every configure, so the
# environment wins over -D / cmake.define, which win over pyproject.toml, and all
# three win over what the build directory remembers from the last configure. To
# change one setting for a single command, put the variable in front of the
# command. An alias applies only when its canonical name is not in the
# environment itself.
#
# ============================================================================
# Build environment variable reference. Vars below that match the forwarding
# rule are passed through by this module; the rest are handled where noted.
@@ -128,12 +135,6 @@
# USE_NINJA select the generator via CMAKE_GENERATOR=Ninja instead
# (ninja is the default when available).
# Additional env vars that are forwarded with a different CMake variable name.
set(_ENV_ALIASES
"CUDNN_LIB_DIR=CUDNN_LIBRARY"
"USE_CUDA_STATIC_LINK=CAFFE2_STATIC_LINK_CUDA"
)
# Additional env vars forwarded with the same name.
set(_ENV_PASSTHROUGH
UBSAN_FLAGS
@@ -172,8 +173,11 @@ set(_ENV_PASSTHROUGH
PYTHON_SIX_SOURCE_DIR
)
# Low-priority aliases: if the canonical var is not set, use the alias.
set(_LOW_PRIORITY_ALIASES
# Env vars forwarded under a different CMake name ("env name=cmake name"). Where
# several aliases name the same variable, the first listed one wins.
set(_ENV_ALIASES
"CUDNN_LIB_DIR=CUDNN_LIBRARY"
"USE_CUDA_STATIC_LINK=CAFFE2_STATIC_LINK_CUDA"
"CUDA_HOST_COMPILER=CMAKE_CUDA_HOST_COMPILER"
"CUDAHOSTCXX=CUDA_HOST_COMPILER"
"CMAKE_CUDA_HOST_COMPILER=CUDA_HOST_COMPILER"
@@ -181,23 +185,6 @@ set(_LOW_PRIORITY_ALIASES
"CUDACXX=CUDA_NVCC_EXECUTABLE"
)
# Forward aliased env vars (env name -> different cmake name)
foreach(_alias IN LISTS _ENV_ALIASES)
string(REPLACE "=" ";" _parts "${_alias}")
list(GET _parts 0 _env_name)
list(GET _parts 1 _cmake_name)
if(DEFINED ENV{${_env_name}} AND NOT DEFINED ${_cmake_name})
set(${_cmake_name} "$ENV{${_env_name}}" CACHE STRING "From env ${_env_name}" FORCE)
endif()
endforeach()
# Forward passthrough env vars (same name)
foreach(_var IN LISTS _ENV_PASSTHROUGH)
if(DEFINED ENV{${_var}} AND NOT DEFINED ${_var})
set(${_var} "$ENV{${_var}}" CACHE STRING "From env ${_var}" FORCE)
endif()
endforeach()
# Forward all BUILD_*, USE_*, CMAKE_* environment variables (plus names ending
# in EXITCODE / EXITCODE__TRYRUN_OUTPUT) into the CMake cache, mirroring the -D
# flags setup.py used to pass.
@@ -288,14 +275,25 @@ if(NOT _envfwd_rc EQUAL 0)
endif()
cmake_language(EVAL CODE "${_envfwd_code}")
# Low-priority aliases
foreach(_alias IN LISTS _LOW_PRIORITY_ALIASES)
# Passthrough names, same semantics as the prefix names above.
foreach(_var IN LISTS _ENV_PASSTHROUGH)
if(DEFINED ENV{${_var}})
_envfwd_apply("${_var}" "$ENV{${_var}}")
endif()
endforeach()
# Aliases apply with the same semantics, but only when the canonical name is not
# in the environment itself (the prefix and passthrough passes above already
# applied it), and the first listed alias of a name wins over later ones.
set(_envfwd_aliased "")
foreach(_alias IN LISTS _ENV_ALIASES)
string(REPLACE "=" ";" _parts "${_alias}")
list(GET _parts 0 _env_name)
list(GET _parts 1 _cmake_name)
if(DEFINED ENV{${_env_name}} AND NOT DEFINED ${_cmake_name})
_envfwd_note("${_cmake_name}")
set(${_cmake_name} "$ENV{${_env_name}}" CACHE STRING "From env alias ${_env_name}" FORCE)
if(DEFINED ENV{${_env_name}} AND NOT DEFINED ENV{${_cmake_name}}
AND NOT "${_cmake_name}" IN_LIST _envfwd_aliased)
list(APPEND _envfwd_aliased "${_cmake_name}")
_envfwd_apply("${_cmake_name}" "$ENV{${_env_name}}")
endif()
endforeach()
+10 -11
View File
@@ -153,13 +153,12 @@ def _report_probe_failure(expr: str, stderr: str, returncode: int) -> None:
def _arch_list() -> str:
"""The arch list THIS build targets, as the CONFIGURE resolved it.
Read from the file cmake/Codegen.cmake records, because that is the only place
the resolved value exists: EnvVarForwarding forwards the environment into the
cache only when the variable is undefined (so -D wins), and Dependencies.cmake
then shadows the cache with the environment for the rest of the configure. Either
source read alone is therefore wrong for one of the two kinds of build. Reading
the record also means a developer who changes the environment without
reconfiguring gets the list the build COMPILED for, not the one they now intend.
Read from the file cmake/Codegen.cmake records, because that is what the
configure resolved: EnvVarForwarding FORCEs a set environment variable into the
cache (the environment wins over -D), and an unset one leaves the cache value,
from -D or an earlier configure, in effect. Reading the record means a developer
who changes the environment without reconfiguring gets the list the build
COMPILED for, not the one they now intend.
Recorded-but-empty is not the same as no record: it means the configure resolved
no arch list, and the on-device path takes over. Absent means nothing configured
@@ -573,10 +572,10 @@ def _refuse_settings_drift(before: dict[str, str] | None) -> None:
"""Refuse a reconfigure that changed a setting the environment forwards.
EnvVarForwarding FORCEs every BUILD_*/USE_*/CMAKE_* environment variable (and the
names in its alias lists, TORCH_CUDA_ARCH_LIST among them) into the cache, so a
stage-2 run in another environment than the build silently relinks torch_cuda
against other settings. Those variables are the whole surface: nothing else here
can change the configuration.
names in its passthrough and alias lists, TORCH_CUDA_ARCH_LIST among them) into
the cache, so a stage-2 run in another environment than the build silently
relinks torch_cuda against other settings. Those variables are the whole surface:
nothing else here can change the configuration.
"""
after = _forwarded_settings()
if after is None:
+130
View File
@@ -0,0 +1,130 @@
"""Precedence of cmake/EnvVarForwarding.cmake, measured with a real configure.
The rule under test: a forwarded environment variable overrides the cache on
every configure (environment > -D > cache), for prefix names, passthrough names
and aliases alike, with an alias yielding to its canonical name.
"""
from __future__ import annotations
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
REPO = Path(__file__).resolve().parents[2]
MODULE = REPO / "cmake" / "EnvVarForwarding.cmake"
PREFIX_NAME = "USE_ENVFWD_PROBE"
PASSTHROUGH_NAME = "BLAS"
PROBED = [
PREFIX_NAME,
PASSTHROUGH_NAME,
"CUDA_NVCC_EXECUTABLE",
"CUDA_HOST_COMPILER",
"CMAKE_CUDA_HOST_COMPILER",
]
# Names the tests set; scrubbed from the ambient environment so a developer's
# shell cannot leak into a configure.
SCRUBBED = PROBED + ["CUDACXX", "CUDAHOSTCXX", "CMAKE_CUDA_COMPILER"]
CMAKELISTS = """
cmake_minimum_required(VERSION 3.27)
project(envfwd NONE)
set(Python_EXECUTABLE "{python}")
include("{module}")
file(WRITE "${{CMAKE_BINARY_DIR}}/resolved.txt" {lines})
"""
@unittest.skipUnless(shutil.which("cmake"), "cmake not found")
class TestEnvVarForwardingPrecedence(unittest.TestCase):
def setUp(self) -> None:
tmp = tempfile.TemporaryDirectory(prefix="envfwd-")
self.addCleanup(tmp.cleanup)
self.tmp = Path(tmp.name)
self.src = self.tmp / "src"
self.build = self.tmp / "build"
self.src.mkdir()
lines = " ".join(f'"{n}=${{{n}}}\\n"' for n in PROBED)
(self.src / "CMakeLists.txt").write_text(
CMAKELISTS.format(
python=Path(sys.executable).as_posix(),
module=MODULE.as_posix(),
lines=lines,
)
)
def configure(
self, env: dict[str, str] | None = None, defines: dict[str, str] | None = None
) -> dict[str, str]:
full_env = {k: v for k, v in os.environ.items() if k not in SCRUBBED}
full_env.update(env or {})
cmd = ["cmake", "-S", str(self.src), "-B", str(self.build)]
cmd += [f"-D{k}={v}" for k, v in (defines or {}).items()]
subprocess.run(cmd, env=full_env, check=True, capture_output=True, text=True)
resolved = (self.build / "resolved.txt").read_text().splitlines()
return dict(line.split("=", 1) for line in resolved)
def forwarded_names(self) -> list[str]:
lines = (self.build / "env_forwarded.txt").read_text().splitlines()
return [line.split("=", 1)[0] for line in lines]
def test_environment_wins_over_flag(self) -> None:
for name in (PREFIX_NAME, PASSTHROUGH_NAME):
with self.subTest(name=name):
got = self.configure(env={name: "env"}, defines={name: "flag"})
self.assertEqual(got[name], "env")
def test_environment_reapplies_over_cache(self) -> None:
for name in (PREFIX_NAME, PASSTHROUGH_NAME):
with self.subTest(name=name):
self.configure(env={name: "first"})
got = self.configure(env={name: "second"})
self.assertEqual(got[name], "second")
def test_cache_keeps_value_when_environment_is_silent(self) -> None:
for name in (PREFIX_NAME, PASSTHROUGH_NAME):
with self.subTest(name=name):
self.configure(env={name: "first"})
got = self.configure()
self.assertEqual(got[name], "first")
def test_flag_updates_cache_when_environment_is_silent(self) -> None:
for name in (PREFIX_NAME, PASSTHROUGH_NAME):
with self.subTest(name=name):
self.configure(env={name: "first"})
got = self.configure(defines={name: "flag"})
self.assertEqual(got[name], "flag")
def test_alias_yields_to_canonical_name(self) -> None:
got = self.configure(env={"CUDACXX": "alias", "CUDA_NVCC_EXECUTABLE": "canon"})
self.assertEqual(got["CUDA_NVCC_EXECUTABLE"], "canon")
def test_alias_reapplies_over_cache(self) -> None:
self.configure(env={"CUDACXX": "first"})
got = self.configure(env={"CUDACXX": "second"})
self.assertEqual(got["CUDA_NVCC_EXECUTABLE"], "second")
def test_first_listed_alias_wins(self) -> None:
# CUDAHOSTCXX precedes CMAKE_CUDA_HOST_COMPILER in _ENV_ALIASES for
# CUDA_HOST_COMPILER; the latter is still forwarded under its own name.
env = {"CUDAHOSTCXX": "one", "CMAKE_CUDA_HOST_COMPILER": "two"}
got = self.configure(env=env)
self.assertEqual(got["CUDA_HOST_COMPILER"], "one")
self.assertEqual(got["CMAKE_CUDA_HOST_COMPILER"], "two")
def test_forwarded_names_are_recorded(self) -> None:
self.configure(env={PREFIX_NAME: "x", PASSTHROUGH_NAME: "y", "CUDACXX": "z"})
names = self.forwarded_names()
self.assertIn(PREFIX_NAME, names)
self.assertIn(PASSTHROUGH_NAME, names)
self.assertIn("CUDA_NVCC_EXECUTABLE", names)
if __name__ == "__main__":
unittest.main()