diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 06c3f32abd5e..cec3ea2f2a00 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -51,6 +51,8 @@ Speed up builds: - Install `ninja` (`pip install ninja`) for faster builds - Use `ccache` for incremental compilation caching +Full list of build environment variables: [`cmake/EnvVarForwarding.cmake`](../cmake/EnvVarForwarding.cmake) + Rebuild specific targets: `(cd build && ninja )` ### Testing diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dd951abdd597..e99eea57a171 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -134,10 +134,12 @@ Follow the instructions for [installing PyTorch from source](https://github.com/ python -m pip install --group dev python -m pip install --no-build-isolation -v -e . ``` - 4. The main step within `python -m pip install -e . -v --no-build-isolation` is running `make` from the `build` directory. If you want to - experiment with some environment variables, you can pass them into the command: + 4. The main step within `python -m pip install -e . -v --no-build-isolation` is the CMake build in the + `build` directory (`ninja` by default). If you want to experiment with some environment variables, you + can pass them into the command (see + [`cmake/EnvVarForwarding.cmake`](./cmake/EnvVarForwarding.cmake) for the environment variables the build forwards): ```bash - ENV_KEY1=ENV_VAL1[, ENV_KEY2=ENV_VAL2]* CMAKE_FRESH=1 python -m pip install --no-build-isolation -v -e . + ENV_KEY1=ENV_VAL1[, ENV_KEY2=ENV_VAL2]* python -m pip install --no-build-isolation -v -e . ``` 5. Try installing PyTorch without build isolation by adding `--no-build-isolation` to the `pip install` command. This will use the current environment's packages instead of creating a new isolated environment for the build. @@ -798,6 +800,9 @@ On the initial build, you can also speed things up by disabling the features you - `USE_CPU_VECTORIZATION=0` will disable building vectorized CPU kernel variants (AVX2, AVX512, VSX, ZVECTOR, SVE). Only the scalar DEFAULT kernels are built. Fine for correctness/dispatch work; not for CPU benchmarking. - `USE_COLORIZE_OUTPUT=1` will colorize compiler output for easier reading. +The full list of build environment variables, what each one does, and how it reaches CMake is +documented at the top of [`cmake/EnvVarForwarding.cmake`](./cmake/EnvVarForwarding.cmake). + For example, a good default for the most minimal build is to add to your bashrc is: ```bash alias BUILD_CONFIG='CMAKE_GENERATOR=Ninja USE_DISTRIBUTED=0 USE_FLASH_ATTENTION=0 USE_MEM_EFF_ATTENTION=0 USE_MKLDNN=0 USE_CUDA=0 BUILD_TEST=0 USE_FBGEMM=0 USE_NNPACK=0 USE_XNNPACK=0 BUILD_LAZY_TS_BACKEND=0 USE_PYTORCH_QNNPACK=0 USE_CPU_VECTORIZATION=0 USE_COLORIZE_OUTPUT=1' @@ -1075,9 +1080,14 @@ Set `TORCH_SHOW_CPP_STACKTRACES=1` to get the C++ stacktrace when an error occur If you are working on the CUDA code, here are some useful CUDA debugging tips: -1. `CUDA_DEVICE_DEBUG=1` will enable CUDA device function debug symbols (`-g -G`). +1. `CUDA_DEVICE_DEBUG` will enable CUDA device function debug symbols (`-g -G`). This will be particularly helpful in debugging device code. However, it will slow down the build process for about 50% (compared to only `DEBUG=1`), so use wisely. + Unlike `DEBUG`, it is read as a CMake variable rather than forwarded from the environment, so + pass it at install time (it has no effect on MSVC builds): + ```bash + python -m pip install -e . -v --no-build-isolation -C cmake.define.CUDA_DEVICE_DEBUG=1 + ``` 2. `cuda-gdb` and `compute-sanitizer` are your best CUDA debugging friends. Unlike`gdb`, `cuda-gdb` can display actual values in a CUDA tensor (rather than all zeros). 3. CUDA supports a lot of C++17/20 features, which include `std::numeric_limits`, `std::nextafter`, diff --git a/README.md b/README.md index d32cd08ebce8..938c15dba315 100644 --- a/README.md +++ b/README.md @@ -209,7 +209,7 @@ If you want to compile with CUDA support, [select a supported version of CUDA fr Note: You could refer to the [cuDNN Support Matrix](https://docs.nvidia.com/deeplearning/cudnn/backend/latest/reference/support-matrix.html) for cuDNN versions with the various supported CUDA, CUDA driver, and NVIDIA hardware. If you want to disable CUDA support, export the environment variable `USE_CUDA=0`. -Other potentially useful environment variables are documented in `cmake/EnvVarForwarding.cmake`. If +Other potentially useful environment variables are documented in [`cmake/EnvVarForwarding.cmake`](./cmake/EnvVarForwarding.cmake). If CUDA is installed in a non-standard location, set PATH so that the nvcc you want to use can be found (e.g., `export PATH=/usr/local/cuda-12.8/bin:$PATH`). @@ -223,7 +223,7 @@ If you want to compile with ROCm support, install By default the build system expects ROCm to be installed in `/opt/rocm`. If ROCm is installed in a different directory, the `ROCM_PATH` environment variable must be set to the ROCm installation directory. The build system automatically detects the AMD GPU architecture. Optionally, the AMD GPU architecture can be explicitly set with the `PYTORCH_ROCM_ARCH` environment variable [AMD GPU architecture](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html#supported-gpus) If you want to disable ROCm support, export the environment variable `USE_ROCM=0`. -Other potentially useful environment variables are documented in `cmake/EnvVarForwarding.cmake`. +Other potentially useful environment variables are documented in [`cmake/EnvVarForwarding.cmake`](./cmake/EnvVarForwarding.cmake). ##### Intel GPU Support If you want to compile with Intel GPU support, follow these @@ -231,7 +231,7 @@ If you want to compile with Intel GPU support, follow these - Intel GPU is supported for Linux and Windows. If you want to disable Intel GPU support, export the environment variable `USE_XPU=0`. -Other potentially useful environment variables are documented in `cmake/EnvVarForwarding.cmake`. +Other potentially useful environment variables are documented in [`cmake/EnvVarForwarding.cmake`](./cmake/EnvVarForwarding.cmake). #### Get the PyTorch Source diff --git a/cmake/EnvVarForwarding.cmake b/cmake/EnvVarForwarding.cmake index a1a4fdb5a457..40f1299e59e2 100644 --- a/cmake/EnvVarForwarding.cmake +++ b/cmake/EnvVarForwarding.cmake @@ -105,6 +105,8 @@ # debug flags (may OOM nvcc). This was always a CMake # option; the setup.py comment that listed it as an env # var was inaccurate -- it was never forwarded. +# CUDA_DEVICE_DEBUG build CUDA device code with -g -G (read in +# cmake/public/cuda.cmake; no effect on MSVC) # # Removed with setup.py (no longer available): # CMAKE_FRESH force a fresh configure. Delete the build/ directory