Files
pytorch/docs/cpp/source/api/nn/activation.md
T
Svetlana Karslioglu 19da3da3d3 Restructure C++ docs (#174096)
The fix-cpp-build branch is a major restructuring that:

- Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives
- Reorganizes notes into a hierarchical API reference with proper categories
- Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs
- Always runs Doxygen instead of gating on an env var

Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096
Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
2026-04-07 17:24:38 +00:00

5.4 KiB

myst
myst
html_meta
description keywords
Activation functions in PyTorch C++ — ReLU, GELU, Sigmoid, Softmax, and more torch::nn activation modules. PyTorch, C++, activation, ReLU, GELU, Sigmoid, Softmax, LeakyReLU, ELU, Mish

Activation Functions

Activation functions introduce non-linearity into neural networks, allowing them to learn complex patterns. Without activations, stacked linear layers would collapse into a single linear transformation.

Common choices:

  • ReLU family (ReLU, LeakyReLU, PReLU, RReLU): Fast, widely used, good default choice
  • ELU family (ELU, SELU, CELU): Smoother than ReLU, can produce negative outputs
  • GELU/SiLU/Mish: Modern activations popular in transformers and advanced architectures
  • Sigmoid/Tanh: Classic activations, useful for output layers (probabilities, bounded outputs)
  • Softmax: Converts logits to probability distribution (classification output)

ReLU

:members:
:undoc-members:
:members:
:undoc-members:

Example:

auto relu = torch::nn::ReLU(torch::nn::ReLUOptions().inplace(true));

LeakyReLU

:members:
:undoc-members:
:members:
:undoc-members:

PReLU

:members:
:undoc-members:
:members:
:undoc-members:

RReLU

:members:
:undoc-members:
:members:
:undoc-members:

ReLU6

Like ReLU but caps the output at 6: min(max(0, x), 6). Commonly used in mobile architectures (MobileNet).

:members:
:undoc-members:
:members:
:undoc-members:

GLU

Gated Linear Unit. Splits the input tensor in half along a dimension, then applies a * sigmoid(b).

:members:
:undoc-members:
:members:
:undoc-members:

LogSigmoid

Applies element-wise log(sigmoid(x)). Numerically more stable than computing log and sigmoid separately.

:members:
:undoc-members:
:members:
:undoc-members:

ELU

:members:
:undoc-members:
:members:
:undoc-members:

SELU

:members:
:undoc-members:
:members:
:undoc-members:

CELU

:members:
:undoc-members:
:members:
:undoc-members:

GELU

:members:
:undoc-members:
:members:
:undoc-members:

SiLU (Swish)

:members:
:undoc-members:
:members:
:undoc-members:

Mish

:members:
:undoc-members:
:members:
:undoc-members:

Sigmoid

:members:
:undoc-members:
:members:
:undoc-members:

Tanh

:members:
:undoc-members:
:members:
:undoc-members:

Softmax

:members:
:undoc-members:
:members:
:undoc-members:

Example:

auto softmax = torch::nn::Softmax(torch::nn::SoftmaxOptions(/*dim=*/1));

Softmax2d

Applies Softmax over features to each spatial location in a 4D input tensor of shape (N, C, H, W).

:members:
:undoc-members:
:members:
:undoc-members:

LogSoftmax

:members:
:undoc-members:
:members:
:undoc-members:

Softmin

:members:
:undoc-members:
:members:
:undoc-members:

Softplus

:members:
:undoc-members:
:members:
:undoc-members:

Softshrink

:members:
:undoc-members:
:members:
:undoc-members:

Softsign

:members:
:undoc-members:
:members:
:undoc-members:

Hardshrink

:members:
:undoc-members:
:members:
:undoc-members:

Hardtanh

:members:
:undoc-members:
:members:
:undoc-members:

Tanhshrink

:members:
:undoc-members:
:members:
:undoc-members:

Threshold

:members:
:undoc-members:
:members:
:undoc-members: