mirror of
https://github.com/pytorch/pytorch.git
synced 2026-11-20 09:04:25 +08:00
19da3da3d3
The fix-cpp-build branch is a major restructuring that: - Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives - Reorganizes notes into a hierarchical API reference with proper categories - Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs - Always runs Doxygen instead of gating on an env var Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096 Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
5.4 KiB
5.4 KiB
myst
| myst | ||||||
|---|---|---|---|---|---|---|
|
Activation Functions
Activation functions introduce non-linearity into neural networks, allowing them to learn complex patterns. Without activations, stacked linear layers would collapse into a single linear transformation.
Common choices:
- ReLU family (ReLU, LeakyReLU, PReLU, RReLU): Fast, widely used, good default choice
- ELU family (ELU, SELU, CELU): Smoother than ReLU, can produce negative outputs
- GELU/SiLU/Mish: Modern activations popular in transformers and advanced architectures
- Sigmoid/Tanh: Classic activations, useful for output layers (probabilities, bounded outputs)
- Softmax: Converts logits to probability distribution (classification output)
ReLU
:members:
:undoc-members:
:members:
:undoc-members:
Example:
auto relu = torch::nn::ReLU(torch::nn::ReLUOptions().inplace(true));
LeakyReLU
:members:
:undoc-members:
:members:
:undoc-members:
PReLU
:members:
:undoc-members:
:members:
:undoc-members:
RReLU
:members:
:undoc-members:
:members:
:undoc-members:
ReLU6
Like ReLU but caps the output at 6: min(max(0, x), 6). Commonly used in
mobile architectures (MobileNet).
:members:
:undoc-members:
:members:
:undoc-members:
GLU
Gated Linear Unit. Splits the input tensor in half along a dimension,
then applies a * sigmoid(b).
:members:
:undoc-members:
:members:
:undoc-members:
LogSigmoid
Applies element-wise log(sigmoid(x)). Numerically more stable than
computing log and sigmoid separately.
:members:
:undoc-members:
:members:
:undoc-members:
ELU
:members:
:undoc-members:
:members:
:undoc-members:
SELU
:members:
:undoc-members:
:members:
:undoc-members:
CELU
:members:
:undoc-members:
:members:
:undoc-members:
GELU
:members:
:undoc-members:
:members:
:undoc-members:
SiLU (Swish)
:members:
:undoc-members:
:members:
:undoc-members:
Mish
:members:
:undoc-members:
:members:
:undoc-members:
Sigmoid
:members:
:undoc-members:
:members:
:undoc-members:
Tanh
:members:
:undoc-members:
:members:
:undoc-members:
Softmax
:members:
:undoc-members:
:members:
:undoc-members:
Example:
auto softmax = torch::nn::Softmax(torch::nn::SoftmaxOptions(/*dim=*/1));
Softmax2d
Applies Softmax over features to each spatial location in a 4D input
tensor of shape (N, C, H, W).
:members:
:undoc-members:
:members:
:undoc-members:
LogSoftmax
:members:
:undoc-members:
:members:
:undoc-members:
Softmin
:members:
:undoc-members:
:members:
:undoc-members:
Softplus
:members:
:undoc-members:
:members:
:undoc-members:
Softshrink
:members:
:undoc-members:
:members:
:undoc-members:
Softsign
:members:
:undoc-members:
:members:
:undoc-members:
Hardshrink
:members:
:undoc-members:
:members:
:undoc-members:
Hardtanh
:members:
:undoc-members:
:members:
:undoc-members:
Tanhshrink
:members:
:undoc-members:
:members:
:undoc-members:
Threshold
:members:
:undoc-members:
:members:
:undoc-members: