mirror of
https://github.com/pytorch/pytorch.git
synced 2026-11-20 09:04:25 +08:00
19da3da3d3
The fix-cpp-build branch is a major restructuring that: - Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives - Reorganizes notes into a hierarchical API reference with proper categories - Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs - Always runs Doxygen instead of gating on an env var Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096 Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
2.6 KiB
2.6 KiB
myst
| myst | ||||||
|---|---|---|---|---|---|---|
|
Transformer Layers
Transformer layers use self-attention mechanisms to process sequences in parallel, enabling efficient training on long sequences. They are the foundation of modern NLP models (BERT, GPT) and increasingly used in vision and other domains.
- Transformer: Complete encoder-decoder architecture
- TransformerEncoder/Decoder: Standalone encoder or decoder stacks
- TransformerEncoderLayer/DecoderLayer: Individual transformer blocks
- MultiheadAttention: Core attention mechanism used throughout
Key parameters:
d_model: Dimension of the model (embedding dimension)nhead: Number of attention headsnum_encoder_layers/num_decoder_layers: Number of stacked layersdim_feedforward: Dimension of feedforward networkdropout: Dropout rate for regularization
Transformer
Complete encoder-decoder transformer architecture.
:members:
:undoc-members:
:members:
:undoc-members:
Example:
auto transformer = torch::nn::Transformer(
torch::nn::TransformerOptions()
.d_model(512)
.nhead(8)
.num_encoder_layers(6)
.num_decoder_layers(6)
.dim_feedforward(2048)
.dropout(0.1));
TransformerEncoder
Stack of encoder layers for processing source sequences.
:members:
:undoc-members:
:members:
:undoc-members:
TransformerDecoder
Stack of decoder layers for generating target sequences.
:members:
:undoc-members:
:members:
:undoc-members:
TransformerEncoderLayer
Single encoder layer with self-attention and feedforward network.
:members:
:undoc-members:
TransformerDecoderLayer
Single decoder layer with self-attention, cross-attention, and feedforward network.
:members:
:undoc-members:
MultiheadAttention
Scaled dot-product attention with multiple parallel heads.
:members:
:undoc-members:
:members:
:undoc-members: