mirror of
https://github.com/pytorch/pytorch.git
synced 2026-11-20 09:04:25 +08:00
19da3da3d3
The fix-cpp-build branch is a major restructuring that: - Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives - Reorganizes notes into a hierarchical API reference with proper categories - Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs - Always runs Doxygen instead of gating on an env var Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096 Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
1.6 KiB
1.6 KiB
myst
| myst | ||||||
|---|---|---|---|---|---|---|
|
Gradient Descent Optimizers
These optimizers use gradient descent with optional enhancements like momentum. They are the foundation of neural network training and work well when you can afford careful hyperparameter tuning.
SGD (Stochastic Gradient Descent)
The classic optimization algorithm. SGD with momentum is often the best choice for convolutional neural networks when properly tuned. While requiring more careful learning rate selection than adaptive methods, it frequently achieves the best final accuracy.
When to use:
- Training CNNs (ResNet, VGG, etc.) where you want maximum accuracy
- When you have time for hyperparameter tuning
- When combined with learning rate schedules (warmup, cosine annealing)
Key parameters:
lr: Learning rate (typical: 0.01-0.1 for CNNs)momentum: Accelerates convergence (typical: 0.9)weight_decay: L2 regularization coefficientnesterov: Use Nesterov momentum (often improves convergence)
:members:
:undoc-members:
Example:
// Standard SGD with momentum - good for CNNs
auto optimizer = torch::optim::SGD(
model->parameters(),
torch::optim::SGDOptions(0.01) // learning rate
.momentum(0.9) // momentum factor
.weight_decay(1e-4) // L2 regularization
.nesterov(true)); // Nesterov momentum