Files
pytorch/docs/cpp/source/api/optim/index.md
T
Svetlana Karslioglu 19da3da3d3 Restructure C++ docs (#174096)
The fix-cpp-build branch is a major restructuring that:

- Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives
- Reorganizes notes into a hierarchical API reference with proper categories
- Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs
- Always runs Doxygen instead of gating on an env var

Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096
Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
2026-04-07 17:24:38 +00:00

2.9 KiB

myst
myst
html_meta
description keywords
PyTorch C++ optimizer API — SGD, Adam, and other optimizers for training neural networks. PyTorch, C++, optimizer, optim, SGD, Adam, training

Optimizers (torch::optim)

The torch::optim namespace provides optimization algorithms for training neural networks. These optimizers update model parameters based on computed gradients to minimize the loss function.

When to use torch::optim:

  • When training neural networks with gradient descent
  • When you need different optimization strategies (SGD, Adam, etc.)
  • When implementing learning rate schedules

Basic usage:

#include <torch/torch.h>

// Create model and optimizer
auto model = std::make_shared<Net>();
auto optimizer = torch::optim::Adam(
    model->parameters(),
    torch::optim::AdamOptions(1e-3));

// Training loop
for (auto& batch : *data_loader) {
    optimizer.zero_grad();                    // Clear gradients
    auto loss = loss_fn(model->forward(batch.data), batch.target);
    loss.backward();                          // Compute gradients
    optimizer.step();                         // Update parameters
}

Header Files

  • torch/csrc/api/include/torch/optim.h - Main optim header
  • torch/csrc/api/include/torch/optim/optimizer.h - Optimizer base class
  • torch/csrc/api/include/torch/optim/sgd.h - SGD optimizer
  • torch/csrc/api/include/torch/optim/adam.h - Adam optimizer

Optimizer Base Class

All optimizers inherit from the Optimizer base class, which provides common functionality for parameter updates, gradient zeroing, and state management.

:members:
:undoc-members:

OptimizerOptions

:members:
:undoc-members:

OptimizerParamGroup

:members:
:undoc-members:

OptimizerParamState

:members:
:undoc-members:

Choosing an Optimizer

Selecting the right optimizer depends on your model architecture, dataset, and training requirements:

:widths: 20 40 40
:header-rows: 1

* - Optimizer
  - Best For
  - Trade-offs
* - **SGD + Momentum**
  - CNNs, well-understood problems, when you can tune hyperparameters
  - Requires careful learning rate tuning; often achieves best final accuracy
* - **Adam/AdamW**
  - General-purpose, transformers, quick prototyping
  - Works well out-of-the-box; AdamW preferred with weight decay
* - **RMSprop**
  - RNNs, non-stationary objectives
  - Good for recurrent architectures; handles varying gradient scales
* - **Adagrad**
  - Sparse data (NLP, embeddings)
  - Learning rate decreases over time; good for infrequent features
* - **LBFGS**
  - Small models, fine-tuning, convex problems
  - Memory-intensive; requires closure function

Optimizer Categories

:maxdepth: 1

gradient_descent
adaptive
second_order
schedulers