Files
pytorch/docs/cpp/source/api/optim/second_order.md
T
Svetlana Karslioglu 19da3da3d3 Restructure C++ docs (#174096)
The fix-cpp-build branch is a major restructuring that:

- Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives
- Reorganizes notes into a hierarchical API reference with proper categories
- Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs
- Always runs Doxygen instead of gating on an env var

Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096
Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
2026-04-07 17:24:38 +00:00

1.6 KiB

myst
myst
html_meta
description keywords
Second-order optimizers in PyTorch C++ — LBFGS optimizer for full-batch optimization. PyTorch, C++, LBFGS, second-order, optimizer, full-batch

Second-Order Optimizers

Second-order methods use curvature information (Hessian or its approximations) to make better optimization steps. They can converge faster but are more computationally expensive and memory-intensive.

LBFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno)

LBFGS is a quasi-Newton method that approximates the inverse Hessian using gradient history. It can converge much faster than first-order methods for smooth, convex-like loss surfaces.

When to use:

  • Small models where memory isn't a concern
  • Fine-tuning pre-trained models
  • Convex or near-convex optimization problems
  • Full-batch training (not mini-batch)

Key parameters:

  • lr: Learning rate (often 1.0 for LBFGS)
  • max_iter: Maximum iterations per step
  • history_size: Number of past gradients to store

Important: LBFGS requires a closure function that recomputes the loss.

:members:
:undoc-members:

Example:

auto optimizer = torch::optim::LBFGS(
    model->parameters(),
    torch::optim::LBFGSOptions(1.0)
        .max_iter(20)
        .history_size(10));

// LBFGS requires a closure that recomputes the model
for (int epoch = 0; epoch < num_epochs; ++epoch) {
    auto closure = [&]() {
        optimizer.zero_grad();
        auto output = model->forward(data);
        auto loss = loss_fn(output, target);
        loss.backward();
        return loss;
    };
    optimizer.step(closure);
}