mirror of
https://github.com/pytorch/pytorch.git
synced 2026-11-20 09:04:25 +08:00
19da3da3d3
The fix-cpp-build branch is a major restructuring that: - Drops Exhale (which auto-generated API docs but caused OOM in CI) in favor of hand-written RST pages using Breathe directives - Reorganizes notes into a hierarchical API reference with proper categories - Adds new content: XPU docs, stable ABI docs, DeviceGuard/Stream docs - Always runs Doxygen instead of gating on an env var Pull Request resolved: https://github.com/pytorch/pytorch/pull/174096 Approved by: https://github.com/janeyx99, https://github.com/seemethere, https://github.com/albanD
1.6 KiB
1.6 KiB
myst
| myst | ||||||
|---|---|---|---|---|---|---|
|
Second-Order Optimizers
Second-order methods use curvature information (Hessian or its approximations) to make better optimization steps. They can converge faster but are more computationally expensive and memory-intensive.
LBFGS (Limited-memory Broyden-Fletcher-Goldfarb-Shanno)
LBFGS is a quasi-Newton method that approximates the inverse Hessian using gradient history. It can converge much faster than first-order methods for smooth, convex-like loss surfaces.
When to use:
- Small models where memory isn't a concern
- Fine-tuning pre-trained models
- Convex or near-convex optimization problems
- Full-batch training (not mini-batch)
Key parameters:
lr: Learning rate (often 1.0 for LBFGS)max_iter: Maximum iterations per stephistory_size: Number of past gradients to store
Important: LBFGS requires a closure function that recomputes the loss.
:members:
:undoc-members:
Example:
auto optimizer = torch::optim::LBFGS(
model->parameters(),
torch::optim::LBFGSOptions(1.0)
.max_iter(20)
.history_size(10));
// LBFGS requires a closure that recomputes the model
for (int epoch = 0; epoch < num_epochs; ++epoch) {
auto closure = [&]() {
optimizer.zero_grad();
auto output = model->forward(data);
auto loss = loss_fn(output, target);
loss.backward();
return loss;
};
optimizer.step(closure);
}