[C++ API Parity] LBFGS optimizer step() update and added closure to the Optimizer step() function (#34564)

Summary:
Follow-ups after this PR:

* Remove `LossClosureOptimizer`, and merge `Optimizer` into `OptimizerBase` (and rename the merged class to Optimizer)
* Merge the LBFGS-specific serialize test function and the generic `test_serialize_optimizer` function, possibly by passing a bool `has_only_global_state` flag into the `test_serialize_optimizer` function to denote whether `size()` should be equal to 1 or 2?
    * https://github.com/pytorch/pytorch/pull/34564#discussion_r393780303
* It seems that we don't have the equivalent `XORConvergence_LBFGS` test like the other optimizers, and it would be good to add one
* Remove mentions of `parameters_` in optimizer.cpp, de-virtualize all functions, and remove the `OptimizerBase(std::vector<Tensor> parameters)` constructor from `OptimizerBase`
Pull Request resolved: https://github.com/pytorch/pytorch/pull/34564

Test Plan: Imported from GitHub, without a `Test Plan:` line.

Differential Revision: D20495701

Pulled By: anjali411

fbshipit-source-id: 6d35286d2decb6f7dff93d9d3e57515770666622
This commit is contained in:
anjali411
2020-03-17 22:23:23 -07:00
committed by Facebook GitHub Bot
parent df20f5b374
commit d7e4a379a0
12 changed files with 1361 additions and 594 deletions
+14 -1
View File
@@ -109,7 +109,8 @@ void check_exact_values(
auto loss = output.sum();
loss.backward();
optimizer.step();
auto closure = []() { return torch::tensor({10}); };
optimizer.step(closure);
if (i % kSampleEvery == 0) {
ASSERT_TRUE(
@@ -301,6 +302,18 @@ TEST(OptimTest, ProducesPyTorchValues_SGDWithWeightDecayAndNesterovMomentum) {
expected_parameters::SGD_with_weight_decay_and_nesterov_momentum());
}
TEST(OptimTest, ProducesPyTorchValues_LBFGS) {
check_exact_values<LBFGS>(
LBFGSOptions(1.0),
expected_parameters::LBFGS());
}
TEST(OptimTest, ProducesPyTorchValues_LBFGS_with_line_search) {
check_exact_values<LBFGS>(
LBFGSOptions(1.0).line_search_fn("strong_wolfe"),
expected_parameters::LBFGS_with_line_search());
}
TEST(OptimTest, ZeroGrad) {
torch::manual_seed(0);