C++ API: torch::nn::BatchNorm1d (#28176)

Summary:
Add torch::nn::BatchNorm1d function/module support for the C++ API.
torch::nn::BatchNorm{2,3}d will be added after this PR is merged.

Related Issue: https://github.com/pytorch/pytorch/issues/25883

Reviewer: yf225

I would like to discuss about below items.

* Necessity of `num_batches_tracked` in `BatchNormImplBase`
  * `num_batches_tracked` is needed to calculate `momentum` when we do not feed `momentum` argument in Python API. But in C++ API, `momentum` argument has a default value.
  * `num_batches_tracked` is only used for counting up `BatchNorm1d::foward()` call. I think it is no necessary for user anymore.
* The design of `BatchNorm{1,2,3}dOptions`
  * We have already `BatchNormOptions` used for deprecated `BatchNorm` module. However, it is hard to use it for `BatchNorm{1,2,3}dOptions` because of the arguments disagreement of each modules.
  * In this PR, I introduce `BatchNormOptionsv2` template class for the `BatchNorm{1,2,3}dOptions`. But I'm not sure this design is good or not.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/28176

Differential Revision: D18196843

Pulled By: yf225

fbshipit-source-id: 667e2b5de4150d5776c41b9088c9e6c2ead24cd4
This commit is contained in:
nuka137
2019-10-29 17:25:56 -07:00
committed by Facebook Github Bot
parent 8f1564b8ab
commit cbc234bceb
11 changed files with 341 additions and 37 deletions
+27
View File
@@ -1266,6 +1266,33 @@ TEST_F(FunctionalTest, Threshold) {
}
}
TEST_F(FunctionalTest, BatchNorm1d) {
int num_features = 5;
double eps = 1e-05;
double momentum = 0.1;
auto input = torch::randn({2, 5});
auto mean = torch::randn(5);
auto variance = torch::rand(5);
auto weight = torch::ones({num_features});
auto bias = torch::zeros({num_features});
auto output = F::batch_norm(
input, mean, variance,
BatchNormOptions().weight(weight).bias(bias).momentum(momentum).eps(eps),
/*training=*/false);
auto expected = (input - mean) / torch::sqrt(variance + eps);
ASSERT_TRUE(output.allclose(expected));
}
TEST_F(FunctionalTest, BatchNorm1dDefaultOptions) {
auto input = torch::randn({2, 5});
auto mean = torch::randn(5);
auto variance = torch::rand(5);
auto output = F::batch_norm(input, mean, variance);
auto expected = (input - mean) / torch::sqrt(variance + 1e-5);
ASSERT_TRUE(output.allclose(expected));
}
TEST_F(FunctionalTest, Interpolate) {
{
// 1D interpolation