Make JIT Serialization support arbitrary std::function<> IO (#28039)

Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/28039

Right now, torch::save() uses std::ostream, which results in unnecessary
data copies in practice. Similar for torch::load().

Adding a std::function<size_t(const void*, size_t)> as an output option,
parallel to the existing filename and std::ostream apis, gives users the
flexibility to emit directly to a backing store.

For a simple case of appending the output to a std::string, we observe
significant benchmark savings (on order of -50%), even with the
minor std::function<> dispatch overhead. The main reason is that
std::ostringstream effectively requires 2 extra copies of the data
beyond a simple string.append lambda.

We also provide a parallel api for the load(), though this one is
slightly more complex due to the need to do arbitrary position reads.

Test Plan:
buck test mode/dev-nosan caffe2/test/...
      (Basic serialization test in caffe2/test/cpp/api/serialize.cpp)
      Benchmark in experimental/jeremyl/c2/SerializationBench.cpp, with D17823443
        (1M time goes from 90ms -> 40ms, albeit with crc patch applied)

Differential Revision: D17939034

fbshipit-source-id: 344cce46f74b6438cb638a8cfbeccf4e1aa882d7
This commit is contained in:
Jeremy Lilley
2019-10-15 22:08:17 -07:00
committed by Facebook Github Bot
parent 9cc4405dc9
commit 2e0294cb39
11 changed files with 194 additions and 41 deletions
+31
View File
@@ -60,6 +60,37 @@ TEST(SerializeTest, BasicToFile) {
ASSERT_TRUE(x.allclose(y));
}
TEST(SerializeTest, BasicViaFunc) {
torch::manual_seed(0);
auto x = torch::randn({5, 5});
std::string serialized;
torch::save(x, [&](const void* buf, size_t n) {
serialized.append(reinterpret_cast<const char *>(buf), n);
return n;
});
torch::Tensor y;
torch::load(y, serialized.data(), serialized.size());
ASSERT_TRUE(y.defined());
ASSERT_EQ(x.sizes().vec(), y.sizes().vec());
ASSERT_TRUE(x.allclose(y));
torch::Tensor z;
torch::load(z, [&](uint64_t pos, void* buf, size_t n) -> size_t {
if (pos >= serialized.size()) return 0;
size_t nbytes = std::min(static_cast<size_t>(pos) + n,
serialized.size()) - pos;
memcpy(buf, serialized.data() + pos, nbytes);
return nbytes;
},
[&]() -> size_t { return serialized.size(); });
ASSERT_TRUE(z.defined());
ASSERT_EQ(x.sizes().vec(), z.sizes().vec());
ASSERT_TRUE(x.allclose(z));
}
TEST(SerializeTest, Resized) {
torch::manual_seed(0);