Commit Graph

9 Commits

Author SHA1 Message Date
Xuehai Pan a229b4526f [BE] Prefer dash over underscore in command-line options (#94505)
Preferring dash over underscore in command-line options. Add `--command-arg-name` to the argument parser. The old arguments with underscores `--command_arg_name` are kept for backward compatibility.

Both dashes and underscores are used in the PyTorch codebase. Some argument parsers only have dashes or only have underscores in arguments. For example, the `torchrun` utility for distributed training only accepts underscore arguments (e.g., `--master_port`). The dashes are more common in other command-line tools. And it looks to be the default choice in the Python standard library:

`argparse.BooleanOptionalAction`: https://github.com/python/cpython/blob/4a9dff0e5adc91cbb1ed68c495dac64ccfe608bd/Lib/argparse.py#L893-L895

```python
class BooleanOptionalAction(Action):
    def __init__(...):
            if option_string.startswith('--'):
                option_string = '--no-' + option_string[2:]
                _option_strings.append(option_string)
```

It adds `--no-argname`, not `--no_argname`. Also typing `_` need to press the shift or the caps-lock key than `-`.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/94505
Approved by: https://github.com/ezyang, https://github.com/seemethere
2023-02-09 20:16:49 +00:00
Elias Ellison 6694fdaccd Clean up profiling mode and profiling executor strategy (#73875)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/73875

Previously we had a few settings:
- getExecutor - which toggled between Profiling Executor and Legacy
- getGraphOptimize - if true, overrides PE/Legacy to run with simple executor (no optimizations)
and then...
- getProfilingMode - which would set PE to 0 specializtions.

The last mode is redundant with getGraphOptimize, we should just remove it and use getGraphOptimize in these cases. It would lead to potentially invalid combinations of logic - what does mean if getProfilingMode is true but getExecutor is set to false ? This would lead to a bug in specialize_autograd_zero in this case, see: https://github.com/pytorch/pytorch/blob/master/torch%2Fcsrc%2Fjit%2Fpasses%2Fspecialize_autogradzero.cpp#L93.

The tests here are failing but get fixed with the PR above it, so i'll squash for landing.

Test Plan: Imported from OSS

Reviewed By: cpuhrsch

Differential Revision: D34938130

Pulled By: eellison

fbshipit-source-id: 1a9c0ae7f6d1cfddc2ed3499a5af611053ae5e1b
(cherry picked from commit cf69ce3d15)
2022-03-29 18:38:51 +00:00
Thomas Viehmann d3d8da7a8e Enable CUDA Fuser for ROCm (#45965)
Summary:
This enables the cuda fuser on ROCm and enables tests for them.

Part of this patch is based on work of Rohith Nallamaddi, thank you.
Errors are my own, of course.

Pull Request resolved: https://github.com/pytorch/pytorch/pull/45965

Reviewed By: seemethere

Differential Revision: D24170457

Pulled By: walterddr

fbshipit-source-id: 3dd25b3501a41d2f00acba3ce8642ce51c49c9a6
2020-10-08 10:41:56 -07:00
Rong Rong 986af53be2 type check for torch.testing._internalcodegen:* (#45368)
Summary:
part of `torch.testing._internal.*` effort

Pull Request resolved: https://github.com/pytorch/pytorch/pull/45368

Reviewed By: malfet

Differential Revision: D23950512

Pulled By: walterddr

fbshipit-source-id: 399f712d12cdd9795b0136328f512c3f86a15f24
2020-09-28 14:04:52 -07:00
Akihiro Nitta f17d7a5556 Fix exception chaining in torch/ (#43836)
Summary:
## Motivation
Fixes https://github.com/pytorch/pytorch/issues/43770.

## Description of the change
This PR fixes exception chaining only in files under `torch/` where appropriate.
To fix exception chaining, I used either:
1. `raise new_exception from old_exception` where `new_exception` itself seems not descriptive enough to debug or `old_exception` delivers valuable information.
2. `raise new_exception from None` where raising both of `new_exception` and `old_exception` seems a bit noisy and redundant.
I subjectively chose which one to use from the above options.

## List of lines containing raise in except clause:
I wrote [this simple script](https://gist.github.com/akihironitta/4223c1b32404b36c1b349d70c4c93b4d) using [ast](https://docs.python.org/3.8/library/ast.html#module-ast) to list lines where `raise`ing in `except` clause.

- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/jit/annotations.py#L35
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/jit/annotations.py#L150
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/jit/annotations.py#L158
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/jit/annotations.py#L231
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/jit/_trace.py#L432
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/nn/utils/prune.py#L192
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/cuda/nvtx.py#L7
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/utils/cpp_extension.py#L1537
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/utils/tensorboard/_pytorch_graph.py#L292
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/utils/data/dataloader.py#L835
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/utils/data/dataloader.py#L849
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/utils/data/dataloader.py#L856
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L186
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L189
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L424
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L1279
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L1283
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L1356
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L1388
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L1391
- [ ] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_utils.py#L1412
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/codegen/random_topo_test.py#L310
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/codegen/random_topo_test.py#L329
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/codegen/random_topo_test.py#L332
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/jit_utils.py#L183
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/testing/_internal/common_nn.py#L4789
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/onnx/utils.py#L367
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/onnx/utils.py#L659
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/onnx/utils.py#L892
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/onnx/utils.py#L897
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/serialization.py#L108
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/serialization.py#L754
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/distributed/rpc/_testing/faulty_agent_backend_registry.py#L76
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/distributed/rpc/backend_registry.py#L260
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/distributed/distributed_c10d.py#L184
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/_utils_internal.py#L57
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/hub.py#L494
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/contrib/_tensorboard_vis.py#L16
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/distributions/lowrank_multivariate_normal.py#L100
- [x] https://github.com/pytorch/pytorch/blob/000739c31ad34909e64124e0d39b2f49249458e9/torch/distributions/constraint_registry.py#L142

Pull Request resolved: https://github.com/pytorch/pytorch/pull/43836

Reviewed By: ailzhang

Differential Revision: D23431212

Pulled By: malfet

fbshipit-source-id: 5f7f41b391164a5ad0efc06e55cd58c23408a921
2020-08-31 20:26:23 -07:00
Stanislau Hlebik b774ce54f8 remediation of S205607
fbshipit-source-id: 798decc90db4f13770e97cdce3c0df7d5421b2a3
2020-07-17 17:19:47 -07:00
Stanislau Hlebik 8fdea489af remediation of S205607
fbshipit-source-id: 5113fe0c527595e4227ff827253b7414abbdf7ac
2020-07-17 17:17:03 -07:00
Jie 07518e120b [nvFuser] add torch.jit.fuser context manager (#38993)
Summary:
1. `torch.jit.fuser(str)` context manager facilitates switch between backend fusers:
  str - 'fuser0' enables only legacy fuser;
  str - 'fuser1' enables only NNC;
  str - 'fuser2' enables only nvFuser;
2. cleanup updated python tests.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/38993

Reviewed By: nairbv, pbelevich

Differential Revision: D21800620

Pulled By: soumith

fbshipit-source-id: 7fe855f5a5b97368e5e84c98c28d04b2e1276c85
2020-06-01 10:52:40 -07:00
Christian Sarofeen 8e69c3be17 [nvFuser] Reduction support in codegen, fp16 support (#38627)
Summary:
Adds reduction  support for the code generator. Reductions are fully supported with split/merge/reorder/rfactor/computeAt/unroll operators. There is also cross thread (intra-block) reduction support.

The two remaining pieces missing for reduction support is:
- Safety: If cross thread reduction was used, child operators shouldn't be able to bind that thread dim anymore
- Cross block reduction: we will want inter-block reduction support to match parity with tensor iterator

PR also provides FP16 support for fusions now. We insert casts on FP16 inputs to FP32, and we insert casts to FP16 on FP16 outputs.

Also working towards reductions and shape inference for reductions in the fusion pass.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/38627

Reviewed By: albanD

Differential Revision: D21663196

Pulled By: soumith

fbshipit-source-id: 3ff2df563f86c39cd5821ab9c1148149e5172a9e
2020-05-21 17:18:39 -07:00