Summary:
Moves torchtext from a required dependency to optional run-time dependency for the part of release note generation which categorizes commits.
torchtext is archived and having it as a dependency makes it hard to use recent PyTorch versions. For now, make the dependency optional. In the future we can probably just remove the classifier (only 5 commits need classification for 2.11, faster to just manually do it) or replace it with a more recent general model.
Test Plan:
```bash
python categorize.py
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/178144
Approved by: https://github.com/atalman
Summary:
Adds two scripts to automate the release note generation process further:
* `parse_cherry_picks.py` takes in a tracker issue (like https://github.com/pytorch/pytorch/issues/170119) and a commitlist.csv file and parses the right github hashes of trunk commits from the tracker issue, validates they match an entry in commitlist.csv, and writes the parsed cherry-pick hashes to a file
* `remove_cherry_picks.py` takes the output of the script above and creates a new commit list with the cherry picks removed
all unexpected behavior is logged for manual inspection
Test Plan:
```
// (before this script - generate results/commitlist.csv)
$ cd scripts/release_notes
// parse cherry picks into 2.10, creates results/cherry_picks_170119.csv
$ python parse_cherry_picks.py --commitlist results/commitlist.csv https://github.com/pytorch/pytorch/issues/170119
// create a new commitlist.csv with cherry picks removed, creates results/commitlist_no_cherry_picks.csv
$ python remove_cherry_picks.py --commitlist results/commitlist.csv --cherry-picks results/cherry_picks_170119.csv
// manually swap the commitlist files
$ mv results/commitlist.csv results/commitlist_before_cherry_pick_removal.csv
$ mv results/commitlist_no_cherry_picks.csv results/commitlist.csv
// inspect dir contents
$ ls results
cherry_pick_removals.log cherry_picks_170119.csv commitlist.csv commitlist_before_cherry_pick_removal.csv
// proceeed with further steps of release notes prep (not shown)
```
Pull Request resolved: https://github.com/pytorch/pytorch/pull/175742
Approved by: https://github.com/atalman
This PR should be merged ~1 week after the copy-scripts PR lands, to give in-flight PRs time to pick up the new script locations.
The previous PR copied the binary build scripts to .ci/pytorch/. This PR updates all workflow references (`.circleci/scripts/` → `.ci/pytorch/`), deletes the old copies, and removes the now-empty `.circleci/` directory.
**Workflows that need the bake time:**
- `build-triton-wheel.yml` — triggers on `pull_request` (with path filters) and uses `checkout-pytorch@main` (head-sha checkout)
- `build-vllm-wheel.yml` — same
These create a workflow-YAML-vs-checkout version mismatch for pre-move PRs.
**Workflows that don't need it** (only trigger on schedule/workflow_dispatch/workflow_call):
- `_binary-build-linux.yml`
- `_binary-upload.yml`
- `_binary-upload-flash-attention-wheel.yml`
- `test-pytorch-binary` composite action
- Generated nightly workflows (windows/macos binary builds)
See PR #175987 for details on the mismatch mechanism.
Authored with Claude.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/175917
Approved by: https://github.com/atalman, https://github.com/malfet
ghstack dependencies: #175915, #175930, #175933
Now that we have bc_lint in CI, this script is no longer needed (nor has it ever been conclusive). I've already updated the Runbook to not need this script.
Suppressing bc_lint as this script is not shipped as a part of torch--it is not user facing! For context, this script is (rarely) used by the release notes manager to ensure BC across releases. It had been broken for at least since 2.6.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/151453
Approved by: https://github.com/albanD, https://github.com/jbschlosser
The `usort` config in `pyproject.toml` has no effect due to a typo. Fixing the typo make `usort` do more and generate the changes in the PR. Except `pyproject.toml`, all changes are generated by `lintrunner -a --take UFMT --all-files`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/127126
Approved by: https://github.com/kit1980
The `usort` config in `pyproject.toml` has no effect due to a typo. Fixing the typo make `usort` do more and generate the changes in the PR. Except `pyproject.toml`, all changes are generated by `lintrunner -a --take UFMT --all-files`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/127126
Approved by: https://github.com/kit1980
ghstack dependencies: #127122, #127123, #127124, #127125
Adds a ruff lint rule to ban raising raw exceptions. Most of these should at the very least be runtime exception, value errors, type errors or some other errors. There are hundreds of instance of these bad exception types already in the codebase, so I have noqa'd most of them. Hopefully this error code will get commiters to rethink what exception type they should raise when they submit a PR.
I also encourage people to gradually go and fix all the existing noqas that have been added so they can be removed overtime and our exception typing can be improved.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/124570
Approved by: https://github.com/ezyang
Update ruff to 0.4.1 .
This version fixes a lot false negatives/false positives, is 20-40% faster, and has various other bug fixes.
Below is a before and after table showing the execution time of ruff lint and ruff format in milliseconds courtesy of https://astral.sh/blog/ruff-v0.4.0
| Repository | Linter (v0.3) | Linter (v0.4) | Formatter (v0.3) | Formatter (v0.4) |
|----------------------------------------------------|---------------|---------------|------------------|------------------|
| [pytorch/pytorch](https://github.com/pytorch/pytorch) | 328.7 | 251.8 | 351.1 | 274.9 |
Pull Request resolved: https://github.com/pytorch/pytorch/pull/124549
Approved by: https://github.com/ezyang
# Summary
This PR made some significant changes to the scripts around Release Scripts. At a high level:
- Turned the quips into docs and updated links
- Update the common.categorizes list in the hopes to make this the source of truth for releases- This is hard since the release_notes labels can be changed at will. An alternative would be to poll from github api. However, I think that is overkill. The notebook does a set compare and will show you knew categories. I think we want this to be manual so that the release note engineer will decided how to categorize.
- Create cateogry group from speaking with folks on distributed and AO that told me these different release categories can be merged.
- I am the newest person to Core and don't use ghstack soo made token getting a lil more generic.
- Added a classifier.py file. This file will train a commit categorizer for you, hopefully with decent accuracy. I was able to achieve 75% accuracy. I drop the highest frequency class - "skip" since this creates a more useful cateogrizer.
- I updated the categorize.py script so that the prompt will be what the classifier thinks, gated by a flag.
- Added a readme that will hopefully help future release notes engineers.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94560
Approved by: https://github.com/albanD
Merges startswith, endswith calls to into a single call that feeds in a tuple. Not only are these calls more readable, but it will be more efficient as it iterates through each string only once.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/96754
Approved by: https://github.com/ezyang
Applies the remaining flake8-comprehension fixes and checks. This changes replace all remaining unnecessary generator expressions with list/dict/set comprehensions which are more succinct, performant, and better supported by our torch.jit compiler. It also removes useless generators such as 'set(a for a in b)`, resolving it into just the set call.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94676
Approved by: https://github.com/ezyang
Preferring dash over underscore in command-line options. Add `--command-arg-name` to the argument parser. The old arguments with underscores `--command_arg_name` are kept for backward compatibility.
Both dashes and underscores are used in the PyTorch codebase. Some argument parsers only have dashes or only have underscores in arguments. For example, the `torchrun` utility for distributed training only accepts underscore arguments (e.g., `--master_port`). The dashes are more common in other command-line tools. And it looks to be the default choice in the Python standard library:
`argparse.BooleanOptionalAction`: https://github.com/python/cpython/blob/4a9dff0e5adc91cbb1ed68c495dac64ccfe608bd/Lib/argparse.py#L893-L895
```python
class BooleanOptionalAction(Action):
def __init__(...):
if option_string.startswith('--'):
option_string = '--no-' + option_string[2:]
_option_strings.append(option_string)
```
It adds `--no-argname`, not `--no_argname`. Also typing `_` need to press the shift or the caps-lock key than `-`.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/94505
Approved by: https://github.com/ezyang, https://github.com/seemethere