Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -783,7 +783,6 @@ Availability:
+def f(x: queue.Queue[int]) -> C:
```


### use `datetime.UTC` alias

Availability:
Expand All @@ -795,3 +794,23 @@ Availability:
-datetime.timezone.utc
+datetime.UTC
```

### Fold nested context managers

Availability:
- `--py310-plus` and higher

```diff
- with foo:
- with bar:
- body
+ with (foo, bar):
+ body
```

```diff
- with (foo as _, bar as _):
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is in scope. there's no reason to write the original code with nonsense as

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a real work around. I know, I maintain code with it. So I added code to remove it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is also useless assigning to underscore so also fits in with things like replacing set() with {} or similar which pyupgrade does.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no it does not. nobody would write code like that because it doesn't help

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.python.org/3.3/reference/compound_stmts.html#the-with-statement

You needed to use the as syntax to stack them in one with. But since you often did not need the name using _ was a real thing.

The options were:

with foo:
    with bar:

or

with foo as f, bar as b:

As you correctly point out, the underscore names are not helpful and with 3.10 there is nothing forcing the programmer to use a name. So it is ok to remove them. Technically.... we could do analysis of the body and following code to see if any of the names are used but that gets ugly fast so I stuck with only removing the useless underscore.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disagree but understood. Will post a revised version shortly.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case we misunderstood each other.

3.3 to 3.9 did require the use of as to nest context managers. with foo as f, bar as b: was the only way.
3.10 remove the requirement for the as naming. Now you can do with (foo, bar): and it follows the same pattern as import statements.

My original PR removed the now unneeded syntax of the as when migrating to 3.10 when the name was an underscore because that is was semantically safe. Any other name may have actually been used. I am moving this logic to another branch because I still want to upgrade code that I have to maintain elsewhere. I can post it as a separate PR for clarity. Or it can remain in a private repo. No worries.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no it did not. I just tried it and it works fine without as

the only thing 3.9 added was trailing commas

Copy link
Copy Markdown
Author

@shaleh shaleh May 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, our code was 3.7 at one point. I wonder if this was changed somewhere before 3.9. Because I know it was failing CI or we would not have added the ugly captures. Either way, it is out of the current PR. Folding the nesting is still valuable.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I did some digging. Looks like that weird underscore form came from a StackOverflow discovery back in like 3.3 or 3.5 days.... definitely worthless now.

This code below passes 3.9 but the parenthesis form at the end does not parse in 3.8 or older. The as syntax does not help.

Thanks for your patience. I have removed one more mislearned thing from my list.

from contextlib import suppress

try:
    with suppress(ValueError) as _, suppress(IndexError) as _:
        print("hello 1")
        raise ValueError("foo")
except ValueError:
    print("Not supressed")

try:
    with suppress(ValueError), suppress(IndexError):
        print("hello 2")
        raise ValueError("foo")
except ValueError:
    print("Not supressed")

try:
    with (
        suppress(ValueError),
        suppress(IndexError)
    ):
        print("hello 3")
        raise ValueError("foo")
except ValueError:
    print("Not supressed")

- body
+ with (foo, bar):
+ body
```
136 changes: 136 additions & 0 deletions pyupgrade/_plugins/fold_nested_context_managers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
from __future__ import annotations

import ast
import functools
import itertools
from collections.abc import Iterable
from typing import Any

from tokenize_rt import Offset
from tokenize_rt import Token

from pyupgrade._ast_helpers import ast_to_offset
from pyupgrade._data import register
from pyupgrade._data import State
from pyupgrade._data import TokenFunc
from pyupgrade._token_helpers import Block


def _expand_item(indent: int, item: ast.AST) -> str:
return '{}{}'.format(' ' * indent, ast.unparse(item))
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ast.unparse does not roundtrip so it is not usable or acceptable

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/Instagram/LibCST is finally being maintained again so I guess @shaleh could use this. Alternatively, and likely recommended by @asottile, use his https://github.com/asottile/tokenize-rt



def _replace_context_managers(
i: int,
tokens: list[Token],
*,
with_items: list[ast.withitem],
body: Iterable[ast.AST],
) -> None:
block = Block.find(tokens, i, trim_end=True)
block_indent = block._minimum_indent(tokens)
replacement = '{}with ({}):\n{}\n'.format(
' ' * block._initial_indent(tokens),
', '.join(ast.unparse(item) for item in with_items),
'\n'.join(_expand_item(block_indent, item) for item in body),
)
tokens[block.start:block.end] = [Token('CODE', replacement)]


def _drop_underscore_names(items: list[ast.withitem]) -> list[ast.withitem]:
"""
Remove unnecessary "_" names.

Returns an empty list if there are no names that need changing.
"""
transformed = []
changed = False
for item in items:
if (
isinstance(item.optional_vars, ast.Name) and
item.optional_vars.id == '_'
):
item.optional_vars = None
changed = True
transformed.append(item)

if changed:
return transformed

return []


def flatten(xs: Iterable[Any]) -> list[Any]:
return list(itertools.chain.from_iterable(xs))
Comment on lines +40 to +41
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not at all acceptable or typesafe



@register(ast.With)
def visit_With_fold_nested(
state: State,
node: ast.With,
parent: ast.AST,
) -> Iterable[tuple[Offset, TokenFunc]]:
"""
Fold nested with statements into one statement.

with foo:
with bar:
body

becomes

with (foo, bar):
body
"""
if state.settings.min_version < (3, 10):
return

with_stmts = []
current: ast.AST = node
while True:
if isinstance(current, ast.With):
with_stmts.append(current)
if len(current.body) == 1:
current = current.body[0]
continue
break

if len(with_stmts) > 1:
with_items = flatten(n.items for n in with_stmts)
yield ast_to_offset(node), functools.partial(
_replace_context_managers,
body=with_stmts[-1].body,
with_items=with_items,
)


@register(ast.With)
def visit_With_drop_unnecessary_underscore_names(
state: State,
node: ast.With,
parent: ast.AST,
) -> Iterable[tuple[Offset, TokenFunc]]:
"""
Drop unnecessary _ names.

If this is a with statement with multiple items, remove any `as _`.
This was a work around before 3.10.

with (foo as _, bar as _):
body

becomes

with (foo, bar):
body
"""
if state.settings.min_version < (3, 10):
return

with_items = _drop_underscore_names(node.items)
if with_items:
yield ast_to_offset(node), functools.partial(
_replace_context_managers,
body=node.body,
with_items=with_items,
)
114 changes: 114 additions & 0 deletions tests/features/fold_nested_context_managers_test.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
from __future__ import annotations

import pytest

from pyupgrade._data import Settings
from pyupgrade._main import _fix_plugins


@pytest.mark.parametrize(
('s', 'version'),
(
pytest.param(
'with foo:\n'
" print('something')\n"
'\n',
(3, 10),
id='simple with expression',
),
pytest.param(
'with foo as bar:\n'
" print('something')\n"
'\n',
(3, 10),
id='simple with expression and captured name',
),
pytest.param(
'with (foo as thing1, bar as thing2):\n'
" print('something')\n"
'\n',
(3, 10),
id='simple with expression and captured names',
),
pytest.param(
'with (foo as _, bar as _):\n'
" print('something')\n"
'\n',
(3, 9),
id='nested with expression with empty name capture workaround',
),
),
)
def test_fold_nested_context_managers_noop(s, version):
assert _fix_plugins(s, settings=Settings(min_version=version)) == s


@pytest.mark.parametrize(
('s', 'expected', 'version'),
(
pytest.param(
'with foo:\n'
' with bar:\n'
" print('something')\n"
" print('another')\n"
'\n',
'with (foo, bar):\n'
" print('something')\n"
" print('another')\n"
'\n',
(3, 10),
id='nested with expression',
),
pytest.param(
'if value:\n'
' with foo:\n'
' with bar:\n'
" print('something')\n"
" print('another')\n"
'\n',
'if value:\n'
' with (foo, bar):\n'
" print('something')\n"
" print('another')\n"
'\n',
(3, 10),
id='nested with expression inside of an if',
),
pytest.param(
'with foo as thing1:\n'
' with bar as thing2:\n'
" print('something')\n"
" print('another')\n"
'\n',
'with (foo as thing1, bar as thing2):\n'
" print('something')\n"
" print('another')\n"
'\n',
(3, 10),
id='nested with expression with named capture',
),
pytest.param(
'with (foo as _, bar as _):\n'
" print('something')\n"
'\n',
'with (foo, bar):\n'
" print('something')\n"
'\n',
(3, 10),
id='nested with expression with unnecessary empty name capture workaround', # noqa: E501
),
pytest.param(
'with (foo as _, bar as thing2):\n'
" print('something')\n"
'\n',
'with (foo, bar as thing2):\n'
" print('something')\n"
'\n',
(3, 10),
id='nested with expression with one unnecessary empty name capture workaround', # noqa: E501
),
),
)
def test_fold_nested_context_managers(s, expected, version):
ret = _fix_plugins(s, settings=Settings(min_version=version))
assert ret == expected
Loading