bpo-43224: Draft implementation of PEP 646#30398
Conversation
Sorry, something went wrong.
Formatted, the generated AST is as follows.
`def f(*args: *Ts): pass`:
```
(
"Module",
[
(
"FunctionDef",
(1, 0, 1, 23),
"f",
(
"arguments",
[],
[],
(
"arg",
(1, 7, 1, 16),
"args",
(
"Starred",
(1, 13, 1, 16),
("Name", (1, 14, 1, 16), "Ts", ("Load",)),
("Load",),
),
None,
),
[],
[],
None,
[],
),
[("Pass", (1, 19, 1, 23))],
[],
None,
None,
)
],
[],
)
```
`def f(*args: *tuple[int, ...]): pass`:
```
(
"Module",
[
(
"FunctionDef",
(1, 0, 1, 36),
"f",
(
"arguments",
[],
[],
(
"arg",
(1, 7, 1, 29),
"args",
(
"Starred",
(1, 13, 1, 29),
(
"Subscript",
(1, 14, 1, 29),
("Name", (1, 14, 1, 19), "tuple", ("Load",)),
(
"Tuple",
(1, 20, 1, 28),
[
("Name", (1, 20, 1, 23), "int", ("Load",)),
("Constant", (1, 25, 1, 28), Ellipsis, None),
],
("Load",),
),
("Load",),
),
("Load",),
),
None,
),
[],
[],
None,
[],
),
[("Pass", (1, 32, 1, 36))],
[],
None,
None,
)
],
[],
)
```
`def f(*args: *tuple[int, *Ts]): pass`:
```
(
"Module",
[
(
"FunctionDef",
(1, 0, 1, 36),
"f",
(
"arguments",
[],
[],
(
"arg",
(1, 7, 1, 29),
"args",
(
"Starred",
(1, 13, 1, 29),
(
"Subscript",
(1, 14, 1, 29),
("Name", (1, 14, 1, 19), "tuple", ("Load",)),
(
"Tuple",
(1, 20, 1, 28),
[
("Name", (1, 20, 1, 23), "int", ("Load",)),
(
"Starred",
(1, 25, 1, 28),
("Name", (1, 26, 1, 28), "Ts", ("Load",)),
("Load",),
),
],
("Load",),
),
("Load",),
),
("Load",),
),
None,
),
[],
[],
None,
[],
),
[("Pass", (1, 32, 1, 36))],
[],
None,
None,
)
],
[],
)
```
Formatted, generated AST is as follows.
`def f() -> tuple[*Ts]: pass`:
```
(
"Module",
[
(
"FunctionDef",
(1, 0, 1, 27),
"f",
("arguments", [], [], None, [], [], None, []),
[("Pass", (1, 23, 1, 27))],
[],
(
"Subscript",
(1, 11, 1, 21),
("Name", (1, 11, 1, 16), "tuple", ("Load",)),
(
"Tuple",
(1, 17, 1, 20),
[
(
"Starred",
(1, 17, 1, 20),
("Name", (1, 18, 1, 20), "Ts", ("Load",)),
("Load",),
)
],
("Load",),
),
("Load",),
),
None,
)
],
[],
)
```
`def f() -> tuple[int, *Ts]: pass`:
```
(
"Module",
[
(
"FunctionDef",
(1, 0, 1, 32),
"f",
("arguments", [], [], None, [], [], None, []),
[("Pass", (1, 28, 1, 32))],
[],
(
"Subscript",
(1, 11, 1, 26),
("Name", (1, 11, 1, 16), "tuple", ("Load",)),
(
"Tuple",
(1, 17, 1, 25),
[
("Name", (1, 17, 1, 20), "int", ("Load",)),
(
"Starred",
(1, 22, 1, 25),
("Name", (1, 23, 1, 25), "Ts", ("Load",)),
("Load",),
),
],
("Load",),
),
("Load",),
),
None,
)
],
[],
)
```
`def f() -> tuple[int, *tuple[int, ...]]: pass`:
```
(
"Module",
[
(
"FunctionDef",
(1, 0, 1, 45),
"f",
("arguments", [], [], None, [], [], None, []),
[("Pass", (1, 41, 1, 45))],
[],
(
"Subscript",
(1, 11, 1, 39),
("Name", (1, 11, 1, 16), "tuple", ("Load",)),
(
"Tuple",
(1, 17, 1, 38),
[
("Name", (1, 17, 1, 20), "int", ("Load",)),
(
"Starred",
(1, 22, 1, 38),
(
"Subscript",
(1, 23, 1, 38),
("Name", (1, 23, 1, 28), "tuple", ("Load",)),
(
"Tuple",
(1, 29, 1, 37),
[
("Name", (1, 29, 1, 32), "int", ("Load",)),
("Constant", (1, 34, 1, 37), Ellipsis, None),
],
("Load",),
),
("Load",),
),
("Load",),
),
],
("Load",),
),
("Load",),
),
None,
)
],
[],
)
```
Formatted, generated AST is as follows:
`x: tuple[*Ts]`:
```
(
"Module",
[
(
"AnnAssign",
(1, 0, 1, 13),
("Name", (1, 0, 1, 1), "x", ("Store",)),
(
"Subscript",
(1, 3, 1, 13),
("Name", (1, 3, 1, 8), "tuple", ("Load",)),
(
"Tuple",
(1, 9, 1, 12),
[
(
"Starred",
(1, 9, 1, 12),
("Name", (1, 10, 1, 12), "Ts", ("Load",)),
("Load",),
)
],
("Load",),
),
("Load",),
),
None,
1,
)
],
[],
)
```
`x: tuple[int, *Ts]`:
```
(
"Module",
[
(
"AnnAssign",
(1, 0, 1, 18),
("Name", (1, 0, 1, 1), "x", ("Store",)),
(
"Subscript",
(1, 3, 1, 18),
("Name", (1, 3, 1, 8), "tuple", ("Load",)),
(
"Tuple",
(1, 9, 1, 17),
[
("Name", (1, 9, 1, 12), "int", ("Load",)),
(
"Starred",
(1, 14, 1, 17),
("Name", (1, 15, 1, 17), "Ts", ("Load",)),
("Load",),
),
],
("Load",),
),
("Load",),
),
None,
1,
)
],
[],
)
```
`x: tuple[int, *tuple[str, ...]]`:
```
(
"Module",
[
(
"AnnAssign",
(1, 0, 1, 31),
("Name", (1, 0, 1, 1), "x", ("Store",)),
(
"Subscript",
(1, 3, 1, 31),
("Name", (1, 3, 1, 8), "tuple", ("Load",)),
(
"Tuple",
(1, 9, 1, 30),
[
("Name", (1, 9, 1, 12), "int", ("Load",)),
(
"Starred",
(1, 14, 1, 30),
(
"Subscript",
(1, 15, 1, 30),
("Name", (1, 15, 1, 20), "tuple", ("Load",)),
(
"Tuple",
(1, 21, 1, 29),
[
("Name", (1, 21, 1, 24), "str", ("Load",)),
("Constant", (1, 26, 1, 29), Ellipsis, None),
],
("Load",),
),
("Load",),
),
("Load",),
),
],
("Load",),
),
("Load",),
),
None,
1,
)
],
[],
)
```
E.g. tuple[*tuple[int, str]].
mrahtz
requested review from
Fidget-Spinner,
gvanrossum,
lysnikolaou,
markshannon and
pablogsal
as code owners
January 4, 2022 12:16
and fix a couple of bugs the tests revealed
Fidget-Spinner
left a comment
Fidget-Spinner
left a comment
There was a problem hiding this comment.
Wow, this is seriously impressive work and a very strong start! I don't have time to review everything right now, so far I've covered genericaliasobject.c and compile.c. I still have typing.py and friends left. I'm hopeless at the parser and grammar so I'll leave those to someone else.
Sorry, something went wrong.
Based on Fidget-Spinner's feedback: 1. Remove the explicit flag for 'exhausted', instead using 'gi->obj == NULL' to tell whether the iterator is exhausted. 2. Construct `starred_tuple` using the `Py_GenericAlias` constructor instead of constructing it manually. 3. Fix reference counting. 4. Make `Py_GenericAliasIterType` static.
JelleZijlstra
left a comment
JelleZijlstra
left a comment
There was a problem hiding this comment.
Just a couple of things I noticed while working on the typing-extensions implementation.
I'm not going to be the one to merge this, but I think it might make sense to split this PR into separate PRs for the grammar changes and the typing.py changes, since they touch very different parts of the codebase and need review from different people.
Sorry, something went wrong.
|
This PR has grown pretty big, and Pradeep agrees with Jelle that it might be better to split this up into separate parts - so I'm going to close this PR, and open a few new ones instead. |
Sorry, something went wrong.
|
@JelleZijlstra @pablogsal @Fidget-Spinner @pradeep90 Oh, gosh, I'm so sorry - I've just realised I'd misunderstood a critical part of GitHub's code review interface! I'd assumed all these comments had been posted weeks ago - I'd missed the part where you actually have to click the 'Submit code review' button to post them! Uff. I'll copy over these threads to the new PRs. |
Sorry, something went wrong.
|
@mrahtz I can review and help with the GC part. Feel free to add me as a reviewer in that PR |
Sorry, something went wrong.
Reviewers
7 participants
PEP 646 (Variadic Generics) hasn't yet been completely approved by the steering council, but here's a draft of its implementation so we can get started on code review.Update 2021-01-19: the SC has approved the PEP, so full steam ahead!
TODO:
typing.get_argsandtyping.get_origin(rather than just checking__args__etc)A couple of notes below.
@pradeep90 @gvanrossum @Fidget-Spinner
Starred tuple types
I realised there's a tricky issue with starred tuple types.
If we have e.g.
tuple[int, *tuple[str, bool]], the obvious thing to do would be to unpack the innertupleat runtime, resulting intuple[int, str, bool]. This would imply thatiter(tuple[str, bool])creates an iterator returningstrthenbool.But this would cause problems when using starred tuple types as the type of
*args, e.g.def foo(*args: *tuple[str, bool]):foo.__annotations__['args']should definitely not betuple[str, bool]. That would instead implydef foo(*args: tuple[str, bool])*args, its iterator should return only a single value. (Or at least, this is the behaviour specified for*args: *Ts; @pradeep90, I've just realised, should we update that section of the PEP to make it explicit that that starred tuples can be also used for*args?)I think the simplest solution to both of these problems is to not unpack tuple types as runtime, having
*tuple[stuff]instead creating an iterator that returns only a single item, an instance of a new 'starred tuple'. (Note that such a type does have to exist anyway, in order to represent e.g.*tuple[int, ...].)For the native
tupletype, I've tentatively implemented this by adding a newstarredflag togaobjectingenericaliasobject.crather than creating a whole new type. This makes implementation easier, but maybe it's suboptimal considering thatgaobjectis used for lots of things other thantuple- feedback appreciated. Fortyping.Tuple, I've implemented a new class intyping.pycalledStarredTuple, which should behave in the same way. (This does mean that*tuple[int] != *Tuple[int], but given thattuple[int] != Tuple[int], this seems fine.)Implementation of
*tupleI'm pretty uncertain about my implementation in
genericaliasobject.c. In particular:typing.py, the implementation is as simple asdef __iter__(self): yield StarredTuple(self). Is there a way to do something similar inga_iter, without the need to also implementga_iternext?https://bugs.python.org/issue43224