Issue 40066: Enum: modify __repr__, __str__; update docs

Serhiy had the idea of having Enum._convert also modify the __str__ and __repr__ of newly created enumerations to display the module name instead of the enumeration name (https://bugs.python.org/msg325007):

--> socket.AF_UNIX
<AddressFamily.AF_UNIX: 1>   ==>  <socket.AF_UNIX: 1>

--> print(socket.AF_UNIX)
AddressFamily.AF_UNIX        ==>  socket.AF_UNIX

Thoughts?

> _in some cases when enum instances are exposed as module globals_

Yes. And repr should be inverse of eval, but it's probably too late for that. :-/

Looks like the `re` module's flags have been updated separately in issue36548:

  >>> import re
  >>> re.I
  re.IGNORECASE

  >>> print(re.I)
  # should also be re.IGNORECASE

  >>> re.I|re.S|re.X
  re.IGNORECASE|re.DOTALL|re.VERBOSE


For stdlib Enum conversions are we happy with that?  Or should __str__ just print the numeric value?

If it's considered to be not too backwards-incompatible, I think it would be nice to have str different from repr. That way we can finetune what exactly we need. But we can already do almost exactly that with *int* instead of *str*, so it's not too compelling.

Much more important thing is the "repr as inverse of eval". Is there any way we can have that for our own enums (as a mixin or a decorator)?

    @module_global(re)
    class RegexFlag(Enum):
        ...

It would be fantastic. :-)

"repr as inverse of eval" is nice to have, but it is not a requirement.

Noone said it is a requirement, I just said it would be nice to have it factored out as a decorator or something instead of having to write __repr__ over and over again.

At this point, the PR has made the following changes:

- normal Enums
  - repr() -> "classname.membername"
  - str()  -> "membername"

- stdlib Enums available as module attributes (RegexFlag, AddressFamily, etc.)
  - repr() -> "modulename.membername"
  - str()  -> "membername"

Python-Dev thread [0], summary below:

> As you may have noticed, Enums are starting to pop up all
> over the stdlib [1].
>
> To facilitate transforming existing module constants to
> IntEnums there is `IntEnum._convert_`.  In Issue36548 [2]
> Serhiy modified the __repr__ of RegexFlag:
>
>   >>> import re
>   >>> re.I
>   re.IGNORECASE
>
> I think for converted constants that that looks nice.
>  For anyone that wants the actual value, it is of course
> available as the `.value` attribute:
>
>   >>> re.I.value
>   2
>
> I'm looking for arguments relating to:
>
> - should _convert_ make the default __repr__ be
>   module_name.member_name?
>
> - should _convert_ make the default __str__ be the same,
>   or be the numeric value?

After discussions with Guido I made a (largely done) PR [3] which:

for stdlib global constants (such as RE)
   - repr() -> uses `module.member_name`
   - str() -> uses `member_name`

for stdlib non-global constants, and enums in general
   - repr() -> uses `class.member_name`
   - str() -> uses `member_name`

The questions I would most appreciate an answer to at this point:

- do you think the change has merit?
- why /shouldn't/ we make the change?

As a reminder, the underlying issue is trying to keep at least the stdlib Enum representations the same for those that are replacing preexisting constants.


[0] https://mail.python.org/archives/list/python-dev@python.org/message/CHQW6THTDYNPPFWQ2KDDTUYSAJDCZFNP/

[1] I'm working on making their creation faster.  If anyone wanted to convert EnumMeta to C I would be grateful.

[2] https://bugs.python.org/issue36548

[3] https://github.com/python/cpython/pull/22392

> - do you think the change has merit?

Absolutely.

> - why /shouldn't/ we make the change?

Well, standard backward compatibility stuff. :-)

New changeset b775106d940e3d77c8af7967545bb9a5b7b162df by Ethan Furman in branch 'master':
bpo-40066: Enum: modify `repr()` and `str()` (GH-22392)
https://github.com/python/cpython/commit/b775106d940e3d77c8af7967545bb9a5b7b162df

> why /shouldn't/ we make the change?

It breaks doctests, and probably some other unit tests, too, e.g. for output formatting.

What should we suggest users to do? Replace

    >>> get_flag(…)
    <app_enums.TrickyFlag: 1>

by

    >>> get_flag(…) == app_enums.TrickyFlag  or get_flag(…)  # (also show result on failure)
    True

and

    assertEqual(
        "You caught the %s flag!" % result,
        "You caught the app_enums.TrickyFlag flag!")

by

    assertEqual(
        ("You caught the %s flag!" % result).replace("app_enums.", ""),
        "You caught the TrickyFlag flag!")

?

Note that using "%r" does not help, since it's also backwards incompatible.

For their own enums, users can probably backport (or forward-port) "__str__()" and "__repr__()" themselves in order to work around this difference. But it's something they would have to do.

I certainly understand the reasoning, and it also makes new Py3.10-only doctests nicer, actually, but IMHO it counts as deliberate breakage.

I'm probably a bit late to the party as well, but as some additional datapoints:

- This broke the tests for my project and I ended up adding a wrapper to stringify enums (since I actually prefer having the enum type in debug logs and such): https://github.com/qutebrowser/qutebrowser/commit/e2c5fe6262564d9d85806bfa9d4486a411cf5045
- This broke tests for pytest, and also changes its test IDs when enums are used for test parametrization, which might cause more breakage downstream: https://github.com/pytest-dev/pytest/issues/8546

Thank you for the feedback.  

The new str() and repr() make more sense for Flag-based enumerations, and I'm hesitant to have different formats for Enum- vs Flag-based enums.

Would it be helpful to have another base Enum class to derive from that maintained the original str() and repr() formats?

It may be helpful for the enum module to come with transitional functions like "pre310_str()" "pre310_repr()" "pre310_flag_str()" etc. so that people who are writing doctests that need to function on both < 3.10 and 3.10+ can temporarily do a "Enum.__str__ = pre310_str" in their test suites (and of course restore it later) until <=3.9 is no longer supported.  Otherwise everyone with doctest suites will be doing this ourselves.

New changeset acf7403f9baea3ae1119fc6b4a3298522188bf96 by Ethan Furman in branch 'main':
bpo-40066:  [Enum] update str() and format() output (GH-30582)
https://github.com/python/cpython/commit/acf7403f9baea3ae1119fc6b4a3298522188bf96

GH-30582 broke doc tests in 3.11 branch:

File "library/enum.rst", line ?, in default
Failed example:
    Color(0)
Exception raised:
    Traceback (most recent call last):
      File "/home/runner/work/cpython/cpython/Lib/doctest.py", line 1346, in __run
        exec(compile(example.source, filename, "single",
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<doctest default[0]>", line 1, in <module>
        Color(0)
        ^^^^^
    NameError: name 'Color' is not defined

This also broke our Solaris build with the following error:

======================================================================
FAIL: testGetaddrinfo (test.test_socket.GeneralModuleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/..../cpython-main/Lib/test/test_socket.py", line 1523, in testGetaddrinfo
    self.assertEqual(repr(type), '<SocketKind.SOCK_STREAM: 1>')
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: '<SocketKind.SOCK_STREAM: 2>' != '<SocketKind.SOCK_STREAM: 1>'
- <SocketKind.SOCK_STREAM: 2>
?                          ^
+ <SocketKind.SOCK_STREAM: 1>
?                          ^

(test.test_socket.GeneralModuleTests fails with the same error).

The issue is almost certainly that on Solaris, SOCK_STREAM is defined as 2 rather than 1; the following simple program confirms that:

#include <stdio.h>
#include <sys/socket.h>

void main() {
	printf("%d\n", SOCK_STREAM);
}

I'm just not sure whether to fix this with `assertRegex` or a special branch for Solaris (though I am not sure whether everybody else uses 1 or it's more varied).

New changeset 42a64c03ec5c443f2a5c2ee4284622f5d1f5326c by Victor Stinner in branch 'main':
Revert "bpo-40066:  [Enum] update str() and format() output (GH-30582)" (GH-30632)
https://github.com/python/cpython/commit/42a64c03ec5c443f2a5c2ee4284622f5d1f5326c

Sorry, I had to revert the change since it broke the CI and it prevented to merge new PRs. Tell me if I can help to get this test fixed and to get this change merged again.

By the way, the PR 30582 was merged even if the Docs CI failed.

After merging in doc fix by kumaraditya303, I'll update tests so Solaris passes.

New changeset 83d544b9292870eb44f6fca37df0aa351c4ef83a by Kumar Aditya in branch 'main':
bpo-40066: [Enum] skip failing doc test (GH-30637)
https://github.com/python/cpython/commit/83d544b9292870eb44f6fca37df0aa351c4ef83a

>    self.assertEqual(repr(type), '<SocketKind.SOCK_STREAM: 1>')

For this one, I suggest to replace the value with "..." doctest pattern.

vstinner wrote:
--------------
>>    self.assertEqual(repr(type), '<SocketKind.SOCK_STREAM: 1>')

> For this one, I suggest to replace the value with "..." doctest pattern.

That bit of code is from the unittest suite, not the doctest suite.

I went with:  

    self.assertEqual(repr(type), '<SocketKind.SOCK_STREAM: %r>' % type.value)

I created https://github.com/python/core-workflow/issues/424 "Should we make the Docs CI mandatory on the Python main branch?".

New changeset 62a6594e66ca955073be2f4e5a40291a39252ef3 by Ethan Furman in branch 'main':
bpo-40066: [Enum] fix tests (GH-30643)
https://github.com/python/cpython/commit/62a6594e66ca955073be2f4e5a40291a39252ef3

messages: + msg410711

stage: resolved -> (no value)

messages: + msg376900
versions: + Python 3.10, - Python 3.9