gh-92810: Reduce memory usage by ABCMeta.__subclasscheck__ (alternative2) by dolfinus · Pull Request #150540 · python/cpython
Alternative implementation of #131914 which does not check __subclasses__ recursively.
To handle cases like:
class Number(ABC): ... class Real(Number): ... class Integral(Real): ... Integral.register(int) assert issubclass(int, Number) is True
which previously were implemented via recursive __subclasses__ check, method cls.register(subclass) is calling super(cls).register(subclass) recursively ("bubble-up" registration to all the parents).
For benchmark from #131914:
sudo ./python -m pyperf system tune taskset -c 0 ./python benchmark.py --metaclass abc.ABCMeta --rounds 3 --classes 5000 taskset -c 0 ./python benchmark.py --metaclass _py_abc.ABCMeta --rounds 3 --classes 5000
| Impl | Max memory before, MB | Max memory after, MB |
|---|---|---|
| _abc | 6331 | 50 |
| _py_abc | 4422 | 61 |
| Impl | Total time before | Total time after |
|---|---|---|
| _abc | 6m 16s | 1m 23s |
| _py_abc | 8m 48s | 3m 53s |
| Check | Impl | before | after | Impl | before | after |
|---|---|---|---|---|---|---|
isinstance(child, Parent) |
_abc | 0.115us 4MiB...15MiB |
0.121us 2MiB...15MiB |
_py_abc | 0.212us 11MiB...24MiB |
0.218us 11MiB...24MiB |
issubclass(Child, Parent) |
_abc | 0.105us 0MiB...1MiB |
0.111us 0MiB...1MiB |
_py_abc | 0.203us 6MiB...8MiB |
0.208us 5MiB...8MiB |
isinstance(child, Grandparent) |
_abc | 0.112us 0MiB...2MiB |
0.119us 1MiB...1MiB |
_py_abc | 0.210us 4MiB...7MiB |
0.216us 4MiB...7MiB |
issubclass(Child, Grandparent) |
_abc | 0.103us 0MiB |
0.108us 0MiB |
_py_abc | 0.201us 0MiB...1MiB |
0.207us 0MiB...1MiB |
not isinstance(child, Sibling) |
_abc | 0.113us 4MiB...14MiB |
0.121us 3MiB...14MiB |
_py_abc | 0.348us 13MiB...23MiB |
0.354us 13MiB...22MiB |
not issubclass(Child, Sibling) |
_abc | 0.105us 1MiB |
0.110us 1MiB...2MiB |
_py_abc | 0.328us 8MiB...10MiB |
0.332us 9MiB...11MiB |
not isinstance(child, Cousin) |
_abc | 0.115us 1MiB...2MiB |
0.121us 1MiB...2MiB |
_py_abc | 0.350us 7MiB...9MiB |
0.354us 7MiB...10MiB |
not issubclass(Child, Cousin) |
_abc | 0.104us 0MiB |
0.110us 0MiB...1MiB |
_py_abc | 0.329us 4MiB |
0.333us 4MiB...5MiB |
not isinstance(child, Uncle) |
_abc | 7.268us 6174MiB...6333MiB |
0.125us 0MiB...1MiB |
_py_abc | 9.957us 4382MiB...4422MiB |
0.360us 6MiB |
not issubclass(Child, Uncle) |
_abc | 7.099us 6171MiB |
0.114us 0MiB |
_py_abc | 9.936us 4380MiB |
0.336us 4MiB...5MiB |
Memory increment is measured during isinstance() / issubclass() calls, not during preparation, like class creation or registration where actual registry allocation is performed. So memory usage in tables below is almost always 0.
Timing drop in _py_abc implementation for 2 first rows is due to if subclass in cls._abc_registry: check added to match _abc.c implementation.
| Check | Impl | before | after | Impl | before | after |
|---|---|---|---|---|---|---|
isinstance(child, Parent.register) |
_abc | 0.273us 0MiB |
0.272us 0MiB |
_py_abc | 0.440us 0MiB |
0.275us 0MiB |
issubclass(Child, Parent.register) |
_abc | 0.154us 0MiB |
0.157us 0MiB |
_py_abc | 0.427us 0MiB |
0.261us 0MiB |
isinstance(child, Grandparent.register) |
_abc | 0.114us 0MiB |
0.120us 0MiB |
_py_abc | 0.253us 0MiB |
0.262us 0MiB |
issubclass(Child, Grandparent.register) |
_abc | 0.103us 0MiB |
0.109us 0MiB |
_py_abc | 0.240us 0MiB |
0.247us 0MiB |
not isinstance(child, Sibling.register) |
_abc | 0.027us 0MiB |
0.030us 1MiB |
_py_abc | 0.028us 0MiB |
0.029us 2MiB |
not issubclass(Child, Sibling.register) |
_abc | 0.018us 0MiB |
0.018us 1MiB |
_py_abc | 0.018us 0MiB |
0.018us 2MiB |
not isinstance(child, Cousin.register) |
_abc | 0.028us 0MiB |
0.028us 2MiB |
_py_abc | 0.028us 0MiB |
0.029us 3MiB |
not issubclass(Child, Cousin.register) |
_abc | 0.018us 0MiB |
0.018us 2MiB |
_py_abc | 0.019us 0MiB |
0.018us 3MiB |
not isinstance(child, Uncle.register) |
_abc | 0.249us 0MiB |
0.233us 2MiB |
_py_abc | 0.843us 0MiB |
0.866us 3MiB |
not issubclass(Child, Uncle.register) |
_abc | 0.238us 0MiB |
0.227us 2MiB |
_py_abc | 0.815us 0MiB |
0.839us 3MiB |
Just to check that nothing is broken:
| Check | Impl | before | after | Impl | before | after |
|---|---|---|---|---|---|---|
not isinstance(child, Unrelated) |
_abc | 0.028us 0MiB |
0.028us 0MiB |
_py_abc | 0.028us 0MiB |
0.029us 0MiB |
not issubclass(Child, Unrelated) |
_abc | 0.018us 0MiB |
0.018us 0MiB |
_py_abc | 0.018us 0MiB |
0.018us 0MiB |
not isinstance(child, UnrelatedABC) |
_abc | 0.118us 0MiB |
0.121us 0MiB |
_py_abc | 0.469us 0MiB |
0.466us 0MiB |
not issubclass(Child, UnrelatedABC) |
_abc | 0.110us 0MiB |
0.112us 0MiB |
_py_abc | 0.442us 0MiB |
0.442us 0MiB |
Flamegraphs for _py_abc impl and test issubclass_uncle (the most time and memory consuming case on main):
main_vs_pr150540.tar.gz