bpo-35879: Fix type comment leaks#11728

gvanrossum

https://bugs.python.org/issue35879

gvanrossum

Since this just fixes a bug introduced by yesterday's merge of GH-11645, I don't think we need a blurb.

Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

pablogsal

LGTM!

Finally! I spent al leat 3 hours tracking this leak. How did you find that PyArena_AddPyObject was the missing piece?

pablogsal

I have run the entire test suite with ref leaks activated and it succeeds. Here is test_type_comments:

~/github/cpython remotes/upstream/fix-type-comment-leaks ✔
❯ ./python -m test test_type_comments -R 10:10
Run tests sequentially
0:00:00 load avg: 0.59 [1/1] test_type_comments
beginning 20 repetitions
12345678901234567890
....................

== Tests result: SUCCESS ==

1 test OK.

Total duration: 168 ms
Tests result: SUCCESS

gvanrossum

How did you find that PyArena_AddPyObject was the missing piece?

I kept reading the source, trying to understand how this was done for other fields. Finally I found that identifiers had the same pattern (a string == PyObject * in the struct) and then realized that those are allocated via new_identifier()/NEW_IDENTIFIER(), which was a pattern very similar to new_type_comment()/NEW_TYPE_COMMENT(), and reading through it found that call at the end. This also solved the mystery why new_type_comment() had an unused struct compiling *c argument that was passed by the NEW_TYPE_COMMENT() macro -- this was copied NEW_IDENTIFIER(), where it was needed for the arena.

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot PPC64 Fedora 3.x has failed when building commit d2b4c19.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/53/builds/2271) and take a look at the build logs.
Check if the failure is related to this commit (d2b4c19) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/53/builds/2271

Click to see traceback logs

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 129, in worker
    put((job, i, result))
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in put
    self._writer.send_bytes(obj)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 134, in worker
    put((job, i, (False, wrapped)))
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in put
    self._writer.send_bytes(obj)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process SpawnPoolWorker-229:
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 129, in worker
    put((job, i, result))
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in put
    self._writer.send_bytes(obj)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 134, in worker
    put((job, i, (False, wrapped)))
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in put
    self._writer.send_bytes(obj)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Process SpawnPoolWorker-230:
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 129, in worker
    put((job, i, result))
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in put
    self._writer.send_bytes(obj)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 302, in _bootstrap
    self.run()
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/pool.py", line 134, in worker
    put((job, i, (False, wrapped)))
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/queues.py", line 368, in put
    self._writer.send_bytes(obj)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
/home/shager/cpython-buildarea/3.x.edelsohn-fedora-ppc64/build/Lib/multiprocessing/semaphore_tracker.py:164: UserWarning: semaphore_tracker: There appear to be 4 leaked semaphores to clean up at shutdown
  warnings.warn('semaphore_tracker: There appear to be %d '

pablogsal

This failure is not related with this PR (is some broken pipe in multiprocessing).

Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

gvanrossum added 2 commits February 1, 2019 13:31

Fix leak for # type: ignore

ef3b9e9

Fix the type comment leak

88975a8

the-knights-who-say-ni added the CLA signed label Feb 1, 2019

bedevere-bot added the awaiting merge label Feb 1, 2019

gvanrossum added the skip news label Feb 1, 2019

gvanrossum pushed a commit to python/typed_ast that referenced this pull request Feb 1, 2019

Fix memory leaks caused by type comment parsing …

6f47b31

Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

pablogsal approved these changes Feb 1, 2019

View reviewed changes

gvanrossum merged commit d2b4c19 into master Feb 1, 2019

bedevere-bot removed the awaiting merge label Feb 1, 2019

gvanrossum deleted the fix-type-comment-leaks branch February 1, 2019 23:28

gvanrossum mentioned this pull request Feb 1, 2019

bpo-35766: Merge typed_ast back into CPython #11645

Merged

gvanrossum pushed a commit to python/typed_ast that referenced this pull request Feb 5, 2019

Fix memory leaks caused by type comment parsing …

b7d582a

Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

gvanrossum mentioned this pull request Feb 5, 2019

Fix memory leaks caused by type comment parsing python/typed_ast#91

Merged

gvanrossum added a commit to python/typed_ast that referenced this pull request Feb 6, 2019

Fix memory leaks caused by type comment parsing (#91) …

c6d7fcb

Echoes python/cpython#11728 (they were found by the CPython refleak detector, see https://bugs.python.org/issue35879).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-35879: Fix type comment leaks#11728

bpo-35879: Fix type comment leaks#11728
gvanrossum merged 2 commits into
masterfrom
fix-type-comment-leaks

gvanrossum commented Feb 1, 2019 •

edited by bedevere-bot

Loading

Uh oh!

gvanrossum commented Feb 1, 2019

Uh oh!

pablogsal left a comment

Uh oh!

pablogsal commented Feb 1, 2019

Uh oh!

gvanrossum commented Feb 1, 2019

Uh oh!

bedevere-bot commented Feb 2, 2019

Uh oh!

pablogsal commented Feb 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

gvanrossum commented Feb 1, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gvanrossum commented Feb 1, 2019

Uh oh!

pablogsal left a comment

Choose a reason for hiding this comment

Uh oh!

pablogsal commented Feb 1, 2019

Uh oh!

gvanrossum commented Feb 1, 2019

Uh oh!

bedevere-bot commented Feb 2, 2019

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

pablogsal commented Feb 2, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gvanrossum commented Feb 1, 2019 •

edited by bedevere-bot

Loading