gh-139871: Optimize small takes in bytearray.take_bytes#141741
Conversation
When less than half the buffer is taken just copy that small part out rather than doing a big alloc + memmove + big shrink.
vstinner
left a comment
There was a problem hiding this comment.
LGTM
Sorry, something went wrong.
|
Hmm, would it make sense to take this path on |
Sorry, something went wrong.
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
I'd need to measure more / not sure. The realloc code does memcpy + malloc if 25% of space can be saved: Line 2676 in ca1e86f Doing a malloc + memcpy + free instead of the memmove + malloc + memcpy + free would be good but not sure how much to tune to the particular allocator constants. My theory for usage pattern has been reading a block of bytes off a stream (network or terminal), appending (or readinto) the bytearray, then taking little chunks out of that as bytes. With that bytearray realloc code should recompact periodically effectively meaning this turns into a reasonably efficient ringbuffer.... (for that use case a "please repack but don't reduce capacity flag may be useful"). Not sure how common that usage is going to be though |
Sorry, something went wrong.
|
That sounds like Anyway, no need to block this PR on that. Thank you! |
Sorry, something went wrong.
e265ce8
into
python:main
Nov 20, 2025
⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️Hi! The buildbot AMD64 Debian root 3.x (tier-1) has failed when building commit e265ce8. What do you need to do:
You can take a look at the buildbot page here: https://buildbot.python.org/#/builders/345/builds/12729 Failed tests:
Failed subtests:
Summary of the results of the build (if available): == Click to see traceback logsTraceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_os/test_os.py", line 4009, in test_timerfd_interval
self.assertEqual(self.read_count_signaled(fd), 1)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 2 != 1
Traceback (most recent call last):
File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_os/test_os.py", line 4017, in test_timerfd_interval
self.assertEqual(self.read_count_signaled(fd), count)
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 4 != 3
|
Sorry, something went wrong.
|
Test failure looks independent of this change, |
Sorry, something went wrong.
…GH-141741) When less than half the buffer is taken just copy that small part out rather than doing a big alloc + memmove + big shrink.
…GH-141741) When less than half the buffer is taken just copy that small part out rather than doing a big alloc + memmove + big shrink.
When less than half the buffer is taken just copy that small part out rather than doing a big alloc + memcpy + big shrink.
cc: @vstinner, @encukou
.take_bytes([n])a zero-copy path frombytearraytobytes#139871