◐ Shell
reader mode source ↗
Skip to content

gh-132983: Don't allow trailer data in ZstdFile#133736

Merged
AA-Turner merged 1 commit into
python:mainfrom
Rogdham:zstdfile-trailer-exception
May 10, 2025
Merged

gh-132983: Don't allow trailer data in ZstdFile#133736
AA-Turner merged 1 commit into
python:mainfrom
Rogdham:zstdfile-trailer-exception

Conversation

@Rogdham

@Rogdham Rogdham commented May 9, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

We previously made sure that an exception is raised when decompressing trailer data with decompress:

>>> from compression.zstd import compress, decompress
>>> invalid = compress(b'xxx') + b'yyy'
>>> decompress(invalid)
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    decompress(invalid)
    ~~~~~~~~~~^^^^^^^^^
  File "/redacted/Lib/compression/zstd/__init__.py", line 157, in decompress
    results.append(decomp.decompress(data))
                   ~~~~~~~~~~~~~~~~~^^^^^^
_zstd.ZstdError: Unable to decompress zstd data: Unknown frame descriptor

Indeed, the Zstandard specification says “Zstandard compressed data is made of one or more frames”, and it does not say that random data can be added at the end.

However, this is not the case in ZstdFile / zstd.open:

>>> from compression.zstd import ZstdFile
>>> from io import BytesIO
>>> ZstdFile(BytesIO(invalid)).read()
b'xxx'

After this PR, the last call becomes:

>>> ZstdFile(BytesIO(invalid)).read()
Traceback (most recent call last):
  File "<python-input-5>", line 1, in <module>
    ZstdFile(BytesIO(invalid)).read()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/redacted/Lib/compression/zstd/_zstdfile.py", line 176, in read
    return self._buffer.read(size)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/redacted/Lib/compression/_common/_streams.py", line 118, in readall
    while data := self.read(sys.maxsize):
                  ~~~~~~~~~^^^^^^^^^^^^^
  File "/redacted/Lib/compression/_common/_streams.py", line 91, in read
    data = self._decompressor.decompress(rawblock, size)
_zstd.ZstdError: Unable to decompress zstd data: Unknown frame descriptor

@bedevere-app bedevere-app Bot mentioned this pull request May 9, 2025
Closed
13 tasks
@Rogdham Rogdham marked this pull request as ready for review May 9, 2025 09:36
@AA-Turner AA-Turner added the needs backport to 3.14 bugs and security fixes label May 9, 2025
@emmatyping

emmatyping commented May 9, 2025

Copy link
Copy Markdown
Member

The current behavior matches LZMA. I think unlike decompress which is handed what is necessarily a zstd stream of one or more frames, with ZstdFile, a user may be parsing a format which has additional information after a zstd stream.

>>> from lzma import LZMAFile, compress
>>> from io import BytesIO
>>> invalid = compress(b'foo') + b'bar'
>>> LZMAFile(BytesIO(invalid)).read()
b'foo'
>>>

@Rogdham

Rogdham commented May 9, 2025

Copy link
Copy Markdown
Contributor Author

You are right this is the case for LZMAFile with format FORMAT_AUTO (which is the default) and also for BZ2File.

However, LZMAFile with format FORMAT_XZ as well as GzipFile raise an exception in that case.

>>> from lzma import LZMAFile, compress, FORMAT_XZ
>>> from io import BytesIO
>>> invalid = compress(b'foo') + b'bar'
>>> LZMAFile(BytesIO(invalid), format=FORMAT_XZ).read()
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    LZMAFile(BytesIO(invalid), format=FORMAT_XZ).read()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/redacted/lzma.py", line 208, in read
    return self._buffer.read(size)
           ~~~~~~~~~~~~~~~~~^^^^^^
  File "/redacted/_compression.py", line 118, in readall
    while data := self.read(sys.maxsize):
                  ~~~~~~~~~^^^^^^^^^^^^^
  File "/redacted/_compression.py", line 99, in read
    raise EOFError("Compressed file ended before the "
                   "end-of-stream marker was reached")
EOFError: Compressed file ended before the end-of-stream marker was reached

emmatyping
emmatyping approved these changes May 9, 2025
View reviewed changes

@emmatyping emmatyping left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Okay this looks good then!

@Rogdham

Rogdham commented May 9, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

In addition, consider decompress(compress(b"xxx") + b"yyy"):

  • returns b"xxx" on: lzma (format FORMAT_AUTO), bz2
  • raises an exception on: lzma (format FORMAT_XZ), gzip

Since for zstd we raise an exception on that, I would say to do the same for ZstdFile to be consistent.

Hide details View details @AA-Turner AA-Turner merged commit 50b5370 into python:main May 10, 2025
48 checks passed
@miss-islington-app

miss-islington-app Bot commented May 10, 2025

Copy link
Copy Markdown

Thanks @Rogdham for the PR, and @AA-Turner for merging it 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request May 10, 2025
@Rogdham @miss-islington
pythongh-132983: Don't allow trailer data in ZstdFile (pythonGH-133736) …
0c55b02 
(cherry picked from commit 50b5370)

Co-authored-by: Rogdham <3994389+Rogdham@users.noreply.github.com>
@bedevere-app

bedevere-app Bot commented May 10, 2025

Copy link
Copy Markdown

GH-133799 is a backport of this pull request to the 3.14 branch.

@bedevere-app bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label May 10, 2025
AA-Turner pushed a commit that referenced this pull request May 10, 2025
@miss-islington @Rogdham
[3.14] gh-132983: Don't allow trailer data in ZstdFile (GH-133736) (#… …
99ca086 
…133799)

gh-132983: Don't allow trailer data in ZstdFile (GH-133736)
(cherry picked from commit 50b5370)

Co-authored-by: Rogdham <3994389+Rogdham@users.noreply.github.com>
@Rogdham Rogdham deleted the zstdfile-trailer-exception branch May 10, 2025 06:45
Pranjal095 pushed a commit to Pranjal095/cpython that referenced this pull request Jul 12, 2025
@Rogdham @Pranjal095
pythongh-132983: Don't allow trailer data in ZstdFile (python#133736)
3882f69
taegyunkim pushed a commit to taegyunkim/cpython that referenced this pull request Aug 4, 2025
@Rogdham @taegyunkim
pythongh-132983: Don't allow trailer data in ZstdFile (python#133736)
11bf99d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@emmatyping emmatyping emmatyping approved these changes

Assignees

No one assigned

Labels

skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Rogdham @emmatyping @AA-Turner