gh-127298: Refactor test_hashlib for better usedforsecurity & openssl fips mode env support.#127492

gpshead

Issue: Ensure builtin hashlib implementations honor usedforsecurity=True when _hashlib is in FIPS mode #127298

…environment support.

gpshead

!buildbot FIPS

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @gpshead for commit 59d9a85 🤖

The command will test the builders whose names match following regular expression: FIPS

The builders matched are:

AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR
AMD64 RHEL8 FIPS No Builtin Hashes PR
AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
AMD64 CentOS9 FIPS No Builtin Hashes PR

gpshead

!buildbot FIPS

bedevere-bot

🤖 New build scheduled with the buildbot fleet by @gpshead for commit bd46651 🤖

The command will test the builders whose names match following regular expression: FIPS

The builders matched are:

AMD64 CentOS9 FIPS Only Blake2 Builtin Hash PR
AMD64 RHEL8 FIPS No Builtin Hashes PR
AMD64 RHEL8 FIPS Only Blake2 Builtin Hash PR
AMD64 CentOS9 FIPS No Builtin Hashes PR

gpshead

#127467 is follow-on work to this that combined gets the FIPS mode buildbots passing in main.

Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>

vstinner

cc @stratakis @encukou

Co-authored-by: Victor Stinner <vstinner@python.org>

stratakis

Thanks for the PR! I'll have a look at it and I can also provide ssh access to the FIPS buildbots if that would make things easier to debug.

vstinner

I tested manually the change on RHEL8. 3 MD5 tests of test_hashlib are failing:

======================================================================
ERROR: test_case_md5_0 (test.test_hashlib.HashLibTestCase.test_case_md5_0)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 547, in test_case_md5_0
    self.check(
    ~~~~~~~~~~^
        'md5', b'', 'd41d8cd98f00b204e9800998ecf8427e',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        usedforsecurity=False
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 427, in check
    self.check_file_digest(name, data, hexdigest)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 447, in check_file_digest
    hashlib.file_digest(buf, digest).hexdigest(), hexdigest
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/hashlib.py", line 211, in file_digest
    digestobj = digest()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

======================================================================
ERROR: test_case_md5_1 (test.test_hashlib.HashLibTestCase.test_case_md5_1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 553, in test_case_md5_1
    self.check(
    ~~~~~~~~~~^
        'md5', b'abc', '900150983cd24fb0d6963f7d28e17f72',
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        usedforsecurity=False
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 427, in check
    self.check_file_digest(name, data, hexdigest)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 447, in check_file_digest
    hashlib.file_digest(buf, digest).hexdigest(), hexdigest
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/hashlib.py", line 211, in file_digest
    digestobj = digest()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

======================================================================
ERROR: test_case_md5_2 (test.test_hashlib.HashLibTestCase.test_case_md5_2)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 559, in test_case_md5_2
    self.check(
    ~~~~~~~~~~^
        'md5',
        ^^^^^^
    ...<2 lines>...
        usedforsecurity=False
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 427, in check
    self.check_file_digest(name, data, hexdigest)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/test/test_hashlib.py", line 447, in check_file_digest
    hashlib.file_digest(buf, digest).hexdigest(), hexdigest
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/vstinner/cpython/Lib/hashlib.py", line 211, in file_digest
    digestobj = digest()
ValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS

----------------------------------------------------------------------

vstinner

I built Python with ./configure --with-pydebug && make. MD5 is available:

[vstinner@python-builder-rhel8-fips cpython]$ ./python 
Python 3.14.0a2+ (heads/pr/127492:3f9ec82db, Dec  5 2024, 06:10:34) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import hashlib
>>> hashlib.md5(usedforsecurity=False).hexdigest()
'd41d8cd98f00b204e9800998ecf8427e'

vstinner

Suggested fix for file digest:

diff --git a/Lib/hashlib.py b/Lib/hashlib.py
index 44656c33a..93d602571 100644
--- a/Lib/hashlib.py
+++ b/Lib/hashlib.py
@@ -192,7 +192,7 @@ def __hash_new(name, data=b'', **kwargs):
     pass
 
 
-def file_digest(fileobj, digest, /, *, _bufsize=2**18):
+def file_digest(fileobj, digest, /, *, usedforsecurity=True, _bufsize=2**18):
     """Hash the contents of a file-like object. Returns a digest object.
 
     *fileobj* must be a file-like object opened for reading in binary mode.
@@ -206,9 +206,9 @@ def file_digest(fileobj, digest, /, *, _bufsize=2**18):
     # On Linux we could use AF_ALG sockets and sendfile() to archive zero-copy
     # hashing with hardware acceleration.
     if isinstance(digest, str):
-        digestobj = new(digest)
+        digestobj = new(digest, usedforsecurity=usedforsecurity)
     else:
-        digestobj = digest()
+        digestobj = digest(usedforsecurity=usedforsecurity)
 
     if hasattr(fileobj, "getbuffer"):
         # io.BytesIO object, use zero-copy buffer
diff --git a/Lib/test/test_hashlib.py b/Lib/test/test_hashlib.py
index 1c1a0396c..a7cca0ba5 100644
--- a/Lib/test/test_hashlib.py
+++ b/Lib/test/test_hashlib.py
@@ -443,11 +443,13 @@ def check_file_digest(self, name, data, hexdigest):
             for digest in digests:
                 buf = io.BytesIO(data)
                 buf.seek(0)
-                self.assertEqual(
-                    hashlib.file_digest(buf, digest).hexdigest(), hexdigest
-                )
+                digestobj = hashlib.file_digest(buf, digest,
+                                                usedforsecurity=False)
+                self.assertEqual(digestobj.hexdigest(), hexdigest)
+
                 with open(os_helper.TESTFN, "rb") as f:
-                    digestobj = hashlib.file_digest(f, digest)
+                    digestobj = hashlib.file_digest(f, digest,
+                                                    usedforsecurity=False)
                 self.assertEqual(digestobj.hexdigest(), hexdigest)
         finally:
             os.unlink(os_helper.TESTFN)

xnox

@gpshead the suggestion from @vstinner looks reasonable in #127492 (comment) will you push that to this pr? Or should that change be done separately, with a separate blurb entry?

xnox

@gpshead hi, my PRs had comments that you are taking over with these changes instead, but they now seem to have stalled. Are you still actively working on this and related PRs?

picnixz

Note that changing how we materialize a digest object should done carefully (see PEP-452). For new(), it should be fine as we're using our own hsahlib.new but for digest(), we should be careful here because not all digest constructors necessarily need to accept usedforsecurity (see the examples in https://docs.python.org/3/library/hashlib.html#hashlib.file_digest):

buf = io.BytesIO(b"somedata")
mac1 = hmac.HMAC(b"key", digestmod=hashlib.sha512)
digest = hashlib.file_digest(buf, lambda: mac1)

xnox

@gpshead Any updates on landing this?

github-actions

This PR is stale because it has been open for 30 days with no activity.

Refactor test_hashlib for better usedforsecurity & openssl fips mode … …

59d9a85

…environment support.

gpshead added tests Tests in the Lib/test dir skip news labels Dec 1, 2024

More hashlib robustness in strange "FIPS mode" environments.

bd46651

gpshead force-pushed the hashlib-fips-oddmode branch from 7a5ab6c to bd46651 Compare December 2, 2024 03:42

gpshead mentioned this pull request Dec 2, 2024

gh-84559: multiprocessing: detect if forkserver cannot work due to missing hmac-sha256 #127467

Draft

2 tasks

remove some obsolete comments

29a2012

gpshead mentioned this pull request Dec 2, 2024

gh-127298: When in FIPS mode ensure builtin hashes check for usedforsecurity=False #127301

Open

remove debug print

e2bf70c

gpshead changed the title ~~Refactor test_hashlib for better usedforsecurity & openssl fips mode env support.~~ Dec 2, 2024

bedevere-app Bot mentioned this pull request Dec 2, 2024

Ensure builtin hashlib implementations honor usedforsecurity=True when _hashlib is in FIPS mode #127298

Open

gpshead self-assigned this Dec 2, 2024

gpshead added stdlib Standard Library Python modules in the Lib/ directory and removed labels Dec 2, 2024

gpshead marked this pull request as ready for review December 2, 2024 04:38

gpshead requested a review from tiran as a code owner December 2, 2024 04:38

bedevere-app Bot added the awaiting core review label Dec 2, 2024

gpshead requested review from vstinner and removed request for tiran December 2, 2024 04:38

nineteendo reviewed Dec 2, 2024

View reviewed changes

docs typo …

3f69411

Co-authored-by: Nice Zombies <nineteendo19d0@gmail.com>

vstinner reviewed Dec 2, 2024

View reviewed changes

gpshead and others added 3 commits December 2, 2024 08:31

remove unnecessary as in except statement, …

2a5b7e8

Co-authored-by: Victor Stinner <vstinner@python.org>

More details in NEWS

a6a75d6

news typo

3f9ec82

ghmj2417 mentioned this pull request Dec 18, 2024

OpenSSL with FIPS enabled causes test_hashlib failures #128071

Closed

github-actions Bot added the stale Stale PR or inactive for long period of time. label Apr 20, 2026

Conversation

gpshead commented Dec 1, 2024 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpshead commented Dec 1, 2024

Uh oh!

bedevere-bot commented Dec 1, 2024

Uh oh!

gpshead commented Dec 1, 2024

Uh oh!

bedevere-bot commented Dec 1, 2024

Uh oh!

gpshead commented Dec 2, 2024

Uh oh!

vstinner left a comment

Choose a reason for hiding this comment

Uh oh!

stratakis commented Dec 3, 2024

Uh oh!

vstinner commented Dec 5, 2024

Uh oh!

vstinner commented Dec 5, 2024

Uh oh!

vstinner commented Dec 5, 2024

Uh oh!

xnox commented Dec 11, 2024

Uh oh!

xnox commented Apr 28, 2025

Uh oh!

picnixz commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xnox commented Sep 8, 2025

Uh oh!

github-actions Bot commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

gpshead commented Dec 1, 2024 •

edited by bedevere-app Bot

Loading

picnixz commented Apr 28, 2025 •

edited

Loading