Issue 46149: FIPS usedforsecurity flag is no longer functional with OpenSSL 3.0.0

I added christian.heimes to the nosy list; I hope that's OK

usedforsecurity flag is based on setting the EVP_MD_CTX_FLAG_NON_FIPS_ALLOW flag. However this flag has no effect in OpenSSL 3.0.0.

    [root@lambada ~]# /opt/opsware/bin/python3
    Python 3.10.0 (default, Dec  8 2021, 17:05:23) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux-x86_64
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import _hashlib                                        
    >>> _hashlib.get_fips_mode()                               
    1
    >>> import hashlib
    >>> hashlib.md5(b'ceva', usedforsecurity=False).hexdigest()
    '970c7956028654ac329b12c10b112058'
    >>> hashlib.md5(b'ceva', usedforsecurity=True).hexdigest()
    '970c7956028654ac329b12c10b112058'

The last call, when usedforsecurity is True, should fail because MD5 is not allowed in FIPS mode.

I will add a GitHub pull request for this issue.

This is a known issue, see #40479. OpenSSL 3.0.0 new provider system requires a major redesign of hashlib module's internals. It's not a trivial change. My first attempt GH-19878 has some flaws and introduces a performance regression.

I was able to get past this issue with minimal changes: https://github.com/florinspatar/cpython/commit/3b16c65eb3e54c0be40413ebabf504356e303e8a
I see the issue you linked, handles multiple OpenSSL 3.0.0 problems.

AFAIK, python 3 no longer supports OpenSSL 1.0.2. Minimum OpenSSL version is 1.1.1, which has no FIPS module, so usedforsecurity flag can only be used with OpenSSL 3. I'm wondering if the issue you linked is still an enhancement or should be treated as an actual bug.

Your patch gets the work done, but it's even slower than my WIP patch set. Hashing is a performance critical path. The new fetch() API in OpenSSL 3.0.0 is substantially slower than the old OpenSSL 1.1.1 APIs.

Python 3.9 and earlier still support OpenSSL 1.0.2. I removed support for OpenSSL < 1.1.1 for Python 3.10. There are approvied FIPS providers for OpenSSL 1.1.1, e.g. RHEL 8 has a certified FIPS module for OpenSSL 1.1.1.