@pitrou @orsenthil @tiran I would very much appreciate to get another round of feedback. I am available for pushing this further, so that maybe we can still land this in 3.7. With all the information present across the issue tracker and this GH conv I want to stress again what's probably easy to miss: this is meant to be a conceptually backwards-compatible change, because the old code path remains intact.

Updated:

• Responded more in-depth to review.
• Merged current master into this branch; manually resolved conflicts (that was required especially after bpo-31346: Use PROTOCOL_TLS_CLIENT/SERVER #3058 and bpo-29464: Rename METH_FASTCALL to METH_FASTCALL|METH_KEYWORDS and make #1955 landed recently).
• Added code comments based on review feedback.
• Built successfully on Fedora 26's gcc version 7.2.1 (new dereferencing pointer to incomplete type errors newly appeared compared to my work from a year ago).
• Confirmed that ./python -m test -v test_ssl passes (one error message emitted by OpenSSL seems to have changed from OpenSSL 1.0.2 to 1.1.0, and I changed one assertRaisesRegex(SSLError, ...) back to simply assertRaises(SSLError))
• Added NEWS entry.

I'm still surprised that this functionality needs so much support code, but I'll let @tiran review it.

I'm still surprised that this functionality needs so much support code, but I'll let @tiran review it.

Gotcha. At the core that's because there is unfortunately no such thing like SSL_CTX_use_certificate_chain_bio() provided by the OpenSSL API: a bio pendant corresponding to SSL_CTX_use_certificate_chain_file() would be super convenient.

That's what this patch adds and what's called _openssl_use_certificate_chain_from_bio().

Thanks for giving this another quick thought @pitrou, much appreciated.

Edit: I've tried to write this before, but I want to retry with more clarity: _openssl_use_certificate_chain_from_bio() is effectively static int use_certificate_chain_file(SSL_CTX *ctx, SSL *ssl, const char *file) straight from the OpenSSL source, just without BIO creation.

OpenSSL's use_certificate_chain_file(SSL_CTX *ctx, SSL *ssl, const char *file) expects a file path as input and then immediately performs BIO creation from a file (via BIO_read_filename(in, file)), and then processes this BIO with the "OpenSSL cert chain loading logic" (a term I will use again below).

In this patch, we create the BIO using the nice existing Python API for Memory BIO management (which was added in Python 3.5). That results in PySSLMemoryBIO objects that hold the OpenSSL-native bio structs internally.

The input to the _openssl_use_certificate_chain_from_bio() added by this patch is precisely the native OpenSSL Bio struct. The function then proceeds with the "OpenSSL cert chain loading logic" copied from OpenSSL' use_certificate_chain_file(SSL_CTX *ctx, SSL *ssl, const char *file).

This is why _openssl_use_certificate_chain_from_bio() is OpenSSL's use_certificate_chain_file(SSL_CTX *ctx, SSL *ssl, const char *file) modulo BIO creation.

That

1. leaves the old use_certificate_chain_file()-based code path intact (for conceptually ensuring backwards compat).
2. should conceptually ensure that the new code path has the same effects on a given SSLContext when using the same input data, just from memory instead of from file.

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@tiran any idea if you'll remove the block on this? More and more people are running into this issue.

It appears the perfect is the enemy of the good here -- this bug is outstanding for nearly a decade and waiting for a full rewrite is a classic engineering tarpit. PEP 543 also appears to be moribund, but that's a separate issue.

+1 for @nimish What need to happen to push the thread forward?
Exposing certificate and private key on host's file system in production increases the risk to expose these credentials.

Putting social pressure on me is not how you are going to land this PR. As I tried to explain earlier I'm not against the feature per se. But I want to a clean and well designed API with a maintainable implementation.

Ultimately I will be the person who has to maintain the code for the next decade or two. We have very few core devs that are familiar with X.509, TLS/SSL, and OpenSSL APIs. I'm doing majority of work and have limited personal (and unpaid) time to contribute to CPython.

If you don't agree with my decision you can ask another core dev to overrule me, take over ownership of the ssl module, or appeal to the Python steering council and ask for mediation.

In the mean time please watch @brettcannon's keynote from PyCon 2018, https://youtu.be/tzFWz5fiVKU?t=2964 .

Exposing certificate and private key on host's file system in production increases the risk to expose these credentials.

I don't agree with your reasoning. If you cannot trust a secure temp directory, then you have more serious problems in your server and application design. Besides private keys should always be encrypted with PKCS#5 v2.0 and a secure passphrase so access to the encrypted key file does not pose a security risk.

@tiran, perhaps a recipe that shows how to use mkstemp() to load certificate that are currently in memory could be useful while work on PEP 543 is ongoing?

@tiran Sure, can you then close this as a WONTFIX? It's in limbo right now.

Your previous comment just refer to developing a PEP 543 implementation. If this is not going to be merged at all in favor of PEP 543 library, then I'd like to ask that this and the BPO bug be closed with that reason vs being left tantalizingly open.

@tiran, perhaps a recipe that shows how to use mkstemp() to load certificate that are currently in memory could be useful while work on PEP 543 is ongoing?

#!/usr/bin/env python3

import tempfile
import ssl

def load_cert_memory(context, certdata, keydata, password):
    if not password:
        raise ValueError("Insecure key")
    with tempfile.NamedTemporaryFile(mode="wt") as f:
        if keydata is not None:
            f.write(keydata)
            if not keydata.endswith("\n"):
                f.write("\n")
        f.write(certdata)
        f.flush()
        context.load_cert_chain(f.name, password=password)


def test():
    passphrase = "somepass"
    with open("Lib/test/ssl_cert.pem") as f:
        testcert = f.read()
    with open("Lib/test/ssl_key.passwd.pem") as f:
        testkey = f.read()
    with open("Lib/test/keycert.passwd.pem") as f:
        combined = f.read()

    ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
    load_cert_memory(ctx, testcert, testkey, passphrase)

    ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
    load_cert_memory(ctx, combined, None, passphrase)


if __name__ == "__main__":
    test()

With PEP 543 marked as Withdrawn, what's the roadmap for getting this sort of functionality?

This PR is stale because it has been open for 30 days with no activity.

So ...
ssl_context.load_cert_chain(io.BytesIO(crtChain.encode("utf8")))
is still not possible? Can only use filenames? (its a sad situation if so ... people want in-memory keys exactly because it's more secure than storing them on the filesystem).

This PR is stale because it has been open for 30 days with no activity.

Unfortunately, closing because of four-year stalemate. The feature is useful by itself but found no support of the SSH part maintainer both from architectural standpoint:

If you cannot trust a secure temp directory, then you have more serious problems in your server and application design. Besides private keys should always be encrypted with PKCS#5 v2.0 and a secure passphrase so access to the encrypted key file does not pose a security risk.

and verification area creep:

I don't [want] to introduce yet another way to load a certificate. The C code is already complicated enough.

We have very few core devs that are familiar with X.509, TLS/SSL, and OpenSSL APIs. I'm doing majority of work and have limited personal (and unpaid) time to contribute to CPython.

Unfortunately, closing because of four-year stalemate

Decade-long stalemate! :) This issue was originally opened in 2012. No hard feelings. Thanks again for all the valuable contributions here everyone.

Please add make this to the standard library of ssl.py for extended functionality especially for context.load_cert_chain(certfile=cert_path, keyfile=key_path) so it can be used with embedded strings or bytes and not only with filepaths!