Issue 39406: Implement os.putenv() with setenv() if available

Currently, os.putenv() is always implemented with putenv(). The problem is that putenv(str) puts directly the string into the environment, the string is not copied. So Python has to keep track of this memory.

In Python 3.9, this string is now cleared at Python exit, without unsetting the environment variable which cause bpo-39395 crash.

I propose to implement os.putenv() with setenv() if available, which avoids bpo-39395 on platforms providing setenv().

setenv() is available on:

* Linux: http://man7.org/linux/man-pages/man3/setenv.3.html
* macOS: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/setenv.3.html
* FreeBSD: https://www.freebsd.org/cgi/man.cgi?query=getenv&sektion=3&apropos=0&manpath=FreeBSD+12.1-RELEASE

It is not available on Windows.

Maybe use SetEnvironmentVariable() on Windows?

> Maybe use SetEnvironmentVariable() on Windows?

I didn't know this function. I updated PR 18095.

> Maybe use SetEnvironmentVariable() on Windows? 

`_wputenv` keeps Python's copy of the environment block in sync with CRT's copy, which in turn calls `SetEnvironmentVariableW` to sync with process environment block. The CRT's copy of the environment excludes and disallows variable names that start with "=". Because the interpreter initializes `os.environ` from the CRT environment, Python has never included or allowed 'hidden' variables such as "=C:". 

Switching to calling `SetEnvironmentVariableW` directly won't affect `os.system` and `os.spawnv[e]`, since the CRT calls `CreateProcessW` without overriding the process environment. However, have you considered the consequences for extension modules and embedding applications that use the CRT environment (e.g. `getenv`, `_wgetenv`) if the interpreter stops syncing with it? 

The concern that `[_w]putenv` doesn't copy the string does not apply to Windows.

This indeed a good argument to continue to use _wputenv. Thank you Eryk, you have saved us from making a mistake.

I created bpo-39413 "Implement os.unsetenv() on Windows" to prepare work on this issue. But Eryk raised the same concern about CRT: https://bugs.python.org/issue39413#msg360404

I wasn't aware of that :-/

> `_wputenv` keeps Python's copy of the environment block in sync with CRT's copy, which in turn calls `SetEnvironmentVariableW` to sync with process environment block. The CRT's copy of the environment excludes and disallows variable names that start with "=". Because the interpreter initializes `os.environ` from the CRT environment, Python has never included or allowed 'hidden' variables such as "=C:". 

My main motivation to use SetEnvironmentVariableW() on Windows is to avoid bpo-39395. Do you mean that Windows _putenv() is not affected by bpo-39395: Python can safely removes the string to _putenv() just after the call?

New changeset 623ed6171eae35af7fd2e804dfd9c832c05c5d48 by Victor Stinner in branch 'master':
bpo-39406: Add PY_PUTENV_DICT macro to posixmodule.c (GH-18106)
https://github.com/python/cpython/commit/623ed6171eae35af7fd2e804dfd9c832c05c5d48

There is also _wputenv_s which affects _spawn, _exec and system.

Why posix_putenv_garbage was renamed to putenv_dict?

> Python can safely removes the string to _putenv() just after the call? 

Windows ucrt copies the buffer that's passed to _[w]putenv. This makes it non-compliant with POSIX but easier to use in this case. Here's an example using ctypes:

    >>> ucrt = ctypes.CDLL('ucrtbase')
    >>> buf = ctypes.create_unicode_buffer('spam=eggs')
    >>> ucrt._wputenv(buf)
    0

Directly modifying the buffer has no effect on the _wgetenv result:

    >>> buf[5] = 'a'
    >>> ucrt._wgetenv.restype = ctypes.POINTER(ctypes.c_wchar)
    >>> ucrt._wgetenv('spam')[:4]
    'eggs'

Linux putenv complies with POSIX, so changing "eggs" to "aggs" in the buffer does affect the getenv result in Linux. 

On the other hand, ucrt's [_w]getenv does not copy the environment string. For example:

    >>> p1 = ucrt._wgetenv('spam')
    >>> p1[0] = 'o'

    >>> p2 = ucrt._wgetenv('spam')
    >>> p2[:4]
    'oggs'

[_w]getenv is not thread safe. Even though all calls that access the ucrt environment are internally synchronized on a common lock, accessing the *result* from [_w]getenv still needs to be synchronized with concurrent _[w]putenv calls in a multithreaded process. Better still, use [_w]getenv_s or _[w]dupenv_s, which returns a copy.

> There is also _wputenv_s which affects _spawn, _exec and system.

The documentation is perhaps misleading. When a variable is set with _[w]putenv[_s], it also sets the variable in the process environment block to stay in sync. This indirectly affects _spawn*, _exec* and system(). common_spawnv (in ucrt\exec\spawnv.cpp), which implements the latter functions, calls CreateProcess with lpEnvironment as NULL (i.e.  inherit the current process environment), unless the caller passes a new environment (e.g. envp of spawnve).

New changeset 0852c7dd52ac42e7843ddfef44571494e4c86070 by Victor Stinner in branch 'master':
bpo-39406: os.putenv() avoids putenv_dict on Windows (GH-18126)
https://github.com/python/cpython/commit/0852c7dd52ac42e7843ddfef44571494e4c86070

New changeset b477d19a6b7751b0c933b239dae4fc96dbcde9c4 by Victor Stinner in branch 'master':
bpo-39406: Implement os.putenv() with setenv() if available (GH-18128)
https://github.com/python/cpython/commit/b477d19a6b7751b0c933b239dae4fc96dbcde9c4

Serhiy:
> Maybe use SetEnvironmentVariable() on Windows?

In bpo-39413, I implemented os.unsetenv() on Windows using SetEnvironmentVariable(), but Eryk reported that SetEnvironmentVariable() does not update the CRT API (_environ, _wenviron, getenv, etc.). So I reverted my change. See bpo-39413 for the details.


Eryk:
> Windows ucrt copies the buffer that's passed to _[w]putenv.

Thank you for checking manually. I pushed commit 0852c7dd52ac42e7843ddfef44571494e4c86070 to avoid putenv_dict on Windows.

The initial issue has been fixed, so I close the issue. Big thanks to Eryk Sun for your great help, thanks also Serhiy Storchaka.

stage: patch review -> resolved