gh-101659: Isolate "obmalloc" State to Each Interpreter by ericsnowcurrently · Pull Request #101660 · python/cpython

In pythongh-102744 we added is_core_module() (in Python/import.c), which relies on get_core_module_dict() (also added in that PR).  The problem is that_PyImport_FixupBuiltin(), which ultimately calls is_core_module(), is called on the builtins module before interp->builtins_copyis set.  Consequently, the builtins module isn't considered a "core" module while it is getting "fixed up" and its module def m_copy erroneously gets set.  Under isolated interpreters this causes problems since sys and builtins are allowed even though they are still single-phase init modules.  (This was discovered while working on pythongh-101660.)

The solution is to stop relying on get_core_module_dict() in is_core_module().

pythongh-103287)

Using the raw allocator for any of the global state makes sense, especially as we move to a per-interpreter obmalloc state (pythongh-101660).

The function is like Py_AtExit() but for a single interpreter.  This is a companion to the atexit module's register() function, taking a C callback instead of a Python one.

We also update the _xxinterpchannels module to use _Py_AtExit(), which is the motivating case.  (This is inspired by pain points felt while working on pythongh-101660.)

…ules (pythongh-102661)

It doesn't make sense to use multi-phase init for these modules. Using a per-interpreter "m_copy" (instead of PyModuleDef.m_base.m_copy) makes this work okay. (This came up while working on pythongh-101660.)

Note that we might instead end up disallowing re-load for sys/builtins since they are so special.

python#102660

…02658)

The error-handling code in new_interpreter() has been broken for a while.  We hadn't noticed because those code mostly doesn't fail.  (I noticed while working on pythongh-101660.)  The problem is that we try to clear/delete the newly-created thread/interpreter using itself, which just failed.  The solution is to switch back to the calling thread state first.

python#98608

…102663)

Aside from sys and builtins, _io is the only core builtin module that hasn't been ported to multi-phase init.  We may do so later (e.g. pythongh-101948), but in the meantime we must at least take care of the module's static types properly.  (This came up while working on pythongh-101660.)

python#94673

…rpreters (pythongh-102925)

This is effectively two changes.  The first (the bulk of the change) is where we add _Py_AddToGlobalDict() (and _PyRuntime.cached_objects.main_tstate, etc.).  The second (much smaller) change is where we update PyUnicode_InternInPlace() to use _Py_AddToGlobalDict() instead of calling PyDict_SetDefault() directly.

Basically, _Py_AddToGlobalDict() is a wrapper around PyDict_SetDefault() that should be used whenever we need to add a value to a runtime-global dict object (in the few cases where we are leaving the container global rather than moving it to PyInterpreterState, e.g. the interned strings dict).  _Py_AddToGlobalDict() does all the necessary work to make sure the target global dict is shared safely between isolated interpreters.  This is especially important as we move the obmalloc state to each interpreter (pythongh-101660), as well as, potentially, the GIL (PEP 684).

python#100227

…Interpreters (pythongh-103084)

Sharing mutable (or non-immortal) objects between interpreters is generally not safe.  We can work around that but not easily. 
 There are two restrictions that are critical for objects that break interpreter isolation.

The first is that the object's state be guarded by a global lock.  For now the GIL meets this requirement, but a granular global lock is needed once we have a per-interpreter GIL.

The second restriction is that the object (and, for a container, its items) be deallocated/resized only when the interpreter in which it was allocated is the current one.  This is because every interpreter has (or will have, see pythongh-101660) its own object allocator.  Deallocating an object with a different allocator can cause crashes.

The dict for the cache of module defs is completely internal, which simplifies what we have to do to meet those requirements.  To do so, we do the following:

* add a mechanism for re-using a temporary thread state tied to the main interpreter in an arbitrary thread
   * add _PyRuntime.imports.extensions.main_tstate` 
   * add _PyThreadState_InitDetached() and _PyThreadState_ClearDetached() (pystate.c)
   * add _PyThreadState_BindDetached() and _PyThreadState_UnbindDetached() (pystate.c)
* make sure the cache dict (_PyRuntime.imports.extensions.dict) and its items are all owned by the main interpreter)
* add a placeholder using for a granular global lock

Note that the cache is only used for legacy extension modules and not for multi-phase init modules.

python#100227

Decref the key in the right interpreter in _extensions_cache_set().

This is a follow-up to pythongh-103084. I found the bug while working on pythongh-101660.

In pythongh-102744 we added is_core_module() (in Python/import.c), which relies on get_core_module_dict() (also added in that PR).  The problem is that_PyImport_FixupBuiltin(), which ultimately calls is_core_module(), is called on the builtins module before interp->builtins_copyis set.  Consequently, the builtins module isn't considered a "core" module while it is getting "fixed up" and its module def m_copy erroneously gets set.  Under isolated interpreters this causes problems since sys and builtins are allowed even though they are still single-phase init modules.  (This was discovered while working on pythongh-101660.)

The solution is to stop relying on get_core_module_dict() in is_core_module().

pythongh-103287)

Using the raw allocator for any of the global state makes sense, especially as we move to a per-interpreter obmalloc state (pythongh-101660).

The function is like Py_AtExit() but for a single interpreter.  This is a companion to the atexit module's register() function, taking a C callback instead of a Python one.

We also update the _xxinterpchannels module to use _Py_AtExit(), which is the motivating case.  (This is inspired by pain points felt while working on pythongh-101660.)