Issue 21412: Solaris/Oracle Studio: Fatal Python error: PyThreadState_Get when built --with-pymalloc

I am porting Python 3.4.0 to Solaris 12.  The Makefile I inherited from my predecessor had --without-pymalloc as an option to be passed to configure.  Curious why, I removed this line, only to find that after python was built, it core dumped:

LD_LIBRARY_PATH=/builds/jbeck/ul-python-3/components/python/python34/build/sparcv9 ./python -E -S -m sysconfig --generate-posix-vars
Fatal Python error: PyThreadState_Get: no current thread
make[3]: *** [pybuilddir.txt] Abort (core dumped)

But if I add the --without-pymalloc line back to my Makefile, everything works fine.
 
Note that although this example was on sparc, the exact same thing occurred on x86.

I searched for a similar bug but did not find out; please feel free to close this as a duplicate if there is one that I missed.  I also suspect I have not provided enough information, out of a desire not to trigger information overload.  But I would be happy to provide whatever specifics might be requested.

On SPARC/suncc the flags in http://bugs.python.org/issue15963#msg170661
appear to work.

Also, we have several Solaris build slaves that don't core dump.
Some are offline, but you can click through to the ./configure
steps of past builds to see the build flags.

http://buildbot.python.org/all/waterfall?category=3.x.stable&category=3.x.unstable

What compiler are you using?.

I compile fine on Solaris with GCC.

Using Oracle Studio 12.3, same as mentioned in http://bugs.python.org/issue15963#msg170661 (as Stefan pointed out).  I am using some of those flags but not all of them.  I will try the others when I have a chance, then report back.

"LD_LIBRARY_PATH=/builds/jbeck/ul-python-3/components/python/python34/build/sparcv9 ./python -E -S -m sysconfig --generate-posix-vars
Fatal Python error: PyThreadState_Get: no current thread"

Could you please run this command in gdb and copy/paste the C traceback (gdb command "where") where the fatal error occurs?

Victor: sure; see attached.

> Victor: sure; see attached.

Ok, so the error occurs when Python tries to import the _heapq dynamic module: PyModule_Create2() calls PyThreadState_Get() to retrieve to current thread, but it fails. There is a current thread because PyModule_Create2() is called indirectly by PyEval_EvalFrameExReal() (and I don't see where the GIL would be released in the call stack).

It looks like a bug in PyThreadState_Get(). This function relies on _Py_atomic_load_relaxed() which is defined in Include/pyatomic.h. This file has an implementation of atomic functions for Intel processors and contains an interesting comment:

...
#else  /* !gcc x86 */
/* Fall back to other compilers and processors by assuming that simple
   volatile accesses are atomic.  This is false, so people should port
   this. */
...

It looks like John tries Python on SPARC which may explain the issue.

This is just a theory. It also looks like we had SPARC buildbots running on Solaris with system C compiler ("/opt/solarisstudio12.3/bin/cc") and it was able to run tests.

I don't understand the link with pymalloc.

@John: Did you try to build Python 3.3? Did it work?

Victor:

* This is not a SPARC-specific issue; the exact same failure occurs
  on x86.

* I had built Python 3.3 (some time ago) but only --without-pymalloc.
  But I tried just now rebuilt Python 3.3 --with-pymalloc, and it
  failed in the exact same way.

"This is not a SPARC-specific issue; the exact same failure occurs on x86."

Ah ok, good to know. To me, it looks like a compiler issue. Did you try Stefan's advices in issue #15963?

You may try to disable compiler optimizations to see if you get the same behaviour.

This appears to be another variation on the problem recently identified in Issue21166, namely that the pybuildir.txt Makefile rule can incorrectly import a shared library module from a previously installed Python instance and, if the ABIs of the installed and being-built Pythons differ, the newly-built interpreter can fail in various ways.  From your supplied trace, one can see that _heapq.so has incorrectly been inported from the installed system Python 3.4 which was probably built with --without-pymalloc:

#7  0x00007ff2f9ee2a6d in PyInit__heapq ()
   from /usr/lib/python3.4/lib-dynload/64/_heapq.so
#8  0x00007ff2f94c7c78 in _PyImport_LoadDynamicModule ()
   from /builds/jbeck/ul-python-3/components/python/python34/build/amd64/libpython3.4m.so.1.0

The fixes for Issue21166, when applied, should prevent this problem.

Ned: yes, I can confirm that the patch from http://bugs.python.org/issue21166 does indeed fix the problem.  Thank you very much!

superseder: Bus error in pybuilddir.txt 'python -m sysconfigure --generate-posix-vars' build step
components: + Build, - Interpreter Core

nosy: + ned.deily
messages: + msg225218
resolution: duplicate
stage: resolved

messages: + msg217804