At exit, the Python finalization calls Python objects finalizers (the __del__() method) and deallocates memory. The daemon threads are a special kind of threads which continue to run during and after the Python finalization. They are causing race conditions and tricky bugs in the Python finalization.
This article covers bugs fixed in the Python finalization in Python 3.2 and Python 3.3 (2009 to 2011), and a backport in Python 2.7.8 (2014).
Drawing: #CoronaMaison by Luppi.
Don't destroy the GIL at exit
In November 2009, Antoine Pitrou implemented a new GIL (Global Interpreter Lock) in Python 3.2: commit 074e5ed9.
In September 2010, he found a crash with daemon threads while stressing test_threading: bpo-9901: GIL destruction can fail. test_finalize_with_trace() failed with:
Fatal Python error: pthread_mutex_destroy(gil_mutex) failed
He pushed a fix for this crash in Python 3.2, commit b0b384b7:
Issue #9901: Destroying the GIL in Py_Finalize() can fail if some other threads are still running. Instead, reinitialize the GIL on a second call to Py_Initialize().
The Python GIL internally uses a lock. If the lock is destroyed while a daemon thread is waiting for it, the thread can crash. The fix is to no longer destroy the GIL at exit.
Revert take_gil() backport to 2.7
In June 2014, Benjamin Peterson (Python 2.7 release manager) backported Antoine's change to Python 2.7: fix included in 2.7.8.
Problem: the Ceph project started to crash with Python 2.7.8.
In November 2014, the change was reverted in Python 2.7.9: see bpo-21963 discussion for the rationale.
In 2014, I already wrote:
Anyway, daemon threads are evil :-( Expecting them to exit cleanly automatically is not good. Last time I tried to improve code to cleanup Python at exit in Python 3.4, I also had a regression (just before the release of Python 3.4.0): see the issue #21788.
Conclusion
Daemon threads caused crashes in the Python finalization, first noticed in 2005.
Python 3.2 (released in February 2011) got a new GIL and also a bugfix for daemon thread. Python 3.3 (released in September 2012) also got a bugfix for daemon threads. The Python finalization became more reliable.
Changing Python finalization is risky. A backport of a bugfix into Python 2.7.8 caused a regression which required to revert the bugfix in Python 2.7.9.