allow per-thread atexit()

[tarekziade]

BPO	14073
Nosy	@tim-one, @pitrou, @vstinner, @tiran, @tarekziade, @zvezdan, @ericsnowcurrently

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-02-21.14:12:07.896>
labels = ['expert-subinterpreters', 'interpreter-core', 'type-feature', 'library']
title = 'allow per-thread atexit()'
updated_at = <Date 2020-06-03.16:42:40.636>
user = 'https://github.com/tarekziade'

bugs.python.org fields:

activity = <Date 2020-06-03.16:42:40.636>
actor = 'vstinner'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)', 'Subinterpreters']
creation = <Date 2012-02-21.14:12:07.896>
creator = 'tarek'
dependencies = []
files = []
hgrepos = []
issue_num = 14073
keywords = []
message_count = 16.0
messages = ['153871', '153905', '153931', '153932', '153933', '153939', '153940', '153943', '153945', '153947', '204439', '204441', '204446', '204494', '236501', '238383']
nosy_count = 11.0
nosy_names = ['tim.peters', 'pitrou', 'vstinner', 'christian.heimes', 'tarek', 'grahamd', 'zvezdan', 'neologix', 'eric.snow', 'Glenn.Maynard', 'Vadim Markovtsev']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue14073'
versions = ['Python 3.5']

[tarekziade]

If you try to run the code below and stop it with ctrl+C, it will lock because atexit is never reached.

Antoine proposed to add a way to have one atexit() per thread, so we can call some cleanup code when the app shuts down and there are running threads.

from wsgiref.simple_server import make_server
import threading
import time
import atexit

class Work(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
        self.running = False

    def run(self):
        self.running = True
        while self.running:
            time.sleep(.2)

    def stop(self):
        self.running = False
        self.join()

worker = Work()


def shutdown():
    # bye-bye
    print 'bye bye'
    worker.stop()


atexit.register(shutdown)


def hello_world_app(environ, start_response):
    status = '200 OK' # HTTP Status
    headers = [('Content-type', 'text/plain')]
    start_response(status, headers)
    return ["Hello World"]

def main():
    worker.start()
    return make_server('', 8000, hello_world_app)

if __name__ == '__main__':
    server = main()
    server.serve_forever()

[grahamd]

My take on this is that if wanting to interact with a thread from an atexit callback, you are supposed to call setDaemon(True) on the thread. This is to ensure that on interpreter shutdown it doesn't try and wait on the thread completing before getting to atexit callbacks.

[tarekziade]

@Grahamd : sometimes you don't own the code that contains the thread, so I think it's better to be able to shutdown properly all flavors of threads.

[grahamd]

Reality is that the way Python behaviour is defined/implemented means that it will wait for non daemonised threads to complete before exiting.

Sounds like the original code is wrong in not setting it to be daemonised in the first place and should be reported as a bug in that code rather than fiddling with the interpreter.

[tarekziade]

Is there any good reason not to add this feature ? what would be the problem ? It does seem to be for the best, I don't see any drawbacks

[grahamd]

At the moment you have showed some code which is causing you problems and a vague idea. Until you show how that idea may work in practice it is a bit hard to judge whether what it does and how it does it is reasonable.

[tarekziade]

Mmm.. you did not say yet why you are against this feature, other than "the lib *should not* use non-daemonized threads"

This sounds like "the lib should not use feature X in Python because it will block everything"

And now we're proposing to remove the limitation and you are telling me I am vague and unreasonable.

Let me try differently then.

Consider this script to be a library I don't control. I need to call the .stop() function when my main application shuts down.

I can't use signals because you forbid it in mod_wsgi

How do I do, since asking the person to daemonize his thread is not an option ?

I see several options:
1 - monkey patch the lib
2 - remove regular threads from Python, or making them always daemonized
3 - add an atexit() option in threads in Python
4 - use signals and drop the usage of mod_wsgi

I think 3- is the cleanest.

[grahamd]

I haven't said I am against it. All I have done so far is explain on the WEB-SIG how mod_wsgi works and how Python currently works and how one would normally handle this situation by having the thread be daemonised.

As for the proposed solution, where is the code example showing how what you are suggesting is meant to work. Right now you are making people assume how that would work. Add an actual example here at least of how with the proposed feature your code would then look.

For the benefit of those who might even implement what you want, which will not be me anyway as I am not involved in Python core development, you might also explain where you expect these special per thread atexit callbacks to be triggered within the current steps for shutting down the interpreter. That way it will be more obvious to those who come later as to what you are actually proposing.

[tarekziade]

Add an actual example here at least of how with the proposed feature your code would then look.

That's the part I am not sure at all about in fact. I don't know at all the internals in the shutdown process in Python and I was hoping Antoine would give us a proposal here.

I would suspect simply adding to the base thread class an .atexit() method that's called when atexit() is called, would do the trick since we'd be able to do things like:

  def atexit(self):
      ... do whatever cleanup needed...
      self.join()

but I have no real experience in these internals.

[grahamd]

Except that calling it at the time of current atexit callbacks wouldn't change the current behaviour. As quoted in WEB-SIG emails the sequence is:

wait_for_thread_shutdown();

/* The interpreter is still entirely intact at this point, and the
 * exit funcs may be relying on that.  In particular, if some thread
 * or exit func is still waiting to do an import, the import machinery
 * expects Py_IsInitialized() to return true.  So don't say the
 * interpreter is uninitialized until after the exit funcs have run.
 * Note that Threading.py uses an exit func to do a join on all the
 * threads created thru it, so this also protects pending imports in
 * the threads created via Threading.
 */
call_py_exitfuncs();

So would need to be done prior to wait_for_thread_shutdown() or by that function before waiting on thread.

The code in that function has:

    PyObject *threading = PyMapping_GetItemString(tstate->interp->modules,
                                                  "threading");

...
result = PyObject_CallMethod(threading, "_shutdown", "");

So calls _shutdown() on the threading module.

That function is aliased to _exitfunc() method of _MainThread.

    def _exitfunc(self):
        self._stop()
        t = _pickSomeNonDaemonThread()
        if t:
            if __debug__:
                self._note("%s: waiting for other threads", self)
        while t:
            t.join()
            t = _pickSomeNonDaemonThread()
        if __debug__:
            self._note("%s: exiting", self)
        self._delete()

So can be done in here.

The decision which would need to be made is whether you call atexit() on all threads before then trying to join() on any, or call atexit() only prior to the join() of the thread.

Calling atexit() on all possibly sounds the better option but I am not sure, plus the code would need to deal with doing two passes like that which may not may not have implications.