extension call backs and the GIL

In article <pU**************@jessikat.fsnet.co.uk>,
Robin Becker <ro***@jessikat.fsnet.co.uk> wrote:

I'm trying to better my understanding of multithreading. I believe a
thread that calls an extension must have the GIL. What happens if the
extension then calls back into Python? Can a function or method somehow
'belong' to a different thread? My understanding was that things should
be OK provided the extension code doesn't mess with the GIL (which it
doesn't).

You need to reacquire the GIL prior to making the callback.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"It is easier to optimize correct code than to correct optimized code."
--Bill Harlan

In article <bl**********@panix3.panix.com>, Aahz <aa**@pythoncraft.com>
writes

In article <pU**************@jessikat.fsnet.co.uk>,
Robin Becker <ro***@jessikat.fsnet.co.uk> wrote:

I'm trying to better my understanding of multithreading. I believe a
thread that calls an extension must have the GIL. What happens if the
extension then calls back into Python? Can a function or method somehow
'belong' to a different thread? My understanding was that things should
be OK provided the extension code doesn't mess with the GIL (which it
doesn't).

You need to reacquire the GIL prior to making the callback.

So does calling an extension release the GIL? I believed the opposite
for some reason.
--
Robin Becker

Robin Becker <ro***@jessikat.fsnet.co.uk> writes:

You need to reacquire the GIL prior to making the callback.

So does calling an extension release the GIL?

No. It would be very inconvenient if it did because what most functions
do first is to call one of the PyArg_ParseTuple functions which could
execute arbitrary python code.

I think it would help if you described a bit more what the actual
problem is or what the user does.

I looked briefly at the sources and assuming tha the accelerator in
question is lib/_rl_accel.c and the callback whatever has been assigned
to _SWRecover, could it be that the problem is that your user tries to
use different callbacks in different threads? _SWRecover is a global
after all and shared by all threads.

Bernhard

--
Intevation GmbH http://intevation.de/
Sketch http://sketch.sourceforge.net/
Thuban http://thuban.intevation.org/

In article <6q************@salmakis.intevation.de>, Bernhard Herzog
<bh@intevation.de> writes

Robin Becker <ro***@jessikat.fsnet.co.uk> writes:

You need to reacquire the GIL prior to making the callback. So does calling an extension release the GIL?

No. It would be very inconvenient if it did because what most functions
do first is to call one of the PyArg_ParseTuple functions which could
execute arbitrary python code.

Thank goodness for that :)

I think it would help if you described a bit more what the actual
problem is or what the user does.

I looked briefly at the sources and assuming tha the accelerator in
question is lib/_rl_accel.c and the callback whatever has been assigned
to _SWRecover, could it be that the problem is that your user tries to
use different callbacks in different threads? _SWRecover is a global
after all and shared by all threads.
I'm not certain, but yes I think that's probably what the problem comes
down to; if so I can claim there are already too many breakages in our
font system to worry about that :(

Bernhard

--
Robin Becker

In article <OW**************@jessikat.fsnet.co.uk>,
Robin Becker <ro***@jessikat.fsnet.co.uk> wrote:

In article <bl**********@panix3.panix.com>, Aahz <aa**@pythoncraft.com>
writes

In article <pU**************@jessikat.fsnet.co.uk>,
Robin Becker <ro***@jessikat.fsnet.co.uk> wrote:

I'm trying to better my understanding of multithreading. I believe a
thread that calls an extension must have the GIL. What happens if the
extension then calls back into Python? Can a function or method somehow
'belong' to a different thread? My understanding was that things should
be OK provided the extension code doesn't mess with the GIL (which it
doesn't).

You need to reacquire the GIL prior to making the callback.

So does calling an extension release the GIL? I believed the opposite
for some reason.

Looks like Bernhard has put you on the right track, but just for the
record: calling an extension does not automatically release the GIL; you
need to explicitly call the Py_BEGIN_ALLOW_THREADS macro. I was just
assuming that because you said "accelerator" you were already releasing
the GIL.
--
Aahz (aa**@pythoncraft.com) <*> http://www.pythoncraft.com/

"It is easier to optimize correct code than to correct optimized code."
--Bill Harlan

In article <bl**********@panix3.panix.com>, Aahz <aa**@pythoncraft.com>
writes
.......

Looks like Bernhard has put you on the right track, but just for the
record: calling an extension does not automatically release the GIL; you
need to explicitly call the Py_BEGIN_ALLOW_THREADS macro. I was just
assuming that because you said "accelerator" you were already releasing
the GIL.

The code is intended to run single thread so we never have done anything
about threading. It's a real pain. In a zope world we ought really to
give every request a fresh context ie each thread would see an entirely
'clean' python and then be forced to load its own fonts/resources. It
just sucks from the overall system viewpoint. How does one ensure that
each thread sees its own version of sys.modules etc etc or is that
considered a dumb question?
--
Robin Becker