Use `PyObject_GetTypeData` without the GIL/attached thread state

[da-woods]

Feature or enhancement

Proposal:

In Cython we've historically let people do things like:

cdef class C:
    cdef int non_object_attribute

def f(C c):
    with nogil:
        value = c.non_object_attribute
        ...

i.e. we've allowed them to get direct access into extension types without the GIL (providing they aren't manipulating Python objects).

When using PEP 697 lookup this becomes a little trickier because access to the attribute has to go through PyObject_GetTypeData. Looking at the implementation of PyObject_GetTypeData, it doesn't appear to need the GIL (except in debug mode where it sanity-checks the type).

I think I can work around this by calculating the offset of the struct myself once at startup (using PyObject_GetTypeData) and then doing all the lookups manually via the offset. It also looks like nanobind use a similar trick in places so I wouldn't be alone in this abuse of the system.

However, it'd be nice not to have to do this and if PyObject_GetTypeData could somehow be exempted from the "C API needs the GIL" rule.

I do understand that people would probably prefer not to do this, so I'm happy to look at other options.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

I asked about this in https://discuss.python.org/t/does-pyobject-gettypedata-need-the-gil/104637/4 - it only generated a small amount of discussion.

[colesbury]

If I understand correctly the options are:

1. Explicitly allow PyObject_GetTypeData to be called without the GIL held. In practice, this might mean getting rid of the assert(PyObject_TypeCheck(obj, cls));
2. Have Cython work around the restriction by computing and caching the offset returned by PyObject_GetTypeData

Neither seems ideal, but I would prefer the first option (don't require the GIL).

Both options could constrain the implementation of PyObject_GetTypeData. In my experience it's easier to maintain and evolve CPython when we are explicit about constraints and downstream dependencies do the straightforward thing. In this case, the second option is more complicated (in Cython) and the constraints are more subtle.

cc @encukou

[da-woods]

Other options are:

• Cython re-acquires the GIL each time it looks up an attribute
• Cython just requires the GIL to access cdef class attributes when in this mode (at the cost that some existing code can't be compiled in this mode - but that's maybe kind of OK - not every feature has to work in the Limited API).

That arguably isn't a huge problem if/when free-threading is the only build (because why even bother releasing the GIL/thread-state then), but it isn't great now.

(Edit:) Or continue precalculating the offset in the short-term, and be prepared to re-acquire the GIL in future if you ever need to break the offset.