Prevent internals destruction before all pybind11 types are destroyed

[XuehaiPan]

Description

Fixes https://github.com/pybind/pybind11/pull/5958/changes#r2717645230

Suggested changelog entry:

• Placeholder.

---

📚 Documentation preview 📚: https://pybind11--5972.org.readthedocs.build/

[rwgk]

@XuehaiPan Is your plan to work on adding a new unit test here after you have the fix? — Short-term the fix is most important, long-term the test will be more important.

I could try to use some LLM to generate a new test, based on what you provided under 5958. Do you think that'd be useful?

[b-pass]

It might be better to mostly revert #5958 if the behavior it has is not what we want. The problem described is expected (but undesirable) behavior from that PR. It is not a use-after-free. It's also possible for OptTree to fix this by manually holding a reference to internals capsule, we could make that easier with something like get_internals_capsule() from this PR.

One option is to go back to something more like the first commit on #5958, before I went into releasing internals itself. The original goal of that PR was to get rid of the leaks of the default_metaclass and instance_base.

[XuehaiPan]

I think releasing some part of the internal but keeping the internal leaked would be a safest choice.

[rwgk]

I think releasing some part of the internal but keeping the internal leaked would be a safest choice.

@XuehaiPan did you see my question above (​https://github.com/pybind/pybind11/pull/5972#issuecomment-3786298624​)?

Subinterpreter support added a lot of complexity; if we don’t capture these issues in unit tests, pybind11 will regress over time. I’m very skeptical of fixes that don’t start from a failing unit test.

[rwgk]

The original goal of that PR was to get rid of the leaks of the default_metaclass and instance_base.

Those leaks are currently dwarfed by CPython leaks documented here (pure‑C reproducer):

#5958 (comment).

Until those are fixed in CPython, or we find a workaround that sidesteps them, the savings from #5958 are marginal. If reverting #5958 is what it takes to ship v3.0.2, that seems like a reasonable short‑term compromise.

[XuehaiPan]

If reverting #5958 is what it takes to ship v3.0.2, that seems like a reasonable short‑term compromise.

This makes sense to me because most users only use the main interpreter. The leak will be freed anyway on program shutdown, while the correctness is more important.

I think releasing some part of the internal but keeping the internal leaked would be a safest choice.

@XuehaiPan did you see my question above (​pybind/pybind11/pull/5972#issuecomment-3786298624​)?

Subinterpreter support added a lot of complexity; if we don’t capture these issues in unit tests, pybind11 will regress over time. I’m very skeptical of fixes that don’t start from a failing unit test.

I am trying to add a repro. It is hard to find a consistent repro for the order bugs since there is no guarantee.