◐ Shell
reader mode source ↗
Skip to content

[WIP, DO NOT MERGE] bpo-41188: Prepare CPython for opague PyObject structure.#21262

Closed
WildCard65 wants to merge 6 commits into
python:masterfrom
WildCard65:bpo-39573
Closed

[WIP, DO NOT MERGE] bpo-41188: Prepare CPython for opague PyObject structure.#21262
WildCard65 wants to merge 6 commits into
python:masterfrom
WildCard65:bpo-39573

Conversation

@WildCard65

@WildCard65 WildCard65 commented Jul 1, 2020

Copy link
Copy Markdown
Contributor

The end goal of this pull request is to make the structure "PyObject" opaque to EVERYTHING except Python core.

To accomplish this, the following must be done:

  • Add new member to 'PyTypeObject' to hold pointer offset to type's data structure.
  • Add implementation defined "slib_python310" static library for end users to link against.
  • Determine if more changes are required.

https://bugs.python.org/issue41188

This member is the companion member required for 'Py_TPFLAGS_USES_OPAQUE_OBJECT' flag.

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Add implementation defined "slib_python310" static library for end users to link against.

I don't understand this.

Change as many types as possible to use 'Py_TPFLAGS_USES_OPAQUE_OBJECT'

Please don't do that in the same PR.

@vstinner

vstinner commented Jul 1, 2020

Copy link
Copy Markdown
Member

I would prefer to open a separated issue on bugs.python.org for this work.

@WildCard65

Copy link
Copy Markdown
Contributor Author

@vstinner The static library will contain '.c' files that reference internal implementations FOR the runtime it's designed for.
The goal of the static library is that end users link against it when building their extensions WITHOUT having access to internals directly.

As it currently stands, the current Py_TYPE still requires direct access to internals for end users while this static library remove that requirement.

example would be:
Py_TYPE macro references a method called "PyObject_GetType", but the definition of the method is not in the header, instead, it's in the static library which will be linked into the final binary.

This allows for implementation independent access while still allowing the C compiler to aggressively inline out the small functions.

@WildCard65 WildCard65 changed the title [WIP, DO NOT MERGE] bpo-39573: Prepare CPython for opague PyObject structure. Jul 1, 2020
…thon's internals and external users.

Note: This static library is incomplete.
@WildCard65

Copy link
Copy Markdown
Contributor Author

@vstinner Before I proceed any further, I want you to review "slib_pythoncore" first.

@WildCard65 WildCard65 requested a review from vstinner July 1, 2020 18:57
@vstinner

vstinner commented Jul 1, 2020

Copy link
Copy Markdown
Member

The goal of the static library is that end users link against it when building their extensions WITHOUT having access to internals directly.

This is the purpose of the limited C API which provides a stable ABI. It's already implemented since Python 3.2. I don't see the point of adding a second flavor of the "libpython" library.

@WildCard65

WildCard65 commented Jul 1, 2020

Copy link
Copy Markdown
Contributor Author

As it stands, functions like _Py_TYPE, _Py_SIZE, _Py_INCREF, _Py_DECREF rely on "PyObject" being a complete type. Opaque objects on the other hand are classified as incomplete types.

Incomplete types can't be used for anything but pointers (*) and references (&), these functions will fail when PyObject goes opaque in the limited API, infact, sizeof() also fails on incomplete types.

The purpose of the static library is not to be a different flavour of Python, but to act as a bridge into Python's internals while respecting the limitations of incomplete types, while also eliminating function call overhead.

As an added bonus, this static library can be "reskinned" for runtimes that take a different approach to PyObject (if any) and descendant built-in types while end users don't have to worry about what the runtime's schematic is.

@WildCard65

Copy link
Copy Markdown
Contributor Author

Added a basic implementation of the flag, but I do not know where all usages of "tp_basicsize" is, I'm also debating on whether or not I need to make GET_TYPE_TOTALSIZE into an inlinable function.

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

IMHO you gone too far in the implementation before discussing the design. You should first propose the design on python-ideas or python-dev list.

@vstinner

vstinner commented Jul 2, 2020

Copy link
Copy Markdown
Member

Py_TPFLAGS_USES_OPAQUE_OBJECT looks useful, but other changes like "slib" looks useless to me.

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

I don't see the point of tp_obj_offset, tp_obj_size and slib.

@bedevere-bot bedevere-bot removed the label Jul 2, 2020
@bedevere-bot

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@WildCard65

WildCard65 commented Jul 2, 2020

Copy link
Copy Markdown
Contributor Author

Consider the following:

struct myobject;

void use_myobject(struct myobect *obj); // Valid, incomplete types can be used as pointers.

size_t get_object_value(struct my object *obj)
{
    return obj->m_value; // ERROR: my_object is an incomplete type!
}

Above demonstrates why functions like _Py_Type will fail in limited api.

'Py_TPFLAGS_OMIT_OBJECT_SIZE' was intended as the stepping stone for making types like PyTypeObject opaque as well, this makes knowing the true offset off the base time a runtime dependant scenario, one with minimal impact to performance, probably less than having _Py_TYPE, etc, exported from python binary.

@WildCard65

WildCard65 commented Jul 2, 2020

Copy link
Copy Markdown
Contributor Author

@vstinner One way or another, PY_TYPE and co need to have their implementations PURGED from the Python header files for a truly opaque PyObject.

Another thing that needs to be purged is "ob_base" usage of objects like PyTupleObject, PyTypeObject, PyListObject to name a few.

I'm willing to replace slib with a precompiled header trio, but that's a can of worms on its own (at least on MSVC).

I'm not budging in regards to "tp_obj_offset" and "tp_obj_size", see above mentioned purging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants