◐ Shell
reader mode source ↗
Skip to content

[3.7] bpo-27987: align PyGC_Head to alignof(long double)#13335

Merged
methane merged 3 commits into
python:3.7from
methane:gc-align
May 25, 2019
Merged

[3.7] bpo-27987: align PyGC_Head to alignof(long double)#13335
methane merged 3 commits into
python:3.7from
methane:gc-align

Conversation

@methane

@methane methane commented May 15, 2019

Copy link
Copy Markdown
Member

@vstinner

Copy link
Copy Markdown
Member

C code:

union {
    struct {
        void *next;
        void *prev;
    } _gc;
    long double alignment;
} PyGC_Head;

int main()
{
    return sizeof(PyGC_Head);
}

With long double:

vstinner@apu$ gcc x.c -o x && ./x; echo $? # 64-bit void*
16
vstinner@apu$ gcc -m32 x.c -o x && ./x; echo $? # 32-bit void*
12

Without long double:

vstinner@apu$ gcc x.c -o x && ./x; echo $? # 64-bit void*
16
vstinner@apu$ gcc -m32 x.c -o x && ./x; echo $? # 32-bit void*
8

I'm not sure that 12 bytes is great for alignment. 32-bit CPU may be more efficient on SIMD instructions with 8 bytes alignment, no?

@vstinner

Copy link
Copy Markdown
Member

@methane also proposed PR #13336 for master, but he rejected it.

@methane

methane commented May 16, 2019

Copy link
Copy Markdown
Member Author

PyGC_Head in master branch is two uintptr_t. It is 8 bytes (not 12 bytes) on x86.

@methane

methane commented May 21, 2019

Copy link
Copy Markdown
Member Author

I'm not sure that 12 bytes is great for alignment. 32-bit CPU may be more efficient on SIMD instructions with 8 bytes alignment, no?

This pull request changes from 12 bytes to 12 or 16 bytes, no?

@vstinner

Copy link
Copy Markdown
Member

Sorry @methane, I don't understand this change :-( What is the effect of PyListObject for example? Is the address of ob_item aligned to 16 bytes on x86 (32-bit)? Is it on x86-64 (64 bit)?

@methane

methane commented May 23, 2019

Copy link
Copy Markdown
Member Author

@vstinner You wrote:

    struct {
        void *next;
        void *prev;
    } _gc;

But it is actually:

    struct {
        void *next;
        void *prev;
        ssize_t gc_refs; /* only in Python < 3.8 */
    } _gc;

This is very big difference between Python 3.7 and 3.8. That's why I closed PR for 3.8 but don't close this.

sizeof(PyGC_Head) is 12 byte on x86, regardless alignment is double or long double. (I can now "gcc -m32" on Ubuntu, thank you)
On amd64, sizeof(PyGC_Head) is 24 byte with double alignment, and 32byte with long double alignment.

What is the effect of PyListObject for example? Is the address of ob_item aligned to 16 bytes on x86 (32-bit)? Is it on x86-64 (64 bit)?

This pull request affects only where PyObject* ((PyObject*)(gc + 1)) is aligned.
This pull request doesn't affect alignment in PyListObject. offsetof(PyListObject, ob_item) is 24.

@vstinner vstinner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

LGTM. I think that I understood the issue and that the fix is correct :-)

@methane methane merged commit ea2b76b into python:3.7 May 25, 2019
@methane methane deleted the gc-align branch May 25, 2019 12:13
@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian PGO 3.7 has failed when building commit ea2b76b.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/128/builds/1189) and take a look at the build logs.
  4. Check if the failure is related to this commit (ea2b76b) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/128/builds/1189

Click to see traceback logs
From https://github.com/python/cpython
 * branch                  3.7        -> FETCH_HEAD
Reset branch '3.7'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [clean] Error 1 (ignored)
Segmentation fault
make[3]: *** [pybuilddir.txt] Error 1
make[2]: *** [build_all_generate_profile] Error 2
make[1]: *** [profile-gen-stamp] Error 2
make: *** [profile-run-stamp] Error 2

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [clean] Error 1 (ignored)

@bedevere-bot

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Debian PGO 3.7 has failed when building commit ea2b76b.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/128/builds/1190) and take a look at the build logs.
  4. Check if the failure is related to this commit (ea2b76b) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/128/builds/1190

Click to see traceback logs
Reset branch '3.7'

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [clean] Error 1 (ignored)
Segmentation fault
make[3]: *** [pybuilddir.txt] Error 1
make[2]: *** [build_all_generate_profile] Error 2
make[1]: *** [profile-gen-stamp] Error 2
make: *** [profile-run-stamp] Error 2

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make: [clean] Error 1 (ignored)

@gpshead

gpshead commented May 25, 2019

Copy link
Copy Markdown
Member

this change needs to be undone. followup on the bug.

gpshead added a commit to gpshead/cpython that referenced this pull request May 25, 2019
gpshead added a commit that referenced this pull request May 25, 2019
@methane

methane commented May 26, 2019

Copy link
Copy Markdown
Member Author

Do you think missing "build" directory is happened by this change?

I thought it just a trouble on the build bot environment.

@gpshead

gpshead commented May 26, 2019

Copy link
Copy Markdown
Member

The resulting python interpreter was crashing on the the PGO bot and a pile of new undefined behavior was happening on the undefined behavior sanitizer (ubsan) bot.

PGO bot:

./python -E -S -m sysconfig --generate-posix-vars ;\
if test $? -ne 0 ; then \
	echo "generate-posix-vars failed" ; \
	rm -f ./pybuilddir.txt ; \
	exit 1 ; \
fi
Segmentation fault
generate-posix-vars failed
Makefile:605: recipe for target 'pybuilddir.txt' failed

undefined behavior sanitizer millions of times:

Objects/funcobject.c:663:5: runtime error: member access within misaligned address 0x7fb4b5a03a08 for type 'union _gc_head', which requires 16 byte alignment
0x7fb4b5a03a08: note: pointer points here
 b4 7f 00 00  70 8d c6 b5 b4 7f 00 00  a0 13 a0 b5 b4 7f 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^ 
Objects/funcobject.c:663:5: runtime error: member access within misaligned address 0x7fb4b5a03a08 for type 'struct (anonymous struct at ./Include/objimpl.h:253:5)', which requires 16 byte alignment
0x7fb4b5a03a08: note: pointer points here
 b4 7f 00 00  70 8d c6 b5 b4 7f 00 00  a0 13 a0 b5 b4 7f 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^ 
Objects/funcobject.c:663:5: runtime error: store to misaligned address 0x7fb4b5a03a08 for type 'union _gc_head *', which requires 16 byte alignment
0x7fb4b5a03a08: note: pointer points here
 b4 7f 00 00  70 8d c6 b5 b4 7f 00 00  a0 13 a0 b5 b4 7f 00 00  00 00 00 00 00 00 00 00  00 00 00 00
              ^ 

methane added a commit that referenced this pull request May 26, 2019
@methane

methane commented May 26, 2019

Copy link
Copy Markdown
Member Author

#13318 fixed it already.

gpshead pushed a commit that referenced this pull request Jun 3, 2019
…H-13581)

This reverts commit 2156fec.

Now that 1b85f4e is in, this change makes sense.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type-bug An unexpected behavior, bug, or error

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants