◐ Shell
reader mode source ↗
Skip to content

WIP: bpo-34990: year 2038 problem in compileall.py#9892

Closed
matrixise wants to merge 12 commits into
python:mainfrom
matrixise:bpo-34990
Closed

WIP: bpo-34990: year 2038 problem in compileall.py#9892
matrixise wants to merge 12 commits into
python:mainfrom
matrixise:bpo-34990

Conversation

@matrixise

@matrixise matrixise commented Oct 15, 2018

Copy link
Copy Markdown
Member

Use 'L' as an unsigned long for the timestamp

https://bugs.python.org/issue34990

Use 'L' as an unsigned long for the timestamp
@bmwiedemann

Copy link
Copy Markdown
Contributor

Lib/test/test_compileall.py also contains 2 instances of <4sll that probably should be updated accordingly.
and you could add the test code provided by xtreak in the bug.

@matrixise

Copy link
Copy Markdown
Member Author

@bmwiedemann yep, I was in the plane from PySS and I just commited my code, and now, I just saw the test of @tirkarthi . I am going to include it.

@tirkarthi

Copy link
Copy Markdown
Member

@matrixise Thanks for the PR. I think it's a NEWS entry worthy fix to add a short note that compileall now supports files with modification time greater than or equal to 19-01-2038 (I guess)

@tirkarthi

Copy link
Copy Markdown
Member

@matrixise Sorry, I just noticed you have added one :)

@matrixise

Copy link
Copy Markdown
Member Author

@tirkarthi please, could you check my PR with the last commits.

@bmwiedemann

Copy link
Copy Markdown
Contributor

btw: the first bad mtime is 0x80000000 or 2147483648 or 2038-01-19T03:14:08+00:00

@tirkarthi

Copy link
Copy Markdown
Member

@matrixise Thanks, Looks good to me from the context of the issue reported but I have limited understanding about the code so I hope someone else might have a better look. Seems there was some discussion about handling 2038 problem as part of the original PR where this code was added : #4575 (comment)

@bmwiedemann

Copy link
Copy Markdown
Contributor

I think the discussion in PR 4575 was about future-proofing the new .pyc format, but it failed to notice that by 2038, only 31 bits are exhausted and bit 32 (which was erroneously interpreted as the sign-bit here), allows to keep the format alive until 2106 - which we can safely assume out-of-scope for today.

@matrixise

Copy link
Copy Markdown
Member Author

This is the reason why I move to 'L' instead of 'l'

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

Check also other parts of the stdlib that work with mtime in pyc files. For example zipimport and checkpyc.py need updates. Maybe there are other parts.

@bedevere-bot

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

And if you don't make the requested changes, you will be poked with soft cushions!

@matrixise

Copy link
Copy Markdown
Member Author

@serhiy-storchaka last update for Tools/scripts/cherrypyc.py by in June 2013, we only read the magic number and timestamp, just that: https://github.com/python/cpython/blob/master/Tools/scripts/checkpyc.py#L45-L46

In https://github.com/python/cpython/blob/master/Lib/zipimport.py#L602 you convert the dword as a uint32 and LGTM.

@matrixise matrixise requested a review from a team October 17, 2018 12:14
@matrixise

Copy link
Copy Markdown
Member Author

@vstinner and @serhiy-storchaka I have fixed mtime with 0xFFFF_FFFF and use an unsigned long, where I found <4slL but for the other files, I am not an expert of the API and I don't know where I could search, zipimport and checkpy.py seem to be good.

@serhiy-storchaka serhiy-storchaka left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hide comment

zipimport needs truncating the timestamp to 32 bit for comparison.

Also update _code_to_timestamp_pyc() in _bootstrap_external.

@matrixise matrixise closed this Oct 17, 2018
13 hidden items Load more…
@matrixise

Copy link
Copy Markdown
Member Author

@serhiy-storchaka can you continue the review with me, because I am a little bit like a chicken without head ;-) I try to understand with 0xFFFF_FFFF ;-)

@matrixise matrixise changed the title bpo-34990: year 2038 problem in compileall.py Oct 25, 2018
@brettcannon

Copy link
Copy Markdown
Member

Due to the WIP in the title I added the DO-NOT-MERGE label.

@matrixise

Copy link
Copy Markdown
Member Author

@brettcannon Thank you, I will continue to work on this issue on the next week, this week I am at the hospital with my daughter.

@bmwiedemann

Copy link
Copy Markdown
Contributor

Only 19 years to go... Can we get this moving again?

@vstinner

Copy link
Copy Markdown
Member

Ping @matrixise: they are still some comments of @serhiy-storchaka that you didn't address.

@matrixise

Copy link
Copy Markdown
Member Author

@vstinner pong, ok thanks for the reminder.

@matrixise

Copy link
Copy Markdown
Member Author

So, I am not serious with this bug, I am going to continue on this bug, I would like to close all my PRs.

@vstinner

Copy link
Copy Markdown
Member

@vstinner pong, ok thanks for the reminder.

FYI I ignore this issue since it's still tagged as "WIP". I prefer to wait until it's no longer a WIP to review it.

@brettcannon brettcannon removed the request for review from a team April 17, 2019 22:45
@brettcannon

Copy link
Copy Markdown
Member

Removing the import team from reviewing as this is still a WIP. We can be added back when it's ready to go.

@bmwiedemann

Copy link
Copy Markdown
Contributor

@matrixise any chance to get this finished this year?

@vstinner

Copy link
Copy Markdown
Member

@matrixise any chance to get this finished this year?

It's ok as soon as it's fixed before 2038 :-D

@bmwiedemann

Copy link
Copy Markdown
Contributor

It's ok as soon as it's fixed before 2038 :-D

I am afraid there are way too many places where software is used unchanged for a long time. Even 18 years. And of course this is not the only timebomb waiting to trigger.

@vstinner

Copy link
Copy Markdown
Member

It was a joke. Hint: see the emoticon.

@ammaraskar

Copy link
Copy Markdown
Member

Since Stéphane said:

I would like to close all my PRs.

I opened up #19651 with the more permanent solution of using a 64-bit timestamp value as Benjamin originally suggested in their PEP 552 PR.

@ammaraskar

Copy link
Copy Markdown
Member

I pinged @matrixise about this and they said I can feel free to take this on since they're too busy to move forward with it. #19708 is a continuation of this PR to make compileall use an unsigned int.

It should be possible to compare the code for both the solutions now.

@tirkarthi

Copy link
Copy Markdown
Member

I guess this can be close since #19708 is merged

@encukou

encukou commented Aug 24, 2021

Copy link
Copy Markdown
Member

Indeed. Thanks for noticing.

@encukou encukou closed this Aug 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants