◐ Shell
reader mode source ↗
Skip to content

Commit 42aa93b

Browse files
benjaminpbenjaminp
authored
closes bpo-31650: PEP 552 (Deterministic pycs) implementation (#4575)
Python now supports checking bytecode cache up-to-dateness with a hash of the source contents rather than volatile source metadata. See the PEP for details. While a fairly straightforward idea, quite a lot of code had to be modified due to the pervasiveness of pyc implementation details in the codebase. Changes in this commit include: - The core changes to importlib to understand how to read, validate, and regenerate hash-based pycs. - Support for generating hash-based pycs in py_compile and compileall. - Modifications to our siphash implementation to support passing a custom key. We then expose it to importlib through _imp. - Updates to all places in the interpreter, standard library, and tests that manually generate or parse pyc files to grok the new format. - Support in the interpreter command line code for long options like --check-hash-based-pycs. - Tests and documentation for all of the above.
1 parent 28d8d14 commit 42aa93b

33 files changed

Lines changed: 3362 additions & 2503 deletions

File tree

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -458,6 +458,12 @@ Glossary
458458
is believed that overcoming this performance issue would make the
459459
implementation much more complicated and therefore costlier to maintain.
460460

461+
462+
hash-based pyc
463+
A bytecode cache file that uses the the hash rather than the last-modified
464+
time of the corresponding source file to determine its validity. See
465+
:ref:`pyc-invalidation`.
466+
461467
hashable
462468
An object is *hashable* if it has a hash value which never changes during
463469
its lifetime (it needs a :meth:`__hash__` method), and can be compared to
Lines changed: 33 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,16 @@ compile Python sources.
8383
If ``0`` is used, then the result of :func:`os.cpu_count()`
8484
will be used.
8585

86+
.. cmdoption:: --invalidation-mode [timestamp|checked-hash|unchecked-hash]
87+
88+
Control how the generated pycs will be invalidated at runtime. The default
89+
setting, ``timestamp``, means that ``.pyc`` files with the source timestamp
90+
and size embedded will be generated. The ``checked-hash`` and
91+
``unchecked-hash`` values cause hash-based pycs to be generated. Hash-based
92+
pycs embed a hash of the source file contents rather than a timestamp. See
93+
:ref:`pyc-invalidation` for more information on how Python validates bytecode
94+
cache files at runtime.
95+
8696
.. versionchanged:: 3.2
8797
Added the ``-i``, ``-b`` and ``-h`` options.
8898

@@ -91,6 +101,9 @@ compile Python sources.
91101
was changed to a multilevel value. ``-b`` will always produce a
92102
byte-code file ending in ``.pyc``, never ``.pyo``.
93103

104+
.. versionchanged:: 3.7
105+
Added the ``--invalidation-mode`` parameter.
106+
94107

95108
There is no command-line option to control the optimization level used by the
96109
:func:`compile` function, because the Python interpreter itself already
@@ -99,7 +112,7 @@ provides the option: :program:`python -O -m compileall`.
99112
Public functions
100113
----------------
101114

102-
.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1)
115+
.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, workers=1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
103116

104117
Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
105118
files along the way. Return a true value if all the files compiled successfully,
@@ -140,6 +153,10 @@ Public functions
140153
then sequential compilation will be used as a fallback. If *workers* is
141154
lower than ``0``, a :exc:`ValueError` will be raised.
142155

156+
*invalidation_mode* should be a member of the
157+
:class:`py_compile.PycInvalidationMode` enum and controls how the generated
158+
pycs are invalidated at runtime.
159+
143160
.. versionchanged:: 3.2
144161
Added the *legacy* and *optimize* parameter.
145162

@@ -156,7 +173,10 @@ Public functions
156173
.. versionchanged:: 3.6
157174
Accepts a :term:`path-like object`.
158175

159-
.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1)
176+
.. versionchanged:: 3.7
177+
The *invalidation_mode* parameter was added.
178+
179+
.. function:: compile_file(fullname, ddir=None, force=False, rx=None, quiet=0, legacy=False, optimize=-1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
160180

161181
Compile the file with path *fullname*. Return a true value if the file
162182
compiled successfully, and a false value otherwise.
@@ -184,6 +204,10 @@ Public functions
184204
*optimize* specifies the optimization level for the compiler. It is passed to
185205
the built-in :func:`compile` function.
186206

207+
*invalidation_mode* should be a member of the
208+
:class:`py_compile.PycInvalidationMode` enum and controls how the generated
209+
pycs are invalidated at runtime.
210+
187211
.. versionadded:: 3.2
188212

189213
.. versionchanged:: 3.5
@@ -193,7 +217,10 @@ Public functions
193217
The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
194218
no matter what the value of *optimize* is.
195219

196-
.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1)
220+
.. versionchanged:: 3.7
221+
The *invalidation_mode* parameter was added.
222+
223+
.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, quiet=0, legacy=False, optimize=-1, invalidation_mode=py_compile.PycInvalidationMode.TIMESTAMP)
197224

198225
Byte-compile all the :file:`.py` files found along ``sys.path``. Return a
199226
true value if all the files compiled successfully, and a false value otherwise.
@@ -213,6 +240,9 @@ Public functions
213240
The *legacy* parameter only writes out ``.pyc`` files, not ``.pyo`` files
214241
no matter what the value of *optimize* is.
215242

243+
.. versionchanged:: 3.7
244+
The *invalidation_mode* parameter was added.
245+
216246
To force a recompile of all the :file:`.py` files in the :file:`Lib/`
217247
subdirectory and all its subdirectories::
218248

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,9 @@ generically as an :term:`importer`) to participate in the import process.
6767
:pep:`489`
6868
Multi-phase extension module initialization
6969

70+
:pep:`552`
71+
Deterministic pycs
72+
7073
:pep:`3120`
7174
Using UTF-8 as the Default Source Encoding
7275

@@ -1327,6 +1330,14 @@ an :term:`importer`.
13271330
.. versionchanged:: 3.6
13281331
Accepts a :term:`path-like object`.
13291332

1333+
.. function:: source_hash(source_bytes)
1334+
1335+
Return the hash of *source_bytes* as bytes. A hash-based ``.pyc`` file embeds
1336+
the :func:`source_hash` of the corresponding source file's contents in its
1337+
header.
1338+
1339+
.. versionadded:: 3.7
1340+
13301341
.. class:: LazyLoader(loader)
13311342

13321343
A class which postpones the execution of the loader of a module until the
Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ byte-code cache files in the directory containing the source code.
2727
Exception raised when an error occurs while attempting to compile the file.
2828

2929

30-
.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1)
30+
.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1, invalidation_mode=PycInvalidationMode.TIMESTAMP)
3131

3232
Compile a source file to byte-code and write out the byte-code cache file.
3333
The source code is loaded from the file named *file*. The byte-code is
@@ -53,6 +53,10 @@ byte-code cache files in the directory containing the source code.
5353
:func:`compile` function. The default of ``-1`` selects the optimization
5454
level of the current interpreter.
5555

56+
*invalidation_mode* should be a member of the :class:`PycInvalidationMode`
57+
enum and controls how the generated ``.pyc`` files are invalidated at
58+
runtime.
59+
5660
.. versionchanged:: 3.2
5761
Changed default value of *cfile* to be :PEP:`3147`-compliant. Previous
5862
default was *file* + ``'c'`` (``'o'`` if optimization was enabled).
@@ -65,6 +69,41 @@ byte-code cache files in the directory containing the source code.
6569
caveat that :exc:`FileExistsError` is raised if *cfile* is a symlink or
6670
non-regular file.
6771

72+
.. versionchanged:: 3.7
73+
The *invalidation_mode* parameter was added as specified in :pep:`552`.
74+
75+
76+
.. class:: PycInvalidationMode
77+
78+
A enumeration of possible methods the interpreter can use to determine
79+
whether a bytecode file is up to date with a source file. The ``.pyc`` file
80+
indicates the desired invalidation mode in its header. See
81+
:ref:`pyc-invalidation` for more information on how Python invalidates
82+
``.pyc`` files at runtime.
83+
84+
.. versionadded:: 3.7
85+
86+
.. attribute:: TIMESTAMP
87+
88+
The ``.pyc`` file includes the timestamp and size of the source file,
89+
which Python will compare against the metadata of the source file at
90+
runtime to determine if the ``.pyc`` file needs to be regenerated.
91+
92+
.. attribute:: CHECKED_HASH
93+
94+
The ``.pyc`` file includes a hash of the source file content, which Python
95+
will compare against the source at runtime to determine if the ``.pyc``
96+
file needs to be regenerated.
97+
98+
.. attribute:: UNCHECKED_HASH
99+
100+
Like :attr:`CHECKED_HASH`, the ``.pyc`` file includes a hash of the source
101+
file content. However, Python will at runtime assume the ``.pyc`` file is
102+
up to date and not validate the ``.pyc`` against the source file at all.
103+
104+
This option is useful when the ``.pycs`` are kept up to date by some
105+
system external to Python like a build system.
106+
68107

69108
.. function:: main(args=None)
70109

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -675,6 +675,33 @@ Here are the exact rules used:
675675
:meth:`~importlib.abc.Loader.module_repr` method, if defined, before
676676
trying either approach described above. However, the method is deprecated.
677677

678+
.. _pyc-invalidation:
679+
680+
Cached bytecode invalidation
681+
----------------------------
682+
683+
Before Python loads cached bytecode from ``.pyc`` file, it checks whether the
684+
cache is up-to-date with the source ``.py`` file. By default, Python does this
685+
by storing the source's last-modified timestamp and size in the cache file when
686+
writing it. At runtime, the import system then validates the cache file by
687+
checking the stored metadata in the cache file against at source's
688+
metadata.
689+
690+
Python also supports "hash-based" cache files, which store a hash of the source
691+
file's contents rather than its metadata. There are two variants of hash-based
692+
``.pyc`` files: checked and unchecked. For checked hash-based ``.pyc`` files,
693+
Python validates the cache file by hashing the source file and comparing the
694+
resulting hash with the hash in the cache file. If a checked hash-based cache
695+
file is found to be invalid, Python regenerates it and writes a new checked
696+
hash-based cache file. For unchecked hash-based ``.pyc`` files, Python simply
697+
assumes the cache file is valid if it exists. Hash-based ``.pyc`` files
698+
validation behavior may be overridden with the :option:`--check-hash-based-pycs`
699+
flag.
700+
701+
.. versionchanged:: 3.7
702+
Added hash-based ``.pyc`` files. Previously, Python only supported
703+
timestamp-based invalidation of bytecode caches.
704+
678705

679706
The Path Based Finder
680707
=====================
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -210,6 +210,20 @@ Miscellaneous options
210210
import of source modules. See also :envvar:`PYTHONDONTWRITEBYTECODE`.
211211

212212

213+
.. cmdoption:: --check-hash-based-pycs default|always|never
214+
215+
Control the validation behavior of hash-based ``.pyc`` files. See
216+
:ref:`pyc-invalidation`. When set to ``default``, checked and unchecked
217+
hash-based bytecode cache files are validated according to their default
218+
semantics. When set to ``always``, all hash-based ``.pyc`` files, whether
219+
checked or unchecked, are validated against their corresponding source
220+
file. When set to ``never``, hash-based ``.pyc`` files are not validated
221+
against their corresponding source files.
222+
223+
The semantics of timestamp-based ``.pyc`` files are unaffected by this
224+
option.
225+
226+
213227
.. cmdoption:: -d
214228

215229
Turn on parser debugging output (for expert only, depending on compilation
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,33 @@ variable is not set in practice.
197197

198198
See :option:`-X` ``dev`` for the details.
199199

200+
Hash-based pycs
201+
---------------
202+
203+
Python has traditionally checked the up-to-dateness of bytecode cache files
204+
(i.e., ``.pyc`` files) by comparing the source metadata (last-modified timestamp
205+
and size) with source metadata saved in the cache file header when it was
206+
generated. While effective, this invalidation method has its drawbacks. When
207+
filesystem timestamps are too coarse, Python can miss source updates, leading to
208+
user confusion. Additionally, having a timestamp in the cache file is
209+
problematic for `build reproduciblity <https://reproducible-builds.org/>`_ and
210+
content-based build systems.
211+
212+
:pep:`552` extends the pyc format to allow the hash of the source file to be
213+
used for invalidation instead of the source timestamp. Such ``.pyc`` files are
214+
called "hash-based". By default, Python still uses timestamp-based invalidation
215+
and does not generate hash-based ``.pyc`` files at runtime. Hash-based ``.pyc``
216+
files may be generated with :mod:`py_compile` or :mod:`compileall`.
217+
218+
Hash-based ``.pyc`` files come in two variants: checked and unchecked. Python
219+
validates checked hash-based ``.pyc`` files against the corresponding source
220+
files at runtime but doesn't do so for unchecked hash-based pycs. Unchecked
221+
hash-based ``.pyc`` files are a useful performance optimization for environments
222+
where a system external to Python (e.g., the build system) is responsible for
223+
keeping ``.pyc`` files up-to-date.
224+
225+
See :ref:`pyc-invalidation` for more information.
226+
200227

201228
Other Language Changes
202229
======================
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#ifndef Py_INTERNAL_HASH_H
2+
#define Py_INTERNAL_HASH_H
3+
4+
uint64_t _Py_KeyedHash(uint64_t, const char *, Py_ssize_t);
5+
6+
#endif
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#ifndef Py_INTERNAL_IMPORT_H
2+
#define Py_INTERNAL_IMPORT_H
3+
4+
extern const char *_Py_CheckHashBasedPycsMode;
5+
6+
#endif
Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,14 @@ PyAPI_DATA(wchar_t *) _PyOS_optarg;
1212

1313
PyAPI_FUNC(void) _PyOS_ResetGetOpt(void);
1414

15-
PyAPI_FUNC(int) _PyOS_GetOpt(int argc, wchar_t **argv, wchar_t *optstring);
15+
typedef struct {
16+
const wchar_t *name;
17+
int has_arg;
18+
int val;
19+
} _PyOS_LongOption;
20+
21+
PyAPI_FUNC(int) _PyOS_GetOpt(int argc, wchar_t **argv, wchar_t *optstring,
22+
const _PyOS_LongOption *longopts, int *longindex);
1623
#endif /* !Py_LIMITED_API */
1724

1825
#ifdef __cplusplus

0 commit comments

Comments
 (0)