> but these are rebuilt when you start the interpreter, aren't they?
No.
Quoting PEP 3147:
Case 4: legacy pyc files and source-less imports
Python will ignore all legacy pyc files when a source file exists next to it. In other words, if a foo.pyc file exists next to the foo.py file, the pyc file will be ignored in all cases
In order to continue to support source-less distributions though, if the source file is missing, Python will import a lone pyc file if it lives where the source file would have been.