◐ Shell
clean mode source ↗

3.12: tokenize adds a newline when it is not there

asottile

Bug report

tokenize adds a concrete newline character when it is not present. this breaks any sort of roundtrip source generation and pycodestyle's end-of-file checker

here's an example of a file containing a single byte (generated via echo -n 1 > t.py)

# hd -c t.py 
00000000  31                                                |1|
0000000   1                                                            
0000001
# python3.12 -m tokenize t.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,1:            NUMBER         '1'            
1,1-1,2:            NEWLINE        '\n'           
2,0-2,0:            ENDMARKER      ''             
# python3.11 -m tokenize t.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,1:            NUMBER         '1'            
1,1-1,2:            NEWLINE        ''             
2,0-2,0:            ENDMARKER      ''

Your environment

  • CPython versions tested on: 3.12 dbd7d7c
  • Operating system and architecture: ubuntu 22.04 LTS x86_64

cc @pablogsal

Linked PRs