Better codecs and fix lots of test_io and other expectedFailures#6997

youknowone

Summary by CodeRabbit

New Features
- Added -X utf8 command-line option to control UTF‑8 mode
- Added a UTF‑32 decode entry to the codecs module
- Enabled pickling support for IO wrapper objects and subclasses
Improvements
- Stronger codec-name validation (NUL and UTF‑8 checks)
- Better IO finalization with ResourceWarning-like warnings for unclosed files
- Improved surrogate/encoding error recovery to allow partial output on failure

coderabbitai

📝 Walkthrough

Walkthrough

Adds partial-output handling to SurrogateEscape encoding; tightens codec-name validation and exposes utf_32_ex_decode; introduces IO finalization/deallocation warnings, per-instance finalizing flags, and __reduce_ex__ pickling support; and adds -X utf8 runtime option parsing.

Changes

Cohort / File(s)	Summary
Surrogate-escape encoding `crates/common/src/encodings.rs`	Adds position tracking and partial-output pathway for SurrogateEscape: emit accumulated bytes and restart from the first non-surrogate code point when a mid-range non-surrogate is encountered; adjusts code-point handling and pushed-byte computation.
Codec validation & UTF-32 exposure `crates/vm/src/stdlib/codecs.rs`	Adds NUL-byte guards to `lookup`/`lookup_error`, rejects non-UTF‑8 names, imports `cstring_error`, exposes `utf_32_ex_decode` as a pyfunction, and refactors Windows code-page encode/decode paths into helper functions separating strict and error-handling flows.
IO finalization, pickling, and encoding `crates/vm/src/stdlib/io.rs`	Adds `iobase_finalize`/`file_closed`, per-instance `finalizing: AtomicBool` fields, `_dealloc_warn` hooks, `__reduce_ex__` implementations for multiple IO classes, richer TextIOWrapper encoding/BOM/newline handling, and changes `BufferedMixin` to require `StaticType`.
Phase doc clarification `crates/vm/src/vm/mod.rs`	Doc comment updated to specify Phase 4 clears module dicts in reverse import order using a 2‑pass algorithm (no behavior change).
Runtime option: utf8 mode `src/settings.rs`	Adds parsing for `-X utf8`, mapping unset/`1`→1, `0`→0, and aborting with a fatal error on other values. No public API signature changes.

Sequence Diagram(s)

sequenceDiagram
    participant GC as GC (finalizer)
    participant Wrapper as TextIOWrapper
    participant Raw as RawBuffer/FileIO
    participant VM as VirtualMachine

    GC->>Wrapper: invoke destructor / finalize
    Wrapper->>Wrapper: check/set finalizing (AtomicBool)
    alt first finalize
        Wrapper->>Raw: call close()/flush()
        Raw-->>Wrapper: success / failure
        alt failure or not closed
            Wrapper->>VM: emit ResourceWarning via iobase_finalize/_dealloc_warn
        end
    else reentrant finalize
        Wrapper-->>GC: skip redundant finalize
    end

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

windows codecs #6337: Related refactor and additions to Windows codec encode/decode paths (overlaps codecs changes).
Add UTF-32 functions #5800: Related addition/exposure of UTF-32 codec entrypoints and delegation logic (utf_32_ex_decode).
Fix test_io on windows #6387: Related IO finalization/destructor changes and deallocation warning wiring in IO subsystem.

Suggested reviewers

ShaharNaveh

Poem

🐇 I nibbled surrogates till bytes hopped free,
NUL guards stood watch where names should be,
IOs whisper warnings as they close their door,
Pickles tuck the stream and save a little more,
A rabbit hops — the runtime hums once more.

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 63.89% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title refers to real aspects of the changeset (codecs improvements and test_io fixes), but does not clearly identify the main change, appearing somewhat broad and vague.	Consider a more specific title that highlights the primary change, such as 'Fix IO finalization and improve codec handling' or 'Add deallocation warnings and codec validation guards'.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

📝 Generate docstrings

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

github-actions

Code has been automatically formatted

The code in this PR has been formatted using:

cargo fmt --all
Please pull the latest changes before pushing again:

git pull origin test_io

coderabbitai

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

crates/vm/src/stdlib/io.rs (1)

2687-2727: ⚠️ Potential issue | 🟠 Major

Route TextIOWrapper encoding parameter through text_encoding() to emit EncodingWarning.

Per PEP 597, TextIOWrapper(..., encoding=None) should emit EncodingWarning when warn_default_encoding is enabled, but currently it bypasses the warning mechanism by calling resolve_encoding() directly. This is inconsistent with io.open(), which correctly uses text_encoding() to emit the warning. The None case in TextIOWrapper::init() should be routed through text_encoding() to comply with PEP 597.

🤖 Fix all issues with AI agents

In `@crates/vm/src/stdlib/io.rs`:
- Around line 2770-2862: The bool_from_index function currently calls
value.try_index() and converts to i32 which breaks truthiness and can overflow;
replace its body to directly use value.try_to_bool(vm) and return that result
(i.e., remove try_index()/try_to_primitive() logic and simply call
value.try_to_bool(vm)?), keeping the function signature (fn
bool_from_index(value: PyObjectRef, vm: &VirtualMachine) -> PyResult<bool>) so
callers (and the pattern used elsewhere with try_to_bool) continue to work.

coderabbitai

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@crates/vm/src/stdlib/codecs.rs`:
- Around line 996-1013: The replacement-handling branch currently rejects
non-ASCII code points; instead, encode the replacement PyStr using the same code
page as the fast path by calling try_encode_code_page_strict on rep_str with
encoding_str and handle errors the same way as other encoding failures: on
success extend output with the returned bytes, on failure return
vm.new_unicode_encode_error_real (preserving s, pos, encoding_str and an
appropriate error message). Replace the manual per-codepoint ASCII check with
this call in the branch that inspects replacement.downcast_ref::<PyStr>() so the
error handler can emit bytes valid for encodings like cp1252.

github-actions

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] lib: cpython/Lib/codecs.py
[ ] test: cpython/Lib/test/test_codecs.py (TODO: 35)
[ ] test: cpython/Lib/test/test_codeccallbacks.py (TODO: 9)
[ ] test: cpython/Lib/test/test_codecencodings_cn.py
[ ] test: cpython/Lib/test/test_codecencodings_hk.py
[ ] test: cpython/Lib/test/test_codecencodings_iso2022.py
[ ] test: cpython/Lib/test/test_codecencodings_jp.py
[ ] test: cpython/Lib/test/test_codecencodings_kr.py
[ ] test: cpython/Lib/test/test_codecencodings_tw.py
[ ] test: cpython/Lib/test/test_codecmaps_cn.py
[ ] test: cpython/Lib/test/test_codecmaps_hk.py
[ ] test: cpython/Lib/test/test_codecmaps_jp.py
[ ] test: cpython/Lib/test/test_codecmaps_kr.py
[ ] test: cpython/Lib/test/test_codecmaps_tw.py
[ ] test: cpython/Lib/test/test_charmapcodec.py
[ ] test: cpython/Lib/test/test_multibytecodec.py

dependencies:

codecs

dependent tests: (106 tests)

codecs: test_charmapcodec test_codeccallbacks test_codecs test_eof test_exceptions test_importlib test_io test_json test_locale test_logging test_os test_plistlib test_str test_sys
- encodings:
  - locale: test__locale test_builtin test_c_locale_coercion test_calendar test_decimal test_format test_re test_regrtest test_time test_types test_utf8_mode
- json: test_subprocess test_sysconfig test_tomllib test_tools test_traceback test_zoneinfo
  - importlib.metadata: test_importlib
- pickle: test_annotationlib test_array test_asyncio test_bytes test_bz2 test_collections test_concurrent_futures test_coroutines test_csv test_ctypes test_defaultdict test_deque test_descr test_dict test_dictviews test_email test_enum test_enumerate test_fractions test_functools test_generators test_genericalias test_http_cookies test_inspect test_ipaddress test_iter test_itertools test_list test_lzma test_memoryio test_memoryview test_opcache test_operator test_ordered_dict test_pathlib test_pickle test_pickletools test_platform test_positional_only_arg test_posix test_random test_range test_set test_shelve test_slice test_socket test_statistics test_string test_trace test_tuple test_typing test_unittest test_uuid test_xml_dom_minicompat test_xml_etree test_zipfile test_zlib test_zoneinfo
  - logging.handlers: test_concurrent_futures
- tokenize: test_linecache test_tabnanny test_tokenize test_unparse
  - inspect: test_abc test_argparse test_asyncgen test_code test_grammar test_ntpath test_patma test_posixpath test_signal test_yield_from test_zipimport

[ ] lib: cpython/Lib/encodings

dependencies:

encodings (native: _codecs_cn, _codecs_hk, _codecs_iso2022, _codecs_jp, _codecs_kr, _codecs_tw, _multibytecodec, binascii, sys, unicodedata, zlib)
- io (native: _io, _thread, errno, sys)
- base64, bz2, codecs, quopri, stringprep

dependent tests: (57 tests)

encodings: test_codecs
- locale: test__locale test_builtin test_c_locale_coercion test_calendar test_decimal test_format test_io test_locale test_os test_re test_regrtest test_sys test_time test_types test_utf8_mode
  - calendar: test_imaplib test_strftime
  - subprocess: test_android test_asyncio test_audit test_bz2 test_cmd_line test_cmd_line_script test_ctypes test_dtrace test_faulthandler test_file_eintr test_gzip test_inspect test_json test_msvcrt test_ntpath test_osx_env test_platform test_plistlib test_poll test_py_compile test_repl test_runpy test_script_helper test_select test_shutil test_signal test_site test_sqlite3 test_subprocess test_support test_sysconfig test_tempfile test_threading test_unittest test_urllib2 test_venv test_wait3 test_webbrowser test_zipfile

[x] lib: cpython/Lib/bz2.py
[x] test: cpython/Lib/test/test_bz2.py (TODO: 1)

dependencies:

bz2

dependent tests: (1 tests)

bz2: test_bz2

[x] lib: cpython/Lib/fileinput.py
[ ] test: cpython/Lib/test/test_fileinput.py

dependencies:

fileinput

dependent tests: (2 tests)

fileinput: test_fileinput test_genericalias

[x] lib: cpython/Lib/gzip.py
[x] test: cpython/Lib/test/test_gzip.py

dependencies:

gzip

dependent tests: (no tests depend on gzip)

[ ] lib: cpython/Lib/io.py
[ ] lib: cpython/Lib/_pyio.py
[ ] test: cpython/Lib/test/test_io.py (TODO: 26)
[ ] test: cpython/Lib/test/test_bufio.py
[ ] test: cpython/Lib/test/test_fileio.py
[ ] test: cpython/Lib/test/test_memoryio.py (TODO: 26)

dependencies:

io

dependent tests: (87 tests)

io: test__colorize test_android test_argparse test_ast test_asyncio test_bufio test_builtin test_bz2 test_calendar test_cmd test_cmd_line_script test_codecs test_compileall test_compiler_assemble test_concurrent_futures test_configparser test_contextlib test_csv test_dbm_dumb test_dis test_email test_enum test_file test_fileinput test_fileio test_ftplib test_getpass test_gzip test_hashlib test_httplib test_httpservers test_importlib test_inspect test_io test_json test_largefile test_logging test_lzma test_mailbox test_marshal test_memoryio test_memoryview test_mimetypes test_optparse test_pathlib test_pickle test_pickletools test_plistlib test_pprint test_print test_pty test_pulldom test_pyexpat test_regrtest test_robotparser test_shlex test_shutil test_site test_smtplib test_socket test_socketserver test_subprocess test_support test_sys test_tarfile test_tempfile test_threadedtempfile test_timeit test_tokenize test_traceback test_typing test_unittest test_univnewlines test_urllib test_urllib2 test_uuid test_wave test_wsgiref test_xml_dom_xmlbuilder test_xml_etree test_xml_etree_c test_xmlrpc test_zipapp test_zipfile test_zipimport test_zoneinfo test_zstd

[ ] lib: cpython/Lib/logging
[ ] test: cpython/Lib/test/test_logging.py (TODO: 9)

dependencies:

logging (native: atexit, errno, logging.handlers, sys, time)
- copy
- pickle (native: itertools, sys)
- traceback (native: collections.abc, itertools, sys)
- types
- io
- functools, os, queue, re, socket, socketserver, string, struct, threading

dependent tests: (9 tests)

logging: test_asyncio test_concurrent_futures test_decimal test_genericalias test_logging test_support test_unittest
- concurrent.futures._base: test_concurrent_futures
- venv: test_venv

[x] lib: cpython/Lib/lzma.py
[x] test: cpython/Lib/test/test_lzma.py (TODO: 50)

dependencies:

lzma

dependent tests: (1 tests)

lzma: test_lzma

[ ] lib: cpython/Lib/plistlib.py
[ ] test: cpython/Lib/test/test_plistlib.py (TODO: 6)

dependencies:

plistlib (native: binascii, itertools, xml.parsers.expat)
- datetime (native: _thread, math, sys, time)
- enum (native: builtins, sys)
- io
- codecs, os, re, struct

dependent tests: (1 tests)

plistlib: test_plistlib

[ ] lib: cpython/Lib/test/libregrtest

dependencies:

regrtest (native: _overlapped, _thread, _winapi, builtins, collections.abc, faulthandler, gc, itertools, math, os.path, sys, test.libregrtest.utils, test.support, test.support.os_helper, time, winreg)
- libregrtest
- argparse (native: sys)
- dataclasses (native: itertools, sys)
- inspect (native: builtins, collections.abc, importlib.machinery, itertools, sys)
- json (native: json.tool, sys)
- linecache
- locale (native: builtins, encodings.aliases, sys)
- platform (native: itertools, sys)
- random (native: _random, itertools, math)
- subprocess (native: builtins, errno, sys, time)
- sysconfig (native: os.path, sys)
- tempfile (native: _thread, errno, sys)
- test (native: main, _ast, _codecs, _ctypes, _datetime, _imp, _interpchannels, _interpreters, _io, _locale, _multibytecodec, _opcode, _operator, _overlapped, _pickle, _pyrepl._module_completer, _pyrepl.console, _pyrepl.input, _pyrepl.keymap, _pyrepl.reader, _pyrepl.readline, _pyrepl.simple_interact, _pyrepl.terminfo, _pyrepl.utils, _ssl, _string, _sysconfig, _testcapi, _testconsole, _testinternalcapi, _testlimitedcapi, _testsinglephase, _thread, _tkinter, _winapi, array, asyncio.events, asyncio.log, asyncio.proactor_events, asyncio.selector_events, asyncio.staggered, atexit, binascii, builtins, cmath, collections.abc, compression._common, compression.zstd, concurrent.futures, concurrent.futures._base, concurrent.futures.interpreter, concurrent.futures.process, concurrent.futures.thread, concurrent.interpreters, concurrent.interpreters._crossinterp, ctypes.macholib.dyld, ctypes.macholib.dylib, ctypes.macholib.framework, ctypes.util, ctypes.wintypes, dbm.dumb, dbm.ndbm, dbm.sqlite3, email._header_value_parser, email._policybase, email.base64mime, email.charset, email.contentmanager, email.errors, email.generator, email.header, email.headerregistry, email.iterators, email.message, email.mime.application, email.mime.audio, email.mime.base, email.mime.image, email.mime.message, email.mime.multipart, email.mime.nonmultipart, email.mime.text, email.parser, email.policy, email.utils, ensurepip._uninstall, errno, faulthandler, fcntl, gc, html.entities, html.parser, http.client, http.cookiejar, http.cookies, http.server, idlelib.idle_test, importlib._bootstrap, importlib._bootstrap_external, importlib.abc, importlib.machinery, importlib.metadata, importlib.readers, importlib.resources, importlib.resources._adapters, importlib.resources.abc, importlib.util, itertools, logging.config, logging.handlers, marshal, math, mmap, msvcrt, multiprocessing.connection, multiprocessing.dummy, multiprocessing.heap, multiprocessing.managers, multiprocessing.pool, multiprocessing.process, multiprocessing.queues, multiprocessing.util, not_a_module, os.path, package.submodule, parse, pyexpat, readline, select, setuptools, sqlite3.main, string.templatelib, sys, sysconfig.main, termios, test._code_definitions, test._crossinterp_definitions, test._test_multiprocessing, test.fork_wait, test.libregrtest, test.libregrtest.filter, test.libregrtest.main, test.libregrtest.result, test.libregrtest.results, test.libregrtest.utils, test.list_tests, test.picklecommon, test.pickletester, test.profilee, test.ssl_servers, test.string_tests, test.support, test.support.ast_helper, test.support.bytecode_helper, test.support.hashlib_helper, test.support.hypothesis_helper, test.support.i18n_helper, test.support.import_helper, test.support.logging_helper, test.support.numbers, test.support.os_helper, test.support.pty_helper, test.support.script_helper, test.support.socket_helper, test.support.testcase, test.support.threading_helper, test.support.venv, test.support.warnings_helper, test.test__interpreters, test.test_ast.snippets, test.test_ast.utils, test.test_asyncio, test.test_asyncio.utils, test.test_capi.test_getargs, test.test_contextlib, test.test_ctypes, test.test_ctypes._support, test.test_dbm, test.test_doctest, test.test_doctest.decorator_mod, test.test_email, test.test_httplib, test.test_httpservers, test.test_import, test.test_import.data.circular_imports.binding, test.test_import.data.circular_imports.binding2, test.test_import.data.circular_imports.import_cycle, test.test_import.data.circular_imports.subpkg2.parent, test.test_import.data.circular_imports.subpkg2.parent.child, test.test_importlib, test.test_importlib.extension.test_loader, test.test_importlib.metadata.fixtures, test.test_importlib.util, test.test_inspect, test.test_json, test.test_math, test.test_module.final_a, test.test_module.final_b, test.test_profile, test.test_py_compile, test.test_pydoc, test.test_pyrepl, test.test_sqlite3, test.test_string._support, test.test_tkinter.support, test.test_tkinter.widget_tests, test.test_tools, test.test_ttk_textonly, test.test_unittest, test.test_unittest.support, test.test_unittest.test_result, test.test_unittest.testmock, test.test_unittest.testmock.support, test.test_warnings.data, test.test_zoneinfo, test.test_zoneinfo._support, test.tracedmodules, test.typinganndata, test.typinganndata.ann_module, test_regrtest_b.util, time, tkinter.colorchooser, tkinter.commondialog, tkinter.messagebox, tkinter.simpledialog, unicodedata, unittest.case, unittest.mock, unittest.util, urllib.error, urllib.parse, urllib.request, urllib.response, urllib.robotparser, winreg, wsgiref.handlers, wsgiref.headers, wsgiref.simple_server, wsgiref.util, wsgiref.validate, xml.dom, xml.dom.minicompat, xml.dom.minidom, xml.etree, xml.etree.ElementTree, xml.parsers, xml.parsers.expat, xml.sax, xml.sax.expatreader, xml.sax.handler, xml.sax.saxutils, xml.sax.xmlreader, xmlrpc.client, xmlrpc.server, zipfile._path)
- trace (native: gc, sys, time)
- typing (native: _typing, collections.abc, sys)
- unittest (native: builtins, os.path, sys, time, unittest.util)
- copy, io, traceback
- _colorize, contextlib, functools, importlib, operator, os, queue, re, shlex, signal, struct, textwrap, threading, warnings

dependent tests: (no tests depend on regrtest)

[ ] test: cpython/Lib/test/test_str.py (TODO: 16)
[ ] test: cpython/Lib/test/test_fstring.py (TODO: 29)
[ ] test: cpython/Lib/test/test_string_literals.py (TODO: 5)

dependencies:

dependent tests: (no tests depend on str)

[ ] lib: cpython/Lib/tarfile.py
[ ] test: cpython/Lib/test/test_tarfile.py (TODO: 3)

dependencies:

tarfile (native: builtins, sys, time)
- shutil (native: errno, sys)
- stat
- copy, io
- os, re, struct

dependent tests: (2 tests)

tarfile: test_shutil test_tarfile

[ ] test: cpython/Lib/test/test_utf8_mode.py (TODO: 6)

dependencies:

dependent tests: (no tests depend on utf8_mode)

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

youknowone force-pushed the test_io branch 2 times, most recently from c160d81 to a10351b Compare February 5, 2026 01:13

youknowone force-pushed the test_io branch from 39736be to e5bc4d8 Compare February 5, 2026 04:39

youknowone commented Feb 5, 2026

View reviewed changes

youknowone force-pushed the test_io branch 2 times, most recently from 0194bb5 to 856c6f1 Compare February 5, 2026 09:16

youknowone changed the title ~~Fix test_io expectedFailures~~ Feb 5, 2026

youknowone marked this pull request as ready for review February 5, 2026 09:18

coderabbitai Bot reviewed Feb 5, 2026

View reviewed changes

youknowone force-pushed the test_io branch from 856c6f1 to 0b9964c Compare February 5, 2026 12:17

coderabbitai Bot reviewed Feb 5, 2026

View reviewed changes

youknowone and others added 4 commits February 6, 2026 00:15

Fix test_io expectedFailures

afea165

Update test_io from v3.14.3

e948314

fix io

258ac74

win codecs

6e09d1b

youknowone force-pushed the test_io branch from af69ae3 to 6e09d1b Compare February 5, 2026 15:15

youknowone deleted the test_io branch February 6, 2026 00:04

This was referenced Feb 6, 2026

Impl more codecs and _codecs._unregister_error #7025

Merged

Refactor warn.rs and _warnings module #7023

Merged

This was referenced Feb 15, 2026

_WindowsConsoleIO #7151

Merged

Add missing _winapi functions #7185

Merged

This was referenced Feb 26, 2026

Remove PyStr::as_str #7218

Merged

Update test_utf8source from v3.14.3 and implement it #7318

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Better codecs and fix lots of test_io and other expectedFailures#6997