ast.NodeVisitor for import tracking#7229
Conversation
Sorry, something went wrong.
📝 WalkthroughWalkthroughThe pull request refactors import parsing logic in Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Sorry, something went wrong.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/update_lib/cmd_todo.py`:
- Line 337: The code that derives lib_name from test_name using chained
removeprefix calls produces a leftover leading underscore for names like
"_test_atexit" (e.g., test_name -> lib_name via lib_name =
test_name.removeprefix("test_").removeprefix("_test")), causing lib_status
lookups to fail; fix by normalizing test_name first (strip leading underscores
or check both "test_" and "_test_" patterns) and then remove the correct prefix
so that lib_name becomes "atexit" (update the logic around lib_name derivation
where test_name is processed, ensuring downstream lib_status and DEPENDENCIES
lookups receive the cleaned library name).
In `@scripts/update_lib/deps.py`:
- Around line 140-142: The docstring for count_rustpython_todo says “Count lines
containing RustPython TODO markers” but the implementation returns
content.count(TODO_MARKER) (counting occurrences); update one to match the
other: either change the docstring to “Count occurrences of TODO_MARKER in
content” or change the implementation in count_rustpython_todo to perform a
line-based count (e.g., iterate over content.splitlines() and sum 1 for lines
that contain TODO_MARKER) so the behavior matches the docstring; references:
function count_rustpython_todo and symbol TODO_MARKER.
- Around line 96-102: In visit_Call, guard access to node.args[0] by first
checking that node.args is non-empty and then ensure the constant's value is a
string before calling endswith; specifically, before "arg = node.args[0]" check
"if not node.args: return" and after confirming isinstance(arg, ast.Constant)
verify "isinstance(arg.value, str)" (or similar) before assigning target/using
target.endswith(".py") so non-string constants or missing args won't raise
IndexError or AttributeError.
- Around line 51-55: The code currently strips module roots by using
name.split(".",1)[-1] and also constructs from-import entries as "module.alias"
which breaks for relative imports (module is None); change the import parsing so
Import names use the top-level component (use name.split(".",1)[0] for
real_name) and change the ImportFrom handling to: if node.module is None
(relative import) use the alias name as the import key, otherwise use the full
module (or module + "." + alias if you need the specific attribute) instead of
blindly synthesizing "module.alias"; update where real_name is added to
self.test_imports / self.lib_imports accordingly.
Sorry, something went wrong.
|
isn't this very slow? I removed ast because it was too slow for interactive tools to parse every test files. |
Sorry, something went wrong.
$ hyperfine --warmup 1 --command-name before --prepare 'git switch main || true' 'python scripts/update_lib/ todo --done --lib ~/Work/cpython/' --command-name after --prepare 'git switch deps-nodevisitor' 'python scripts/update_lib/ todo --done --lib ~/Work/cpython' --export-markdown ~/out.md
It's actually faster this way 🤷 |
Sorry, something went wrong.
It's within the margin of error. |
Sorry, something went wrong.
youknowone
merged commit 804a7e4
into
RustPython:main
Feb 26, 2026
This reverts commit 804a7e4.
This reverts commit 804a7e4.
|
@ShaharNaveh I reverted this in #7230 due to CI. Let's fix update_libs and merge this again. Sorry for inconvenience! |
Sorry, something went wrong.
…stPython#7230) This reverts commit a47572c.
…tPython#7230) This reverts commit 804a7e4.
* Reapply "`ast.NodeVisitor` for import tracking (RustPython#7229)" (RustPython#7230) This reverts commit a47572c.
New output:
abc(18 dependents) | 2025-06-29test_abc(1 TODO)struct(17 dependents) | 2026-01-21test_struct(7 TODO)stat(13 dependents) | 2026-02-14test_statoperator(7 dependents) | 2026-01-23test_operatorenum(7 dependents) | 2026-02-15test_enum(4 TODO)signal(6 dependents) | 2024-04-26test_signal(1 TODO)selectors(6 dependents) | 2025-07-20test_selectorskeyword(5 dependents) | 2024-06-21test_keywordtoken(5 dependents) | 2026-01-22shlex(4 dependents) | 2026-02-02test_shlex(4 TODO)__future__(3 dependents) | 2024-04-23test_future_stmt(8 TODO)bisect(3 dependents) | 2023-12-24test_bisectnumbers(3 dependents) | 2025-09-11test_abstract_numbersreprlib(3 dependents) | 2026-01-21test_reprlib(2 TODO)annotationlib(2 dependents) | 2026-02-24test_annotationlibtty(2 dependents) | 2026-01-05test_tty(2 TODO)cmd(2 dependents) | 2026-01-24test_cmdstringprep(1 dependents) | 2019-12-21test_stringprep_ios_support(1 dependents) | 2025-03-10__hello__| 2023-02-25__phello__| 2023-02-25colorsys| 2025-04-21test_colorsysthis| 2019-06-22graphlib| 2026-01-18test_graphlibopcode| 2026-01-26 Δ313test__opcode(2 TODO)test_opcodesre(59 dependents) | 2026-01-20test_re(14 TODO)argparse(26 dependents) | 2026-02-07test_argparsefunctools(26 dependents) | 2026-01-20test_functools(9 TODO)types(25 dependents) | 2026-02-18test_types(10 TODO)threading(20 dependents) | 2026-01-18test_threading(18 TODO)test_threadedtempfiletest_threading_local(3 TODO)socket(17 dependents) | 2026-02-01test_socket(16 TODO)contextlib(16 dependents) | 2025-08-01test_contextlib(2 TODO)test_contextlib_async(2 TODO)collections(15 dependents) | 2026-02-14 Δ61test_collections(3 TODO)test_deque(3 TODO)test_defaultdict(1 TODO)test_ordered_dict(8 TODO)weakref(14 dependents) | 2026-01-30test_weakref(20 TODO)test_weaksetcopy(11 dependents) | 2026-02-10test_copybase64(10 dependents) | 2026-01-19test_base64getopt(10 dependents) | 2026-02-08test_getopttempfile(10 dependents) | 2026-01-04 Δ26test_tempfile(1 TODO)posixpath(9 dependents) | 2026-02-13test_posixpathpickle(8 dependents) | 2026-02-05 Δ8test_pickle(21 TODO)test_picklebuffer(12 TODO)test_pickletools(8 TODO)ssl(8 dependents) | 2025-10-28 Δ4test_ssl(21 TODO)textwrap(7 dependents) | 2026-01-30test_textwrapcodecs(7 dependents) | 2026-01-31test_codecs(12 TODO)test_codeccallbacks(9 TODO)test_codecencodings_cn(4 TODO)test_codecencodings_hk(1 TODO)test_codecencodings_iso2022(5 TODO)test_codecencodings_jp(7 TODO)test_codecencodings_kr(3 TODO)test_codecencodings_tw(1 TODO)test_codecmaps_cn(3 TODO)test_codecmaps_hk(1 TODO)test_codecmaps_jp(6 TODO)test_codecmaps_kr(3 TODO)test_codecmaps_tw(3 TODO)test_charmapcodectest_multibytecodec(untracked)locale(7 dependents) | 2025-08-20 Δ11test_localetest__localeast(7 dependents) | 2026-02-02test_ast(54 TODO)test_unparsetest_type_comments(15 TODO)fnmatch(6 dependents) | 2026-02-13test_fnmatchbz2(6 dependents) | 2026-01-19test_bz2(1 TODO)runpy(5 dependents) | 2025-09-11test_runpy(2 TODO)json(5 dependents) | 2026-01-18 Δ14test_json(14 TODO)random(5 dependents) | 2024-11-11 Δ149test_randomcopyreg(4 dependents) | 2026-01-24test_copyregdifflib(4 dependents) | 2026-02-08test_diffliblzma(4 dependents) | 2026-01-19test_lzma(49 TODO)queue(4 dependents) | 2026-02-04test_queue_colorize(4 dependents) | 2026-01-17test__colorizeheapq(3 dependents) | 2026-01-18test_heapqhmac(3 dependents) | 2026-02-15test_hmac(14 TODO)string(3 dependents) | 2026-01-17test_stringtest_userstringcontextvars(3 dependents) | 2026-01-18socketserver(3 dependents) | 2026-01-03test_socketserver(1 TODO)gzip(3 dependents) | 2026-01-19test_gzippkgutil(3 dependents) | 2026-01-04 Δ57test_pkgutil(1 TODO)zipfile(3 dependents) | 2026-02-24test_zipfile(15 TODO)test_zipfile64ntpath(3 dependents) | 2026-01-25test_ntpathpathlib(3 dependents) | 2026-02-13test_pathlibcodeop(2 dependents) | 2026-02-07test_codeop(4 TODO)genericpath(2 dependents) | 2025-08-04test_genericpathglob(2 dependents) | 2026-02-13test_globhtml(2 dependents) | 2026-02-15test_htmltest_htmlparserplistlib(2 dependents) | 2026-01-04 Δ2test_plistlib(6 TODO)code(2 dependents) | 2026-02-01test_code(11 TODO)test_code_module(3 TODO)py_compile(2 dependents) | 2026-01-30test_py_compile(3 TODO)quopri(2 dependents) | 2026-01-24test_quopribdb(2 dependents) | 2025-12-30 Δ271test_bdb(33 TODO)mimetypes(2 dependents) | 2026-01-21test_mimetypespstats(2 dependents)test_pstats(untracked)encodings(2 dependents) | 2026-02-06ipaddress(1 dependents) | 2026-01-18test_ipaddressnetrc(1 dependents) | 2025-08-20test_netrcpyclbr(1 dependents) | 2025-07-25test_pyclbr(2 TODO)secrets(1 dependents) | 2025-07-17test_secretssite(1 dependents) | 2026-01-17 Δ29test_site(4 TODO)filecmp(1 dependents) | 2026-01-03test_filecmpdataclasses(1 dependents) | 2026-02-08test_dataclasses(10 TODO)getpass(1 dependents) | 2026-02-08test_getpassrlcompleter(1 dependents) | 2025-07-17test_rlcompleter(1 TODO)configparser(1 dependents) | 2026-02-07test_configparserdbm(1 dependents) | 2026-01-01test_dbm(2 TODO)test_dbm_dumbtest_dbm_gnu(untracked)test_dbm_ndbm(untracked)test_dbm_sqlite3sqlite3(1 dependents) | 2026-01-19test_sqlite3(82 TODO)zipimport(1 dependents) | 2026-01-24test_zipimport(2 TODO)test_zipimport_support(untracked)tarfile(1 dependents) | 2025-05-07 Δ527test_tarfile(3 TODO)csv(1 dependents) | 2026-02-18test_csv(27 TODO)_apple_support| 2025-03-10nturl2path| 2026-01-25test_nturl2pathzipapp| 2025-08-06 Δ14test_zipappfileinput| 2025-04-30test_fileinputfractions| 2026-01-30test_fractions(2 TODO)optparse| 2026-01-30test_optparsewave| 2026-02-13test_wave_android_support| 2025-10-22 Δ7modulefindertest_modulefinder(untracked)sched| 2025-04-21test_schedshelve| 2026-01-01test_shelvetimeit| 2026-02-01test_timeitcursestest_curses(untracked)pty| 2026-01-05 Δ51test_pty(4 TODO)xmlrpc| 2026-02-14test_xmlrpc(5 TODO)test_docxmlrpccompression| 2026-01-19symtable| 2026-01-30test_symtable(17 TODO)tomllib| 2026-02-08test_tomllibos(84 dependents) | 2026-02-03test_os(3 TODO)test_popenwarnings(53 dependents) | 2026-01-18test_warnings(12 TODO)io(42 dependents) | 2026-02-13test_io(21 TODO)test_bufiotest_fileio(1 TODO)test_memoryio(28 TODO)traceback(17 dependents) | 2026-02-06test_traceback(34 TODO)inspect(14 dependents) | 2026-02-07test_inspect(46 TODO)subprocess(14 dependents) | 2026-02-04 Δ2test_subprocess(4 TODO)linecache(12 dependents) | 2026-02-09test_linecachetokenize(11 dependents) | 2022-08-09 Δ357test_tokenize(2 TODO)datetime(7 dependents) | 2026-02-24test_datetimetest_strptime(untracked)calendar(5 dependents) | 2026-01-17test_calendarhashlib(5 dependents) | 2026-02-14test_hashlib(7 TODO)dis(5 dependents) | 2026-01-26test_dis(43 TODO)webbrowser(4 dependents) | 2025-04-21 Δ49test_webbrowserpprint(3 dependents) | 2026-02-06test_pprintemail(3 dependents) | 2026-01-17 Δ238test_email(12 TODO)typing(2 dependents) | 2026-02-11test_typing(4 TODO)test_type_aliasestest_type_annotations(1 TODO)test_type_params(6 TODO)test_genericaliastabnanny(1 dependents) | 2026-02-02test_tabnanny(5 TODO)ftplib(1 dependents) | 2026-02-02test_ftplib(4 TODO)smtplib(1 dependents) | 2025-12-19 Δ11test_smtplibtest_smtpnetprofile(1 dependents)test_profile(untracked)test_cprofile(untracked)tracemalloc(1 dependents)test_tracemalloc(untracked)antigravity| 2023-02-28poplibtest_poplib(untracked)pickletools| 2026-02-06gettext| 2026-01-30test_gettext(7 TODO)compileall| 2026-02-25test_compileall(2 TODO)mailbox| 2026-02-14test_mailboxdecimal| 2026-01-30test_decimal(1 TODO)wsgiref| 2026-01-03 Δ7test_wsgiref(1 TODO)statistics| 2026-02-11test_statistics(1 TODO)xml| 2025-08-21 Δ35test_xml_etree(55 TODO)test_xml_etree_ctest_minidom(untracked)test_pulldom(4 TODO)test_pyexpat(28 TODO)test_sax(untracked)test_xml_dom_minicompattest_xml_dom_xmlbuildershutil(12 dependents) | 2026-02-15test_shutiltkinter(2 dependents) | 2025-04-06 Δ279test_tkinter(untracked)test_ttk(untracked)test_ttk_textonly(untracked)test_tcl(untracked)test_idle(untracked)doctest(1 dependents) | 2026-02-11test_doctest(9 TODO)cProfilesysconfig(6 dependents) | 2025-10-22 Δ265test_sysconfig(8 TODO)test__osx_supportplatform(4 dependents) | 2026-01-04 Δ124test_platformctypes(3 dependents) | 2026-02-18test_ctypes(11 TODO)test_stable_abi_ctypesuuid(1 dependents) | 2026-02-02test_uuidvenv| 2026-01-17 Δ29test_venv(4 TODO)trace| 2025-07-25 Δ15test_trace(14 TODO)ensurepip| 2026-01-17test_ensurepipimaplib| 2026-02-01test_imaplib(1 TODO)zoneinfo| 2025-09-07 Δ24test_zoneinfo(3 TODO)http| 2026-01-16 Δ198test_httplibtest_http_cookiejartest_http_cookiestest_httpservers(1 TODO)turtletest_turtle(untracked)logging(7 dependents) | 2025-07-20 Δ77test_logging(5 TODO)unittest(2 dependents) | 2026-01-18 Δ102test_unittest(15 TODO)urllib(1 dependents) | 2026-02-13test_urllibtest_urllib2test_urllib2_localnettest_urllib2nettest_urllibnettest_urlparsetest_urllib_responsetest_robotparserconcurrent| 2026-01-09 Δ1012test_concurrent_futures(18 TODO)test_interpreters(untracked)test__interpreters(untracked)test__interpchannels(untracked)test_crossinterp(untracked)pydoc(4 dependents) | 2026-02-13test_pydoc(36 TODO)importlib(4 dependents) | 2026-02-05test_importlib(16 TODO)multiprocessing(2 dependents) | 2026-02-04 Δ314test_multiprocessing_fork(35 TODO)test_multiprocessing_forkserver(10 TODO)test_multiprocessing_spawn(13 TODO)test_multiprocessing_main_handlingpdb(1 dependents) | 2026-02-11 Δ2601test_pdb(untracked)_pyrepl| 2025-04-11 Δ2534asyncio(2 dependents) | 2026-02-02 Δ26test_asyncio(38 TODO)idlelibStandalone Tests
test_atexit_test_atexittest_eintr_test_eintr(6 TODO)_test_embed_structseq(untracked)_test_gc_fast_cycles(untracked)_test_monitoring_shutdown(untracked)_test_multiprocessing(15 TODO)_test_venv_multiprocessingtest___all__test_androidtest_appletest_array(55 TODO)test_asdl_parser(untracked)test_asyncgen(4 TODO)test_audittest_augassigntest_exceptions(25 TODO)test_baseexceptiontest_except_star(1 TODO)test_exception_group(3 TODO)test_exception_hierarchy(2 TODO)test_exception_variationstest_bigaddrspacetest_bigmem(4 TODO)test_binascii(1 TODO)test_binoptest_booltest_buffer(11 TODO)test_build_details(untracked)test_builtin(25 TODO)test_bytes(17 TODO)test_c_locale_coerciontest_call(1 TODO)test_capi(untracked)test_cext(untracked)test_class(15 TODO)test_genericclasstest_subclassinittest_clinic(untracked)test_cmathtest_cmd_line(24 TODO)test_cmd_line_script(15 TODO)test_comparetest_compile(24 TODO)test_compiler_assembletest_compiler_codegentest_peepholer(31 TODO)test_complex(2 TODO)test_contains(1 TODO)test_context(6 TODO)test_coroutines(19 TODO)test_cppext(untracked)test_decorators(1 TODO)test_descr(47 TODO)test_descrtut(3 TODO)test_devpolltest_dict(6 TODO)test_dictcomps(1 TODO)test_dictviews(2 TODO)test_userdicttest_dtrace(8 TODO)test_dynamic(1 TODO)test_dynamicclassattributetest_embed(untracked)test_enumeratetest_eof(6 TODO)test_epolltest_errnotest_extcall(8 TODO)test_external_inspection(untracked)test_faulthandler(4 TODO)test_fcntltest_ioctltest_filetest_largefiletest_file_eintr(1 TODO)test_fileutils(untracked)test_finalization(untracked)test_float(6 TODO)test_strtod(6 TODO)test_flufl(4 TODO)test_fork1(1 TODO)test_format(5 TODO)test_frame(untracked)test_free_threading(untracked)test_frozentest_str(16 TODO)test_fstring(19 TODO)test_string_literals(5 TODO)test_funcattrs(6 TODO)test_gctest_gdb(untracked)test_generated_cases(untracked)test_generators(12 TODO)test_genexps(untracked)test_generator_stop(untracked)test_yield_from(6 TODO)test_getpath(untracked)test_global(3 TODO)test_grammar(18 TODO)test_grptest_hash(4 TODO)test_import(3 TODO)test_indextest_int(7 TODO)test_long(4 TODO)test_int_literaltest_isinstancetest_iter(1 TODO)test_iterlentest_itertools(6 TODO)test_keywordonlyargtest_kqueuetest_launchertest_list(5 TODO)test_listcomps(1 TODO)test_userlist(1 TODO)test_lltrace(untracked)test_longexptest_marshal(21 TODO)test_mathtest_math_propertytest_memoryview(9 TODO)test_metaclass(10 TODO)test_mmap(2 TODO)test_module(4 TODO)test_monitoring(untracked)test_msvcrttest_named_expressions(12 TODO)test_numeric_towertest_opcachetest_openptytest_optimizer(untracked)test_osx_envtest_patma(20 TODO)test_peg_generator(untracked)test_pep646_syntax(12 TODO)test_perf_profiler(untracked)test_perfmaps(untracked)test_pkgtest_select(3 TODO)test_poll(1 TODO)test_positional_only_arg(4 TODO)test_posix(4 TODO)test_powtest_print(6 TODO)test_propertytest_pwdtest_pyrepl(untracked)test_repltest_raisetest_range(1 TODO)test_readline(untracked)test_regrtest(10 TODO)test_remote_pdb(untracked)test_resource(2 TODO)test_richcmptest_scope(1 TODO)test_support(3 TODO)test_script_helpertest_set(5 TODO)test_setcompstest_slice(1 TODO)test_sort(2 TODO)test_source_encoding(untracked)test_startfile(untracked)test_timetest_strftimetest_structseqtest_sundrytest_super(4 TODO)test_syntax(26 TODO)test_sys(8 TODO)test_syslog(2 TODO)test_sys_setprofile(14 TODO)test_sys_settrace(85 TODO)test_termios(15 TODO)test_threadtest_thread_local_bytecode(untracked)test_threadsignalstest_timeouttest_toolstest_tstring(5 TODO)test_tuple(1 TODO)test_type_cache(untracked)test_typecheckstest_unicodedata(9 TODO)test_unicode_filetest_unicode_file_functionstest_unicode_identifiers(1 TODO)test_ucn(3 TODO)test_unarytest_univnewlinestest_unpack(1 TODO)test_unpack_ex(11 TODO)test_utf8_mode(6 TODO)test_utf8source(2 TODO)test_wait3test_wait4test_winapitest_winconsoleiotest_winregtest_winsoundtest_with(1 TODO)test_wmitest_xpickle(untracked)test_xxlimited(untracked)test_xxtestfuzz(untracked)test_zlib(2 TODO)test_zstdUntracked Files
Original Files
Summary by CodeRabbit