gh-140601: Add ResourceWarning to iterparse when not closed#140603

osamakader

When iterparse() opens a file by filename and is not explicitly closed, emit a ResourceWarning to alert developers of the resource leak.

This implements the TODO comment at line 1270 of ElementTree.py which has been requesting this feature since the close() method was added.

Add _closed flag to IterParseIterator to track state
Emit ResourceWarning in del if not closed
Add comprehensive test cases
Update existing tests to properly close iterators

Issue: xml.etree.ElementTree.iterparse() should emit ResourceWarning when not explicitly closed #140601

python-cla-bot

All commit authors signed the Contributor License Agreement.

bedevere-app

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

When iterparse() opens a file by filename and is not explicitly closed, emit a ResourceWarning to alert developers of the resource leak. This implements the TODO comment at line 1270 of ElementTree.py which has been requesting this feature since the close() method was added. - Add _closed flag to IterParseIterator to track state - Emit ResourceWarning in __del__ if not closed - Add comprehensive test cases - Update existing tests to properly close iterators Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

cmaloney

Please don't force push to CPython PRs; it's preferred to keep commit history intact for easier review. See: https://devguide.python.org/getting-started/pull-request-lifecycle/#quick-guide

cmaloney

cc: @serhiy-storchaka who added the close() ResourceWarning comment

serhiy-storchaka

These tests were intentionally added to check that the resource is closed without explicit close(). Don't add close(). Just replace check_no_resource_warning() with assertWarns(ResourceWarning) and add gc_collect(). There are other tests that explicitly call close().

- Use nonlocal close_source instead of _closed flag - Bind warnings.warn as default parameter in __del__ - Use assertWarns(ResourceWarning) instead of adding close() calls - Use support.gc_collect() for consistent test behavior - Remove broad exception handling Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

F-strings can fail during cleanup. Use % formatting like subprocess.Popen does for safer __del__ behavior. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

Check that warning message contains 'unclosed file' and the filename to ensure it's from iterparse, not the file destructor. Addresses review feedback from serhiy-storchaka. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

The ResourceWarning tests are already covered in test_iterparse() as noted by serhiy-storchaka in review. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

Removing source=self prevents unraisable exception in C extension. The warning message still contains the filename via %r formatting. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

osamakader

Hi @serhiy-storchaka

I've addressed all the review feedback:
All tests pass locally. However, CI shows "Unraisable exception"
warnings in the C extension tests (test_xml_etree_c). The ResourceWarning
is still being emitted correctly, but it's reported as "unraisable".

Is this expected behavior for warnings in __del__ with C extensions,
or do I need to adjust the implementation?

Thanks!

serhiy-storchaka

No, it is not an expected behavior.

I have fixed the warnings. The underlying file is closed when the iterator has been exhausted or raised an exception, so no explicit close() is needed in this case. And many tests depended on this. I have removed a ResourceWarning in such case.

Please add a versionchanged directive in the module documentation. I am not sure about the What's New document. This is a minor change, so it may not need a What's New entry.

Document that iterparse now emits ResourceWarning in Python 3.15 when not explicitly closed. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

osamakader

Thanks @serhiy-storchaka for the support.

serhiy-storchaka

I did not merge this PR yet, because there is an important question -- should the warning be emitted if the generator was automatically closed after iterating to the end (or after parsing error)? I silenced them, otherwise many existing tests would need an explicit close().

But now I think that the warning should be emitted even if the file was closed. Yes, this will force users to add an explicit close() after iteration (or use with closing(iterparse(...))) in the working code, but this will save them in exceptional cases -- when the iteration was interrupted and never finished.

cmaloney

👍 to always emitting the warning. I think more critical in potential data loss cases (ex. there's a write buffer) but it's also worthwhile when it helps prevent hard to track down easy to resolve bugs

Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

* main: (463 commits) pythongh-140601: Add ResourceWarning to iterparse when not closed (pythonGH-140603) pythongh-137969: Fix double evaluation of `ForwardRef`s which rely on globals (python#140974) pythongh-139109: A new tracing JIT compiler frontend for CPython (pythonGH-140310) pythongh-141004: Document `PyErr_RangedSyntaxLocationObject` (python#141521) pythongh-140873: Add support of non-descriptor callables in functools.singledispatchmethod() (pythonGH-140884) pythongh-139653: Add PyUnstable_ThreadState_SetStackProtection() (python#139668) pythongh-141004: Document `PyCode_Optimize` (pythonGH-141378) pythongh-141004: Document C APIs for dictionary keys, values, and items (pythonGH-141009) pythongh-137959: Fix `TIER1_TO_TIER2` macro name in JIT InternalDocs (pythonGH-141496) pythongh-139871: Add `bytearray.take_bytes([n])` to efficiently extract `bytes` (pythonGH-140128) pythongh-140601: Refactor ElementTree.iterparse() tests (pythonGH-141499) pythongh-135801: Add the module parameter to compile() etc (pythonGH-139652) pythongh-140260: fix data race in `_struct` module initialization with subinterpreters (python#140909) pythongh-137109: refactor warning about threads when forking (python#141438) pythongh-141004: Document `PyRun_InteractiveOneObject` (pythonGH-141405) pythongh-124111: Fix TCL 9 thread detection (pythonGH-128103) pythongh-141442: Add escaping to iOS testbed arguments (python#141443) pythongh-140936: Fix JIT assertion crash at finalization if some generator is alive (pythonGH-140969) Add details about JIT build infrastructure and updating dependencies to `Tools/jit` (python#141167) pythongh-141412: Use reliable target URL for urllib example (pythonGH-141428) ...

* 'main' of github.com:python/cpython: (464 commits) pythongh-140601: Add ResourceWarning to iterparse when not closed (pythonGH-140603) pythongh-137969: Fix double evaluation of `ForwardRef`s which rely on globals (python#140974) pythongh-139109: A new tracing JIT compiler frontend for CPython (pythonGH-140310) pythongh-141004: Document `PyErr_RangedSyntaxLocationObject` (python#141521) pythongh-140873: Add support of non-descriptor callables in functools.singledispatchmethod() (pythonGH-140884) pythongh-139653: Add PyUnstable_ThreadState_SetStackProtection() (python#139668) pythongh-141004: Document `PyCode_Optimize` (pythonGH-141378) pythongh-141004: Document C APIs for dictionary keys, values, and items (pythonGH-141009) pythongh-137959: Fix `TIER1_TO_TIER2` macro name in JIT InternalDocs (pythonGH-141496) pythongh-139871: Add `bytearray.take_bytes([n])` to efficiently extract `bytes` (pythonGH-140128) pythongh-140601: Refactor ElementTree.iterparse() tests (pythonGH-141499) pythongh-135801: Add the module parameter to compile() etc (pythonGH-139652) pythongh-140260: fix data race in `_struct` module initialization with subinterpreters (python#140909) pythongh-137109: refactor warning about threads when forking (python#141438) pythongh-141004: Document `PyRun_InteractiveOneObject` (pythonGH-141405) pythongh-124111: Fix TCL 9 thread detection (pythonGH-128103) pythongh-141442: Add escaping to iOS testbed arguments (python#141443) pythongh-140936: Fix JIT assertion crash at finalization if some generator is alive (pythonGH-140969) Add details about JIT build infrastructure and updating dependencies to `Tools/jit` (python#141167) pythongh-141412: Use reliable target URL for urllib example (pythonGH-141428) ...

…thonGH-140603) When iterparse() opens a file by filename and is not explicitly closed, emit a ResourceWarning to alert developers of the resource leak. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com> Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-app Bot added the awaiting review label Oct 25, 2025

bedevere-app Bot mentioned this pull request Oct 25, 2025

xml.etree.ElementTree.iterparse() should emit ResourceWarning when not explicitly closed #140601

Closed

osamakader force-pushed the issue-140601-iterparse-warning branch 5 times, most recently from 7d1d79a to 54742a1 Compare October 25, 2025 20:34

osamakader force-pushed the issue-140601-iterparse-warning branch from 54742a1 to bf3f888 Compare October 25, 2025 20:50

This comment was marked as duplicate.

Sign in to view

ashm-dev reviewed Oct 26, 2025

View reviewed changes

serhiy-storchaka reviewed Oct 26, 2025

View reviewed changes

osamakader added 3 commits October 26, 2025 22:55

remove extra whitespace …

53f63b9

Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

remove unused import …

c8ea776

Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

serhiy-storchaka reviewed Oct 26, 2025

View reviewed changes

osamakader added 4 commits October 26, 2025 23:24

Use % formatting instead of f-strings in __del__ …

fcd7ca0

F-strings can fail during cleanup. Use % formatting like subprocess.Popen does for safer __del__ behavior. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

Remove redundant test_iterparse_resource_warning function …

6e3b071

The ResourceWarning tests are already covered in test_iterparse() as noted by serhiy-storchaka in review. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

Remove source=self from ResourceWarning …

86fc36b

Removing source=self prevents unraisable exception in C extension. The warning message still contains the filename via %r formatting. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

serhiy-storchaka added 3 commits October 27, 2025 11:29

Add explicit close().

329c09c

Reorganize tests.

dd20441

Exhausted or failed iterators are auto-closed.

b1f67a4

Add versionchanged directive for ResourceWarning …

15c0c5b

Document that iterparse now emits ResourceWarning in Python 3.15 when not explicitly closed. Signed-off-by: Osama Abdelkader <osama.abdelkader@gmail.com>

osamakader requested a review from serhiy-storchaka October 29, 2025 14:08

osamakader requested a review from ashm-dev November 6, 2025 19:38

serhiy-storchaka added 2 commits November 12, 2025 22:20

Merge branch 'main' into issue-140601-iterparse-warning

a7ef9f2

Always emit a warning. Add tests for exhausted not closed iterator.

766d510

serhiy-storchaka requested a review from AA-Turner as a code owner November 12, 2025 21:16

cmaloney reviewed Nov 13, 2025

View reviewed changes

ashm-dev suggested changes Nov 13, 2025

View reviewed changes

bedevere-app Bot added awaiting core review and removed awaiting review labels Nov 13, 2025

serhiy-storchaka and others added 3 commits November 13, 2025 11:50

Apply suggestions from code review …

53880a6

Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>

Refactor tests.

8a37683

Merge branch 'main' into issue-140601-iterparse-warning

6c5c5e8

bedevere-app Bot removed the awaiting core review label Nov 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-140601: Add ResourceWarning to iterparse when not closed#140603

gh-140601: Add ResourceWarning to iterparse when not closed#140603
serhiy-storchaka merged 17 commits into
python:mainfrom
osamakader:issue-140601-iterparse-warning

osamakader commented Oct 25, 2025 •

edited by bedevere-app Bot

Loading

Uh oh!

python-cla-bot Bot commented Oct 25, 2025 •

edited

Loading

Uh oh!

bedevere-app Bot commented Oct 25, 2025

Uh oh!

cmaloney commented Oct 26, 2025

Uh oh!

cmaloney commented Oct 26, 2025

Uh oh!

This comment was marked as duplicate.

Uh oh!

serhiy-storchaka left a comment

Uh oh!

osamakader commented Oct 26, 2025

Uh oh!

serhiy-storchaka commented Oct 27, 2025

Uh oh!

osamakader commented Oct 27, 2025

Uh oh!

serhiy-storchaka commented Nov 12, 2025

Uh oh!

cmaloney commented Nov 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

osamakader commented Oct 25, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

python-cla-bot Bot commented Oct 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app Bot commented Oct 25, 2025

Uh oh!

cmaloney commented Oct 26, 2025

Uh oh!

cmaloney commented Oct 26, 2025

Uh oh!

This comment was marked as duplicate.

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

osamakader commented Oct 26, 2025

Uh oh!

serhiy-storchaka commented Oct 27, 2025

Uh oh!

osamakader commented Oct 27, 2025

Uh oh!

serhiy-storchaka commented Nov 12, 2025

Uh oh!

cmaloney commented Nov 13, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

osamakader commented Oct 25, 2025 •

edited by bedevere-app Bot

Loading

python-cla-bot Bot commented Oct 25, 2025 •

edited

Loading