Issue 43960: test_pdb fails when only some tests are run

Created on 2021-04-27 21:01 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (10) msg392131 - (view) Author: STINNER Victor (vstinner) * (Python committer)

Date: 2021-04-27 21:01

test_pdb fails with the following commands. I tested on Linux, same/similar failure on Windows.

See also bpo-41914: "test_pdb fails".


$ cat bisect 
test.test_pdb.PdbTestCase.test_run_module
test.test_pdb.test_next_until_return_at_return_event
test.test_pdb.test_pdb_next_command_in_generator_for_loop

$ ./python -m test test_pdb  --matchfile=bisect -v
== CPython 3.10.0a7+ (heads/master:6bd9288b80, Apr 27 2021, 22:16:25) [GCC 11.0.1 20210324 (Red Hat 11.0.1-0)]
== Linux-5.11.15-300.fc34.x86_64-x86_64-with-glibc2.33 little-endian
== cwd: /home/vstinner/python/master/build/test_python_7720æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 1.31 Run tests sequentially
0:00:00 load avg: 1.31 [1/1] test_pdb
test_run_module (test.test_pdb.PdbTestCase) ... ok
test_next_until_return_at_return_event (test.test_pdb)
Doctest: test.test_pdb.test_next_until_return_at_return_event ... ok
test_pdb_next_command_in_generator_for_loop (test.test_pdb)
Doctest: test.test_pdb.test_pdb_next_command_in_generator_for_loop ... FAIL

======================================================================
FAIL: test_pdb_next_command_in_generator_for_loop (test.test_pdb)
Doctest: test.test_pdb.test_pdb_next_command_in_generator_for_loop
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vstinner/python/master/Lib/doctest.py", line 2205, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test.test_pdb.test_pdb_next_command_in_generator_for_loop
  File "/home/vstinner/python/master/Lib/test/test_pdb.py", line 1182, in test_pdb_next_command_in_generator_for_loop

----------------------------------------------------------------------
File "/home/vstinner/python/master/Lib/test/test_pdb.py", line 1195, in test.test_pdb.test_pdb_next_command_in_generator_for_loop
Failed example:
    with PdbTestInput(['break test_gen',
                       'continue',
                       'next',
                       'next',
                       'next',
                       'continue']):
        test_function()
Expected:
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(3)test_function()
    -> for i in test_gen():
    (Pdb) break test_gen
    Breakpoint 1 at <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>:1
    (Pdb) continue
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>(2)test_gen()
    -> yield 0
    (Pdb) next
    value 0
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>(3)test_gen()
    -> return 1
    (Pdb) next
    Internal StopIteration: 1
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(3)test_function()
    -> for i in test_gen():
    (Pdb) next
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(5)test_function()
    -> x = 123
    (Pdb) continue
Got:
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(3)test_function()
    -> for i in test_gen():
    (Pdb) break test_gen
    Breakpoint 2 at <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>:1
    (Pdb) continue
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>(2)test_gen()
    -> yield 0
    (Pdb) next
    value 0
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[0]>(3)test_gen()
    -> return 1
    (Pdb) next
    Internal StopIteration: 1
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(3)test_function()
    -> for i in test_gen():
    (Pdb) next
    > <doctest test.test_pdb.test_pdb_next_command_in_generator_for_loop[1]>(5)test_function()
    -> x = 123
    (Pdb) continue


----------------------------------------------------------------------

Ran 3 tests in 0.186s

FAILED (failures=1)
test test_pdb failed
test_pdb failed

== Tests result: FAILURE ==

1 test failed:
    test_pdb

Total duration: 480 ms
Tests result: FAILURE

msg392144 - (view) Author: Irit Katriel (iritkatriel) * (Python committer)

Date: 2021-04-27 22:47

I have a fix, I'll make a PR in a bit.

The issue here is the breakpoint number:

Breakpoint 1 at <doctest ...

vs 

Breakpoint 2 at <doctest ...

The breakpoint number is global state, so tests can impact each other.

I recently added a reset_Breakpoint() function that can be used to clear the global state at the beginning of the test: https://github.com/python/cpython/pull/21989/files

So I will make this test use it.

msg392146 - (view) Author: Irit Katriel (iritkatriel) * (Python committer)

Date: 2021-04-27 22:57

I scanned the file and I think this was the only test where this was missing.  test_pdb_continue_in_bottomframe doesn't have a reset, but it doesn't expect a specific breakpoint number, it does:

Breakpoint ... at <doctest test

So with this patch I think the tests shouldn't interfere with each other anymore.

msg392181 - (view) Author: STINNER Victor (vstinner) * (Python committer)

Date: 2021-04-28 10:38

New changeset 21b02b5f4018474620676be04310f7d230a464ea by Irit Katriel in branch 'master':
bpo-43960: test_pdb resets breakpoints (GH-25673)
https://github.com/python/cpython/commit/21b02b5f4018474620676be04310f7d230a464ea

msg392186 - (view) Author: STINNER Victor (vstinner) * (Python committer)

Date: 2021-04-28 11:30

Irit: the automated backported failed, test_pdb fails with:

   NameError: name 'reset_Breakpoint' is not defined

Do you want to try to backport manually the change to Python 3.9 (and then I can automate the backport to 3.8).

test_pdb on Python 3.8 and 3.9 are also affected by this issue.

msg392188 - (view) Author: Irit Katriel (iritkatriel) * (Python committer)

Date: 2021-04-28 12:08

We didn't backport PR21989 because it was changing the way breakpoints hare maintained, so reset_Breakpoint is not there before 3.10. I can add that to the test file manually.

msg392201 - (view) Author: STINNER Victor (vstinner) * (Python committer)

Date: 2021-04-28 14:02

> We didn't backport PR21989 because it was changing the way breakpoints hare maintained, so reset_Breakpoint is not there before 3.10. I can add that to the test file manually.

Ah yes, it sounds reasonable to me to add reset_Breakpoint() to test_pdb with the bdb clearBreakpoints() code from master.

msg392217 - (view) Author: STINNER Victor (vstinner) * (Python committer)

Date: 2021-04-28 15:21

New changeset 2dc6b1789ec86dc80ea290fe33edd61140e47f6f by Irit Katriel in branch '3.9':
bpo-43960: test_pdb resets breakpoints to make tests deterministic (GH-25691)
https://github.com/python/cpython/commit/2dc6b1789ec86dc80ea290fe33edd61140e47f6f

msg392225 - (view) Author: STINNER Victor (vstinner) * (Python committer)

Date: 2021-04-28 15:43

New changeset b52cc7c5f1a6c5b48d51cd718719a766c37d6e38 by Miss Islington (bot) in branch '3.8':
bpo-43960: test_pdb resets breakpoints to make tests deterministic (GH-25691) (GH-25692)
https://github.com/python/cpython/commit/b52cc7c5f1a6c5b48d51cd718719a766c37d6e38

msg392226 - (view) Author: STINNER Victor (vstinner) * (Python committer)

Date: 2021-04-28 15:43

Thanks for the fix Irit!

History Date User Action Args 2022-04-11 14:59:44adminsetgithub: 88126 2021-04-28 15:43:32vstinnersetmessages: + msg392226 2021-04-28 15:43:10vstinnersetmessages: + msg392225 2021-04-28 15:27:55vstinnersetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.8, Python 3.9 2021-04-28 15:21:28miss-islingtonsetpull_requests: + pull_request24382 2021-04-28 15:21:20vstinnersetmessages: + msg392217 2021-04-28 14:07:39iritkatrielsetpull_requests: + pull_request24381 2021-04-28 14:02:38vstinnersetmessages: + msg392201 2021-04-28 12:08:06iritkatrielsetmessages: + msg392188 2021-04-28 11:30:14vstinnersetmessages: + msg392186 2021-04-28 10:39:54miss-islingtonsetpull_requests: + pull_request24372 2021-04-28 10:38:57vstinnersetmessages: + msg392181 2021-04-28 10:38:48miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request24371
2021-04-27 22:57:51iritkatrielsetmessages: + msg392146 2021-04-27 22:52:18iritkatrielsetkeywords: + patch
stage: patch review
pull_requests: + pull_request24364 2021-04-27 22:47:30iritkatrielsetnosy: + iritkatriel
messages: + msg392144
2021-04-27 21:01:37vstinnercreate