{{ message }}
gh-137242: Add a --no-randomize option, and use it in Android CI#138303
Closed
mhsmith wants to merge 3 commits into
Closed
gh-137242: Add a --no-randomize option, and use it in Android CI#138303mhsmith wants to merge 3 commits into
--no-randomize option, and use it in Android CI#138303mhsmith wants to merge 3 commits into
Conversation
Member
Sorry, something went wrong.
mhsmith
added
skip news
bugs and security fixes
needs backport to 3.14
bugs and security fixes
labels
Aug 31, 2025
Member
Author
It looks like some tests are already doing that on some runners. I'll switch this PR back to draft until this has been resolved. |
Sorry, something went wrong.
vstinner
reviewed
Sep 3, 2025
vstinner
left a comment
vstinner
left a comment
Member
There was a problem hiding this comment.
I dislike changing CI defaults.
Sorry, something went wrong.
mhsmith
changed the title
--randomize, add --fail-rerun
vstinner
reviewed
Sep 8, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.
Original PR title: "Change CI arguments: remove
--randomize, add--fail-rerun"--randomizecan discover real problems, but they're often ordering dependencies between tests, which are difficult to diagnose, and usually have nothing to do with the PR on which they occur.I assume this was the reason for removing
--fail-rerunfrom the--fast-ciand--slow-ciarguments in #110849. However, this renders--randomizealmost useless, because the failing test will usually pass on the rerun in a fresh process, and nobody will ever know that there was a failure.Also,
--rerunwithout--fail-rerunmeans that a test which ALWAYS fails the first time and passes the second time will still be treated as a pass. This seems unsafe.So I propose removing
--randomizeand restoring--fail-rerun.This will also allow iOS and Android to switch to
--fast-cion GitHub Actions and--slow-cion the buildbots. They were previously unable to do this because of the frequent failures caused by--randomize, which were not hidden on the rerun because these platforms use--single-processmode.