gh-137242: Add Android CI job#137186

mhsmith

See the linked issue for details:

Issue: Android CI and binary releases #137242

mhsmith

The logs are showing a number of "Scudo OOM" warnings similar to those discussed in #121595. However, none of them have been fatal yet.

In API level 35 the memory allocator no longer prints such an alarming message when this happens. Instead it will just be a one line message like this (llvm/llvm-project#68444):

I/scudo   : Can't populate more pages for size class 8720.

Luckily, in API level 35 they've also added the ability to print the allocator statistics at any time, with code like this (llvm/llvm-project@5759e3c):

import ctypes
ctypes.CDLL(None).mallopt(-205, 0)

Based on those statistics, it doesn't look like Python has a memory leak, as the "inuse" column after the test is roughly the same as it was before. So it's probably just temporary high memory use during the test, causing allocations to be directed to the next highest size class.

On the buildbot about a year ago, these warnings were often associated with crashes. As described in Android/README.md, we worked around this by running the emulator once to create it, then editing its configuration files to increase the emulator RAM from 2 GB to 4 GB. After that, the warnings continued, but the crashes stopped.

But that isn't such a good option on GitHub Actions, because the runner is not persistent, so the extra emulator restart would add about a minute to every CI run. In theory you should be able to avoid this by creating the emulator from a custom hardware profile, but in my experiments, the emulator always ended up with 2 GB of RAM regardless of what it said in the profile.

The tests that trigger this are:

test.test_ast.test_ast.AST_Tests.test_ast_recursion_limit
test.test_compile.TestSpecifics.test_compiler_recursion_limit
test.test_compile.TestSpecifics.test_stack_overflow

These tests all involve parsing an extremely long Python statement. Although they're all skipped on WASM because of limited stack space, reducing the recursion limit does not prevent the warning. What does prevent it is reducing the size of the Python statement, e.g. from 500,000 repetitions to 200,000.

Since I haven't seen these tests crash on GitHub Actions yet, I'm not going to change them just now. If they do start crashing, we can change them to reduce the statement size on Android alone, or even on all platforms if it doesn't significantly reduce the strength of the test.

freakboy3742

One fairly minor comment inline; my other comment is an extension of that one.

Although the action is broken up into steps, the context of those steps is lost when it's embedded in the workflow. Aside from the general ergonomics of being able to follow what stage the workflow is up to, there's very little visibility on the fact that some steps aren't be in invoked - unless you pay close attention, it's not clear that the arm64 run isn't running the tests. I know it's not, and why - but there's no external visibility of that fact (like a clearly skipped build step).

If the constraint against using a workflow is that we would need to reference a specific commit - isn't that commit hash one of the inputs to the release workflow? Does uses not accept variable expansions in workflow references?

bedevere-app

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

webknjaz

Does uses not accept variable expansions in workflow references?

@freakboy3742 it does not. But I don't think that it's neccessary or that composite actions would be any different in this regard.

hugovk

Thank you!

mhsmith

@hugovk: Are you able to merge this?

miss-islington-app

Thanks @mhsmith for the PR, and @hugovk for merging it 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14.
🐍🍒⛏🤖

miss-islington-app

Sorry, @mhsmith and @hugovk, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker f660ec37531b5e368a27ba065f73d31ff6fb6680 3.14

hugovk

Yes, thanks!

miss-islington-app

Sorry, @mhsmith and @hugovk, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker f660ec37531b5e368a27ba065f73d31ff6fb6680 3.13

(cherry picked from commit f660ec3) Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Russell Keith-Magee <russell@keith-magee.com> Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

bedevere-app

GH-137683 is a backport of this pull request to the 3.14 branch.

bedevere-app

GH-137684 is a backport of this pull request to the 3.13 branch.

(cherry picked from commit f660ec3) Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Russell Keith-Magee <russell@keith-magee.com> Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

bedevere-bot

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot aarch64 Android 3.x (tier-3) has failed when building commit f660ec3.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1594/builds/3046) and take a look at the build logs.
Check if the failure is related to this commit (f660ec3) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1594/builds/3046

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 9, done.        
remote: Counting objects:  11% (1/9)        
remote: Counting objects:  22% (2/9)        
remote: Counting objects:  33% (3/9)        
remote: Counting objects:  44% (4/9)        
remote: Counting objects:  55% (5/9)        
remote: Counting objects:  66% (6/9)        
remote: Counting objects:  77% (7/9)        
remote: Counting objects:  88% (8/9)        
remote: Counting objects: 100% (9/9)        
remote: Counting objects: 100% (9/9), done.        
remote: Compressing objects:  14% (1/7)        
remote: Compressing objects:  28% (2/7)        
remote: Compressing objects:  42% (3/7)        
remote: Compressing objects:  57% (4/7)        
remote: Compressing objects:  71% (5/7)        
remote: Compressing objects:  85% (6/7)        
remote: Compressing objects: 100% (7/7)        
remote: Compressing objects: 100% (7/7), done.        
remote: Total 9 (delta 2), reused 2 (delta 2), pack-reused 0 (from 0)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to 'f660ec37531b5e368a27ba065f73d31ff6fb6680'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at f660ec37531 gh-137242: Add Android CI job (#137186)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  192k  100  192k    0     0  1753k      0 --:--:-- --:--:-- --:--:-- 1753k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 42455  100 42455    0     0   469k      0 --:--:-- --:--:-- --:--:--  469k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 5041k  100 5041k    0     0  33.3M      0 --:--:-- --:--:-- --:--:-- 33.3M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

 73 1257k   73  927k    0     0  10.2M      0 --:--:-- --:--:-- --:--:-- 10.2M
100 1257k  100 1257k    0     0  13.5M      0 --:--:-- --:--:-- --:--:--  160M
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  635k  100  635k    0     0  5278k      0 --:--:-- --:--:-- --:--:-- 5278k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100  479k  100  479k    0     0  5203k      0 --:--:-- --:--:-- --:--:-- 5203k
../../configure: line 4071: pkg-config: command not found
configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

../../Python/fileutils.c:460:1: warning: unused function 'decode_current_locale' [-Wunused-function]
  460 | decode_current_locale(const char* arg, wchar_t **wstr, size_t *wlen,
      | ^~~~~~~~~~~~~~~~~~~~~
../../Python/fileutils.c:679:1: warning: unused function 'encode_current_locale' [-Wunused-function]
  679 | encode_current_locale(const wchar_t *text, char **str,
      | ^~~~~~~~~~~~~~~~~~~~~
2 warnings generated.
../../Modules/_localemodule.c:195:1: warning: unused function 'is_all_ascii' [-Wunused-function]
  195 | is_all_ascii(const char *str)
      | ^~~~~~~~~~~~
1 warning generated.
../../Modules/pwdmodule.c:69:16: warning: unused variable 'pwd_db_mutex' [-Wunused-variable]
   69 | static PyMutex pwd_db_mutex = {0};
      |                ^~~~~~~~~~~~
1 warning generated.
../../Modules/_hacl/Lib_Memzero0.c:66:6: warning: "Your platform does not support any safe implementation of memzero -- consider a pull request!" [-W#warnings]
   66 |     #warning "Your platform does not support any safe implementation of memzero -- consider a pull request!"
      |      ^
1 warning generated.

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  8784  100  8784    0     0   203k      0 --:--:-- --:--:-- --:--:--  204k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  2894  100  2894    0     0  78256      0 --:--:-- --:--:-- --:--:-- 80388
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 43504  100 43504    0     0  1436k      0 --:--:-- --:--:-- --:--:-- 1464k

mhsmith

The aarch64 Android buildbot has been failing intermittently today with this message when building the testbed app:

Execution failed for task ':app:buildCMakeDebug[arm64-v8a]'.
> com.android.ide.common.process.ProcessException: ninja: Entering directory `/Users/android/buildarea/3.x.mhsmith-android-aarch64/build/Android/testbed/app/.cxx/Debug/4e684x38/arm64-v8a'
  [1/2] Building C object CMakeFiles/main_activity.dir/main_activity.c.o
  [2/2] Linking C shared library /Users/android/buildarea/3.x.mhsmith-android-aarch64/build/Android/testbed/app/build/intermediates/cxx/Debug/4e684x38/obj/arm64-v8a/libmain_activity.so
  FAILED: /Users/android/buildarea/3.x.mhsmith-android-aarch64/build/Android/testbed/app/build/intermediates/cxx/Debug/4e684x38/obj/arm64-v8a/libmain_activity.so 
  : && /Users/android/android-sdk/ndk/27.3.13750724/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang --target=aarch64-none-linux-android24 --sysroot=/Users/android/android-sdk/ndk/27.3.13750724/toolchains/llvm/prebuilt/darwin-x86_64/sysroot -fPIC -g -DANDROID -fdata-sections -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -D__BIONIC_NO_PAGE_SIZE_MACRO -D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security  -fno-limit-debug-info  -static-libstdc++ -Wl,-z,max-page-size=16384 -Wl,--build-id=sha1 -Wl,--no-rosegment -Wl,--no-undefined-version -Wl,--fatal-warnings -Wl,--no-undefined -Qunused-arguments -shared -Wl,-soname,libmain_activity.so -o /Users/android/buildarea/3.x.mhsmith-android-aarch64/build/Android/testbed/app/build/intermediates/cxx/Debug/4e684x38/obj/arm64-v8a/libmain_activity.so CMakeFiles/main_activity.dir/main_activity.c.o -L/Users/android/buildarea/3.x.mhsmith-android-aarch64/build/cross-build/aarch64-linux-android/prefix/lib -llog  -lpython3.15  -latomic -lm && :
  ld.lld: error: /Users/android/buildarea/3.x.mhsmith-android-aarch64/build/cross-build/aarch64-linux-android/prefix/lib/libpython3.15.so: unknown file type

The only explanation I can think of is that it's running low on disk space. I would hope for a more obvious error message if that was the case, but anyway, I've freed up some space and I'll keep an eye on it.

EDIT: Looks like it's OK now, so either it was the disk space, or it was something else that was somehow fixed by restarting the worker process.

(cherry picked from commit f660ec3) Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Russell Keith-Magee <russell@keith-magee.com> Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Russell Keith-Magee <russell@keith-magee.com> Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

This is a follow-up for python#137186 aiming to make the initially added automation live in its own "module" rather than the top-level workflow and make it possible to reuse it externally.

webknjaz

and I could send a follow-up with such a change.

#137768 / python/release-tools#270

Co-authored-by: Russell Keith-Magee <russell@keith-magee.com> Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

…137683) Co-authored-by: Malcolm Smith <smith@chaquo.com> Co-authored-by: Russell Keith-Magee <russell@keith-magee.com> Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

hugovk

[I've reported @zinzoch to GH]

Add Android CI job

97ffb3e

StanFromIreland added the infra CI, GitHub Actions, buildbots, Dependabot, etc. label Jul 28, 2025

mhsmith added 4 commits July 28, 2025 20:28

Add shell to each step of the composite action

bd7b642

Use single quotes

1d8dc2f

Use different artifact names in each matrix job

51ee923

Use similar arguments to --fast-ci

0d1b6d1

mhsmith added the skip news label Jul 28, 2025

mhsmith added 2 commits July 30, 2025 14:56

Suppress make install output by default

67c7af3

Strip debug information from release package by default

16f1230

mhsmith added a commit to mhsmith/release-tools that referenced this pull request Jul 30, 2025

Temporarily point inputs at python/cpython#137186

081e3be

mhsmith changed the title ~~Add Android CI job~~ Jul 30, 2025

bedevere-app Bot mentioned this pull request Jul 30, 2025

Android CI and binary releases #137242

Closed

mhsmith added needs backport to 3.13 bugs and security fixes needs backport to 3.14 bugs and security fixes labels Jul 30, 2025

mhsmith marked this pull request as ready for review July 30, 2025 16:30

mhsmith requested review from AA-Turner, AlexWaygood, ezio-melotti, freakboy3742 and hugovk as code owners July 30, 2025 16:30

bedevere-app Bot added the awaiting review label Jul 30, 2025

freakboy3742 requested changes Jul 31, 2025

View reviewed changes

bedevere-app Bot added awaiting changes and removed awaiting review labels Jul 31, 2025

freakboy3742 mentioned this pull request Jul 31, 2025

Add Android support python/release-tools#265

Merged

webknjaz reviewed Jul 31, 2025

View reviewed changes

hugovk approved these changes Aug 11, 2025

View reviewed changes

bedevere-app Bot removed the awaiting merge label Aug 12, 2025

bedevere-app Bot removed the needs backport to 3.14 bugs and security fixes label Aug 12, 2025

bedevere-app Bot removed the needs backport to 3.13 label Aug 12, 2025

hugovk mentioned this pull request Aug 13, 2025

gh-137400: Fix a crash when disabling profiling across all threads #137471

Merged

webknjaz reviewed Aug 14, 2025

View reviewed changes

webknjaz mentioned this pull request Aug 14, 2025

gh-137242: Build Android artifacts in a reusable workflow #137768

Open

mhsmith mentioned this pull request Sep 9, 2025

gh-137242: Add a --no-randomize option, and use it in Android CI #138649

Merged

This comment was marked as abuse.

Sign in to view

mhsmith mentioned this pull request Dec 4, 2025

gh-141794: Reduce size of compiler stress tests to fix Android warnings #142263

Merged

mhsmith mentioned this pull request Feb 3, 2026

Android testbed emulator needs more RAM #144418

Closed

Conversation

mhsmith commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhsmith commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

freakboy3742 left a comment

Choose a reason for hiding this comment

Uh oh!

bedevere-app Bot commented Jul 31, 2025

Uh oh!

webknjaz commented Jul 31, 2025

Uh oh!

hugovk left a comment

Choose a reason for hiding this comment

Uh oh!

mhsmith commented Aug 12, 2025

Uh oh!

Uh oh!

miss-islington-app Bot commented Aug 12, 2025

Uh oh!

miss-islington-app Bot commented Aug 12, 2025

Uh oh!

hugovk commented Aug 12, 2025

Uh oh!

miss-islington-app Bot commented Aug 12, 2025

Uh oh!

bedevere-app Bot commented Aug 12, 2025

Uh oh!

bedevere-app Bot commented Aug 12, 2025

Uh oh!

bedevere-bot commented Aug 12, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

mhsmith commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

webknjaz commented Aug 14, 2025

Uh oh!

This comment was marked as abuse.

Uh oh!

hugovk commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

mhsmith commented Jul 28, 2025 •

edited

Loading

mhsmith commented Jul 30, 2025 •

edited

Loading

mhsmith commented Aug 12, 2025 •

edited

Loading