Ports to aarch64 and ppc64le systems with OS pages 65536 bytes big#165

Karm

Hello,

This PR surpasses #161.
This PR depends on lmdbjava/native#9
Notable changes:

PAGE_SIZE env and sys property configurable at runtime
allocations aligned to whole pages
size of DBs chnages according to what is needed when big pages are in use

Notable documentation these changes are based on:

 *    https://github.com/LMDB/lmdb/blob/LMDB_0.9.24/libraries/liblmdb/mdb.c#L498:L514
 *    https://github.com/LMDB/lmdb/blob/LMDB_0.9.24/libraries/liblmdb/lmdb.h#L813:L845

How it was tested

git clone https://github.com/Karm/native.git
pushd native
git checkout  secondaryArches
git clone https://github.com/LMDB/lmdb.git
pushd lmdb
git checkout LMDB_0.9.24
popd
mvn clean install
popd
git clone https://github.com/Karm/lmdbjava.git 
pushd lmdbjava
git checkout secondaryArches
export PAGE_SIZE=$(getconf PAGESIZE)
mvn clean install

RHEL 8 aarch64, 4 CPUs, 6G RAM

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.14 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.002 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.873 s - in org.lmdbjava.DbiTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.152 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.494 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.029 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.118 s - in org.lmdbjava.TutorialTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0

RHEL 8 ppc64le, 50 CPUs, 252G RAM

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.209 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.275 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.331 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 926.308 s - in org.lmdbjava.DbiTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.52 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 132.677 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.294 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.511 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.649 s - in org.lmdbjava.TutorialTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0

Fedora, amd64, 8 CPUs, 32G RAM

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.463 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.129 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.002 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.026 s - in org.lmdbjava.TutorialTest
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.219 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.768 s - in org.lmdbjava.DbiTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0

Note about ppc64le

As you can see, Dbi test takes a very long time on the, ironically, most powerful ppc64le box, see:

[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 926.308 s - in org.lmdbjava.DbiTest

I do not know why it takes so much time, but it happened more or less in the same way all three times I tried.

* Added more distinguished names for x86_64, aarch64, ppc64le and unified arch64 for all three linuxes * Renamed unifying x86_64 varable to make spotbugs happy

- Page size is configurable via env and sys prop PAGE_SIZE - Allocation happens page by page, not by an arbitrary chunk

codecov-commenter

Codecov Report

Merging #165 into master will decrease coverage by 0.52%.
The diff coverage is 53.57%.

@@             Coverage Diff              @@
##             master     #165      +/-   ##
============================================
- Coverage     90.93%   90.40%   -0.53%     
- Complexity      383      385       +2     
============================================
  Files            30       30              
  Lines          1346     1366      +20     
  Branches        118      124       +6     
============================================
+ Hits           1224     1235      +11     
- Misses           58       63       +5     
- Partials         64       68       +4

Impacted Files	Coverage Δ	Complexity Δ
src/main/java/org/lmdbjava/Library.java	`69.35% <27.27%> (-3.86%)`	`3.00 <0.00> (ø)`
src/main/java/org/lmdbjava/Env.java	`90.47% <70.58%> (-2.39%)`	`41.00 <3.00> (+2.00)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 26d76be...dacfdc8. Read the comment docs.

lgtm-com

This pull request introduces 1 alert when merging dacfdc8 into 26d76be - view on LGTM.com

new alerts:

1 for Result of multiplication cast to wider type

Karm

?

judovana

?

 -   private long mapSize = 1_024 * 1_024;
 +    // On a system with pages 4096 bytes big, this translates to default max map size of 4 MB
 +    private long mapSize = PAGE_SIZE * 1_024;
 => NEWPotential overflow in int multiplication before it is converted to long by use in an assignment context.

It is correct, but not deserved.

bp-alex

It is correct, but not deserved.

There's more information in the LGTM "Result of multiplication cast to wider type" article. Try making PAGE_SIZE a long.

judovana

It is correct, but not deserved.

There's more information in the LGTM "Result of multiplication cast to wider type" article. Try making PAGE_SIZE a long.

Yup. I understand that, and long is direct fix. But considering how the page_size is used in java, jni and lmdb... I'm quite unable to judge to consequences of the retypeing.

On contrary, we can safely assume, that PAGE_SIZE * 1_024 will remain in the int range

bp-alex

As Env.getPageSize() guarantees the page size will never exceed 32768, I agree the mapSize maximum of 33,554,432 (32786 * 1024) is well within the Java int range as you observed.

The Env.mapSize field is a Java long for two reasons:

Its value is eventually passed to the LMDB function mdb_env_set_mapsize(MDB_env *env, size_t size) (API docs). The size_t on a 64-bit machine is of course larger than a Java int.
Users need the ability to specify they require a larger map via Env.Builder.setMapSize(long mapSize). Otherwise their LMDB maps would be limited to 2,147,483,647 bytes (ie about 2 Gigabytes).

Hope this clears things up. I see no issue with performing the cast.

JulienBe

Hello ! Any activity ongoing on this one ? Just waiting on the cast ?
Thanks for your time :3

judovana

I'm afraid nothing si happening here:(

judovana

And afaik should be merged :)

Flole998

Can this be rebased and then merged please?

bp-alex

As per #217 a Zig-based cross-compilation process has now been added to LmdbJava and deprecates the old LmdbJava Native project.

I would welcome an update to this PR or a new PR that provides or improves any cross-platform support.

wardle

The lmdb library states that the map size should be a multiple of the page size.

The failing tests are examples which succeed when tested on systems with conventional page sizes but fail with larger page sizes.

The two options are a) massage the requested map size to a "proper" size in lmdbjava or b) adopt the same view as lmdb and document that it is the responsibility of client to choose a map size and simply alter tests to request a larger map size.

See https://github.com/LMDB/lmdb/blob/LMDB_0.9.24/libraries/liblmdb/lmdb.h#L813:L845 referenced in above PR.

wardle

I've taken the view that if LMDB feels setting the right map size is the job of the client, then lmdbjava should do the same.

As such, I've created a minimal change PR #218 to make the tests run on architectures with a different page size.

bp-alex

Closing this ticket as #218 has added the required functionality and has been merged.

judovana and others added 2 commits September 25, 2020 09:35

Added usage of aarch64 build of native llmdb …

bc0ccbc

* Added more distinguished names for x86_64, aarch64, ppc64le and unified arch64 for all three linuxes * Renamed unifying x86_64 varable to make spotbugs happy

Porting to systems with large pages …

dacfdc8

- Page size is configurable via env and sys prop PAGE_SIZE - Allocation happens page by page, not by an arbitrary chunk

Karm mentioned this pull request Sep 25, 2020

Added support for aarch64 and ppc64le binaries #161

Closed

bp-alex mentioned this pull request Mar 13, 2021

SIGSEGV when running verifier on Raspberry Pi #163

Closed

bp-alex force-pushed the master branch 2 times, most recently from 0a78414 to a1162e1 Compare December 10, 2022 01:29

bp-alex force-pushed the master branch from 7a9b30e to 0dc6780 Compare February 4, 2023 04:40

bp-alex mentioned this pull request Apr 25, 2023

Add support for Linux and Mac ARM platforms #216

Closed

bp-alex closed this Apr 25, 2023

Conversation

Karm commented Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it was tested

RHEL 8 aarch64, 4 CPUs, 6G RAM

RHEL 8 ppc64le, 50 CPUs, 252G RAM

Fedora, amd64, 8 CPUs, 32G RAM

Note about ppc64le

Uh oh!

codecov-commenter commented Sep 25, 2020

Codecov Report

Uh oh!

lgtm-com Bot commented Sep 25, 2020

Uh oh!

Karm commented Oct 23, 2020

Uh oh!

judovana commented Oct 24, 2020

Uh oh!

bp-alex commented Oct 24, 2020

Uh oh!

judovana commented Oct 24, 2020

Uh oh!

bp-alex commented Oct 25, 2020

Uh oh!

JulienBe commented May 19, 2021

Uh oh!

judovana commented May 19, 2021

Uh oh!

judovana commented May 19, 2021

Uh oh!

Flole998 commented Jan 10, 2023

Uh oh!

bp-alex commented Apr 24, 2023

Uh oh!

wardle commented Apr 25, 2023

Uh oh!

wardle commented Apr 25, 2023

Uh oh!

bp-alex commented Apr 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Karm commented Sep 25, 2020 •

edited

Loading