◐ Shell
clean mode source ↗

Ports to aarch64 and ppc64le systems with OS pages 65536 bytes big by Karm · Pull Request #165 · lmdbjava/lmdbjava

judovana

Hello,

This PR surpasses #161.
This PR depends on lmdbjava/native#9
Notable changes:

  • PAGE_SIZE env and sys property configurable at runtime
  • allocations aligned to whole pages
  • size of DBs chnages according to what is needed when big pages are in use

Notable documentation these changes are based on:

 *    https://github.com/LMDB/lmdb/blob/LMDB_0.9.24/libraries/liblmdb/mdb.c#L498:L514
 *    https://github.com/LMDB/lmdb/blob/LMDB_0.9.24/libraries/liblmdb/lmdb.h#L813:L845

How it was tested

git clone https://github.com/Karm/native.git
pushd native
git checkout  secondaryArches
git clone https://github.com/LMDB/lmdb.git
pushd lmdb
git checkout LMDB_0.9.24
popd
mvn clean install
popd
git clone https://github.com/Karm/lmdbjava.git 
pushd lmdbjava
git checkout secondaryArches
export PAGE_SIZE=$(getconf PAGESIZE)
mvn clean install

RHEL 8 aarch64, 4 CPUs, 6G RAM

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.14 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.002 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.086 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.013 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.018 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 35.873 s - in org.lmdbjava.DbiTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.152 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.494 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.029 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.118 s - in org.lmdbjava.TutorialTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0

RHEL 8 ppc64le, 50 CPUs, 252G RAM

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.209 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.275 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.331 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.015 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 926.308 s - in org.lmdbjava.DbiTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.52 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 132.677 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.022 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.294 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.511 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.03 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.649 s - in org.lmdbjava.TutorialTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0

Fedora, amd64, 8 CPUs, 32G RAM

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.lmdbjava.ResultCodeMapperTest
[INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.113 s - in org.lmdbjava.ResultCodeMapperTest
[INFO] Running org.lmdbjava.TxnTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.463 s - in org.lmdbjava.TxnTest
[INFO] Running org.lmdbjava.LibraryTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.LibraryTest
[INFO] Running org.lmdbjava.MetaTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.lmdbjava.MetaTest
[INFO] Running org.lmdbjava.EnvTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.129 s - in org.lmdbjava.EnvTest
[INFO] Running org.lmdbjava.CursorIterableTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.036 s - in org.lmdbjava.CursorIterableTest
[INFO] Running org.lmdbjava.VerifierTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.002 s - in org.lmdbjava.VerifierTest
[INFO] Running org.lmdbjava.MaskedFlagTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.lmdbjava.MaskedFlagTest
[INFO] Running org.lmdbjava.TutorialTest
[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.026 s - in org.lmdbjava.TutorialTest
[INFO] Running org.lmdbjava.KeyRangeTest
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 s - in org.lmdbjava.KeyRangeTest
[INFO] Running org.lmdbjava.CursorParamTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.219 s - in org.lmdbjava.CursorParamTest
[INFO] Running org.lmdbjava.ComparatorTest
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.019 s - in org.lmdbjava.ComparatorTest
[INFO] Running org.lmdbjava.ByteBufferProxyTest
[INFO] Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in org.lmdbjava.ByteBufferProxyTest
[INFO] Running org.lmdbjava.CursorTest
[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.014 s - in org.lmdbjava.CursorTest
[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.768 s - in org.lmdbjava.DbiTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 187, Failures: 0, Errors: 0, Skipped: 0

Note about ppc64le

As you can see, Dbi test takes a very long time on the, ironically, most powerful ppc64le box, see:

[INFO] Running org.lmdbjava.DbiTest
[INFO] Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 926.308 s - in org.lmdbjava.DbiTest

I do not know why it takes so much time, but it happened more or less in the same way all three times I tried.