RasterRef should not read HDFS scheme with GDAL reader.#319
Conversation
Signed-off-by: Jason T. Brown <jason@astraea.earth>
Signed-off-by: Jason T. Brown <jason@astraea.earth>
Sorry, something went wrong.
Signed-off-by: Jason T. Brown <jason@astraea.earth>
|
Still having some class not found errors in travis. Unsure why. |
Sorry, something went wrong.
|
@vpipkt I think something's wrong with the implementation, because in this test case the scheme resolution should not be attempting to use the GDAL reader if it's not available, yet it's still selecting it over the default reader. https://travis-ci.org/locationtech/rasterframes/builds/580393491#L832-L866 |
Sorry, something went wrong.
metasim
left a comment
There was a problem hiding this comment.
I think there's a bug causing GDAL reader to be selected when GDAL not available and JVM one sufficient.
Sorry, something went wrong.
|
@vpipkt Not 100% sure what's going on here, but don't want to be extra sure the JVM fallback reader still gets selected in non-GDAL scenarios. Wonder if we should leverage this sort of logic? rasterframes/.circleci/config.yml Line 126 in 36be700 Also odd: not seeing it here: |
Sorry, something went wrong.
|
Now you can see in Circle CI, it is passing. https://circleci.com/gh/s22s/rasterframes/tree/fix%2F314 |
Sorry, something went wrong.
Signed-off-by: Jason T. Brown <jason@astraea.earth>
|
@vpipkt Once this weirdness is resolved, should we be trying to use this (if memory serves, written by Azavea folk!)?: https://gdal.org/user/virtual_file_systems.html#vsihdfs-hadoop-file-system |
Sorry, something went wrong.
I am here to help if you run into any trouble with that. Please be advised that some users have run into issues when trying to access HDFS filesystems through GDAL from a JVM process. The problem appears to be related to the fact that GDAL spawns its own inferior JVM to interface with HDFS, and that is done from within a JVM process running native GDAL code (more here). EDIT: Specifically, please see this comment and surrounding comments. There does not appear to be a problem on OpenJDK 11, but there is a problem with OpenJDK 8. |
Sorry, something went wrong.
|
@jamesmcclain Thanks for the background color... sounds to me like we shouldn't go that route if we don't have to. |
Sorry, something went wrong.
You are welcome. |
Sorry, something went wrong.
844c524 to
a5c6446
Compare
January 7, 2021 14:50
And enforce other logic like can't read a gdal only format (mrf, jp2) outside the gdal reader.
Needs a close look and maybe more extensive unit tests. Possibly also more extensive / detailed error messages.
For #314
This also relates, from the user perspective, to #267