I was wondering whether anyone had managed to use the SNAP Engine with Hadoop, I’m trying to access files via HDFS to open with ProductIO.readProduct() but I’m having no such luck. The method takes in a Java File object but I’m struggling to go from HDFS -> File -> SNAP.
We use Hadoop on our Calvalus cluster too.
Most of our readers can’t utilise an InputStream and this is what you get from HDFS, so I understand your problem.
So we have a little utility class which copies the file from HDFS to the local node into a temporary directory and then we feed this local file copy into SNAP. When the processing is done we clean up the temporary dir.
Works pretty good.
A specific format is not needed. But the file name and its extension is important for several readers. Therefor the createTempFile method will not work.
If the read method returns null it means in most cases that the format is not supported and maybe a plugin is missing. Maybe because if the tempFile.
I’ve managed to get past the file issue, I’m now experiencing a similar issue to before where the operators I need are not available, yet locally it runs OK.
Do you do anything to ensure the operators are available up in Hadoop ‘world’?
As far as I know (I’m not one of the main developer of our cluster) all jars are in one directory and we put them on the class path. So actually not different to the local use case.
Interesting, its strange that they seem to not be available as soon as I run it on Hadoop, for reference I had to do this to get them working locally, I didn’t seem to manage to get it running automatically.
def provisionSentinel1ProductReader(): Unit = {
val plugins = ProductIOPlugInManager.getInstance().getAllReaderPlugIns
var hasSentinel1ProductReader = false
while (plugins.hasNext) {
val plugin = plugins.next()
if (plugin.isInstanceOf[Sentinel1ProductReaderPlugIn]) {
hasSentinel1ProductReader = true
}
}
if (!hasSentinel1ProductReader) {
val manager = ProductIOPlugInManager.getInstance()
manager.addReaderPlugIn(new Sentinel1ProductReaderPlugIn)
manager.addWriterPlugIn(new BigGeoTiffProductWriterPlugIn)
}
@marpet, just ran getOperatorSpis() locally and get a set of size 60 back, looking inside I can see that it contains the ‘Calibration’ operator.
However, if I run the same code on the cluster, I get a set of size 12. If I run GPF.getDefaultInstance.getOperatorSpiRegistry.loadOperatorSpis() and check the size I still only get 12 operators on the cluster.
The operators I get on the cluster are:
Write
Resample
Subset
WriteRGB
BandMaths
Read
Merge
Reproject
Import-Vector
PassThrough
Mosaic
ProductSet-Reader
I thought that might be the case, unsure how this is happening though, the S1 modules are available when ran locally… The code is the exact same when run on the cluster. Any idea to a work around for this? Or is this possibly something with how I’m running it?
Yes. The cause for this must be how you instantiate the VM and provide the class path. It’s not in the source.
I assume that you also can’t create any instance of a S1 class like CalibrationOp.
Try
Operator tempOp = new org.esa.s1tbx.calibration.gpf.CalibrationOp();
val calibrationSpi: OperatorSpi = new org.esa.s1tbx.calibration.gpf.CalibrationOp().getSpi
GPF.getDefaultInstance.getOperatorSpiRegistry.addOperatorSpi(calibrationSpi)
I then get:
Exception in thread "main" org.esa.snap.core.gpf.OperatorException: Operator 'CalibrationOp': Value for 'Source Band' is invalid: 'Intensity_VH'
I don’t want to do this manually though as locally I already have that operator available. I know however that those parameters for Source Band are correct as again locally it runs fine.
For reference, we believe now that a call in DefaultServiceRegistry in com.bc.ceres.core to getService(serviceName) is not returning the Operator, whilst debugging we can see that the services Hashmap is indeed only 12 large. Not sure how this gets populated and whether it is system dependent?
Meanwhile I have a guess. It seems that you compile the classes differently than we do.
Do you consider als the file in resources/META-INF/services
There are the OperatorSPIs defined.
See