Error with band creation for S3 subset/land-mask

[BenL]

In reference to an older post:

Error when using the BandMaths operator via GPT

We are currently trying to switch over to the s1tbx version with snap 2.0. However, we are experiencing several problems when running graph-files via the GPT. First of all we noticed that the xml-tags for some operators have changed rendering our sets of test-graphfiles useless. Although annoying, this is something we can work out, but in the process of converting them to the new syntax we also found the following strange issue. When trying to execute the following xml via the gpt: 20151130T153…

I am having a recurrence of this issue on SNAP 5.0, S3tbx 5.0.3 on OSx 10.10. I am trying to run a simple XML graph that subsets, land-masks and reprojects a level 2 S3 OLCI scene. The same XML has been confirmed as working on two other OSx machines (10.10 and 10.12).

The error log is below (collected from a python logger, with gpt called as a sub-process with the -e option). Can you please help with how to proceed to correct this? I can’t find any reference to modern occurrences.

Thanks

Ben

===================================
INFO:root:INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters

INFO:root:INFO: org.esa.s2tbx.dataio.gdal.activator.GDALDistributionInstaller: No distribution folder found on Macintosh.

INFO:root:Executing processing graph

INFO:root:The Product ‘Subset_S3A_OL_2_WFR____20170705T063341_20170705T063641_20170706T150012_0179_019_291_1799_MAR_O_NT_002.SEN3’ already contains a band with the name ‘latitude’.

INFO:root: done.

INFO:root:org.esa.snap.core.gpf.graph.GraphException: [NodeId: Land-Sea-Mask] The Product ‘Subset_S3A_OL_2_WFR____20170705T063341_20170705T063641_20170706T150012_0179_019_291_1799_MAR_O_NT_002.SEN3’ already contains a band with the name ‘latitude’.

INFO:root: at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:79)

INFO:root: at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)

INFO:root: at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)

INFO:root: at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)

INFO:root: at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)

INFO:root: at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:162)

INFO:root: at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:91)

INFO:root: at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:64)

INFO:root: at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:130)

INFO:root: at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)

INFO:root: at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)

INFO:root: at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)

INFO:root: at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)

INFO:root: at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)

INFO:root: at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)

INFO:root: at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)

INFO:root: at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)

INFO:root: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO:root: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

INFO:root: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

INFO:root: at java.lang.reflect.Method.invoke(Method.java:498)

INFO:root: at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)

INFO:root: at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)

INFO:root: at org.esa.snap.runtime.Launcher.run(Launcher.java:51)

INFO:root: at org.esa.snap.runtime.Launcher.main(Launcher.java:31)

INFO:root: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

INFO:root: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

INFO:root: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

INFO:root: at java.lang.reflect.Method.invoke(Method.java:498)

INFO:root: at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:62)

INFO:root: at com.install4j.runtime.launcher.UnixLauncher.main(UnixLauncher.java:57)

INFO:root:Caused by: org.esa.snap.core.gpf.OperatorException: The Product ‘Subset_S3A_OL_2_WFR____20170705T063341_20170705T063641_20170706T150012_0179_019_291_1799_MAR_O_NT_002.SEN3’ already contains a band with the name ‘latitude’.

INFO:root: at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUtils.java:421)

INFO:root: at org.esa.snap.raster.gpf.masks.CreateLandMaskOp.initialize(CreateLandMaskOp.java:103)

INFO:root: at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)

INFO:root: at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)

INFO:root: at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)

INFO:root: at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)

INFO:root: … 30 more

INFO:root:

INFO:root:Error: [NodeId: Land-Sea-Mask] The Product ‘Subset_S3A_OL_2_WFR____20170705T063341_20170705T063641_20170706T150012_0179_019_291_1799_MAR_O_NT_002.SEN3’ already contains a band with the name ‘latitude’.

[marpet]

Probably the difference between the three computer is that you have enabled the pixel based geo-codings in the options.

And this is good. It increases the accuracy.
The problem is this issue.
If you specify all the bands as source bands, except the latitude and longitude bands the graph should also work for you.

[falahfakhri]

Dear @marpet Sorry for today annoying,

I think I’m facing up the already exist problem, I got the following error,

C:\Users\FALAH FAKHRI\AppData\Local\Temp>“C:\Users\FALAH FAKHRI.snap\auxdata\sen-et-conda-Win64\sen-et-snap-scripts\…\python.exe” “C:\Users\FALAH FAKHRI.snap\auxdata\sen-et-conda-Win64\sen-et-snap-scripts\warp_to_template.py” --resample_algorithm cubicspline --output D:\Sen-ET\tests3.dim --source D:\Sen-ET\DATA\RAW_DATA\S3\S3A_SL_2_LST____20211029T065758_20211029T070058_20211030T131752_0179_078_063_2340_LN2_O_NT_004.SEN3\xfdumanifest.xml --template D:\Sen-ET\DATA\RAW_DATA\S2\S2A_MSIL2A_20211021T075951_N0301_R035_T37SGU_20211021T103900.zip
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL 3.0.2 found on system. JNI driver will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
WARNING: org.esa.s2tbx.dataio.gdal.GDALInstaller: Failed to delete the GDAL distribution folder ‘C:\Users\FALAH FAKHRI.snap\auxdata\gdal’.
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.C:\Users\FALAH FAKHRI.snap\auxdata\gdal\environment-variables.dll (The process cannot access the file because it is being used by another process)
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Installed GDAL 3.0.2 set to be used by SNAP.
WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: D:\Sen-ET\DATA\RAW_DATA\S3\S3A_SL_2_LST____20211029T065758_20211029T070058_20211030T131752_0179_078_063_2340_LN2_O_NT_004.SEN3 (Access is denied)
WARNING: org.esa.s3tbx.dataio.s3.AbstractProductFactory: Could not find ‘’.
ERROR:java.lang.IllegalArgumentException: The Product ‘S3A_SL_2_LST____20211029T065758_20211029T070058_20211030T131752_0179_078_063_2340_LN2_O_NT_004.SEN3’ already contains a band with the name ‘latitude_in’.
Finished tool execution in 8 seconds

[marpet]

Hi Falah,

Does this happen only when using sen-et or is this error also raised when you simply open the data in SNAP? Are you using SNAP7 or SNAP8?
Maybe you can update the script not manually copy the latitude and longitude bands.
Alternatively, you can disable the per-pixel geo-coding option shown in the post above.

[falahfakhri]

Dear Marco,
Very nice good morning, and start of the week,

"Does this happen only when using sen-et or is this error also raised when you simply open the data in SNAP?"

S2 has been processed successfully using sen_et plugin without any error, also I don’t face up any error in SNAP while open any products,

"Are you using SNAP7 or SNAP8?"

I think the SNAP V. is up to date as you see below,

However, I faced this error up while trying to implement Warp to template which is within SEN_ET,
It’s worth mentioning that the plugin and its bundle was successfully installed as you can see below,

"Maybe you can update the script not manually copy the latitude and longitude bands."

I did quite understand this point, because the error has occurred after a few seconds of running the warp to template and gives the foregoing message.

Alternatively, you can disable the per-pixel geo-coding option shown in the post above.

Both options was tired up, and the same error was given.

**By the way, **

Is the input of Warp to template Source S3 the output of • !OUTPUT_LST! - product containing land surface temperature data, OR the source S3 without in preprocessing?

But in general while run SNAP, I got the following error, Does this has any relation to the issue?,

However, this error doesn’t affect opining of any product.

java.util.ServiceConfigurationError: javax.xml.stream.XMLInputFactory: Provider com.ctc.wstx.stax.WstxInputFactory not found
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:372)
at java.util.ServiceLoader$LazyIterator.access$700(ServiceLoader.java:323)
at java.util.ServiceLoader$LazyIterator$2.run(ServiceLoader.java:407)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:409)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at javax.xml.stream.FactoryFinder$1.run(FactoryFinder.java:353)
at java.security.AccessController.doPrivileged(Native Method)
at javax.xml.stream.FactoryFinder.findServiceProvider(FactoryFinder.java:341)
Caused: java.lang.RuntimeException: Provider for class javax.xml.stream.XMLInputFactory cannot be created
at javax.xml.stream.FactoryFinder.findServiceProvider(FactoryFinder.java:367)
Caused: javax.xml.stream.FactoryConfigurationError: Provider for class javax.xml.stream.XMLInputFactory cannot be created
at javax.xml.stream.FactoryFinder.findServiceProvider(FactoryFinder.java:370)
at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:313)
at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:227)
at javax.xml.stream.XMLInputFactory.newInstance(XMLInputFactory.java:154)
at gov.nasa.worldwind.util.WWXML.openEventReaderStream(WWXML.java:353)
at gov.nasa.worldwind.util.WWXML.openEventReader(WWXML.java:502)
at gov.nasa.worldwind.util.WWXML.openEventReader(WWXML.java:464)
at gov.nasa.worldwind.ogc.OGCCapabilities.createReader(OGCCapabilities.java:89)
at gov.nasa.worldwind.ogc.OGCCapabilities.(OGCCapabilities.java:70)
at gov.nasa.worldwind.ogc.wms.WMSCapabilities.(WMSCapabilities.java:71)
at gov.nasa.worldwind.util.SessionCacheRetrievalPostProcessor.handleWMSCapabilitiesContent(SessionCacheRetrievalPostProcessor.java:248)
at gov.nasa.worldwind.util.SessionCacheRetrievalPostProcessor.handleContent(SessionCacheRetrievalPostProcessor.java:238)
at gov.nasa.worldwind.util.SessionCacheRetrievalPostProcessor.onRetrievalSuceeded(SessionCacheRetrievalPostProcessor.java:193)
at gov.nasa.worldwind.util.SessionCacheRetrievalPostProcessor.run(SessionCacheRetrievalPostProcessor.java:177)
at gov.nasa.worldwind.retrieve.URLRetriever.end(URLRetriever.java:352)
at gov.nasa.worldwind.retrieve.URLRetriever.call(URLRetriever.java:280)
at gov.nasa.worldwind.retrieve.URLRetriever.call(URLRetriever.java:27)
at gov.nasa.worldwind.util.SessionCacheUtils.retrieveSessionData(SessionCacheUtils.java:80)
at gov.nasa.worldwind.util.SessionCacheUtils.getOrRetrieveSessionCapabilities(SessionCacheUtils.java:170)
at gov.nasa.worldwind.layers.BasicTiledImageLayer.retrieveResources(BasicTiledImageLayer.java:723)
at gov.nasa.worldwind.layers.BasicTiledImageLayer$2.run(BasicTiledImageLayer.java:811)
[catch] at java.lang.Thread.run(Thread.java:748)

[marpet]

@falahfakhri

“Maybe you can update the script not manually copy the latitude and longitude bands.”

I did quite understand this point, because the error has occurred after a few seconds of running the warp to template and gives the foregoing message.

Wasn’t well written by me. The error indicates that the bands containing the geographic information are copied twice. Once automatically by SNAP and again by the SEN-ET script. That’s why I thought you could change the script not to copy those bands.
But as you have tried to disable the per-pixel geo-coding option and it still does not work the origin of this problem is something else.

@falahfakhri

Is the input of Warp to template Source S3 the output of • !OUTPUT_LST! - product containing land surface temperature data, OR the source S3 without in preprocessing?

This I can’t answer. I don’t know SEN-ET. But from the manual it seems to be a generic operation. But I think, you want to use the output of the S2 pre-processing graph

The XMLInputFactory error is known and is not related to the problem you experience.
See here:

After update: Provider com.ctc.wstx.stax.WstxInputFactory not found

So, it is not related to windows update or the OS itself. I just updated my SNAP installation with the latest updates and the World View is still working. But I get also the exception “javax.xml.stream.XMLInputFactory: Provider com.ctc.wstx.stax.WstxInputFactory not found” In your log I see: WARNING [org.netbeans.ProxyClassLoader]: Will not load class com.ctc.wstx.stax.WstxInputFactory arbitrarily from one of ModuleCL@6e751c5f[org.esa.snap.snap.product.library] and ModuleCL@4fc19dfb[org.esa.…

[falahfakhri]

Of course I’m thankful to you Marco, because you always give the hints, after two days I found the S3 should be re-projected to the same S2 projection, because simply the tutorial says that the template is the preprocessed S2 band reflectance, and thus it has different projection, the input S3 is Band_observation_geometry

[abruescas]

Hi Falah,
I managed to do the whole Sen_ET process a year ago. I took some notes because the official tutorial is not very helpful. I can try to find them and send them to you via email. If they work fine, maybe we could prepare a short manual to share here.What do you think?

[falahfakhri]

Dear Ana,

@abruescas

maybe we could prepare a short manual to share here.What do you think?

Very great idea, and I’m very enthusiastic to do so, I sent my email to you as a private message.

[Estella]

Hi, can you provide me a SEN_ET note? Because I’m also working on this recently, thanks a lot

[marpet]

You can find the note Ana and Falah did in the tutorials section.
Tutorials – STEP (esa.int)
Just search for sen-et or check the ‘External’ subsection.