Java heap errors when BackGeoCoding certain specific S1B products via GPT

Hi Luis,

thanks for the graph. I tried running it, after changing the inputs and outputs to point at the correct files on my system, and found that my systems chokes on it because the required Precise Orbits are not available on my system and also the corresponding zip-file for download from http://step.esa.int/auxdata/orbits/Sentinel-1/POEORB/ does not exist (yet). Even after manual downloading the corresponding Precise Orbit files from https://qc.sentinel1.eo.esa.int/aux_poeorb/ the problem persists… and snap keeps complaining about not being able to download the 2017-2.zip.

Nevertheless, for our application, we have to use RESORBS anyway since the Precise Orbits become available to late. So what I did was changing:

<orbitType>Sentinel Precise (Auto Download)</orbitType>

to

<orbitType>Sentinel Restituted (Auto Download)</orbitType>

in both “Apply-Orbit” Operators.

After this, the Orbit File download errors disappeared, but to be replaced again by the earlier Java Heap Error that started this topic (see again error log below).

Therefore, may I ask you to run your own test-graph once more, just replacing the Precise Orbits by the Restituted Orbits?

Just to make sure that all this is not just caused by some anomalies in Restituted Orbit files…

Many thanks,
Sven.

Error log:

Java heap space
Java heap space
org.esa.snap.core.gpf.OperatorException: Java heap space
at org.esa.snap.core.gpf.graph.GraphProcessor$GPFImagingListener.errorOccurred(GraphP rocessor.java:373)
at com.sun.media.jai.util.SunTileScheduler.sendExceptionToListener(SunTileScheduler.j ava:1646)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:921)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.cobbleFloat(PlanarImage.java:3254)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2181)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
at org.esa.snap.core.datamodel.Band.readRasterData(Band.java:311)
at org.esa.snap.core.dataio.ProductSubsetBuilder.readBandRasterDataRegion(ProductSubs etBuilder.java:328)
at org.esa.snap.core.dataio.ProductSubsetBuilder.readBandRasterDataImpl(ProductSubset Builder.java:297)
at org.esa.snap.core.dataio.AbstractProductReader.readBandRasterData(AbstractProductR eader.java:250)
at org.esa.snap.core.gpf.common.SubsetOp.computeTile(SubsetOp.java:268)
at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:80)
at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at javax.media.jai.PlanarImage.getData(PlanarImage.java:2085)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java: 406)
at org.esa.snap.core.gpf.internal.OperatorContext.getSourceTile(OperatorContext.java: 392)
at org.esa.snap.core.gpf.internal.OperatorImage.computeRect(OperatorImage.java:73)
at javax.media.jai.SourcelessOpImage.computeTile(SourcelessOpImage.java:137)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
at javax.media.jai.OpImage.getTile(OpImage.java:1129)
at com.sun.media.jai.util.RequestJob.compute(SunTileScheduler.java:247)
at com.sun.media.jai.util.WorkerThread.run(SunTileScheduler.java:468)
Caused by: org.esa.snap.core.gpf.OperatorException: Java heap space
at org.esa.snap.engine_utilities.gpf.OperatorUtils.catchOperatorException(OperatorUti ls.java:389)
at org.esa.s1tbx.sentinel1.gpf.BackGeocodingOp.computeTileStack(BackGeocodingOp.java: 476)
at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeRect(OperatorImageTil eStack.java:116)
at org.esa.snap.core.gpf.internal.OperatorImageTileStack.computeTile(OperatorImageTil eStack.java:85)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(SunTileScheduler.java:904)
… 23 more

Error: Java heap space

Well, the orbit file problem is because you are using an older version. A while ago already we started downloading the orbit files directly from the qc website so they are as up to date as possible.
I re-ran it with the restitute orbits and it was still fine. I also ran it without the subset.

From my point of view I believe there is no problem with the data.
We just ran interferogram between those two products and I have obtained something nice. This exlcudes issues with the data timing.

Also there are very little chances to have a problem with the orbit file. An orbit file of wrong quality will lead to not perfect geocoding or terrain correction but not to a straight failure unless it is completly wrong which I exclude (in first place).

Thanks to @lveci and @nuno.miranda for running those tests. I have been running the tests on my side using the 5.0.0 (downloadable) release version, but without the updates since this release. To rule out the option that this is the origin of my issues I will rerun my tests on a mint VM with fresh install of SANP with all available updates…

I will report back here later.
Sven.

I am extracting 6-day repeat coherence for exactly these S1A and S1B scene combinations (orbit 88 and also 37) since late September 2016 and have not had any processing failures, other than the odd SNAP issue with not finding SRTM 1’’ HGT tiles that are already on disk at the terrain correction stage (rerunning that step usually solves that).

Hi All,

after a nice day of troubleshooting, debugging and testing I can report the following on my findings with respect to the reported issue. In fact it turned out to be two issues:

  1. The Java Heap Error reported above appears to originate from some legacy content left behind in the .snap folder from earlier SNAP versions and update cycles. Completely, clearing out the .snap folder and starting out with a new mint one removed the Java Heap error and enabled me to reproduce the results from @lveci graph file. What remains a mystery to me is why this only affected the processing of certain specific S1B products.

  2. With the Java Heap error out of the way, it turned out that my orginal issue as reported here was still there. It now seems that intermediate products (beam dimaps) generated earlier with older SNAP versions are no longer fully compatible with the current toolbox version (at least not for some S1B products???). I managed to get rid off the warning/error by reprocessing everything in the latest version… So at first I thought this solved my issue but I just found that instead of the toolbox throwing an error I now do get output, but the output itself contains patches with NaN’s …

So progress today, but unfortunately no full solution yet :unamused:

Sven.

Hi @lveci,

like posted earlier today, although I do not get error msgs anymore, I am now stuck with strange artifacts in my output data/images rendering them useless for further processing (an example is shown below). Would you have any idea what could be going on here? Causing patches of NaN to pop-up across my output files?

Many thanks in advance!
SVen.

Dear @lveci,

although I still have not found a solution for my issue, I do have now a better understanding about which particular settings and circumstances seem to generate it. I therefore hope you are available to run a few more tests with me which should enable us to far better pinpoint the root-cause of this issue. For this I like to consider the following three test-cases:

All processing done with toolbox instance checked out from the master branch today!

  1. Processing Graph Producing Patches of NaN’s (Example_creating_patches_of_NaNs.xml (6.6 KB))
    In this case, we execute a simple processing sequence that illustrates the issue at hand. It is relevant to note here that the “disableReramp” parameter of the BackGeoCodingOp is set to “true”. The resulting band: Intensity_IW1_VH_slv1_27Feb2017 looks as follows:


    Note that additional output is generated by the graph just after TOPSAR DeburstOp, which does not contain any NaNs (not shown here).

  2. Almost identical processing graph not producing NaN’s (Example_NOT_creating_patches_of_NaNs.xml (6.6 KB)
    In this case, exactly the same processing as in 1 is done except now the parameter “disableReramp” is set to “false”. Now the NaN-patches disappear and the Intensity_IW1_VH_slv1_27Feb2017 band looks as:


    Note that, without the ramp removed we do get a lot of interpolation artifacts from the TerrainCorrection step.

  3. Same processing steps as 1 and 2 but starting from preprocessed data (Example_preprocessed_data_disableReramp_no_NaNs.xml (4.9 KB))
    In this case, instead of starting from the downloaded SLC products, we take as input a stack of two pre-processed subswaths. These pre-processed subswaths have been obtained from the processing steps: Read -> TOPSAR-Split -> Apply-Orbit executed with a SNAP 4.x.x installation. The execution of the attached graph is done with the latest toolbox version and in addition the “disableReramp” is set to “true”. Maybe to some of yours surprise this example generates the following ouput band:


    Note that we do not observe NaN patches nor interpolation artifacts here. This is in fact the processing outcome that we fully rely on.

So, some observations:

  • The appearance of NaN patches after Terrain Correction seem to be related to the “disableReramp” parameter in the earlier BackGeoCoding step.
  • We require “disableReramp” set to “true”, because we need to be able to interpolate the complex data later on without interpolation artifacts.
  • Using input data that is pre-processed with an earlier version (SNAP 4.x.x), the issue can not be reproduced. Using pre-processed data generated with the SNAP 5.x.x versions, does reproduce the issue.
  • The fact that snap 4.x.x pre-processed data does not reproduce the issue, makes me conclude that changes are introduced in SNAP 5.x.x in Operators TOPSAR-Split and/or Apply_Orbit that are somehow related to the observed issue.

I hope that this extensive input will allow you to pinpoint and resolve the issue. Let me know if there is anything else I can do to help.

Best regards,
Sven.

1 Like

Hi Sven,

I tried your graphs and was able to replicate the issue.

I hope it gets resolved soon.

Esteban

I encountered similar problems when I updated my SNAP5 to 5.0.4. If I used 5.0 everything seems fine. Did you update your SNAP to the lastest version too? You could try to uninstall the current version and reinstall the older version (for example 5.0).

Hi Yul,

thanks for thinking along and making your suggestion. Unfortunately, running the example scripts above using the 5.0.0 version does not remove the observed issue of NaN patches.

Sven.

Did you try with all the updates installed for 5.0.0?

Yes, the images above showing the issue were generated with version 5.0.4 and additionally we also tried using a checked-out version of the current the master branch leading to exactly the same results.

Sven.

Hi All,

The problem was caused by the No-Data Value setting in the source product. Basically, after opening a product in the SNAP and right clicking on the source band and selecting Properties, you will see that the No-Data Value Used checkbox is selected and the No-Data Value is set to 0.0. That means all pixels (valid or not) with pixel values being 0.0 will be treated as No-Data Value and their values will not be used in the later on processing and that causes all kinds of problems. Therefore, you should always deselect the “No-Data Value Used” option in the band Properties before starting your processing. By doing so, the problem reported above should not happen.

Jun

1 Like

Hi Junlu,

thanks for your investigations and proposed solution. However, we never open our products via the SNAP GUI. Our processing is fully automated using the GPT running on graph-files like provided above. Consequently, your solution involving right-clicking on the band and deselecting that option does not apply to our situation. In our use-case, where (in which operator) can I set the “No-Data-Value-Used” option in the graph-file?

Sven.

Hi Sven,

We have modified the code so that No-Data Value will not be set to 0.0 by default for S-1 SLC product. Therefore, the problem that you’ve encountered should not happen. We’ve also created a new operator under Raster -> Data Conversion that users can use it to enable or disable the No-Data Value usage and set the No-Data Value. All these fixes should be in SNAP V6.0.

Jun

1 Like

Hi @junlu,

many thanks! We will verify on our side if it completely removes the issues we were experiencing. When is the official release of version 6.0 expected?

Best regards,
Sven.

6.0 official is expected in September - due to the holiday-season it’s impossible to be more precise than this for the moment.

1 Like

Hi Junlu,

I have retried my test-case posted above using the SNAP 6.0 Beta 4 preview. Unfortunately, using that SNAP install the issue remains. Is this as expected or should it have been resolved in this beta preview version? Please advise, should I retry by building the master to verify is it is resolved there?

Sven.

I have just also tried it with a checkout from the current master branch. There the NaN Patches are gone and the issue does seem to be resolved!

Many thanks to @junlu and his team!

1 Like