Resample processing time

probably if you create a graph file with this processing chain. It computes the intermediate products in the memory and only writes the final product to a physical file.
You can then apply this graph on various images.

The “virtual” products are only faster because no data has been computed when they are created. The computation takes place only when the data is requested (image is displayed). 4 hours seems to be a long time. I agree. How much memory does your PC have?

Creating a graph, as @ABraun said, and let it run from the command line is the best way of computing such a bunch of data.

Thank you very much @ABraun and @marpet, the graph worked perfectly and only took 40 minutes (1 image).

How easy and quick
I wish I had that How-to one year ago or more !!
Anyway I still don’t understand why GrassGis does this : resample , band math ,and image displayed in a few minutes -versus 1hour with Snap for the same size image

So many thanks
bruno

you can not set the resolution.it is ok.

Dear all,

Could u help to make more explanation about Resample (Resampling)? For instance, Sentinel-2 imagery have 10m, 20m (depend on each band) and we could Resample in to a commmon resolution 10m.

  • Could we Resample all band into a common high resolution (i.e 1m resolution)?
  • If we could do that (resample into 1m), what are the differences between original high resolution imagery (i.e Worldview) and resampled 1m resolution imagery?

Thank you very much!!!

Resampling is only of technical nature. It changes the pixel size to a desired degree, under a given interpolation method.
You can technically resample Sentinel-2 imagery to 1 meter but you will not increase the quality of the data because it was acquired at that resolution. Instead of one black pixel with 10m² you will get 100 black pixels with a size of 1 m². The information below the 10 meter resolution (or 20 and 60 in other bands) cannot be restored.

WorldView or other VHR satellites take images at way higher spatial resolution, but at the cost of spatial coverage. As the satellite constantly moves, images are always a compromise between spatial detail and coverage.
Large image - low detail OR Small image - high detail.
Sentinel-2 is surely something in between but no technique will increase the level of detail present in the original image.

2 Likes

Hi!
I made this graph file which I run using gpt. I want to run resample without storing the file physically. But it seems that it does. Am I doing something wrong?

And another question, how can I run idepix in best way without making it take too much time? The resample resolution is 10 m.

gpt.exe S2Resample_Idepix.xml -p resample_10m.properties -t Idepix_S2A_MSIL1C_20170116T105401_N0204_R051_T32VNM_20170116T105355.nc S2A_MSIL1C_20170116T105401_N0204_R051_T32VNM_20170116T105355.SAFE

With the graph you show you should only get a single output file. The one from Idepix. The resample result is not written by directly passed to the Idepix processor. Where do you see the resampled output?

Processing S2 data at 10m takes always quite some time.
The S2Resampling is not the fastes. If you are not interested in accurate angle interpolation then you can use Resample instead of S2Resampling.

Idepix is an complex algorithm too. Especially when enabling cloud shadow detection. But this you have already disabled. Maybe disabling mountain shadows can also save some computation time. But this is probably not suitable for your application.

So there is not much you can do besides lowering the resolution and or equipping your computer with more ram.

You can also check if you have some resources left. You can change the memory settings for SNAP. I took the following from another document.

For the SNAP Desktop application, you can increase the amount of memory available to SNAP.

In the ‘etc’ folder of the SNAP installation directory, you’ll find a file named snap.conf. Open it in a text editor.

There is the line which starts with ‘default_options=’

In this line you’ll find an option like -J-Xmx5G . Increase the value. You could use something like -J-Xmx13G , if you have enough memory in your computer. By default, it is set to ~75% of the maximum value. This is usually a good choice.

If you experience the error on the command line with gpt or pconvert you need to change different files.

You need to change the corresponding vmoptions files, either gpt.vmoptions or pconvert.vmoptions.

Change the value after -Xmx in the last line.

If your computer has 16GB the Xmx-value is 11GB, by default. You can try to set to 14GB. But then you have not much memory left for other things.
If your computer has only 8GB of RAM, then it is just not powerful enough for processing S2 data at 10m.

1 Like

Thank you! Then it works ok. If I want to do the atmospheric correction afterwards using C2RCC, then it is probably best to do that with the same approach as above, not physically save resample and idepix but only the final product? My computer has 32 GB RAM. I would really like to keep the mountain shadow.

Another question, sometimes I get a message which say somethings like “Database is closed”. What does that mean?

You can safely ignore the “database closed” message. It is only a log output which should not be there. A database is used for an internal purpose of the S2 reader and closed by intention. Nothing wrong.

Yes, you can add C2RCC as the third node to your graph.

I tried to add the C2RCC as a third node using idepix as a sourceproduct. I have checked the spelling and if nodes doesn’t match, but I don’t understand the error message. I didn’t do anything different just adding this node. Can I use the output from idepix into valid expression? My node for C2RCC looks like this

logoutput.txt (8.0 KB)

For better readability, I’ve extracted the log output into a separate file.

I think your xml is not valid. Maybe you have the tag at the wrong place or it is missing at all.
I’ve attached an example file. The variable references follow a different notation because I copied it from a different context.

c2rcc-S2Idepix-graph.xml (2.8 KB)

Thank you! I actually found several errors in the last node added, but I handn’t included the
sourceProduct refid=“idepix”.
So the xml graph runs, but the error message I now get is a bit weird. I haven’t given any input on width or heigth. Is it maybe due to some error in the sourceproduct or the output from the resample node?

java.lang.IllegalArgumentException: Width (-9586) and height (-9586) must be > 0
at java.awt.image.SampleModel.(Unknown Source)
at java.awt.image.ComponentSampleModel.(Unknown Source)
at javax.media.jai.ComponentSampleModelJAI.(ComponentSampleModelJAI.java:73)
at org.esa.snap.core.util.jai.SingleBandedSampleModel.(SingleBandedSampleModel.java:62)
at org.esa.snap.core.util.ImageUtils.createSingleBandedSampleModel(ImageUtils.java:274)
at org.esa.snap.core.image.ImageManager.createSingleBandedImageLayout(ImageManager.java:135)
at org.esa.snap.core.image.ImageManager.createSingleBandedImageLayout(ImageManager.java:127)
at org.esa.snap.core.image.ImageManager.createSingleBandedImageLayout(ImageManager.java:119)
at org.esa.snap.core.gpf.internal.OperatorImage.(OperatorImage.java:37)
at org.esa.snap.core.gpf.internal.OperatorImageTileStack.(OperatorImageTileStack.java:48)
at org.esa.snap.core.gpf.internal.OperatorContext.initTargetImages(OperatorContext.java:724)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:489)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.GPF.createProductNS(GPF.java:318)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:293)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:272)
at org.esa.s2tbx.s2msi.idepix.algorithms.sentinel2.S2IdepixPostProcessOp.initialize(S2IdepixPostProcessOp.java:84)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.GPF.createProductNS(GPF.java:318)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:293)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:272)
at org.esa.s2tbx.s2msi.idepix.algorithms.sentinel2.S2IdepixOp.computePostProcessProduct(S2IdepixOp.java:223)
at org.esa.s2tbx.s2msi.idepix.algorithms.sentinel2.S2IdepixOp.processSentinel2(S2IdepixOp.java:164)
at org.esa.s2tbx.s2msi.idepix.algorithms.sentinel2.S2IdepixOp.initialize(S2IdepixOp.java:146)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:162)
at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:91)
at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:64)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:128)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:62)
at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:101)
at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:16)

Error: Width (-9586) and height (-9586) must be > 0

Your product has unusually high sun zenith values. The cloud shadow computation does not work well with these. Also, it does not perform well on 10m products, so I suggest you set computeCloudShadow to false.
It will be possible to compute it (also for high sun zenith values on 10m) with SNAP 7.

Ok. That is due to it being a january scene. I need to do a filtering.
But how can I/you tell by the error message?

I couldn’t. I downloaded the product and had a close look at the idepix implementation.

Which products does cloud shadow work best with? 20 m, 60 m or S-3 300 m? I will set it to false for now

I tried to save the output as NetCDF4-BEAM first, and then NetCDF4-CF, but both failed. It seems saving as dim file works, no errors yet and size of file increasing. But why doesn’t the netcdf works? It would be better to work with netcdf. can I easily change the file from .dim to .nc using snappy in python for instance later on?

For S2, Cloud Shadow does only work well on 60m, but that won’t solve the issue with the high sun zenith angle.
Saving to netcdf should work, we’ll look into this.