Hello,
I need to resample 50 or 60 Sentinel 2 images. It takes almost 4 hours for one image. Is it normal?? Or it is just my computer??
Any other way to do it faster??
Thanks in advance,
Hi,
at what resolution are you resampling the images? Have you tried to subset the images before writting them? If you are not going to use some bands, if you remove them using the subset operator, it should be faster (for example, there are two angle bands for every spectral band and perhaps you are not going to use them)
It is also worth to mention that there is a kind of preliminary (or virtual) resampling for S2 which is offered before many operations:
If you directly select “Yes” you are led to the Resampling operator but the output product path is greyed out:
don’t select “save as…” and leave it like that.
Make your choices in the second tab and hit “Run”
The product is created within 1 second because it is not physically stored as a raster on your hard drive but only as a logical condition defined by your resampling.
You can then work with the virtual product, for example to make a spatial subset (or any other operation). The resampling process is started then only for the selected bands and pixels.
Should take much less time.
3 Likes
Thank you very much, that will save us a lot of time.
One related question: I have several intermediate steps before the final product. Can I work with virtual products and only save as BEAM-DIMAP the last product?? Virtual products are much faster to create and It would be a great solution since I have to work with 50-60 images.
These are the steps:
- resample to 10m
- subsets of bands 2, 3, 4, 8 and 11
- reprojection to ETRS 89, 31N
- land/sea mask (with an imported shp vector)
- kmeans cluster analysis
- export to GepTiff
Any help will be much appreciated.
probably if you create a graph file with this processing chain. It computes the intermediate products in the memory and only writes the final product to a physical file.
You can then apply this graph on various images.
The “virtual” products are only faster because no data has been computed when they are created. The computation takes place only when the data is requested (image is displayed). 4 hours seems to be a long time. I agree. How much memory does your PC have?
Creating a graph, as @ABraun said, and let it run from the command line is the best way of computing such a bunch of data.
Thank you very much @ABraun and @marpet, the graph worked perfectly and only took 40 minutes (1 image).
How easy and quick
I wish I had that How-to one year ago or more !!
Anyway I still don’t understand why GrassGis does this : resample , band math ,and image displayed in a few minutes -versus 1hour with Snap for the same size image
So many thanks
bruno
you can not set the resolution.it is ok.
Dear all,
Could u help to make more explanation about Resample (Resampling)? For instance, Sentinel-2 imagery have 10m, 20m (depend on each band) and we could Resample in to a commmon resolution 10m.
- Could we Resample all band into a common high resolution (i.e 1m resolution)?
- If we could do that (resample into 1m), what are the differences between original high resolution imagery (i.e Worldview) and resampled 1m resolution imagery?
Thank you very much!!!
Resampling is only of technical nature. It changes the pixel size to a desired degree, under a given interpolation method.
You can technically resample Sentinel-2 imagery to 1 meter but you will not increase the quality of the data because it was acquired at that resolution. Instead of one black pixel with 10m² you will get 100 black pixels with a size of 1 m². The information below the 10 meter resolution (or 20 and 60 in other bands) cannot be restored.
WorldView or other VHR satellites take images at way higher spatial resolution, but at the cost of spatial coverage. As the satellite constantly moves, images are always a compromise between spatial detail and coverage.
Large image - low detail OR Small image - high detail.
Sentinel-2 is surely something in between but no technique will increase the level of detail present in the original image.
2 Likes
Hi!
I made this graph file which I run using gpt. I want to run resample without storing the file physically. But it seems that it does. Am I doing something wrong?
And another question, how can I run idepix in best way without making it take too much time? The resample resolution is 10 m.
gpt.exe S2Resample_Idepix.xml -p resample_10m.properties -t Idepix_S2A_MSIL1C_20170116T105401_N0204_R051_T32VNM_20170116T105355.nc S2A_MSIL1C_20170116T105401_N0204_R051_T32VNM_20170116T105355.SAFE
With the graph you show you should only get a single output file. The one from Idepix. The resample result is not written by directly passed to the Idepix processor. Where do you see the resampled output?
Processing S2 data at 10m takes always quite some time.
The S2Resampling is not the fastes. If you are not interested in accurate angle interpolation then you can use Resample instead of S2Resampling.
Idepix is an complex algorithm too. Especially when enabling cloud shadow detection. But this you have already disabled. Maybe disabling mountain shadows can also save some computation time. But this is probably not suitable for your application.
So there is not much you can do besides lowering the resolution and or equipping your computer with more ram.
You can also check if you have some resources left. You can change the memory settings for SNAP. I took the following from another document.
For the SNAP Desktop application, you can increase the amount of memory available to SNAP.
In the ‘etc’ folder of the SNAP installation directory, you’ll find a file named snap.conf. Open it in a text editor.
There is the line which starts with ‘default_options=’
In this line you’ll find an option like
-J-Xmx5G. Increase the value. You could use something like-J-Xmx13G, if you have enough memory in your computer. By default, it is set to ~75% of the maximum value. This is usually a good choice.If you experience the error on the command line with gpt or pconvert you need to change different files.
You need to change the corresponding vmoptions files, either gpt.vmoptions or pconvert.vmoptions.
Change the value after -Xmx in the last line.
If your computer has 16GB the Xmx-value is 11GB, by default. You can try to set to 14GB. But then you have not much memory left for other things.
If your computer has only 8GB of RAM, then it is just not powerful enough for processing S2 data at 10m.
1 Like
Thank you! Then it works ok. If I want to do the atmospheric correction afterwards using C2RCC, then it is probably best to do that with the same approach as above, not physically save resample and idepix but only the final product? My computer has 32 GB RAM. I would really like to keep the mountain shadow.
Another question, sometimes I get a message which say somethings like “Database is closed”. What does that mean?
You can safely ignore the “database closed” message. It is only a log output which should not be there. A database is used for an internal purpose of the S2 reader and closed by intention. Nothing wrong.
Yes, you can add C2RCC as the third node to your graph.
I tried to add the C2RCC as a third node using idepix as a sourceproduct. I have checked the spelling and if nodes doesn’t match, but I don’t understand the error message. I didn’t do anything different just adding this node. Can I use the output from idepix into valid expression? My node for C2RCC looks like this
logoutput.txt (8.0 KB)
For better readability, I’ve extracted the log output into a separate file.
I think your xml is not valid. Maybe you have the tag at the wrong place or it is missing at all.
I’ve attached an example file. The variable references follow a different notation because I copied it from a different context.
c2rcc-S2Idepix-graph.xml (2.8 KB)
Thank you! I actually found several errors in the last node added, but I handn’t included the
sourceProduct refid=“idepix”.
So the xml graph runs, but the error message I now get is a bit weird. I haven’t given any input on width or heigth. Is it maybe due to some error in the sourceproduct or the output from the resample node?
java.lang.IllegalArgumentException: Width (-9586) and height (-9586) must be > 0
at java.awt.image.SampleModel.(Unknown Source)
at java.awt.image.ComponentSampleModel.(Unknown Source)
at javax.media.jai.ComponentSampleModelJAI.(ComponentSampleModelJAI.java:73)
at org.esa.snap.core.util.jai.SingleBandedSampleModel.(SingleBandedSampleModel.java:62)
at org.esa.snap.core.util.ImageUtils.createSingleBandedSampleModel(ImageUtils.java:274)
at org.esa.snap.core.image.ImageManager.createSingleBandedImageLayout(ImageManager.java:135)
at org.esa.snap.core.image.ImageManager.createSingleBandedImageLayout(ImageManager.java:127)
at org.esa.snap.core.image.ImageManager.createSingleBandedImageLayout(ImageManager.java:119)
at org.esa.snap.core.gpf.internal.OperatorImage.(OperatorImage.java:37)
at org.esa.snap.core.gpf.internal.OperatorImageTileStack.(OperatorImageTileStack.java:48)
at org.esa.snap.core.gpf.internal.OperatorContext.initTargetImages(OperatorContext.java:724)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:489)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.GPF.createProductNS(GPF.java:318)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:293)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:272)
at org.esa.s2tbx.s2msi.idepix.algorithms.sentinel2.S2IdepixPostProcessOp.initialize(S2IdepixPostProcessOp.java:84)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.GPF.createProductNS(GPF.java:318)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:293)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:272)
at org.esa.s2tbx.s2msi.idepix.algorithms.sentinel2.S2IdepixOp.computePostProcessProduct(S2IdepixOp.java:223)
at org.esa.s2tbx.s2msi.idepix.algorithms.sentinel2.S2IdepixOp.processSentinel2(S2IdepixOp.java:164)
at org.esa.s2tbx.s2msi.idepix.algorithms.sentinel2.S2IdepixOp.initialize(S2IdepixOp.java:146)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.graph.NodeContext.initTargetProduct(NodeContext.java:77)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:195)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
at org.esa.snap.core.gpf.graph.GraphContext.initNodeContext(GraphContext.java:178)
at org.esa.snap.core.gpf.graph.GraphContext.initOutput(GraphContext.java:162)
at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:91)
at org.esa.snap.core.gpf.graph.GraphContext.(GraphContext.java:64)
at org.esa.snap.core.gpf.graph.GraphProcessor.executeGraph(GraphProcessor.java:128)
at org.esa.snap.core.gpf.main.DefaultCommandLineContext.executeGraph(DefaultCommandLineContext.java:86)
at org.esa.snap.core.gpf.main.CommandLineTool.executeGraph(CommandLineTool.java:534)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraph(CommandLineTool.java:388)
at org.esa.snap.core.gpf.main.CommandLineTool.runGraphOrOperator(CommandLineTool.java:287)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:188)
at org.esa.snap.core.gpf.main.CommandLineTool.run(CommandLineTool.java:121)
at org.esa.snap.core.gpf.main.GPT.run(GPT.java:54)
at org.esa.snap.core.gpf.main.GPT.main(GPT.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.esa.snap.runtime.Launcher.lambda$run$0(Launcher.java:55)
at org.esa.snap.runtime.Engine.runClientCode(Engine.java:189)
at org.esa.snap.runtime.Launcher.run(Launcher.java:51)
at org.esa.snap.runtime.Launcher.main(Launcher.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at com.exe4j.runtime.LauncherEngine.launch(LauncherEngine.java:62)
at com.exe4j.runtime.WinLauncher.main(WinLauncher.java:101)
at com.install4j.runtime.launcher.WinLauncher.main(WinLauncher.java:16)
Error: Width (-9586) and height (-9586) must be > 0
Your product has unusually high sun zenith values. The cloud shadow computation does not work well with these. Also, it does not perform well on 10m products, so I suggest you set computeCloudShadow to false.
It will be possible to compute it (also for high sun zenith values on 10m) with SNAP 7.