Hello! I am trying to process S2 L1 data using gpt (SNAP command-line). My goal is to resample the data and then use the idepix. I have turned off cloud shadow mask, mountain shadow mask, cloud buffer mask, but it still hangs. I am using the latest version of SNAP and idepix. I assume my code is ok, because it starts processing but it hangs after one loading dot. I can see that my computer memory and cpus are 100% used. What should I do, because I have 100 of images to process?
The processing can be quite demanding.
How long have you waited? To which resolution have you tried to resample the data and which resampler are you using? The general or the S2 Resampler.
Can you provide the graph you are using?
Also, important to know would be which version are you using?
The configuration of your computer is also of interest. Especially, how much memory do you have?
Thank you for fast response. I have waited 2 hours without any progress (and as the memory is full, then I don’t expect any progress as well, am I right?). I want to resample the data to 10 meters. If I try to only resample the data without idepix, then it works fine. I am using S2 resampler. I have SNAP version 9, resampler version 1.0 and idepix version 9.0.1. And my computer have 15.6 gb of memory. I added the graph in attached files. I hope it helps to solve my problem.
resampling_idepix.xml (1.2 KB)
Is you CLI output similar to this?
gpt D:\_UserSupport\avea\resampling_idepix.xml -t D:\_UserSupport\avea\idepix.dim E:\SENTINEL2\S2B_MSIL1C_20220125T105229_N0400_R051_T31TCF_20220125T133326.SAFE\MTD_MSIL1C.xml
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: GDAL not found on system. Internal GDAL 3.2.1 from distribution will be used. (f0)
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.2.1 set to be used by SNAP.
SEVERE: org.esa.s2tbx.dataio.gdal.GDALLoader: Failed to initialize GDAL native drivers. GDAL readers and writers were disabled.org.esa.s2tbx.jni.EnvironmentVariablesNative.getenv(Ljava/lang/String;)Ljava/lang/String;
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.2.1 set to be used by SNAP.
Executing processing graph
INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32631
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [metadata_level] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist
WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist
INFO: org.hsqldb.persist.Logger: dataFileCache open start
100% done.
Initializing Elevation Model
100% done.
100% done.
100% done.
..INFO: org.esa.snap.core.dataop.dem.ElevationFile: http retrieving https://download.esa.int/step/auxdata/dem/SRTM90/tiff/srtm_37_04.zip
Do you see the Elevation data download? Maybe the processing hangs at downloading the data.
What is your internet connection speed?
One option to solve the issue is to split the processing. First do the resampling and on the result do the idepix. Both processes are quite demanding and in combination it is getting worth.
But we are aware of it and working on solutions. Unfortunately, this takes some time.
C:\Users\ave94\Desktop\toprocess\processDataset.bat C:\Users\ave94\Desktop\toprocess\resampling_idepix.xml C:\Users\ave94\Desktop\toprocess C:\Users\ave94\Desktop\toprocess\output resampled_idepix_
setlocal ENABLEDELAYEDEXPANSION
set gptPath=“C:\Program Files\snap\bin\gpt.exe”
set graphXmlPath=C:\Users\ave94\Desktop\toprocess\resampling_idepix.xml
set sourceDirectory=C:\Users\ave94\Desktop\toprocess
if s == \ set sourceDirectory=C:\Users\ave94\Desktop\toproces
set targetDirectory=C:\Users\ave94\Desktop\toprocess\output
if t == \ set targetDirectory=C:\Users\ave94\Desktop\toprocess\outpu
set targetFilePrefix=resampled_idepix_
md C:\Users\ave94\Desktop\toprocess\output
A subdirectory or file C:\Users\ave94\Desktop\toprocess\output already exists.
for / F in (S2*.SAFE) do ( echo. set sourceFile=~fF
echo Processing !sourceFile!
set targetFile=C:\Users\ave94\Desktop\toprocess\output\resampled_idepix__~nF.dim set procCmd="C:\Program Files\snap\bin\gpt.exe" C:\Users\ave94\Desktop\toprocess\resampling_idepix.xml -e -t "!targetFile!" "!sourceFile!" call !procCmd! ) ( echo. set sourceFile=C:\Users\ave94\Desktop\toprocess\S2A_MSIL1C_20170830T093031_N0205_R136_T35VNE_20200809T215515.SAFE echo Processing !sourceFile! set targetFile=C:\Users\ave94\Desktop\toprocess\output\resampled_idepix__S2A_MSIL1C_20170830T093031_N0205_R136_T35VNE_20200809T215515.dim set procCmd="C:\Program Files\snap\bin\gpt.exe" C:\Users\ave94\Desktop\toprocess\resampling_idepix.xml -e -t "!targetFile!" "!sourceFile!" call !procCmd! ) Processing C:\Users\ave94\Desktop\toprocess\S2A_MSIL1C_20170830T093031_N0205_R136_T35VNE_20200809T215515.SAFE INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience. Executing processing graph INFO: org.esa.s2tbx.dataio.s2.ortho.S2OrthoProductReaderPlugIn: Building product reader - EPSG:32635 WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [granuleidentifier] does not exist WARNING: org.esa.snap.core.metadata.GenericXmlMetadata: Metadata: the path to element [bandid] does not exist INFO: org.hsqldb.persist.Logger: dataFileCache open start 100 done.
Initializing Elevation Model
100% done.
100% done.
100% done.
.
I think it is the same? I will try tomorrow, with better internet connection. I thought about splitting the process, but I hoped to find better solution. At least I know now, it is the overall problem. Thank you!
Unfortunately, it is not better with good internet connection. However, another problem raised, when I split these two processes. Resampling is ok, I saved the files to .nc format. When I try to process resampled data with idepix, then nothing happens. I only get these lines below and that’s it.
C:\Users\ave94\Desktop\toprocess\processDataset.bat C:\Users\ave94\Desktop\toprocess\idepix.xml C:\Users\ave94\Desktop\toprocess\output C:\Users\ave94\Desktop\toprocess\output\out idepix_
setlocal ENABLEDELAYEDEXPANSION
set gptPath=“C:\Program Files\snap\bin\gpt.exe”
set graphXmlPath=C:\Users\ave94\Desktop\toprocess\idepix.xml
set sourceDirectory=C:\Users\ave94\Desktop\toprocess\output
if t == \ set sourceDirectory=C:\Users\ave94\Desktop\toprocess\outpu
set targetDirectory=C:\Users\ave94\Desktop\toprocess\output\out
if t == \ set targetDirectory=C:\Users\ave94\Desktop\toprocess\output\ou
set targetFilePrefix=idepix_
md C:\Users\ave94\Desktop\toprocess\output\out
for / %F in (*.nc) do (
echo.
set sourceFile=%~fF
echo Processing !sourceFile!
set targetFile=C:\Users\ave94\Desktop\toprocess\output\out\idepix__%~nF.nc
set procCmd=“C:\Program Files\snap\bin\gpt.exe” C:\Users\ave94\Desktop\toprocess\idepix.xml -e -t “!targetFile!” “!sourceFile!”
call !procCmd!
)
idepix.xml (750 Bytes)
processDataset.bat (1.7 KB)
What could possibly be wrong?
Can you help me with executing resampling and idepix processes separately? I am able to resample the images, but I don’t know how to process with idepix then? I used netcdf format (calling the output file name as .nc), unfortunately nothing happens, when I use the input as *.nc. What could possibly be wrong? (I added my scripts in the previous comment). Or I should use .dim format still? How to name the input in idepix process then?
Hello,
That nothing happens is quite strange.
Probably your loop is not looping.
Try to replace the header with this one:
for %%F in (%sourceDirectory%\*.nc) do (
The loop header is quite specifying for looping recursively over folders in a given directory.
I’ll update this section on the wiki page:
Bulk Processing with GPT - SNAP - Confluence (atlassian.net)
Yes, it works now!
Thank you!
Another question raised for reducing the memory usage. Is it possible to subset all the images before using idepix, when I have defined, for example, several polygons or roi’s before. I assume it is not possible to define, which date or which satellite images have which polygons?
For example satelliteimage1 (roi1,roi2,roi3), satelliteimage2 (roi4,roi1), satelliteimage3 (roi5).
Thank you in advance!
Ave
Yes, this should be possible. Before subsetting, you should resample the S2 data.
In a batch process, or python script you can use the several ROIs. But you need to use one ROI after the other. A subsetting step can only handle one ROI at a time.