GPT processing from command prompt

chiarasalv23 · November 22, 2021, 11:12am

Hello everyone!

I’m trying to batch process multiple S1 SLC data. I created a graph on SNAP and basically followed this procedure: SNAP - graph processing with GPT - YouTube

I tried with one product and worked perfectly (and faster than I expected) but what should I do if I need to ‘extend’ this procedure to 20 data or more? There is something like an iteration or a loop to do?

Hope this post finds you well.

Best regards,

CS

marpet · November 22, 2021, 11:24am

If you want to do the processing on multiple products you have to options.
You can use a command line script which iterates your input data.
This is described here:

Bulk Processing with GPT
SNAP CommandLine Tutorial
But you can also use the Graph Processing within SNAP Desktop.

image671×572 22.8 KB

There are also videos which explain its use:

chiarasalv23 · November 23, 2021, 11:04am

Hello,

thank you for the advice!
I’m trying to follow the procedure explained here https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT#Known-Limitations-of-the-Scripts but I can’t find the ‘parameter file’. Where it is located?

Thanks in advance!

marpet · November 23, 2021, 11:14am

You can find it at the beginning:

In the example section it is also shown.

chiarasalv23 · November 23, 2021, 12:15pm

I’m trying but it is not working. I read all the guides but I don’t know where I’m wrong.

I’m new to this kind of processing, sorry.

marpet · November 23, 2021, 12:50pm

Probably you need to update the head of the loop so that your Sentinel-1 is considered:
You should also rename the property file. The name should match the content and you not resampling to 20m.

:: double '%' in batch file and only a single '%' on command line
:: '/D' is for directories like Sentinel data. Remove '/D' when you open files.
for /D /R %sourceDirectory% %%F in (S2*.SAFE) do (

chiarasalv23 · November 23, 2021, 1:44pm

I tried but it is not working.

Where I’m wrong?

marpet · November 23, 2021, 2:17pm

It might be helpful if you attach the modified batch script.
Also, it would be good if you share the output of the script not as image but as text.

I just tried it and for it iterates of my data.

That’s my command line call:
H:\_other\bulkProcess>processDataset.bat resample_s2.xml resample_20m.properties H:\SENTINEL2\sharp_shadow H:\_other\bulkProcess\out resampled20m

And that’s the output

H:\_other\bulkProcess>setlocal ENABLEDELAYEDEXPANSION

H:\_other\bulkProcess>set gptPath="C:\Program Files\snap\bin\gpt.exe"

H:\_other\bulkProcess>set graphXmlPath=resample_s2.xml

H:\_other\bulkProcess>set parameterFilePath=resample_20m.properties

H:\_other\bulkProcess>set sourceDirectory=H:\SENTINEL2\sharp_shadow

H:\_other\bulkProcess>if w == \ set sourceDirectory=H:\SENTINEL2\sharp_shado

H:\_other\bulkProcess>set targetDirectory=H:\_other\bulkProcess\out

H:\_other\bulkProcess>if t == \ set targetDirectory=H:\_other\bulkProcess\ou

H:\_other\bulkProcess>set targetFilePrefix=resampled20m

H:\_other\bulkProcess>md H:\_other\bulkProcess\out
Ein Unterverzeichnis oder eine Datei mit dem Namen "H:\_other\bulkProcess\out" existiert bereits.

H:\_other\bulkProcess>for / %F in (S2*.SAFE) do (
echo.
 set sourceFile=%~fF
 echo Processing !sourceFile!
 set targetFile=H:\_other\bulkProcess\out\resampled20m_%~nF.dim
 set procCmd="C:\Program Files\snap\bin\gpt.exe" resample_s2.xml -e -p resample_20m.properties -t "!targetFile!" "!sourceFile!"
 call !procCmd!
)

H:\_other\bulkProcess>(
echo.
 set sourceFile=H:\SENTINEL2\sharp_shadow\S2A_MSIL1C_20211024T145311_N0301_R082_T22VER_20211024T170119.SAFE
 echo Processing !sourceFile!
 set targetFile=H:\_other\bulkProcess\out\resampled20m_S2A_MSIL1C_20211024T145311_N0301_R082_T22VER_20211024T170119.dim

 set procCmd="C:\Program Files\snap\bin\gpt.exe" resample_s2.xml -e -p resample_20m.properties -t "!targetFile!" "!sourceFile!"
 call !procCmd!
)

Processing H:\SENTINEL2\sharp_shadow\S2A_MSIL1C_20211024T145311_N0301_R082_T22VER_20211024T170119.SAFE
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph

At the end you see that it is finding the product S2A_MSIL1C_20211024T145311_N0301_R082_T22VER_20211024T170119.SAFE
For Sentinel-1 data it works the same. I just changed in the loop header S2 to S1 and used, of course, a different input directory.

ABraun · November 23, 2021, 5:10pm

“gia esistente” indicates that the target products are already there, could this be a problem or should SNAP/GPT overwrite existing files?

marpet · November 24, 2021, 6:58am

This message is okay. It results from the ‘md’ command which tries to create the target directory. If it already exists this message is prompted but the script executes normally.
In my output you can see:

Ein Unterverzeichnis oder eine Datei mit dem Namen “H:_other\bulkProcess\out” existiert bereits.

ABraun · November 24, 2021, 7:15am

Oh I see, thank you for clarification.

chiarasalv23 · November 30, 2021, 10:00am

Hello,

I spent much time trying to do this processing, and I decided to try this way: http://step.esa.int/docs/tutorials/SNAP_CommandLine_Tutorial.pdf .

The only problem now is this error:

Error: Not able to write product file: 'C:\Program Files\snap\bin\"C:\gpt_processing_SNAP\out\S1A_IW_GRDH_1SDV_20150712T174032_20150712T174057_006781_0091C2_E940_finalprod.dim"'

I don’t know how to resolve this problem! Is maybe something wrong in my graph?

<graph id="Graph">
  <version>1.0</version>
  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>C:\gpt_processing_SNAP\data\S1A_IW_GRDH_1SDV_20150712T174032_20150712T174057_006781_0091C2_E940.zip</file>
    </parameters>
  </node>
  <node id="Calibration">
    <operator>Calibration</operator>
    <sources>
      <sourceProduct refid="Subset"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <auxFile>Product Auxiliary File</auxFile>
      <externalAuxFile/>
      <outputImageInComplex>false</outputImageInComplex>
      <outputImageScaleInDb>false</outputImageScaleInDb>
      <createGammaBand>false</createGammaBand>
      <createBetaBand>false</createBetaBand>
      <selectedPolarisations/>
      <outputSigmaBand>true</outputSigmaBand>
      <outputGammaBand>false</outputGammaBand>
      <outputBetaBand>false</outputBetaBand>
    </parameters>
  </node>
  <node id="Speckle-Filter">
    <operator>Speckle-Filter</operator>
    <sources>
      <sourceProduct refid="Calibration"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <filter>Lee Sigma</filter>
      <filterSizeX>3</filterSizeX>
      <filterSizeY>3</filterSizeY>
      <dampingFactor>2</dampingFactor>
      <estimateENL>true</estimateENL>
      <enl>1.0</enl>
      <numLooksStr>1</numLooksStr>
      <windowSize>7x7</windowSize>
      <targetWindowSizeStr>3x3</targetWindowSizeStr>
      <sigmaStr>0.9</sigmaStr>
      <anSize>50</anSize>
    </parameters>
  </node>
  <node id="Import-Vector">
    <operator>Import-Vector</operator>
    <sources>
      <sourceProduct refid="Speckle-Filter"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <vectorFile>C:\prove snap\baie de somme GRDH\features_param_idrvis.shp</vectorFile>
      <separateShapes>false</separateShapes>
    </parameters>
  </node>
  <node id="Subset">
    <operator>Subset</operator>
    <sources>
      <sourceProduct refid="Read"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <region>0,0,0,0</region>
      <referenceBand/>
      <geoRegion>POLYGON ((1.3836134672164917 50.06991195678711, 1.7917132377624512 50.06991195678711, 1.7917132377624512 50.395450592041016, 1.3836134672164917 50.395450592041016, 1.3836134672164917 50.06991195678711, 1.3836134672164917 50.06991195678711))</geoRegion>
      <subSamplingX>1</subSamplingX>
      <subSamplingY>1</subSamplingY>
      <fullSwath>false</fullSwath>
      <tiePointGridNames/>
      <copyMetadata>true</copyMetadata>
    </parameters>
  </node>
  <node id="Write">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="Import-Vector"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>"C:\gpt_processing_SNAP\out\S1A_IW_GRDH_1SDV_20150712T174032_20150712T174057_006781_0091C2_E940_finalprod.dim"</file>
      <formatName>BEAM-DIMAP</formatName>
    </parameters>
  </node>
  <applicationData id="Presentation">
    <Description/>
    <node id="Read">
            <displayPosition x="51.0" y="53.0"/>
    </node>
    <node id="Calibration">
      <displayPosition x="129.0" y="134.0"/>
    </node>
    <node id="Speckle-Filter">
      <displayPosition x="250.0" y="137.0"/>
    </node>
    <node id="Import-Vector">
      <displayPosition x="366.0" y="140.0"/>
    </node>
    <node id="Subset">
      <displayPosition x="140.0" y="57.0"/>
    </node>
    <node id="Write">
            <displayPosition x="505.0" y="143.0"/>
    </node>
  </applicationData>
</graph>

marpet · November 30, 2021, 11:08am

I think you only need to remove the quotes from the file path in the write node.

chiarasalv23 · November 30, 2021, 11:23am

It worked but it overwrited all the data in the folder.

marpet · November 30, 2021, 11:26am

It has overwritten the data in C:\gpt_processing_SNAP\out ?

chiarasalv23 · November 30, 2021, 11:29am

Yes. The loop worked but the output file was just a single file continuously overwritten.

marpet · November 30, 2021, 11:36am

Yes, the output file path needs to be updated for the different inputs. You cannot specify a constant path.
This is explained in the section “Using a Graph Created by the GraphBuilder” in the document you have linked.

chiarasalv23 · November 30, 2021, 11:49am

I read the tutorial but I didn’t understand. So there is no solution to this problem? How do you suggest I proceed?

marpet · December 2, 2021, 8:01am

Or this tutorial might help you, in particular, the Exercise 4.
SNAP Sentinel-1 Toolbox Course - Satellite Applications Catapult

You might try the batch processing in the GUI.
SNAP - Using batch processing - YouTube

chiarasalv23 · December 12, 2022, 11:20am

I forgot to keep this post updated on what concerns this solution.
I found really useful tutorials that may help someone with the same issues I had!

SNAP massive processing
GitHub

Thank you so much for helping! I hope this comment will be appreciated!