Problem with band merge in Gpt. Defining the second input?

GeorgeB · July 16, 2021, 10:56am

Hello everybody! I hope that you are safe in this strange times.

I want to process some LAI from sentinel 2 and for that iI need cloud free S2 images. I use IDEPIX cloud mask for masking out clouds, because it gives better results. Because IDEPIX it can’t be calculated from L2A products and for LAI computing I need L2A products. I decided 2 preprocess data individually, each image is atmospheric corrected and cloud mask computed individually. I want to merge the bands from L2A product with the mask from L1C product to continue my computation.
The problem is I don’t know how to introduce a second product for batch processing with GPT.

Anyone thinks that can help me to define the second product? I must define it as a variable? (with one product I developed many work fluxes)

marpet · July 16, 2021, 11:23am

Hi,

just a quick reply. Here are two examples.

merge.xml (768 Bytes)
merge_two_nameless_products.xml (971 Bytes)

Both files contain at the top an example gpt call.
Probably they help you already

GeorgeB · July 20, 2021, 7:41am

Hi @marpet !

The problem is that I must configure this .bat file that is described in Bulk Processing with GPT - SNAP - SNAP Wiki (atlassian.net). I tried to modify it to introduce the second source product (slave one) but i didn’t find a way to do it.

::@echo off
:: enable delayed expansion - used to resolve variable in loop
:: variable has to be used with '!' instead of '%'
setlocal ENABLEDELAYEDEXPANSION

::::::::::::::::::::::::::::::::::::::::::::
:: User Configuration
::::::::::::::::::::::::::::::::::::::::::::

:: adapt this path to your needs
set gptPath="C:\Program Files\snap\bin\gpt.exe"

::::::::::::::::::::::::::::::::::::::::::::
:: Command line handling
::::::::::::::::::::::::::::::::::::::::::::

:: first parameter is a path to the graph xml
set graphXmlPath=%1


:: use third parameter for path to source products
set sourceDirectory=%2
:: if sourceDirectory ends with '\' remove it
if %sourceDirectory:~-1%==\ set sourceDirectory=%sourceDirectory:~0,-1%

:: use third parameter for path to target products
set targetDirectory=%3
:: if targetDirectory ends with '\' remove it
if %targetDirectory:~-1%==\ set targetDirectory=%targetDirectory:~0,-1%

set targetFilePrefix=L1C_IDEP_

:: Create the target directory
md %targetDirectory%
::::::::::::::::::::::::::::::::::::::::::::
:: Main processing
::::::::::::::::::::::::::::::::::::::::::::

:: double '%' in batch file and only a single '%' on command line
:: '/D' is for directories like Sentinel data. Remove '/D' when you open files.
for /D /R %sourceDirectory% %%F in (S2*.SAFE) do (
  echo.
  :: '~fF' means abolute path of 'F'
  set sourceFile=%%~fF
  set sourceFile2=%%~fF
  echo Processing !sourceFile!
  :: '~nF' means filename without extension of 'F'
  set targetFile=%targetDirectory%\%targetFilePrefix%%%~nF.dim
  set procCmd=%gptPath% %graphXmlPath% -e -t "!targetFile!" "!sourceFile!" "!sourceFile2!"
  call !procCmd! 
)

But you can see i don’t get it how to put another folder to the second input, i basically iterate on the same files. Do you have some ideas for this? Or maybe some resources to find by myself?

GeorgeB · July 20, 2021, 1:49pm

I resolved one part of the problem with the idea presented in this post OLCI C2RCC bulk processing - s3tbx - STEP Forum (esa.int). Now the problem it is that with a *.dim file the gpt won’t start, but the same process will start with a *.SAFE file…
Below I’ll describe and write the whole process.

THIS IS THE WINDOWS BATCH

::@echo off
:: enable delayed expansion - used to resolve variable in loop
:: variable has to be used with '!' instead of '%'
setlocal ENABLEDELAYEDEXPANSION

::::::::::::::::::::::::::::::::::::::::::::
:: User Configuration
::::::::::::::::::::::::::::::::::::::::::::

:: adapt this path to your needs
set gptPath="C:\Program Files\snap\bin\gpt.exe"

::::::::::::::::::::::::::::::::::::::::::::
:: Command line handling
::::::::::::::::::::::::::::::::::::::::::::

:: first parameter is a path to the graph xml
set graphXmlPath=%1


:: use third parameter for path to source products
set sourceDirectory=%2
:: if sourceDirectory ends with '\' remove it
if %sourceDirectory:~-1%==\ set sourceDirectory=%sourceDirectory:~0,-1%

:: use third parameter for path to source products
set sourceDirectory2=%3
:: if sourceDirectory ends with '\' remove it
if %sourceDirectory2:~-1%==\ set sourceDirectory=%sourceDirectory2:~0,-1%


:: use third parameter for path to target products
set targetDirectory=%4
:: if targetDirectory ends with '\' remove it
if %targetDirectory:~-1%==\ set targetDirectory=%targetDirectory:~0,-1%


set targetFilePrefix=L1C_IDEP_


:: Create the target directory
::md %targetDirectory%
::::::::::::::::::::::::::::::::::::::::::::
:: Main processing
::::::::::::::::::::::::::::::::::::::::::::

:: double '%' in batch file and only a single '%' on command line
:: '/D' is for directories like Sentinel data. Remove '/D' when you open files.
for /D /R %sourceDirectory% %%F in (S2*.SAFE) do (
for /D /R %sourceDirectory2% %%N in (*.dim) do (
  echo
  :: '~fF' means abolute path of 'F'
  set sourceFile=%%~fF
  set sourceFile2=%%~fN
  echo sourceFile2=%sourceDirectory2%\%%~fN

  echo Processing !sourceFile!
  :: '~nF' means filename without extension of 'F'
  set targetFile=%targetDirectory%\%targetFilePrefix%%%~nF.dim
  set procCmd=%gptPath% %graphXmlPath% -e -t "!targetFile!" "!sourceFile!" "!sourceFile2!"
  call !procCmd! 
))

HERE IS THE .XML GRAPH FOR MERGING BANDS FROM DIFFERENT PRODUCTS AND ALSO RESAMPLE THE NEW PRODUCT IN THE END

<graph id="Graph">
  <version>1.0</version>
  <node id="mergeNode">
        <operator>Merge</operator>
        <sources>
            <masterProduct>${sourceProduct1}</masterProduct>
            <level2>${sourceProduct2}</level2>
        </sources>
        <parameters>
            <includes>
                <include>
                    <productId>masterProduct</productId>
                    <name>B1</name>
					<newName>B1</newName>
				</include>
				<include>
					<productId>masterProduct</productId>
					<name>B2</name>
					<newName>B2</newName>
                </include>
                <include>
                    <productId>level2</productId>
                    <name>B7</name>
					<newName>B7</newName>
                </include>
            </includes>
        </parameters>
  </node>
  <node id="Resample">
    <operator>Resample</operator>
    <sources>
      <sourceProduct refid="mergeNode"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <referenceBand/>
      <targetWidth/>
      <targetHeight/>
      <targetResolution>10</targetResolution>
      <upsampling>Nearest</upsampling>
      <downsampling>First</downsampling>
      <flagDownsampling>First</flagDownsampling>
      <resamplingPreset/>
      <bandResamplings/>
      <resampleOnPyramidLevels>true</resampleOnPyramidLevels>
    </parameters>
  </node>
</graph>

THE FINAL BATCH COMMAND TO CREATE A MERGED AND RESAMPLED PRODUCT

PATH_TO_BATCH_FILE\gpt_S2_merge.bat PATH_TO_MERGE_GRAPH\graph_merge_test_inverse.xml PATH_TO_FIRST_PRODUCT\S2L2A_brute 
PATH_TO_SECOND_PRODUCT\S2_Idepix 
PATH_TO_RESULTED_PRODUCT\S2_final\test

This whole workflow works with to .SAFE files in the configuration batch but not with a .SAFE and .DIM file. Anyone knows what the problem might be?

marpet · July 26, 2021, 10:43am

Is there an error message when you use the dimap file?
At least I see no obvious error in your batch file.
Another way of processing several files is using excel to create the batch file.
Copy the path the path to the files file in separate columns. For each pair of files you have now a row.
In the following columns you can specify the values of the parameters. In the last column you concatenate all columns to the command.
Because you can copy the values easily in excel, it is quite quick to create the commands.
The values of the command column can be copied in a batch file. A list of files names ca be retrieved in Windows ono the command line by calling:

dir /B > fileNames.txt
This will write all file names into the specified text file.

GeorgeB · July 26, 2021, 11:14am

Hi @marpet! It’s no error when I try the DIMAP format, just nothing happens. I really not understand why, cause with SAFE it’s just working as expected. I’ll try with the excel trick even if I found it quite complicated now. Thank you, I’ll post an answear as soon as I’ll have one.

Many thanks!