GPT collocation of more than 2 products

Hi,
I am trying to collocate two Sentinel-2 products with a Sentinel-1 product using graph processing tool. Earlier I have used it for two products (One Sentinel-1 and Sentinel-2). I had a look at the forum answers but couldn’t find a way to do it. For two products, I did it in the following manner

Collocate
</sources>
<parameters class="com.bc.ceres.binding.dom.XppDomElement">
  <sourceProductPaths/>
  <masterProductName>${Collocate_master}_resampled</masterProductName>
  <targetProductName>_collocated</targetProductName>
  <targetProductType>COLLOCATED</targetProductType>
  <renameMasterComponents>true</renameMasterComponents>
  <renameSlaveComponents>true</renameSlaveComponents>
  <masterComponentPattern>${ORIGINAL_NAME}_M</masterComponentPattern>
  <slaveComponentPattern>${ORIGINAL_NAME}_S${SLAVE_NUMBER_ID}</slaveComponentPattern>
  <resamplingType>NEAREST_NEIGHBOUR</resamplingType>
</parameters>

.

But when I am using this for more than two Sentinel-2 products, I think that the band names for two Sentinel-2 products are same and it is not able to run successfully. For more than two products, I am doing it as follows

Collocate
</sources>
<parameters class="com.bc.ceres.binding.dom.XppDomElement">
  <sourceProductPaths/>
  <masterProductName>${Collocate_master}_resampled</masterProductName>
  <targetProductName>_collocated</targetProductName>
  <targetProductType>COLLOCATED</targetProductType>
  <renameMasterComponents>true</renameMasterComponents>
  <renameSlaveComponents>true</renameSlaveComponents>
  <masterComponentPattern>${ORIGINAL_NAME}_M</masterComponentPattern>
  <slaveComponentPattern>${ORIGINAL_NAME}_S${SLAVE_NUMBER_ID}</slaveComponentPattern>
  <resamplingType>NEAREST_NEIGHBOUR</resamplingType>
</parameters>

Any help in this regards would be highly appreciated. Thank you very much.

Kind regards,
Priti

And this is not working? Probably, otherwise you wouldn’t ask.
Can you tell what is not working? Describe the error you get.

Btw. both snippets you present look the same.

Have you seen this tutorial: ESA Synergetic use of S1 and S2 data.
it focuses on the GUI, but might be helpful. You can save the parameter setting from the GUI and use it in the graph XML.

Hi Marco,
Thank you very much for your quick reply as always. I am trying to bring mutltiemporal S1, their coherence and multitemporal S2 together. I am using the following graph

I was trying to extract a few bands at the end using the BandExtractor Operator, But when I was passing values as B1_M, B1_S0, it was giving me an error for the B1_S0. I assumed that this is due to naming of the master and slave components in the Collocate Operator. But it turns out that it was due to an extra space in parameters of BandExtractorOp. I should have passed ‘B1_M,B1_S0’ instead of ‘B1_M, B1_S0’.

Apologies that you had to look into it, though it was a silly mistake on my part. I really appreciate the efforts you and your team are putting to resolve all our issues.

Kind regards,
Priti

1 Like

Hi,

I am attempting to use GPT to collocate 2,000+ subsetted .nc files using one master product. I’m doing this so all my subsetted images will have the same geographical raster so I can use xr.open_mfdataset in python which requires the same dimension sizes. Let me know if there is a better way to do this…

Here is what I have so far in my .xml graph:

  <graph id="Graph">
  <version>1.0</version>
  <node id="Collocate">
    <operator>Collocate</operator>
    <sources>
      <sourceProduct>${sourceProduct}</sourceProduct>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceProductPaths/>
      <masterProductName>${Collocate_master}_resampled</masterProductName>
      <masterProductName/>
      <targetProductName>_collocated</targetProductName>
      <targetProductType>COLLOCATED</targetProductType>
      <renameMasterComponents>true</renameMasterComponents>
      <renameSlaveComponents>true</renameSlaveComponents>
      <masterComponentPattern>${ORIGINAL_NAME}_M</masterComponentPattern>
      <slaveComponentPattern>${ORIGINAL_NAME}_S${SLAVE_NUMBER_ID}</slaveComponentPattern>
      <resamplingType>NEAREST_NEIGHBOUR</resamplingType>
    </parameters>
  </node>
</graph>

For other gpt calls, I have used this call via command line: ./processDataset.bash xml_directory parameters.properties_directory target_scene_directory source_scene_directory

However, where should I be adding my ‘Master’ file? @preeti28192 @marpet could you provide some guidance? Thanks!

Anna

I think you don’t want to collocate them, because this will merge the products into one. I guess you want to keep them as separate files.
Instead, you can use the reprojection tool.
If you have already a product which reprojected and defines the grid as you want it you can specify it as source product “collocateWith”

If not, you can specify the CRS and the output parameters. for the output parameters it needs some maths to calculate the coordinates of the reference pixels and the width and height.
But with these settings all products are compatible afterwards.

1 Like

Hi @marpet, what would the call for the xml you provided be?

I usually do this:
./processDataset.bash xml_directory parameters.properties_directory target_scene_directory source_scene_directory

Where would the collocateWith product go?

I would hardcode the path to the collocateWith product in the bash file.
Then you can add it to the commandline line call with the paramter:

-ScollocateWith=collocate_Path

You could also add it as a parameter of the script, in the same way as the other parameters are defined.

I would suggest you first find out how to create the correct graph file and how to call it with gpt without the script. Afterwards you can probably update the script yourself.

I’m familiar using the bash file to run gpt. Can you advise where I should add -ScollocateWith=collocate_Path in the main processing? processDataset.bash (1.5 KB)

And as for the graph file, how would I find the information to fill out the parameters?

Hi there,

First time posting!
I was following the Bulk Processing with GPT Tutorial (apparently cant put the link), and I managed to make it work with one input with the following part of code for the XML:

    <sources>
      <sourceProduct>${sourceProduct}</sourceProduct>
    </sources>

And for the .bash:

for F in $(ls -1d "${sourceDirectory}"/S2*.zip); do
  sourceFile="$(realpath "$F")"
  targetFile="${targetDirectory}/${targetFilePrefix}_$(removeExtension "$(basename ${F})").dim"
  gpt ${graphXmlPath} -e -t ${targetFile} ${sourceFile}

For two inputs, I edited the XML as follows:

    <sources>
      <sourceProduct> ${sourceProduct}</sourceProduct>
      <sourceProduct.1> ${sourceProduct}</sourceProduct.1>
    </sources>

But I’m getting an error "duplicated source node id ". I think maybe I should change the name but its referring to a variable or parameter right?

Could someone please help me? I couldn’t find this information on this topic (despite being called collocation of more than 2 products) neither tutorials. Also, I’m using Ubuntu.

Thanks in advance,
Andrea.

Delete the second sourceProduct.1 line. The bash is defining the list of products that start with S2, so you do not need it,

Many thanks @abruescas for your reply.

What I did was, instead of using the sourceProduct, using the read and write nodes, and set two different nodes for each file in the XML with a parameter for each:

<node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <useAdvancedOptions>false</useAdvancedOptions>
      <file>$Input1</file>
      <copyMetadata>true</copyMetadata>
      <bandNames/>
      <pixelRegion>0,0,10980,10980</pixelRegion>
      <maskNames/>
    </parameters>
  </node>

<node id="Read(2)">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <useAdvancedOptions>false</useAdvancedOptions>
      <file>$Input2</file>
      <copyMetadata>true</copyMetadata>
      <bandNames/>
      <pixelRegion>0,0,10980,10980</pixelRegion>
      <maskNames/>
    </parameters>
  </node>

and edit my bash file as follows:

#!/bin/bash

# adapt this path
export PATH=/home/avinuales/snap/bin/:$PATH
gptPath="gpt"

# set path and other variables
input_folder=/home/avinuales/FIRE/SierraMorena-Kri/tst/in/
out_folder=/home/avinuales/FIRE/SierraMorena-Kri/tst3/out/
name_start=multimosaic_
name_end=.dim
dates=$(ls -1d "${input_folder}"S2*.zip | cut -d '_' -f 3 | sort | uniq)

# loop for all dates 
for date in $dates; do
  # Collect files for the current date
  echo $date
  files=($(ls -1d "${input_folder}"S2*_"$date"_*.zip))
    for ((i=0; i<${#files[@]}; i+=2)); do
      file1=${files[i]}
      echo $file1
      file2=${files[i+1]}
      echo $file2
      out_pathname=$out_folder$name_start$date$name_end
      echo $out_pathname
      gpt /home/avinuales/FIRE/SierraMorena-Kri/tst3/mult_siz.xml -PInput1=$file1 -PInput2=$file2 -POutput=$out_pathname
    done
done

When I run it from the command prompt, I don’t get any error, it takes some time (like if the PC were processing) but when it’s done, I check the output folder where the output files are supposed to be saved and it’s empty. I have checked the paths and they are correct. I also tested something similar but with two outputs and one input and it worked well. I don’t know where the problem is.

Please, let me know if this is a good approach, or instead, I try again with the sourceProduct way.

Regards,
Andrea.