Batch processing/graph file problem

[jbramich]

Hi, I am having trouble with batch processing using a graph file. Due to some gui issues my graph file was made initially with the GUI graph builder and then some parameters were added in using the ‘display xml’ feature .

I open a bunch of sentinel 2 products (I have also tried with just one) in the batch processing window, load the graph file and click run. I get a dialog box with “Null”.

Here is the graph file, I appreciate any advice:

<graph id="Graph">
  <version>1.0</version>
  <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>/Users/james/phd-local/exp2/sat-img/sent2/S2A_MSIL1C_20161209T001212_N0204_R073_T55GDQ_20161209T001325.SAFE</file>
      <formatName>SENTINEL-2-MSI-MultiRes-UTM55S</formatName>
    </parameters>
  </node>
  <node id="S2Resampling">
    <operator>S2Resampling</operator>
    <sources>
      <sourceProduct refid="Read"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <referenceBand>B2</referenceBand>
      <upsampling>Nearest</upsampling>
      <downsampling>First</downsampling>
      <flagDownsampling>First</flagDownsampling>
      <resampleOnPyramidLevels>true</resampleOnPyramidLevels>
    </parameters>
  </node>
  <node id="Subset">
    <operator>Subset</operator>
    <sources>
      <sourceProduct refid="S2Resampling"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <region>0,0,0,0</region>
      <geoRegion>POLYGON ((147.05 -41.43, 147.10 -41.43, 147.10 -41.48, 147.05 -41.48, 147.05 -41.43))</geoRegion>
      <subSamplingX>1</subSamplingX>
      <subSamplingY>1</subSamplingY>
      <fullSwath>false</fullSwath>
      <tiePointGridNames/>
      <copyMetadata>true</copyMetadata>
    </parameters>
  </node>
  <node id="RayleighCorrection">
    <operator>RayleighCorrection</operator>
    <sources>
      <sourceProduct refid="Subset"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBandNames/>
      <computeTaur>false</computeTaur>
      <computeRBrr>false</computeRBrr>
      <computeRtoaNg>false</computeRtoaNg>
      <computeRtoa>true</computeRtoa>
      <addAirMass>false</addAirMass>
      <s2MsiTargetResolution>10</s2MsiTargetResolution>
      <s2MsiSeaLevelPressure>1013.25</s2MsiSeaLevelPressure>
      <s2MsiOzone>300.0</s2MsiOzone>
    </parameters>
  </node>
  <node id="Write">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="RayleighCorrection"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>/Users/james/sat-img/sent2_rrc_10m/sent2_rrc_10m/target.dim</file>
      <formatName>BEAM-DIMAP</formatName>
    </parameters>
  </node>
  <applicationData id="Presentation">
    <Description/>
    <node id="Read">
            <displayPosition x="66.0" y="20.0"/>
    </node>
    <node id="S2Resampling">
      <displayPosition x="65.0" y="66.0"/>
    </node>
    <node id="Subset">
      <displayPosition x="67.0" y="112.0"/>
    </node>
    <node id="RayleighCorrection">
      <displayPosition x="67.0" y="162.0"/>
    </node>
    <node id="Write">
            <displayPosition x="68.0" y="220.0"/>
    </node>
  </applicationData>
</graph>

[marpet]

Probably it happens because the parameter in the RayleighCorrection is not set. This one does not take all bands by default.

[jbramich]

Thank you for the quick reply Marco.

I updated the file with:

<sourceBandNames>B1,B2,B3,B4,B5,B6,B7,B8,B8A,B9,B10,B11,B12</sourceBandNames>

I still get the same error. I was wondering if I need to have the and tags at the top when I am processing multiple files?

[marpet]

It could be that the GUI does not pick up the manually added parameters. I suggest to take this graph and run it from the command line. You can remove the Read and the Write nodes and provide the paths to the source and target files in the command line call.
See also this page: https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT

[jbramich]

Thanks Marco, that worked a treat. In case it helps others I would add the following notes:

• I couldn’t get the processDataset bash script to work on OSX (I didn’t try linux)

• The windows batch file worked fine with my graph. I put all the parameters into the xml so just entered the name of an empty text file as the parameter file name.

The graph file I used to Resample -> Spatial Subset -> Raleigh Correct is below for reference:

<graph id="Graph">
  <version>1.0</version>
  <!-- READ NODE NOT REQUIRED WHEN PROCESSING WITH GPT SCRIPT
    <node id="Read">
    <operator>Read</operator>
    <sources/>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>/Users/james/phd-local/exp2/sat-img/sent2/S2A_MSIL1C_20161209T001212_N0204_R073_T55GDQ_20161209T001325.SAFE</file>
      <formatName>SENTINEL-2-MSI-MultiRes-UTM55S</formatName>
    </parameters>
  </node>-->
  <node id="S2Resampling">
    <operator>S2Resampling</operator>
    <sources>
      <sourceProduct>${sourceProduct}</sourceProduct>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <!--GPT (on windows) DID NOT LIKE THE REFERENCE BAND TAG, BUT SPECIFYING THE RESOLUITON WAS FINE -->
      <!--<referenceBand>B2</referenceBand>-->
      <targetResolution>10</targetResolution>
      <upsampling>Nearest</upsampling>
      <downsampling>First</downsampling>
      <flagDownsampling>First</flagDownsampling>
      <resampleOnPyramidLevels>true</resampleOnPyramidLevels>
    </parameters>
  </node>
  <node id="Subset">
    <operator>Subset</operator>
    <sources>
      <sourceProduct refid="S2Resampling"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <region>0,0,0,0</region>
      <geoRegion>POLYGON ((147.05 -41.43, 147.10 -41.43, 147.10 -41.48, 147.05 -41.48, 147.05 -41.43))</geoRegion>
      <subSamplingX>1</subSamplingX>
      <subSamplingY>1</subSamplingY>
      <fullSwath>false</fullSwath>
      <tiePointGridNames/>
      <copyMetadata>true</copyMetadata>
    </parameters>
  </node>
  <node id="RayleighCorrection">
    <operator>RayleighCorrection</operator>
    <sources>
      <sourceProduct refid="Subset"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBandNames>B1,B2,B3,B4,B5,B6,B7,B8,B8A,B9,B10,B11,B12</sourceBandNames>
      <computeTaur>false</computeTaur>
      <computeRBrr>false</computeRBrr>
      <computeRtoaNg>false</computeRtoaNg>
      <computeRtoa>true</computeRtoa>
      <addAirMass>false</addAirMass>
      <s2MsiTargetResolution>10</s2MsiTargetResolution>
      <s2MsiSeaLevelPressure>1013.25</s2MsiSeaLevelPressure>
      <s2MsiOzone>300.0</s2MsiOzone>
    </parameters>
  </node>
 <!-- WRITE & APPLICATIONDATA NODES NOT NEEDED FOR USE WITH GPT BATCH PROCESSING SCRIPT
    <node id="Write">
    <operator>Write</operator>
    <sources>
      <sourceProduct refid="RayleighCorrection"/>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <file>/Users/james/sat-img/sent2_rrc_10m/sent2_rrc_10m/target.dim</file>
      <formatName>BEAM-DIMAP</formatName>
    </parameters>
  </node>
  <applicationData id="Presentation">
    <Description/>
    <node id="Read">
            <displayPosition x="66.0" y="20.0"/>
    </node>
    <node id="S2Resampling">
      <displayPosition x="65.0" y="66.0"/>
    </node>
    <node id="Subset">
      <displayPosition x="67.0" y="112.0"/>
    </node>
    <node id="RayleighCorrection">
      <displayPosition x="67.0" y="162.0"/>
    </node>
    <node id="Write">
            <displayPosition x="68.0" y="220.0"/>
    </node>
  </applicationData>-->
</graph>

[snasonova]

Hello!

I am attempting a similar exercise but using Sentinel-1 data. I am using the processDataset.batch file from here:
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT

I’ve also edited my graph xml file and removed the Read and Write nodes. As Jbramich suggested I input an empty .properties file because all the specification are in the xml file.

I use the following Windows command to call the scripts:
processDataset.bat S1_backscatter_final_noTC.xml S1_preprocessing.properties Z:\data\Sentinel1 Z:\data\Sentinel1\outputs

But I get the following error:
[NodeId: ThermalNoiseRemoval] Operator ‘Sentinel1RemoveThermalNoiseOp’: Mandatory source product (field ‘sourceProduct’) not set.

My end goal is to be able to run this script on about a 100 Sentinel-1 images. Any help would be much appreciated!

Thank you!

S1_backscatter_final_noTC.xml (2.4 KB)

[TonioF]

Did you adapt the processDataset.batch file to S2 data? In line 46, you will find it loops over data in (S2*.SAFE). Replace the 2 with a 1 (and .SAFE with .zip if you have zipped data in your directory).

[snasonova]

Yep, I tried that. I tried S1*.zip and unzipped the data and tried S1*.SAFE. The processing starts, it even says “processing such and such file”. But it terminates because it can’t find the source product. Is there something wrong with the way I set up the xml file?

[kkop]

Could this help?

Mandatory source product (field 'sourceProduct') not set

got it working by myself. Somehow, there was a .1 inserted in the sourceProduct tag when creating the xml-file. I removed it in the xml, saved and imported the graph again. Now it works. Maybe a bug? The error message occured when I deleted a connection in the old graph (remove source) to insert the additional module TerrainFlattening. Could be that there went something wrong internally. These are the xml files: working <node id="Terrain-Correction"> <operator>Terrain-Correction</operator> <s…

Mandatory source product (field 'sourceProduct') not set

I investigated this some more: The problem re-occurs every time you insert a new module into an existing graph and place it between two connected modules, and remove the old link respectively. This somehow adds the .1 in the source product tag in the XML file which makes the graph incomplete.