Batch process subset, idepix, c2rcc on OLCI

I am trying to use an .xml to batch process 1200 OLCI files. I’d like to subset, apply idepix, and then process using c2rcc. I have tried the below .xml file on 10 files, but they do not seem to be subsetted. Can you let me know what I am doing wrong?

> 
> <graph id="Graph">
>   <version>1.0</version>
> 
>   <!-- Subset the product by geographic polygon. -->
>   <node id="Subset">
>     <operator>Subset</operator>
>     <sources>
>       <source>${source}</source>
>     </sources>
>     <parameters class="com.bc.ceres.binding.dom.XppDomElement">
>       <sourceBands/>
>       <geoRegion>POLYGON ((-77.37755584716797 39.660213470458984, -75.18029022216797 39.660213470458984, -75.18029022216797 36.92744827270508, -77.37755584716797 36.92744827270508, -77.37755584716797 39.660213470458984, -77.37755584716797 39.660213470458984))</geoRegion>
>       <subSamplingX>1</subSamplingX>
>       <subSamplingY>1</subSamplingY>
>       <fullSwath>false</fullSwath>
>       <tiePointGridNames/>
>       <copyMetadata>true</copyMetadata>
>     </parameters>
>   </node>
> 
>   <!-- Calculate IdePix flags across subset scene; these are retained by NOT applied. -->
>   <node id="Idepix.Olci">
>     <operator>Idepix.Olci</operator>
>     <sources>
>       <sourceProduct refid="Subset"/>
>     </sources>
>     <parameters class="com.bc.ceres.binding.dom.XppDomElement">
>       <radianceBandsToCopy/>
>       <reflBandsToCopy/>
>       <outputSchillerNNValue>false</outputSchillerNNValue>
>       <computeCloudBuffer>true</computeCloudBuffer>
>       <cloudBufferWidth>2</cloudBufferWidth>
>       <useSrtmLandWaterMask>true</useSrtmLandWaterMask>
>     </parameters>
>   </node>
> 
>   <!-- Merge IdePix flags back into subset product. -->
>   <node id="BandMerge">
>     <operator>BandMerge</operator>
>     <sources>
>       <sourceProduct refid="Subset"/>
>       <sourceProduct.1 refid="Idepix.Olci"/>
>     </sources>
>     <parameters class="com.bc.ceres.binding.dom.XppDomElement">
>       <sourceBands/>
>       <geographicError>1.0E-5</geographicError>
>     </parameters>
>   </node>
> 
>   <!-- Calculate C2RCC across subset scene flags. Ocean and inland water pixels retained. IdePix flags not applied. -->
>   <node id="c2rcc.olci">
>     <operator>c2rcc.olci</operator>
>     <sources>
>       <sourceProduct refid="BandMerge"/>
>     </sources>
>     <parameters>
>       <outputAsRrs>true</outputAsRrs>
>       <validPixelExpression>!quality_flags.invalid AND (!quality_flags.land || quality_flags.fresh_inland_water)</validPixelExpression>
>     </parameters>
>   </node>
> 
> </graph>

Also attached is my .bash fileprocessDataset.bash (1.5 KB)

Maybe the problems are the quotes in “Subset” in your source product for Idepix. I have been using this format successfully (example with Resample after Subset):

I have have an extra line in my Subset, my not sure if that is the issue:

Hi @abruescas, thanks for that fix. My .xml now looks like this:

<graph id="Graph">
  <version>1.0</version>

  <!-- Subset the product by geographic polygon. -->
  <node id="Subset">
    <operator>Subset</operator>
    <sources>
      <source>${source}</source>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <geoRegion>POLYGON((-77.37755584716797 39.660213470458984, -75.18029022216797 39.660213470458984, -75.18029022216797 36.92744827270508, -77.37755584716797 36.92744827270508, -77.37755584716797 39.660213470458984, -77.37755584716797 39.660213470458984))</geoRegion>
      <subSamplingX>1</subSamplingX>
      <subSamplingY>1</subSamplingY>
      <fullSwath>false</fullSwath>
      <tiePointGridNames/>
      <copyMetadata>true</copyMetadata>
    </parameters>
  </node>

  <!-- Calculate IdePix flags across subset scene; these are retained by NOT applied. -->
  <node id="Idepix.Olci">
    <operator>Idepix.Olci</operator>
    <sources>
 <!-- <sourceProduct refid="Subset"/> -->
      <sourceProduct>Subset</sourceProduct>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <radianceBandsToCopy/>
      <reflBandsToCopy/>
      <outputSchillerNNValue>false</outputSchillerNNValue>
      <computeCloudBuffer>true</computeCloudBuffer>
      <cloudBufferWidth>2</cloudBufferWidth>
      <useSrtmLandWaterMask>true</useSrtmLandWaterMask>
    </parameters>
  </node>

  <!-- Merge IdePix flags back into subset product. -->
  <node id="BandMerge">
    <operator>BandMerge</operator>
    <sources>
 <!-- <sourceProduct refid="Subset"/> -->
      <sourceProduct>Subset</sourceProduct>
 <!-- <sourceProduct.1 refid="Idepix.Olci"/> -->
      <sourceProduct.1>Idepix.Olci</sourceProduct.1>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <geographicError>1.0E-5</geographicError>
    </parameters>
  </node>

  <!-- Calculate C2RCC across subset scene flags. Ocean and inland water pixels retained. IdePix flags not applied. -->
  <node id="c2rcc.olci">
    <operator>c2rcc.olci</operator>
    <sources>
 <!-- <sourceProduct refid="BandMerge"/> -->
      <sourceProduct>BandMerge</sourceProduct>
    </sources>
    <parameters>
      <outputAsRrs>true</outputAsRrs>
      <validPixelExpression>!quality_flags.invalid AND (!quality_flags.land || quality_flags.fresh_inland_water)</validPixelExpression>
    </parameters>
  </node>

</graph>

I still don’t think the subset is working. This is the polygon I’d like it to subset:


but when I pull the images into SNAP, they are not subsetted to that polygon:
image

Any ideas why?

First thing that comes to my mind is that some of your scenes do not cover the whole area, but I guess you checked that already.

But the scenes that do cover the area like the one above are not even getting subsetted. If they don’t cover the whole area, will subset not work?

It will subset until the dimensions of the scenes allows it. What I do not understand is why the ones with the whole area are not subsetted properly. Could you pass me the name of one of your files and I check?

Here is the file for the picture I showed above: S3A_OL_1_EFR____20160714T153555_20160714T153755_20171002T000405_0119_006_225______MR1_R_NT_002.nc

hi @abruescas, were you able to run and test the .xml file on this image?

I am not able not find your scene in SciHub, the closest one is this one:
S3A_OL_1_EFR____20160714T153730_20160714T154030_20180223T031940_0179_006_225_2340_LR2_R_NT_002.SEN3
Subset works on that one.

From where did you download it?

I used CODARep, the EUMETSAT Data Center, and CODA to download these images. This file was downloaded from CODARep since it is from 2016. Would this cause the issue of the .xml file not working correctly?

I have downloaded it and you have only one part of the bay in that scene. Did I misunderstand something?

So I’m wondering why the the subset polygon (picture below) isn’t working on this image? Is it because the entire image isn’t included and therefore, it won’t subset?

I think it is doing the subset, but not on the south because the scene stops in lat 38º 41’ N more or less.

Okay, I see now. I’ve compared more of the original scenes to the subsetted images and it does seem to be subsetting. It didn’t look like it was based on the polygon, but I guess because projections are different, it’s going to look a little different. thanks for your help