Batch process subset, idepix, c2rcc on OLCI

aewindle · June 12, 2020, 12:03pm

I am trying to use an .xml to batch process 1200 OLCI files. I’d like to subset, apply idepix, and then process using c2rcc. I have tried the below .xml file on 10 files, but they do not seem to be subsetted. Can you let me know what I am doing wrong?

> 
> <graph id="Graph">
>   <version>1.0</version>
> 
>   <!-- Subset the product by geographic polygon. -->
>   <node id="Subset">
>     <operator>Subset</operator>
>     <sources>
>       <source>${source}</source>
>     </sources>
>     <parameters class="com.bc.ceres.binding.dom.XppDomElement">
>       <sourceBands/>
>       <geoRegion>POLYGON ((-77.37755584716797 39.660213470458984, -75.18029022216797 39.660213470458984, -75.18029022216797 36.92744827270508, -77.37755584716797 36.92744827270508, -77.37755584716797 39.660213470458984, -77.37755584716797 39.660213470458984))</geoRegion>
>       <subSamplingX>1</subSamplingX>
>       <subSamplingY>1</subSamplingY>
>       <fullSwath>false</fullSwath>
>       <tiePointGridNames/>
>       <copyMetadata>true</copyMetadata>
>     </parameters>
>   </node>
> 
>   <!-- Calculate IdePix flags across subset scene; these are retained by NOT applied. -->
>   <node id="Idepix.Olci">
>     <operator>Idepix.Olci</operator>
>     <sources>
>       <sourceProduct refid="Subset"/>
>     </sources>
>     <parameters class="com.bc.ceres.binding.dom.XppDomElement">
>       <radianceBandsToCopy/>
>       <reflBandsToCopy/>
>       <outputSchillerNNValue>false</outputSchillerNNValue>
>       <computeCloudBuffer>true</computeCloudBuffer>
>       <cloudBufferWidth>2</cloudBufferWidth>
>       <useSrtmLandWaterMask>true</useSrtmLandWaterMask>
>     </parameters>
>   </node>
> 
>   <!-- Merge IdePix flags back into subset product. -->
>   <node id="BandMerge">
>     <operator>BandMerge</operator>
>     <sources>
>       <sourceProduct refid="Subset"/>
>       <sourceProduct.1 refid="Idepix.Olci"/>
>     </sources>
>     <parameters class="com.bc.ceres.binding.dom.XppDomElement">
>       <sourceBands/>
>       <geographicError>1.0E-5</geographicError>
>     </parameters>
>   </node>
> 
>   <!-- Calculate C2RCC across subset scene flags. Ocean and inland water pixels retained. IdePix flags not applied. -->
>   <node id="c2rcc.olci">
>     <operator>c2rcc.olci</operator>
>     <sources>
>       <sourceProduct refid="BandMerge"/>
>     </sources>
>     <parameters>
>       <outputAsRrs>true</outputAsRrs>
>       <validPixelExpression>!quality_flags.invalid AND (!quality_flags.land || quality_flags.fresh_inland_water)</validPixelExpression>
>     </parameters>
>   </node>
> 
> </graph>

Also attached is my .bash fileprocessDataset.bash (1.5 KB)

abruescas · June 13, 2020, 10:21am

Maybe the problems are the quotes in “Subset” in your source product for Idepix. I have been using this format successfully (example with Resample after Subset):

I have have an extra line in my Subset, my not sure if that is the issue:

aewindle · June 15, 2020, 1:16pm

Hi @abruescas, thanks for that fix. My .xml now looks like this:

<graph id="Graph">
  <version>1.0</version>

  <!-- Subset the product by geographic polygon. -->
  <node id="Subset">
    <operator>Subset</operator>
    <sources>
      <source>${source}</source>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <geoRegion>POLYGON((-77.37755584716797 39.660213470458984, -75.18029022216797 39.660213470458984, -75.18029022216797 36.92744827270508, -77.37755584716797 36.92744827270508, -77.37755584716797 39.660213470458984, -77.37755584716797 39.660213470458984))</geoRegion>
      <subSamplingX>1</subSamplingX>
      <subSamplingY>1</subSamplingY>
      <fullSwath>false</fullSwath>
      <tiePointGridNames/>
      <copyMetadata>true</copyMetadata>
    </parameters>
  </node>

  <!-- Calculate IdePix flags across subset scene; these are retained by NOT applied. -->
  <node id="Idepix.Olci">
    <operator>Idepix.Olci</operator>
    <sources>
 <!-- <sourceProduct refid="Subset"/> -->
      <sourceProduct>Subset</sourceProduct>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <radianceBandsToCopy/>
      <reflBandsToCopy/>
      <outputSchillerNNValue>false</outputSchillerNNValue>
      <computeCloudBuffer>true</computeCloudBuffer>
      <cloudBufferWidth>2</cloudBufferWidth>
      <useSrtmLandWaterMask>true</useSrtmLandWaterMask>
    </parameters>
  </node>

  <!-- Merge IdePix flags back into subset product. -->
  <node id="BandMerge">
    <operator>BandMerge</operator>
    <sources>
 <!-- <sourceProduct refid="Subset"/> -->
      <sourceProduct>Subset</sourceProduct>
 <!-- <sourceProduct.1 refid="Idepix.Olci"/> -->
      <sourceProduct.1>Idepix.Olci</sourceProduct.1>
    </sources>
    <parameters class="com.bc.ceres.binding.dom.XppDomElement">
      <sourceBands/>
      <geographicError>1.0E-5</geographicError>
    </parameters>
  </node>

  <!-- Calculate C2RCC across subset scene flags. Ocean and inland water pixels retained. IdePix flags not applied. -->
  <node id="c2rcc.olci">
    <operator>c2rcc.olci</operator>
    <sources>
 <!-- <sourceProduct refid="BandMerge"/> -->
      <sourceProduct>BandMerge</sourceProduct>
    </sources>
    <parameters>
      <outputAsRrs>true</outputAsRrs>
      <validPixelExpression>!quality_flags.invalid AND (!quality_flags.land || quality_flags.fresh_inland_water)</validPixelExpression>
    </parameters>
  </node>

</graph>

I still don’t think the subset is working. This is the polygon I’d like it to subset:

but when I pull the images into SNAP, they are not subsetted to that polygon:

Any ideas why?

abruescas · June 17, 2020, 11:02am

First thing that comes to my mind is that some of your scenes do not cover the whole area, but I guess you checked that already.

aewindle · June 17, 2020, 11:56am

But the scenes that do cover the area like the one above are not even getting subsetted. If they don’t cover the whole area, will subset not work?

abruescas · June 17, 2020, 2:43pm

It will subset until the dimensions of the scenes allows it. What I do not understand is why the ones with the whole area are not subsetted properly. Could you pass me the name of one of your files and I check?

aewindle · June 17, 2020, 2:47pm

Here is the file for the picture I showed above: S3A_OL_1_EFR____20160714T153555_20160714T153755_20171002T000405_0119_006_225______MR1_R_NT_002.nc

aewindle · June 23, 2020, 11:13am

hi @abruescas, were you able to run and test the .xml file on this image?

abruescas · June 27, 2020, 3:10pm

I am not able not find your scene in SciHub, the closest one is this one:
S3A_OL_1_EFR____20160714T153730_20160714T154030_20180223T031940_0179_006_225_2340_LR2_R_NT_002.SEN3
Subset works on that one.

From where did you download it?

aewindle · June 29, 2020, 10:38am

I used CODARep, the EUMETSAT Data Center, and CODA to download these images. This file was downloaded from CODARep since it is from 2016. Would this cause the issue of the .xml file not working correctly?

abruescas · June 29, 2020, 11:19am

I have downloaded it and you have only one part of the bay in that scene. Did I misunderstand something?

aewindle · June 29, 2020, 11:23am

So I’m wondering why the the subset polygon (picture below) isn’t working on this image? Is it because the entire image isn’t included and therefore, it won’t subset?

abruescas · June 29, 2020, 11:45am

I think it is doing the subset, but not on the south because the scene stops in lat 38º 41’ N more or less.

aewindle · June 29, 2020, 12:05pm

Okay, I see now. I’ve compared more of the original scenes to the subsetted images and it does seem to be subsetting. It didn’t look like it was based on the polygon, but I guess because projections are different, it’s going to look a little different. thanks for your help

aewindle · July 23, 2020, 2:10pm

I am trying to use the same .xml file as above but have added vicarious calibration gains to the beginning- see attached. When running in on some test files, it gives the error “Error: [NodeId: Idepix.Olci] Selected cloud screening algorithm cannot be used with given input product.” This did not occur before when I ran it without the SVC gains. Do I need to put the SVC somewhere else in the .xml file?

c2rcc_config_with_gains.xml (14.5 KB)

marpet · July 23, 2020, 2:57pm

Probably you got the template for the graph from the S3VT forum.
Please check it again. The product where the gains are applied needs to be merged with a subset of the original OLCI product before further passed to another processor.
Otherwise necessary bands are missing.

aewindle · July 23, 2020, 3:07pm

Hi @marpet. I successfully ran a subset and SVC .xml on 10 files and confirmed that the band values were different from the input data. I am trying now to add on to that .xml with idepix and c2rcc, see attached. subset_SVC_idepix_c2rcc.xml (14.4 KB)
Do I need to apply IdePix and c2rcc to subset, and then merge them with vicarious?

marpet · July 27, 2020, 6:44am

What is important is to add the to merge the source product with the output of the vicarious step.

...    
<node id="subset"> <!-- remove the calibrated bands-->
    <operator>Subset</operator>
    <sources>
        <source>${sourceProduct}</source>
    </sources>
    <parameters>
        <sourceBands>altitude,latitude,longitude,detector_index,FWHM_band_1,FWHM_band_2,FWHM_band_3,FWHM_band_4,FWHM_band_5,FWHM_band_6,FWHM_band_7,FWHM_band_8,FWHM_band_9,FWHM_band_10,FWHM_band_11,FWHM_band_12,FWHM_band_13,FWHM_band_14,FWHM_band_15,FWHM_band_16,FWHM_band_17,FWHM_band_18,FWHM_band_19,FWHM_band_20,FWHM_band_21,frame_offset,lambda0_band_1,lambda0_band_2,lambda0_band_3,lambda0_band_4,lambda0_band_5,lambda0_band_6,lambda0_band_7,lambda0_band_8,lambda0_band_9,lambda0_band_10,lambda0_band_11,lambda0_band_12,lambda0_band_13,lambda0_band_14,lambda0_band_15,lambda0_band_16,lambda0_band_17,lambda0_band_18,lambda0_band_19,lambda0_band_20,lambda0_band_21,solar_flux_band_1,solar_flux_band_2,solar_flux_band_3,solar_flux_band_4,solar_flux_band_5,solar_flux_band_6,solar_flux_band_7,solar_flux_band_8,solar_flux_band_9,solar_flux_band_10,solar_flux_band_11,solar_flux_band_12,solar_flux_band_13,solar_flux_band_14,solar_flux_band_15,solar_flux_band_16,solar_flux_band_17,solar_flux_band_18,solar_flux_band_19,solar_flux_band_20,solar_flux_band_21,quality_flags</sourceBands>
    </parameters>
</node>

<node id="merged"> <!-- remove the calibrated bands-->
    <operator>Merge</operator>
    <sources>
        <masterProduct>subset</masterProduct>
        <vicar>vicarious</vicar>
    </sources>
    <parameters>
        <includes>
            <include>
                <productId>vicar</productId>
                <namePattern>Oa.*radiance</namePattern>
            </include>
        </includes>
    </parameters>
</node>
...

I’ve updated you graph. It should work, but I have not tested it.
subset_SVC_idepix_c2rcc_update.xml (16.2 KB)

irgan · August 4, 2020, 8:06am

Hello,
I am trying a similar gpt batch process using read -> subset -> resample -> c2rcc for Sentinel 2 MSI products.
My graph runs successfully when I have subset -> resample. Resample -> c2rcc also works fine, delivering result for the entire image. If I merge all three steps I get the error:

Error: [NodeId: c2rccNode] Source must be a resampled S2 MSI L1C product

Of course I could run two batch processes, but it would be nice if I could get it with one.
Here’s my graph subset-resample-c2rcc_v2.xml (2.2 KB)
Any advise would be appreciated.
Best,
I

abruescas · August 4, 2020, 8:51am

Hi there,

In the c2rccNode, you have to write “Resample” in the source products. That is the name you gave the product in the resample node. Please, let us know if that is the issue.