Trying to run OLCI IdePix via command line

Hi all,

I’m new to SNAP and trying to do some cloud masking for Sentinel 3 OLCI. I have got this working with the GUI (using IdePix) but am having some difficulties via command line (want to process thousands of files) so thought I’d post as I am sure I am missing something simple. I found a guide in the GitHub page I’m using for a basis.

First, I made this file and saved as graph-file.xml in my working directory:

<graph id="IdepixOlciTest">
  <version>1.0</version>
  <node id="idepix">
    <operator>Idepix.Olci</operator>
      <sources>
        <source>${sourceProduct}</source>
      </sources>
  </node>
</graph>

Then I invoke from the command line like this:

asayer@gs616-payara:~/Desktop/tmp_test_idepix$ /accounts/asayer/snap/bin/gpt graph-file.xml -SsourceProduct=/accounts/asayer/Desktop/tmp_test_idepix/S3A_OL_1_ERR____20170601T000119_20170601T000547_20180423T154402_0268_018_187______LR2_R_NT_002.SEN3/xfdumanifest.xml

The first time I ran it I got an error that LD_LIBRARY_PATH had to include the current working directory, so I added that. The next time I ran I got this:

INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
Executing processing graph
 done.

Error: [NodeId: idepix] Operator 'IdepixOlciOp': Mandatory source product (field 'sourceProduct') not set.

I had thought it weird that the command line example said -SsourceProduct not -sourceProduct so thought I’d try changing to this:

asayer@gs616-payara:~/Desktop/tmp_test_idepix$ /accounts/asayer/snap/bin/gpt graph-file.xml -sourceProduct=/accounts/asayer/Desktop/tmp_test_idepix/S3A_OL_1_ERR____20170601T000119_20170601T000547_20180423T154402_0268_018_187______LR2_R_NT_002.SEN3/xfdumanifest.xml

However now I get this error:

Error: Unknown option '-sourceProduct=/accounts/asayer/Desktop/tmp_test_idepix/S3A_OL_1_ERR____20170601T000119_20170601T000547_20180423T154402_0268_018_187______LR2_R_NT_002.SEN3/xfdumanifest.xml'

Then I thought, ok, maybe the line in the xml file needs to say SsourceProduct not sourceProduct. So I changed to that in the xml file, but I gives this error:

Error: Missing source '${SsourceProduct}' in node 'idepix'

Does someone know what I am doing wrong here? Or is there a better way to run the cloud maksing routine from command line? Any advice appreciated!

Your first try was the closest.
Change the XML tag from

<source>${sourceProduct}</source>
to
<sourceProduct>${sourceProduct}</sourceProduct>

Then it should work.

Hi @amsayer,

If you get things working with IdePix I would be glad to know of it - I have had trouble with this plugin.

I can get it to run with a simple XML like you provide, but if I try to do much else bugs arise. Specifically,

Per-pixel geocoding does not seem to work SNAP generates 3-band GeoTIFFs when using per pixel geocoding? . Note that without it, when you write out your GeoTIFFs, you should see wobble up to 10 pixels. If this is an issue for you, and you find a workaround, please post back here.

Also, we are moving more and more of our workflow out of SNAP - for example I cannot reproject anything if I run the IdePix plugin. IdePix.olci hangs so now we do our reprojection in other tools. Again, if you find a fix for this, please post.

Alternatively, if you try to do other things and run into bugs, just be aware that it may be tied in to IdePix. Please post so we can get this very nice and useful plugin some attention.

Thanks,

-k.

Thanks, Marcos! I can confirm the correction you suggested worked. I appreciate the advice. I have two follow-up questions:

  1. The output consists of a directory target.data containing a whole bunch of stuff. None of these appear to be hdf or NetCDF files. I infer that the pixel classification might be in the file “target.data/pixel_classif_flags.img”. However, I am not sure what the format of this file is to make sure I read it correctly… is there some documentation? (I checked the .hdf file to get dimensions, but am not sure if I should read it as a byte or int or similar, or how to decode values to meanings.)

  2. What is the best reference to understand the working of this routine, and cite in e.g. a scientific paper or tech report? (I have found mention of a 2012 conference proceedings, but contacted ESA and they couldn’t find a copy, and emailed Brockmann consulting direct but didn’t get a response).

@mankoff: my goal is a simple batch processing to use this as a cloud mask for algorithms I am developing/running outside of SNAP. But yes, I will post as other issues arise.

Ok, I think this post may answer my first question: Idepix pixel_classif_flags interpretation

For others: it is an integer field. When I read it in, I think it must be big-endian (otherwise I get negative values). And the interpretation key is in the file “target.dim”.