IdePix.olci hangs

Is anyone here successfully running Idepix.olci with S3 scenes?

This may be related to the issue here, where IdePix.olci runs overnight, consumes all the RAM, but doesn’t produce anything: Sentinel 3 IDEPIX mask in snappy

I’m not using python just the SNAP GUI. When I load an S3 scene and run IdePix, it returns results fairly quickly. If I then try to display a cloud mask, nothing happens - the progress bar runs for a few hours before I kill it.

I’m running the latest SNAP as of today (6 Feb 2020).

System:

Linux geus3064linuxwsm 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

32 GB RAM

Plugin version:
| org.esa.snap.idepix.olci | 7.0.1 | Enabled |

Steps:

  1. Open S3 scene
  2. Run idepix with default options
  3. Try to display a MASK.

You tried another scenes also or just this one? I’ve observed that indeed on some scenes idepix it’s just crazy but i don’t think it’s something that applies in general. Can you provide the scene whole id to download it?

Hi George,

It does work in snap GUI on most scenes. But the same scenes where it works in the GUI it does not work on GPT - it just hangs. Have you gotten Idepix to work with gpt? I can provide a scene and a GPT xml if that helps…

Thanks,

-k.

To be honest I didn’t try to use Idepix with gpt. I will try now on my laptop and tomorrow at the office. I will update you as soon as possibile. I’ve got some error with displaying IDEPIX masks too but just for some scenes.

Have a nice day,
George

I can track this down to reproject failing after idexpix. This works:

Read > Idepix.Olci -> BandMaths -> Write

This does not work:

Read > Idepix.Olci -> Reproject -> BandMaths -> Write

For the latter, I’m running this on scene S3A_OL_1_EFR____20180816T104417_20180816T104717_20180817T144146_0179_034_322_1800_LN1_O_NT_002.SEN3 and with the following XML file for GPT:

 <graph id="S3">
  <version>1.0</version>
  
  <node id="Read_OLCI">
    <operator>Read</operator>
    <parameters>
      <file>${OLCIsource}/xfdumanifest.xml</file>
      <formatName>Sen3</formatName>
      <!-- https://forum.step.esa.int/t/error-when-selecting-multiple-bands-from-slstr-file/15319 -->
      <!-- Sen3 ensures that SLSTR is read as multisize -->
    </parameters>
  </node>
  
  <node id="Idepix_OLCI">
    <operator>Idepix.Olci</operator>
    <sources>
      <sourceProduct>Read_OLCI</sourceProduct>
    </sources>
  </node>

  <node id="Reproject_Idepix">
    <operator>Reproject</operator>
    <sources><source>Idepix_OLCI</source></sources>
    <parameters>
      <crs>EPSG:3413</crs>
      <resampling>Nearest</resampling>
      <noDataValue>NaN</noDataValue>
      <includeTiePointGrids>true</includeTiePointGrids>
    </parameters>
  </node>
 
  <node id="BandMaths_Idepix_cloud">
    <operator>BandMaths</operator>
    <sources><sourceProducts>Reproject_Idepix</sourceProducts></sources>
    <parameters>
      <targetBands><targetBand>
        <name>IDEPIX_CLOUD</name><type>Int16</type>
        <expression>IDEPIX_CLOUD</expression>
      </targetBand></targetBands>
    </parameters>
  </node>
  <node id="Write_Idepix_cloud">
    <operator>Write</operator>
    <sources><sourceProduct>BandMaths_Idepix_cloud</sourceProduct></sources>
    <parameters>
      <file>${targetFolder}/idepix_cloud_x.tif</file><formatName>GeoTIFF</formatName>
    </parameters>
  </node>
</graph>
1 Like

Hi Ken,

Thanks for the additional info. We are looking into this issue. Actually in v7.0.0 of Idepix-Olci was an error which caused the problem you describe. With the update to Idepix-OLCI 7.0.1 it should behave better.
We’re checking this.

Hi @marpet,

I tried the .xml proposed by Ken also on two PC’s. The weakeast one would just stop when all the resources were 100%. The one with a proper configuration just stopped after 5 seconds with some long error. The Idepix.OLCI is 7.0.1 version. I attached the files with the error and the .xml graph used.

I hope it helps someway.
Cheers!

idepix.xml (2.8 KB)
error.txt (19.4 KB)

1 Like

Hi

@GeorgeB
You must have a strange/corrupted SNAP installation.
Your error log contains pointers into the source code which doesn’t exist anymore.
Maybe you reinstalling SNAP and Idepix can help.

We let Idepix tun and it works well on Windows. On Linux, there is an issue. When cloud shadow detection is enabled, it runs very slow. Without cloud-shadow, the processing is done in a few minutes.

@mankoff
Ken, maybe it is an option for you to disable the cloud-shadow detection.
Change the idepix node in your graph to:

<node id="Idepix_OLCI">
<operator>Idepix.Olci</operator>
<sources>
  <sourceProduct>Read_OLCI</sourceProduct>
</sources>
<parameters>
  <computeCloudShadow>false</computeCloudShadow>   
</parameters>
</node>

Thanks Marco. I will properly configure the SNAP on the other PC. It’s not mine and in the first time I won’t get how it’s corrupted cause on snappy and GUI works just fine. Thank you very much for all the answears.

Cheers!

@marpet Yes! It appears to work if I turn off the cloud shadow option. Thank you for the fix.

Strange that the Idepix module ran with that option as default (true) or set to false, but only hung when reproject was added to it…

Hi @marpet,

GPT seems to run IdePix if cloud shadow is disabled as you suggest, but I cannot reproject Idepix outputs. GPT hangs. The message is:

"....10%....20%..INFO: org.hsqldb.persist.Logger: Database closed"

And the CPU keeps churning but nothing else happens. If the reproject node is removed, gpt runs fairly quickly. We’re using the following XML. Do you have any suggestions how to get Idepix to work?

Thanks,

-k.

<graph id="S3">
  <version>1.0</version>

  <node id="Read_OLCI">
    <operator>Read</operator>
    <parameters>
      <file>${OLCIsource}/xfdumanifest.xml</file>
      <formatName>Sen3</formatName>
      <!-- https://forum.step.esa.int/t/error-when-selecting-multiple-bands-from-slstr-file/15319 -->
      <!-- Sen3 ensures that SLSTR is read as multisize -->
    </parameters>
  </node>

  <node id="Idepix_OLCI">
    <operator>Idepix.Olci</operator>
    <sources>
      <sourceProduct>Read_OLCI</sourceProduct>
    </sources>
    <parameters>
      <!-- <outputSchillerNNValue>boolean</outputSchillerNNValue> -->
      <computeCloudShadow>false</computeCloudShadow>
      <!-- <alternativeNNDirPath>string</alternativeNNDirPath> -->
      <!-- <outputCtp>boolean</outputCtp> -->
      <computeCloudBuffer>false</computeCloudBuffer>
      <!-- <cloudBufferWidth>2</cloudBufferWidth> -->
      <!-- <useSrtmLandWaterMask>boolean</useSrtmLandWaterMask> -->
    </parameters>
  </node>
  
  <node id="Reproject_Idepix">
    <operator>Reproject</operator>
    <sources><source>Idepix_OLCI</source></sources>
    <parameters>
      <crs>EPSG:3413</crs>
      <resampling>Nearest</resampling>
      <noDataValue>NaN</noDataValue>
      <includeTiePointGrids>true</includeTiePointGrids>
    </parameters>
  </node>
  
  <node id="BandMaths_Idepix_cloud">
    <operator>BandMaths</operator>
    <sources><sourceProducts>Reproject_Idepix</sourceProducts></sources>
    <parameters>
      <targetBands><targetBand>
        <name>IDEPIX_CLOUD</name><type>Int16</type>
        <expression>IDEPIX_CLOUD</expression>
      </targetBand></targetBands>
    </parameters>
  </node>
  <node id="Write_Idepix_cloud">
    <operator>Write</operator>
    <sources><sourceProduct>BandMaths_Idepix_cloud</sourceProduct></sources>
    <parameters>
      <file>${targetFolder}/idepix_cloud_x.tif</file><formatName>GeoTIFF</formatName>
    </parameters>
  </node>
</graph>

Hi,

I have a very similar issue. However mine seems to come from the cloud buffer computation. When I turn cloud buffer computation off everything works as expected. So that’s also my workaround for the moment.

May you let me know whether that works for you too?