IdePix.olci hangs

mankoff · February 6, 2020, 10:08pm

Is anyone here successfully running Idepix.olci with S3 scenes?

This may be related to the issue here, where IdePix.olci runs overnight, consumes all the RAM, but doesn’t produce anything: Sentinel 3 IDEPIX mask in snappy

I’m not using python just the SNAP GUI. When I load an S3 scene and run IdePix, it returns results fairly quickly. If I then try to display a cloud mask, nothing happens - the progress bar runs for a few hours before I kill it.

I’m running the latest SNAP as of today (6 Feb 2020).

System:

Linux geus3064linuxwsm 4.15.0-76-generic #86-Ubuntu SMP Fri Jan 17 17:24:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

32 GB RAM

Plugin version:
| org.esa.snap.idepix.olci | 7.0.1 | Enabled |

Steps:

Open S3 scene
Run idepix with default options
Try to display a MASK.

GeorgeB · February 7, 2020, 5:24am

You tried another scenes also or just this one? I’ve observed that indeed on some scenes idepix it’s just crazy but i don’t think it’s something that applies in general. Can you provide the scene whole id to download it?

mankoff · February 9, 2020, 1:54am

Hi George,

It does work in snap GUI on most scenes. But the same scenes where it works in the GUI it does not work on GPT - it just hangs. Have you gotten Idepix to work with gpt? I can provide a scene and a GPT xml if that helps…

Thanks,

-k.

GeorgeB · February 9, 2020, 9:12am

To be honest I didn’t try to use Idepix with gpt. I will try now on my laptop and tomorrow at the office. I will update you as soon as possibile. I’ve got some error with displaying IDEPIX masks too but just for some scenes.

Have a nice day,
George

mankoff · February 9, 2020, 9:58pm

I can track this down to reproject failing after idexpix. This works:

Read > Idepix.Olci -> BandMaths -> Write

This does not work:

Read > Idepix.Olci -> Reproject -> BandMaths -> Write

For the latter, I’m running this on scene S3A_OL_1_EFR____20180816T104417_20180816T104717_20180817T144146_0179_034_322_1800_LN1_O_NT_002.SEN3 and with the following XML file for GPT:

 <graph id="S3">
  <version>1.0</version>
  
  <node id="Read_OLCI">
    <operator>Read</operator>
    <parameters>
      <file>${OLCIsource}/xfdumanifest.xml</file>
      <formatName>Sen3</formatName>
      <!-- https://forum.step.esa.int/t/error-when-selecting-multiple-bands-from-slstr-file/15319 -->
      <!-- Sen3 ensures that SLSTR is read as multisize -->
    </parameters>
  </node>
  
  <node id="Idepix_OLCI">
    <operator>Idepix.Olci</operator>
    <sources>
      <sourceProduct>Read_OLCI</sourceProduct>
    </sources>
  </node>

  <node id="Reproject_Idepix">
    <operator>Reproject</operator>
    <sources><source>Idepix_OLCI</source></sources>
    <parameters>
      <crs>EPSG:3413</crs>
      <resampling>Nearest</resampling>
      <noDataValue>NaN</noDataValue>
      <includeTiePointGrids>true</includeTiePointGrids>
    </parameters>
  </node>
 
  <node id="BandMaths_Idepix_cloud">
    <operator>BandMaths</operator>
    <sources><sourceProducts>Reproject_Idepix</sourceProducts></sources>
    <parameters>
      <targetBands><targetBand>
        <name>IDEPIX_CLOUD</name><type>Int16</type>
        <expression>IDEPIX_CLOUD</expression>
      </targetBand></targetBands>
    </parameters>
  </node>
  <node id="Write_Idepix_cloud">
    <operator>Write</operator>
    <sources><sourceProduct>BandMaths_Idepix_cloud</sourceProduct></sources>
    <parameters>
      <file>${targetFolder}/idepix_cloud_x.tif</file><formatName>GeoTIFF</formatName>
    </parameters>
  </node>
</graph>

marpet · February 10, 2020, 8:00am

Hi Ken,

Thanks for the additional info. We are looking into this issue. Actually in v7.0.0 of Idepix-Olci was an error which caused the problem you describe. With the update to Idepix-OLCI 7.0.1 it should behave better.
We’re checking this.

GeorgeB · February 10, 2020, 10:49am

Hi @marpet,

I tried the .xml proposed by Ken also on two PC’s. The weakeast one would just stop when all the resources were 100%. The one with a proper configuration just stopped after 5 seconds with some long error. The Idepix.OLCI is 7.0.1 version. I attached the files with the error and the .xml graph used.

I hope it helps someway.
Cheers!

idepix.xml (2.8 KB)
error.txt (19.4 KB)

marpet · February 10, 2020, 12:18pm

Hi

@GeorgeB
You must have a strange/corrupted SNAP installation.
Your error log contains pointers into the source code which doesn’t exist anymore.
Maybe you reinstalling SNAP and Idepix can help.

We let Idepix tun and it works well on Windows. On Linux, there is an issue. When cloud shadow detection is enabled, it runs very slow. Without cloud-shadow, the processing is done in a few minutes.

@mankoff
Ken, maybe it is an option for you to disable the cloud-shadow detection.
Change the idepix node in your graph to:

<node id="Idepix_OLCI">
<operator>Idepix.Olci</operator>
<sources>
  <sourceProduct>Read_OLCI</sourceProduct>
</sources>
<parameters>
  <computeCloudShadow>false</computeCloudShadow>   
</parameters>
</node>

GeorgeB · February 10, 2020, 1:26pm

Thanks Marco. I will properly configure the SNAP on the other PC. It’s not mine and in the first time I won’t get how it’s corrupted cause on snappy and GUI works just fine. Thank you very much for all the answears.

Cheers!

mankoff · February 10, 2020, 2:39pm

@marpet Yes! It appears to work if I turn off the cloud shadow option. Thank you for the fix.

Strange that the Idepix module ran with that option as default (true) or set to false, but only hung when reproject was added to it…

mankoff · February 13, 2020, 2:57pm

Hi @marpet,

GPT seems to run IdePix if cloud shadow is disabled as you suggest, but I cannot reproject Idepix outputs. GPT hangs. The message is:

"....10%....20%..INFO: org.hsqldb.persist.Logger: Database closed"

And the CPU keeps churning but nothing else happens. If the reproject node is removed, gpt runs fairly quickly. We’re using the following XML. Do you have any suggestions how to get Idepix to work?

Thanks,

-k.

<graph id="S3">
  <version>1.0</version>

  <node id="Read_OLCI">
    <operator>Read</operator>
    <parameters>
      <file>${OLCIsource}/xfdumanifest.xml</file>
      <formatName>Sen3</formatName>
      <!-- https://forum.step.esa.int/t/error-when-selecting-multiple-bands-from-slstr-file/15319 -->
      <!-- Sen3 ensures that SLSTR is read as multisize -->
    </parameters>
  </node>

  <node id="Idepix_OLCI">
    <operator>Idepix.Olci</operator>
    <sources>
      <sourceProduct>Read_OLCI</sourceProduct>
    </sources>
    <parameters>
      <!-- <outputSchillerNNValue>boolean</outputSchillerNNValue> -->
      <computeCloudShadow>false</computeCloudShadow>
      <!-- <alternativeNNDirPath>string</alternativeNNDirPath> -->
      <!-- <outputCtp>boolean</outputCtp> -->
      <computeCloudBuffer>false</computeCloudBuffer>
      <!-- <cloudBufferWidth>2</cloudBufferWidth> -->
      <!-- <useSrtmLandWaterMask>boolean</useSrtmLandWaterMask> -->
    </parameters>
  </node>
  
  <node id="Reproject_Idepix">
    <operator>Reproject</operator>
    <sources><source>Idepix_OLCI</source></sources>
    <parameters>
      <crs>EPSG:3413</crs>
      <resampling>Nearest</resampling>
      <noDataValue>NaN</noDataValue>
      <includeTiePointGrids>true</includeTiePointGrids>
    </parameters>
  </node>
  
  <node id="BandMaths_Idepix_cloud">
    <operator>BandMaths</operator>
    <sources><sourceProducts>Reproject_Idepix</sourceProducts></sources>
    <parameters>
      <targetBands><targetBand>
        <name>IDEPIX_CLOUD</name><type>Int16</type>
        <expression>IDEPIX_CLOUD</expression>
      </targetBand></targetBands>
    </parameters>
  </node>
  <node id="Write_Idepix_cloud">
    <operator>Write</operator>
    <sources><sourceProduct>BandMaths_Idepix_cloud</sourceProduct></sources>
    <parameters>
      <file>${targetFolder}/idepix_cloud_x.tif</file><formatName>GeoTIFF</formatName>
    </parameters>
  </node>
</graph>

scheunro · March 18, 2020, 7:58am

Hi,

I have a very similar issue. However mine seems to come from the cloud buffer computation. When I turn cloud buffer computation off everything works as expected. So that’s also my workaround for the moment.

May you let me know whether that works for you too?

mankoff · April 9, 2020, 12:34am

Nope - this doesn’t work for me. I can run IdePix w/o cloud buffer, shadow, reproject, or per-pixel geocoding, but disabling reproject and ppgc isn’t an option. We’ll wait for a working idepix implementation.

Tereza_S · May 18, 2020, 11:55am

I am running into problems with Idepix as well, for me SNAP immediatelly crahes when I try to run IdePix, whether i try to run it directly or import it in a graph. I have a Linux OS with all updates installed and I have even reinstalled Idepix again just to check. On Windows the process sees to run through but takes very long time to complete if run directly. In graph it also fails. Any ideas why this could be?

BaptisteVandecrux · March 2, 2021, 7:07am

Hi all,
I run into similar issue with the latest SNAP8 and IdePix_v8.0.0.

When I run this simple graph (2.0 KB) with

gpt test_idepix.xml  -POLCIsource="S3A_OL_1_EFR____20180601T115221_20180601T115521_20180602T170821_0179_032_009_1620_LN1_O_NT_002.SEN3"

then gpt hangs.

If I switch off

<computeCloudShadow>false</computeCloudShadow>  
<computeCloudBuffer>false</computeCloudBuffer>

then the gpt call above, without per-pixel geocoding, works:

If I switch on pixel geocoding, I get pixels set to zero throughout the Idepix output:

gpt test_idepix.xml  -POLCIsource="S3A_OL_1_EFR____20180601T115221_20180601T115521_20180602T170821_0179_032_009_1620_LN1_O_NT_002.SEN3"   -Ds3tbx.reader.olci.pixelGeoCoding=true

This seems to be somehow related to Pixels set to NaN when using pixel geocoding

fucongju · October 12, 2022, 7:53am

Hi, Marpet
When I use python to call .xml to process S3 data, I get an error: SPI not found for operator"Idepix.Olci".
When I checked SNAP, I didn’t find the idepix.olci plugin either, how do I download and install this plugin please?

fucongju · October 12, 2022, 8:50am

I have solved this problem.

Quan · September 7, 2023, 12:27pm

Hi fucongju
please tell me how to resolve the issue about "SPI not found for operator “Idepix.Olci” . I have got this problem! thanks very much!!!