Reduce cloud and shadows

besi8469 · March 27, 2017, 12:44am

Hello,
Is there a solution to reduce the clouds and their shadows on sentinel 2 sattelite images?
Thank you,
Simon

marpet · March 27, 2017, 7:03am

Not really reducing the clouds and shadows, but the IDEPIX is detecting them. And you can exclude them in your further processing.
You find it in the menu at Optical / Preprocessing / Masking / IDEPIX / S2.
You need to resample the S2 product first. If you want to run IDEPIX on a subset you should remove the masks first. I’ve explained it recently how to do it.

atteggiani · February 5, 2018, 1:59pm

Hello @marpet, I’m trying to run Idepix through snappy on a L1 S-2 granule too.
What I wanna do is:

mask each granule with the Idepix Op
mosaic the granules (total of 6 in my case)
take only some specific bands I need (B2,B3,B4 and B8 to be more precise)
subsetting to a specific area.

Now I have two main questions:

I don’t know if there’s a specifical order to be followed for the process to work, or if it’s merely the same thing to do one step first rather than another one (with excepts for resampling that I reckon it must be done ahead of the rest), so which is the best sequence of operations you would suggest?
I tried to run idepix on snap gui to see if the entire process would work before implementing it on snappy. I’ve done the following steps:

open a S2 granule;
resample the granule to 10m;
run the idepix on the resampled image;
fuse some cloud idepix masks (cirrus sure, cloud sure, cloud ambiguous and cloud buffer) to the mask I named CLOUDS;
apply bandmath to create new bands with values of the cloud pixels replaced by NaNs (as an example, for B2 I created B2_mask by putting this expression in the bandmaths: “If CLOUDS == 0 then B2 else NaN”)
I removed all the other bands except for B2_mask created;
Saved the product

Now if I close the product and open it again, if I try to open band B2_mask previously created it shows me this error:
expression: Undefined symbol 'pixel_classif_flags.F_INVALID'

What am I missing? How can I create such products? I only need some bands (B2,B3,B4 and B8) to which I wanna apply idepix and replace with NaN the values corresponding to clouds.
Thank you for your time!
Davide

marpet · February 12, 2018, 3:35pm

I would do it in the following way.

Resample
Idepix
Subset (Band subset, maybe a regional subset)

This would be my “pre-processing” chain. I would create a graph XML file and let it run from the command line on my input data set (see Bulk Processing with GPT).

Then I would define the mosaic processing and use the pre-processed data as input. Maybe also as a graph. Depends on how often you plan to do it. Here I would specify my area of interest.

Maybe you have taken the ‘pixel_classif_flags.F_INVALID’ from an older IdePix version. The flags have been renamed. Now, in version 6, the flag names start with ‘IDEPIX’. So, it should be ‘IDEPIX_INVALID’.

atteggiani · February 13, 2018, 10:39am

Thank you for your reply,
You are right, I noticed I hadn’t yet updated my SNAP to version 6.
It works now ! thank you.

Speaking about SNAP 6 I’ve noticed you implemented the possibility to open .zip S-2 products directly. Is it possible to do it in python snappy too? Do I have to update the snappy module as well?
I tried to open a .zip S-2 product but I get this error
RuntimeError: java.lang.NoSuchMethodError: org.hsqldb.DatabaseURL.parseURL(Ljava/lang/String;ZZ)Lorg/hsqldb/persist/HsqlProperties;

marpet · February 13, 2018, 12:11pm

Actually, it should also work with snappy. I’m not sure if this error message is somehow related to the S2 reader.
When installing the new version, do you have uninstalled the old SNAP before and also checked the option to remove all SNAP user data?
@obarrilero Do you use HSQLDB in the reader?

atteggiani · February 13, 2018, 1:36pm

I did not remove the old SNAP before but just overwrite the already existent one.
Yes I checked the option to remove SNAP user data but did not check the one to remove all SNAP data

obarrilero · February 13, 2018, 3:00pm

In the S2 reader we do not use hsqldb directly, but we use geotools and I think geotools use it (if it is not geotools, another library uses it, because some log messages are shown when opening the product).
On the other hand, I have just tested snappy with a zipped product and it seems to be working properly, perhaps something is corrupted during your installation, @atteggiani. Could you please try installing the last version from scratch and check if it works properly?

atteggiani · February 13, 2018, 3:48pm

What do you mean?

Ok so do you suggest me to uninstall my current version of SNAP prior to installing the new one?
Is anythin supposed to be modified in the snappy module directory? Or the SNAP update should be able to make all the updated functions usable by snappy as well?

obarrilero · February 13, 2018, 3:51pm

Yes, I suggest to:

Uninstall your current version of SNAP
Install from scratch version 6
Configure again snappy by using snappy-conf
If you had copied your snappy folder to your python installation, replace it by the new one that will be created in the step3.

atteggiani · February 14, 2018, 10:46am

I’ve tried to do your suggested procedure but, when running snappy-config (I preferred doing the configuration manually after SNAP installation), I get an error producing the following log file:

INFO: Installing from Java module 'C:\Program Files\snap\snap\modules\org-esa-snap-snap-python.jar'
INFO: Installing jpy...
INFO: Extracting 'lib/jpy-0.8.0-cp34-none-win_amd64.whl' from 'C:\Program Files\snap\snap\modules\org-esa-snap-snap-python.jar'
INFO: Unzipping 'C:\ProgramData\Anaconda\envs\SNAP\snappy\lib\jpy-0.8.0-cp34-none-win_amd64.whl'
INFO: Configuring jpy...
INFO: jpy Python API configuration written to 'C:\ProgramData\Anaconda\envs\SNAP\snappy\jpyconfig.py'
INFO: jpy Java API configuration written to 'C:\ProgramData\Anaconda\envs\SNAP\snappy\jpyconfig.properties'
INFO: Configuring snappy...
INFO: snappy configuration written to 'C:\ProgramData\Anaconda\envs\SNAP\snappy\snappy.ini'
INFO: Importing snappy for final test...
ERROR: Configuration failed
Traceback (most recent call last):
  File ".\snappyutil.py", line 266, in _main
    force=args.force)
  File ".\snappyutil.py", line 213, in _configure_snappy
    __import__('snappy')
  File "C:\ProgramData\Anaconda\envs\SNAP\snappy\..\snappy\__init__.py", line 380, in <module>
    Engine.start()
RuntimeError: java.lang.UnsatisfiedLinkError: org.gdal.gdalconst.gdalconstJNI.GDT_Unknown_get()I

I was running the command prompt in administrator mode and I deleted the old snappy directory prior to launching snappy-conf.
Can you help me with this?
Thank you

EDIT
Solved
Even if I get the same error if I try to launch snappy-config, now that I reinstalled SNAP from scratch it lets me open .zip files directly even from snappy. Maybe the issue was to be found in the user files deletion.

Jackarles · February 10, 2021, 12:00pm

Hello everyone,

As I’m trying to detect shadows in S2 products, I want to test the Idepix operator, but there is only “MERIS” available into Optical/Preprocessing/Masking. And I can’t find any Idepix operator with “gpt -h”.
(I’m using SNAP 8.0.3).

Thank you in advance.

marpet · February 10, 2021, 12:44pm

Maybe your installation of Idepix modules is not complete.
Please check the plugin manager (Tools / Plugins).
In the Available tab, you should see multiple Idepix modules which you can install.

Jackarles · February 10, 2021, 1:18pm

Thank you very much!
From your point of view, is that a correct tool to detect big tree/buildings shadows as well?

marpet · February 10, 2021, 2:36pm

No, those shadows are not detected.
S2 MSI is the sensor with the highest resolution Idepix supports. With this resolution big buildings and their shadows could be detected but trees are difficult. However, such shadow detection is not implemented at all.