How to apply user defined operation on multiple cropped sentinel-2 images using gpt script?

Hi.
I’m quite new to SNAP and I have an issue regarding gpt.
How to apply user defined operation on multiple cropped sentinel 2 images using gpt script?
Here is the process:

  1. I have ~200 sentinel-2 images (.zip files).
  2. I cropped each one of them using gpt Subset script (=area of interest). And got .dim and .data files for each image.
  3. I want to apply an operation on each of the cropped images (.dim files) that is not included in the gpt operations (=my operation is: Biological Crust Index = 1 - (RED-BLUE)/(RED+BLUE) ).
    For this I used band-math and saved the xml file named “CrustIndex.xml”.
  4. I want to apply this new operation=CrustIndex.xml in a gpt script that will go through all ~200 sentinel-2 cropped images (.dim files). How can I do it? What should I change in the “CrustIndex.xml” file?

*** Here is the relevant part in my script which is not working yet (f=the cropped .dim file, target=t=.tif file):
./gpt CrustIndex.xml -Pfile=<dollar_sign>f -t <dollar_sign>{DIR}/${name}.tif

*** What should I write in the CrustIndex.xml file between the of the “read” and “write” part?

Thanks.

Hello rickysh,

How you can do batch processing is explained on this wiki page:
Bulk Processing with GPT - SNAP Wiki
You can use the BandMath op to define your operation

I attach an example of a graph using BandMaths: band_math_s2.xml (1.3 KB)

Dear marpet,

when I try your Bandmaths first, it not works. I want calibration first, resample 20m and c2rcc. actually, resample and c2rcc, it works well. but I try put the bandmaths, it not work well. could you give a hand? thanks

Best

Binbin

msi_calibration_res20_c2rcc_bin.xml (1.9 KB) error.txt (8.7 KB)

Could you tell us what you are trying to obtain? Why do you apply factors to the bands? What is the idea behind this calibration? Have to tried first in the GUI?

Dear abruescas,

the satellite sentinel2 should calibration at first. because the band and response of sensor after the launch, it will be a little bit changed (April 2020 sentinel2 report). so why should calibration. thanks
of course, GUI must be works.

Best

Binbin

Correct me if I am wrong, but the calibration procedure is done before calculating TOA reflectances. I do not think you need to apply those factors to the L1C product.

thank you so much. I am not sure it is right or not.

But my supervisor ask me to do vicarious calibration. and also Nima Pahlevan’s paper also did it. so why I try to do it. any way, thanks a lot.

Best

Binbin

2017 Pah Sentinel-2 MultiSpectral Instrument (MSI) data processing for aquatic science applications_ Demonstrations and validations _ Elsevier Enhanced Reader.pdf (1.3 MB)

@Jan Is this calibration necessary or advisable?

In the Pahlevan’s paper they do a vicarious calibration to the L2 data (Rrs), and you are trying to apply factors to the L1C (TOA reflectances).

1 Like

yes you are right. here is the other one for L1C (TOA reflectances) calibration.

Best

Binbin

2019 Pah Sentinel-2_Landsat-8 product consistency and implications for monitoring aquatic systems _ Elsevier Enhanced Reader (3).pdf (753.4 KB)

Dear marpet,

the other atmospheric correction like (seaDAS polymer acolite), they could setup the vrcarious calibration. so did SNAP has any setup? or use “BandMath”? thanks

Best

Binbin

Hello
II’m sorry @marpet - I didn’t see your request.

Vicarious calibration of Sentinel-2 is undertaken as part of the monitoring activities of the Mission. It is carried out by ESL (Expert Support Labs) that report every week to ESA on the status of the instrument. The radiometric calibration of the instruments is undertaken monthly, and lead to the release of updated Absolute Calibration and On Ground Equalisation GIPPs that are assimilated into PDGS processing every month (we will be releasing the April GIPPs shortly, for entry into production on Tuesday morning).

Thus, the instruments are already calibrated prior to production of data.

More on the calibration of the instrument can be found here https://sentinels.copernicus.eu/web/sentinel/technical-guides/sentinel-2-msi/calibration and the status of the Cal/Val is reported in the Sentinel-2 L1C Data Quality Report

Cheers

Jan

S2 MPC Operations Manager

2 Likes

Thanks @Jan. That’s what I had in mind, but wasn’t sure.

Dear marpet,

is that means I should not calibration again with Nima’s paper or 2020 report? is that right? thanks

Best

Binbin

Yes, you do not need to calibrate the data, because they are regularly calibrated.

Dear marpet,

our cluster changed. so now we use the SLURM workload manager. The code run very well ago. but now it get the error, detail below. Could you give me a hand? thanks

Best

Binbin

out1.txt (6.9 KB)
when err1.txt (2.1 MB)

This sounds like the process does not have write privileges to /home/bjiang/.snap/product-library.
But I don’t know your system and can’t tell you what’s wrong.
Maybe @oana_hogoiu has an idea what could be wrong.

From the logs you povided it seems that /home/bjiang/.snap is read-only, and this is causing several crashes:

SEVERE: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: /home/bjiang/.snap/auxdata/tool-adapters: Read-only file system
SEVERE: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Cannot create user folder for external tool adapter extensions
SEVERE: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Failed to scan /home/bjiang/.snap/auxdata/tool-adapters [reason: /home/bjiang/.snap/auxdata/tool-adapters]
SEVERE: org.esa.snap.lib.openjpeg.activator.OpenJPEGInstaller: OpenJPEG configuration error: failed to create /home/bjiang/.snap/auxdata/openjpeg/8.0.0
SEVERE: org.esa.snap.dataio.netcdf.NetCdfActivator: Native libraries for NetCDF could not be extracted to/home/bjiang/.snap/auxdata/netcdf_natives/8.0.0
org.h2.message.DbException: Log file error: “/home/bjiang/.snap/product-library/products.trace.db”, cause: “org.h2.message.DbException: Error while creating file “”/home/bjiang/.snap/product-library”" [90062-197]" [90034-197]
at org.h2.message.DbException.get(DbException.java:168)

Are you running SNAP using sudo?

Dear oana_hogoiu,

when i install “SNAP” software, the error :“an instance of the program cannot access specified user directory. This is a serious problem that prevent the program to function properly. make sure home/bjiang/.snap/system is writable. click ok to continue in spite of previous error”. thanks. it looks like not install successful. Could you help me ?

Best

Binbin

Dear oana_hogoiu,

i try sudo, but i am not sudoer. what should i do next? thanks

Best

Binbin

[bjiang@katahdin bin]$ sudo ./snap
[sudo] password for bjiang:
bjiang is not in the sudoers file. This incident will be reported.