Subset batch process problem

I’m trying to batch process 105 sentinel-1 products. First of all I lanched the subset process in batch mode using SNAP. After 5.30 hours I have got process completed from the software but, If I look in the folder, only 17 products are processed. The products are S1_IW_GRDH_1SDV_ .zip file directly downloaded from SciHub_download command line. Why the process cannot go thru all the products?

Hi @nicola.sangiorgi,

Are you sure the process is completed? Maybe SNAP crashes at that point and that’s way you only see 17 products. If you are using the GUI to process large datasets in batch processing you most probably will face memory issues. If you have a good machine this can be OK but 105 sentinel-1 products can fill up your computer.

Did you try GPT? That would be a good and faster solution. While the GUI interface does not release the memory when it finishes processing one input, GPT does it. Which are the specifications of your machine? Check the performance during the process to see how the resources are managed.

M

Thank you MCG, I will try the GPT! My machine has 60 Gb RAM and 8 core processor.

@MCG have you some instruction or tutorial to made this process in GPT? I never use SNAP thru the command line.

Thank’s

You can have a look at these wiki pages:
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503590/Creating+a+GPF+Graph
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT

But basically you open the SNAP Command line and then do

gpt Subset <Parameters> <Path_to_Product>

You can also type

gpt -h

or

gpt Subset -h 

for help

Thank you @marpet,
I’m not a computer programmer, so writing code in gpt mode is quite difficult for me.
I need to make the process in batch mode, that seem complicate a lot the code needed. I’m wrong?

It is not straight forward, especially for people who are not very familiar with programming. But even our GIS people here got used to it. So it is not too hard to learn.

as an example, if you want to cut out a geo-region just do:

gpt Subset -PgeoRegion=POLYGON((<lon1> <lat1>, <lon2> <lat2>, ..., <lon1> <lat1>)) -t <Path_to_target_product> <Path_to_source _product>

The geo-region is given in WKT format. This you can get from a geometry you have drawn in SNAP desktop.
Of course, you can do a lot more fancy stuff, but maybe this is all you need at the moment.

1 Like

Yes @marpet , but that code is for a single product. I need to do that process for 105 products. The single product process is quite easy but I need the batch mode.

@nicola.sangiorgi

I really suggest you to take some time to learn how to use GPT. It will make your work with Sentinel data much better. Start by learning how to use GPT for a single operation (like subset as suggested by @marpet). Move to process one image with a graph containing several steps (e.g.: resample and subset). It is also convenient to understand the structure of the xml graph files. This will help you to solve errors in the future.

Batch processing in GPT is definitely your last move. For that, you need a code (in bash, python, R, or whatever you prefer) that changes the input parameter and run the command. That is why it is not straight forward. You can find this code on the internet but you will need to understand the script to really get what is doing and which are the limitations and possible adaptations to your specific case.

Check the links of @marpet and this material as a starting point:


Don’t be afraid about coding, each hour you spend learning will save days of working time in the future.

M

1 Like

@marpet I am trying to batch subset L2 OLCI files (.SEN3 files). When I run this call:
./processDataset.bash subset_4.xml “/media/silsbelab/S31/S3_L2/S3A_20160216_20190724” “/media/silsbelab/S31/S3_L2/Subset_S3_L2” subset_S3_L2

It says “ls: cannot access ‘/media/silsbelab/S31/S3_L2/S3A_20160216_20190724/*.SEN3.nc’: No such file or directory”. Is this not the right call to batch process images?

Update: I got this code to work (I had to change something in the processDataset.bash file)
./processDataset.bash subset_4.xml parameters.properties “/media/silsbelab/S31/S3_L2/S3A_20160216_20190724/” “/media/silsbelab/S31/S3_L2/Subset_S3_L2/” subset_S3_L2

But when it runs, it goes through and says “Error: [NodeId: ReadOp@sourceProduct] Specified ‘file’ [/media/silsbelab/S31/S3_L2/Oa02_reflectance.nc] does not exist.” It seems to be searching for all products listed in OLCI L1 data, how do I change it to L2 products?