I’m trying to batch process 105 sentinel-1 products. First of all I lanched the subset process in batch mode using SNAP. After 5.30 hours I have got process completed from the software but, If I look in the folder, only 17 products are processed. The products are S1_IW_GRDH_1SDV_ .zip file directly downloaded from SciHub_download command line. Why the process cannot go thru all the products?
Are you sure the process is completed? Maybe SNAP crashes at that point and that’s way you only see 17 products. If you are using the GUI to process large datasets in batch processing you most probably will face memory issues. If you have a good machine this can be OK but 105 sentinel-1 products can fill up your computer.
Did you try GPT? That would be a good and faster solution. While the GUI interface does not release the memory when it finishes processing one input, GPT does it. Which are the specifications of your machine? Check the performance during the process to see how the resources are managed.
M
Thank you MCG, I will try the GPT! My machine has 60 Gb RAM and 8 core processor.
@MCG have you some instruction or tutorial to made this process in GPT? I never use SNAP thru the command line.
Thank’s
You can have a look at these wiki pages:
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503590/Creating+a+GPF+Graph
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/70503475/Bulk+Processing+with+GPT
But basically you open the SNAP Command line and then do
gpt Subset <Parameters> <Path_to_Product>
You can also type
gpt -h
or
gpt Subset -h
for help
Thank you @marpet,
I’m not a computer programmer, so writing code in gpt mode is quite difficult for me.
I need to make the process in batch mode, that seem complicate a lot the code needed. I’m wrong?
It is not straight forward, especially for people who are not very familiar with programming. But even our GIS people here got used to it. So it is not too hard to learn.
as an example, if you want to cut out a geo-region just do:
gpt Subset -PgeoRegion=POLYGON((<lon1> <lat1>, <lon2> <lat2>, ..., <lon1> <lat1>)) -t <Path_to_target_product> <Path_to_source _product>
The geo-region is given in WKT format. This you can get from a geometry you have drawn in SNAP desktop.
Of course, you can do a lot more fancy stuff, but maybe this is all you need at the moment.
1 Like
Yes @marpet , but that code is for a single product. I need to do that process for 105 products. The single product process is quite easy but I need the batch mode.
I really suggest you to take some time to learn how to use GPT. It will make your work with Sentinel data much better. Start by learning how to use GPT for a single operation (like subset as suggested by @marpet). Move to process one image with a graph containing several steps (e.g.: resample and subset). It is also convenient to understand the structure of the xml graph files. This will help you to solve errors in the future.
Batch processing in GPT is definitely your last move. For that, you need a code (in bash, python, R, or whatever you prefer) that changes the input parameter and run the command. That is why it is not straight forward. You can find this code on the internet but you will need to understand the script to really get what is doing and which are the limitations and possible adaptations to your specific case.
Check the links of @marpet and this material as a starting point:
https://sa.catapult.org.uk/wp-content/uploads/2016/05/SNAP-Sentinel-1_TrainingCourse_Exercise4.pdf
Don’t be afraid about coding, each hour you spend learning will save days of working time in the future.
M
1 Like
@marpet I am trying to batch subset L2 OLCI files (.SEN3 files). When I run this call:
./processDataset.bash subset_4.xml “/media/silsbelab/S31/S3_L2/S3A_20160216_20190724” “/media/silsbelab/S31/S3_L2/Subset_S3_L2” subset_S3_L2
It says “ls: cannot access ‘/media/silsbelab/S31/S3_L2/S3A_20160216_20190724/*.SEN3.nc’: No such file or directory”. Is this not the right call to batch process images?
Update: I got this code to work (I had to change something in the processDataset.bash file)
./processDataset.bash subset_4.xml parameters.properties “/media/silsbelab/S31/S3_L2/S3A_20160216_20190724/” “/media/silsbelab/S31/S3_L2/Subset_S3_L2/” subset_S3_L2
But when it runs, it goes through and says “Error: [NodeId: ReadOp@sourceProduct] Specified ‘file’ [/media/silsbelab/S31/S3_L2/Oa02_reflectance.nc] does not exist.” It seems to be searching for all products listed in OLCI L1 data, how do I change it to L2 products?
Hi,
I have successfully
resampled a set of S2 imagery according to the example given by Marco Peters to
bulk processing with GPT. Now I am adapting it to apply for a subset process.
I have got some
doubts about the .xml file and Parameters File. I attach them.
xml file:
parametersFile:
I haven´t
got results applying the ProcessDataset and I don´t get a warning.
Are
correctly settings these files?
Marcela
The parameters look good.
You can remove the spaces in front of the Polygon parameter. But this shouldn’t be the issue.
That you don’t get a warning or an error and also not processed output is strange.
Is something logged? Maybe there is in the script something wrong so the gpt is not correctly called?
Marco,
I have removed the space and now I have gotten an error: Error: [NodeId: Subset] Operator ‘SubsetOp’:
I have seen the same problem in another post but it wasn´t resolved.
Can you post the complete polygon in textual form?
By the way, in the console output it is stated that you still run SNAP 6.
It might work better with SNAP 7.
Yes.
POLYGON=((-1.7246453762054443 39.6510009765625, -1.2131989002227783 39.6510009765625, -1.2131989002227783 39.1479377746582, -1.7246453762054443 39.1479377746582, -1.7246453762054443 39.6510009765625, -1.7246453762054443 39.6510009765625))
Also, I attach you the three files that I am applying.
myGraph_T30_SXJ_resto.xml (645 Bytes) processDataset_resto.bat (2.2 KB) subset.properties (284 Bytes).
Thanks
Marcela.
1 Like
I found the issue with the geometry.
The WKT syntax starts with POLYGON, so your property needs to be:
POLYGON=POLYGON((-1.7246453762054443 39.6510009765625, -1.2131989002227783 39.6510009765625, -1.2131989002227783 39.1479377746582, -1.7246453762054443 39.1479377746582, -1.7246453762054443 39.6510009765625, -1.7246453762054443 39.6510009765625))
The updated property file:subset.properties (291 Bytes)
2 Likes
Thanks a lot Marco. Now it is running.
Marcela
Is it possible to install and use gpt in Ubuntu Linux? Please share the corresponding links for download and installation instructions.
Yes, this is possible.
Simply get the Unix 64-bit installer from this page: SNAP Download
Execute it and follow the instructions shown.