I am processing Sentinel-2 tiles by resampling bands at 10 m and then calculating 2 biophysical products (LAI and FVC). I am trying various methodologies because I have many tiles to process, including 1. manual steps and 2. Creating a graph and trying and use the batch processor (but seems i need to manually define output file name - so i am not sure if that will improve efficiency).
My question is regarding processing speed. Does anyone know approximately how long it should take to process one full tile for LAI and FVC? Seems resampling might take a couple of minutes on my system, but the LAI/FVC calculation may take multiple hours for one tile.
Does anyone have experience with this type of operation? What machine specs impact processing speed? Should i expect it to take multiple hours to process one tile to LAI/FVC at this resolution?
I am running from a virtual machine and the specs are below:
Host Machine:
Chipset: Intel® Xeon® CPU E5-2620 v4 @ 2.10GHz
16 cores
32GB RAM
I would pay attention to the RAM usage when running your graph. Because the graph stays in memory between steps (instead of saving directly to the disk after each step), if you run out of memory it will start caching to the disk, and end up taking longer as it then has to keep reading and writing to the disk cache.
Unfortunately I do not have a solution but I don’t think that this is RAM dependent. It can take many hours (up to half a day per scene) to run the Sentinel-2 biophysical processor on a Linux machine with 35 GB of RAM allocated to SNAP/GPT. If you find a method to speed this up then please share!
That LAI is not the fastest processing operation is true.
But there is also an issue in the current SNAP version.
This has been discussed in this thread. I also sketch a workaround for this issue.
With the next version of SNAP this issue should be fixed. *fingerscrossed*
As mentioned by @marpet, SNAP may not be taking advantage of the multiple CPU’s.
If you have many simple jobs (e.g., an “embarrassingly parallel” task), you can turn a
bug into a feature by running several jobs at a time.
You don’t mention the OS, but each OS supported by SNAP has tools to help monitor resource usage, including CPU and memory. You may be able to speed up your processing by running several jobs at a time. I have found GNU parallel works well on macOS and linux. To use GNU parallel I generally create a shell script that takes one argument (typically a file name) and uses that to construct the GPT command and XML file for one job. Then I just pass a list of arguments to GNU parallel. To monitor CPU usage while experimenting with the configuration you may find bpytop useful.
You may also want to consider ways to remove I/O bottlenecks. If you can structure jobs so they don’t read and write to the same storage unit (including scratch I/O) you may see useful speedups.
Just FYI - we confirmed the parallelization issue (this was SNAP version7), by running the graph and checking the CPU usage, which indicated only one CPU was utilized at a time when running the graph and trying batch processing. We did not experiment with the -q option (was not sure how to invoke that) but instead used SNAP installed on multiple (Virtual machines on linux host) machines simultaneously to accomplish the task. Thanks for tips! For a complete-ish scene, with the scene resampled at 10-meters, our systems took about 1-2 hours to process LAI and FVC, 4-12 CPU machines, RAM = 8GB+