Hi,
I am running Sentinel-2 biophysical processor through GPT in Linux. In SNAP 7 it was very slow and processing one scene would take many hours. Things improved greatly with release of SNAP 8 and the processing time went down to minutes. However, somewhere between Sentinel-2 Toolbox 8.0.0 and 8.0.4 the execution time degraded again and now it takes hours again.
I did a test using T32UPG scene from 16/06/2021 and two Docker images, with the only difference between them being either installation of just SNAP 8 (RUN ./esa-snap_sentinel_unix_8_0.sh -q) or SNAP 8 and all the updates (RUN ./esa-snap_sentinel_unix_8_0.sh -q followed by RUN snap --nosplash --nogui --modules --update-all). With SNAP 8 the biophysical processor took 4 minutes to complete, with the updates it took 4 hours.
There are not many fixes between Sentinel-2 Toolbox 8.0.0 and 8.0.4 but two of them concern the biophysical processor and one cache management. Could one of those be the reason for the regression?
Is this difference reproducible? If the processing needs some internet resource, one instance might have encountered an internet glitch – maybe just checking that some component is a current version so very little data being transferred . It could be helpful to collect additional information on CPU load and memory usage to compare the two SNAP versions, using e.g., bpytop.
I’ve been sticking to SNAP 8.0.0 because of this issue, but now with the new S2 format I switched to SNAP 8.0.9 (S2TBX 8.0.5). And unfortunately the issue persists. With a semi-randomly chosen S2 L2A scene biophysical processor takes 3-4 minutes with SNAP 8.0.0 and 20-25 minutes with SNAP 8.0.9.
With the update 8.0.4 of s2tbx the operator changed.
In the related SNAP 8.0.4 release the following has changed: Issue navigator - JIRA (atlassian.net)
But I can’t identify which of the issue could have an impact on the performance of the biophysical operator.
Hi,
The decrease of performance isn’t due to the modification processor, but the most of part is due to the addition of the metadata of the definition domain grids that it was missing. The addition of metadata introduce many input slower tests.
It have to improve the processing of these input tests.
The issue is tracked here [SIITBX-481] Execution speed of Sentinel-2 Biophysical Processor - JIRA
I hope release a fix on the next s2tbx 8.0.6 release.
Just want to add that from a quick inspection of output LAI maps the results produced by SNAP 8.0.0 and 8.0.9 seem identical. I.e. the extra metadata is not affecting results.
Yes, the biophysical processing will be faster than 8.0.4 in the next release 8.0.7 ASAP.
I have improve the manage of the output range testing that it was very slow. It is not faster than 8.0.0, but the computing time is reasonable ( 9 min 8.0.7 vs 50 min 8.0.4, the result seems closed to 8.0.0 with in addition the correct ouput range testing).
Thanks @FlorianD! In my test Biophysical processor in S2TBX 8.0.7 took about 8 minutes, so significant improvement compared to 8.0.6.
I am wondering what you meant by
in addition the correct ouput range testing
I quickly compared LAI produced with S2TBX 8.0.7 and 8.0.0 and they seem identical, including negative LAI values. The output masks (e.g. lai_output_too_low) also look the same. What is the expected effect of the output range testing?
You could consult ATBD page 48 (step.esa.int/docs/extra/ATBD_S2ToolBox_V2.1.pdf) for more details.
There was several test metadata missing
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/FAPAR/FAPAR_DefinitionDomain_Grid
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/FAPAR/FAPAR_DefinitionDomain_MinMax
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/FAPAR/FAPAR_TestCases
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/FCOVER/FCOVER_DefinitionDomain_Grid
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/FCOVER/FCOVER_DefinitionDomain_MinMax
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/FCOVER/FCOVER_TestCases
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/LAI/LAI_DefinitionDomain_Grid
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/LAI/LAI_DefinitionDomain_MinMax
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/LAI/LAI_TestCases
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/LAI_Cab/LAI_Cab_DefinitionDomain_Grid
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/LAI_Cab/LAI_Cab_DefinitionDomain_MinMax
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/LAI_Cab/LAI_Cab_TestCases
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/LAI_Cw/LAI_Cw_DefinitionDomain_Grid
A s2tbx-biophysical/src/main/resources/auxdata/3_0/S2A/LAI_Cw/LAI_Cw_DefinitionDomain_MinMax
Further, other metadata files has been modified and our regression testing data reference values has been updated caused the some outputs changed (not many).
I guess it concerns some limits cases.
Having being involved in development and validation of the biophysical processor algorithm using in the ST2BX during the VALSE2 project please note my presentation at the 4th sentinel 2 validation team meeting that indicated the definition domain is over constrained generally leading to instances with much fewer retrievals than possible (there is also a bug in the implementation of the actual retrieval algorithm) https://az659834.vo.msecnd.net/eventsairwesteuprod/production-nikal-public/398d3817bd554833a33d07cb20126bcb
I have worked with Marie Weiss (the author of the ATBD) to implement a corrected algorithm. It is available in Google Earth Engine and is capable of processing about 1000 MSI scenes for visualization and can export a scene in about 1-2 minutes (GitHub - rfernand387/LEAF-Toolbox) . The actual networks and code are also available is S2TBX would like to update their code.
Richard Fernandes, Canada Centre for Remote Sensing