Dears,
I have made off observations of the processing performance. I know that these are not new, and actions have been taken to optimise it. And progress was made. But still in some situations the processing performance is strange.
The issue cannot be the memory. I’ve used 20GB for the process. and during the process the limit wasn’t hit.
First, I observed unexplainable slowness for a specific data format when processing with my current operator under development.
Observing this, analysing my code and decided that the issue is not in my code. I decided to deep dive and spent some hours on it. I started a series of test runs without my operator. I only use different read / write methods of SNAP and use the two data formats
- Standard S2 L2A
- Theia S2 L2A
Theia has roughly twice the size, so should also take roughly twice the time.
Output format is always the ZNAP format, no big difference to BEAM-DIMAP observed. For Theia I used the unpacked data, for the Standard S2 I used the zipped data.
Here are my observation:
| Methodology | Standard | Theia |
|---|---|---|
| Export in SNAP Desktop *1 | 2:50 | 4:35 |
| PIO-Read/GPF-Write *2 | 34:50 | 29:00 |
| PIO-Read/PIO-Write *3 | 2:40 | 4:48 |
| GPF-Read/GPF-Write *4 | stopped after 8 min | stopped after 13 min |
*1 Simply open and exported the product to the ZNAP format
*2 Reading the Product with ProductIO.readProduct() and writing with GPF.writeProduct()
*3 Reading the Product with ProductIO.readProduct() and writing with ProductIO.writeProduct()
*4 Reading the Product with GPF.createProduct("Read") and writing with GPF.writeProduct()
As soon as GPS comes into play the processing seems to slow down.
My initial processing observation, including my operator was
| Methodology | Standard | Theia |
|---|---|---|
| PIO-Read/PIO-Write | 3:00 | 59:00 |
| PIO-Read/GPF-Write | 9:00 | 60:00 |
Here also the version without GPF writing is faster, but only for the standard product. For the Theia product it doesn’t have an effect. But the difference between standard and Theia is huge.
This still puzzles me.
It gets even more puzzling. When using gpt directly with my operator the numbers get more confusing. The operator has an option to include the bands of the source product.
| Methodology | Standard | Theia |
|---|---|---|
| new bands only | 7:51 | 8:11 |
| incl. source bands | 20:16 | 10:40 |
Suddenly the standard product with less bands is slower. It could be that I made a mistake and mixed up the numbers, but I double checked all.
I guess this needs more thorough investigation. But I stop here for now. Maybe you can use it as starting point for digging deeper.