SNAP new Version BIG Problem with data gaps!

Hello SNAP-Team,
I am currently teaching around 100 students using SNAP. Since a few days - I think it corresponds to the newest version - several students of my class report about big problems with SNAP when calculating new bands with the band math tool. The new calculated band sometimes has gaps in between. It looks as if a tile of the image is not calculated at all. There are whole rectangular squares missing in the new band.

Here an example from a calculated mask. This mask was simply calculated using a water index band and the band math equation water_index > 0. I never had these missing tiles before.

It is really important for my teaching, that this error is fixed, as my students have to work on a marked project at the end of the class and they will use SNAP. It would be a big problem, if this error remains.

Best regards,
Geo

could it be a matter of visualization?
Sometimes, if you zoom in, the data successively is shown. Band Maths expressions are created on the fly (even if you disable the “virtual” option) and only stored when they are saved physically, so it might be a matter of system capabilities.

You can use the pixel info and move your cursor to test if these areas contain valid pixel values or if these are really empty.

We are working with SNAP 8 in this semester (around 50 students) and have not experienced this so far. At least no one has reported something similar to me yet.

2 Likes

Hello ABraun,
thanks a lot for your fast reply.
It is not a problem of visualization. I know the “zoom” topic in SNAP since years, because it is quite old. But this time the data is not “refreshed visually” when zooming into the data. Using the “Pixel Info” reveals NAN values for the squared areas.

I also know about the difference between virtual and “hard drive saved” bands. The band is not virtual. It is also not a problem of system capacities. I also have the problem here on my computer and my maschine is quite up do date (AMD Ryzen 3700X, 32 GB-RAM, Harddrive M.2 NVMe with more than 50% recources left - so defnitly not a hardware problem).

This problem is very new. So I would say not older than 5 to 7 days. I also never had this problem before. I don`t think it is a problem of the users or the maschines. But… maybe?

Best regards,
Geo

Hello,
I tested a bit and I could manage to reproduce the problem!

I use a subset from a Landsat 8 scene, but it should work with any other data too!

  1. calculate the NDVI with the band math tool
  2. calculate a second band showing only positive NDVI values as 1 (formula NDVI > 0). Lets call this new band NDVI mask.
  3. After the calculation, the two new bands (NDVI and NDVI_mask) should be displayed in the views and everything seems to be okay.
  4. CLOSE the whole product now and reopen it in SNAP
  5. Display the two new bands. NDVI should now be okay, but NDVI mask should contain gaps.

This is not good…

Update:

I deinstalled SNAP 8 and installed the older SNAP 7. Everything is okay in SNAP 7!
I used the same data, the same methods and the same calculations. No gaps in the data.

So it must be something with SNAP 8.

Best
Geo

thank you for your description. I just have to suggest these basic things because in 80% of cases, problems are solved by the more obvious things.

I tested with a Sentinel-2 image in SNAP 8 according to your steps and cannot reproduce it:

are the latest updates installed?

Hello ABraun,
I now downloaded and installed SNAP 8 (newest version) once again.

I tested it with a first scene and everything seemed to be fine.
But already on a second scene, the same problem occured once again.
I therefore repeated the same calculation several times on the same data and found, that the error seems to be random. Sometimes I have a gap - even already in the NDVI band.

Best regards,
Geo

ABraun,
in your example you used virtual bands. I did not use virtual bands. I therefore don`t know, whether the problem would also occur on virtual bands. Bur I think, it should also occur on your S2 data.

Thanks for your fast reply and help.

Best regards,
Geo

you’re right. I have tested it again, creating Band Maths with the “virtual” option disabled. I closed the product and opened it again, but still I am not able to reproduce it.

What kind of data are you using?
Also, which file type is the data?

I am currently using Landsat 8 Collection 2 Level 2 data. The data is converted to the *.dim format. But the problem also occured with Landsat 7 Level 1 data in my class. I have to mention, that the Landsat 7 data (with no changes) is used for the teaching since several years. I used the data last year and the year before. So I am totally shure, the data is okay! I did not do any changes to the data compared to the years before.

Now: I re installed SNAP 7: calculated the NDVI several times, calculated the mask several times. No Problems with SNAP 7.

No problem with SNAP7:

Can you maybe share the dataset so we can test this?

@marpet is there something new in the tiling/writing of the products?

@Geonow Are these gaps of a certain pixel size, for example 512x512 pixels?
I am wondering if it is related to this setting: Maybe it is worth testing what happens if you reduce the tile size in SNAP 8.

I will test your suggested settings in a minute.

First I want to show you, what I did in the last minutes.
I reinstalled SNAP 8 now and used the same scene shown for SNAP 7.
Again I calculated the NDVI four times and afterwards calculated the masks from the NDVIs.

I found two things of importance:

  1. the problem does not always occur. Only few calculations messed up.
  2. It must be a problem with data writing/reading? Because if you look at my screenshot, the NDVI8 is corrupt, but the mask calculated from the NDVI8 band is okay. It should at least suffer from the same NANs.

Hi,
I had a look at the size of one square.
The width is 520 px
The height is 572 px.

I currently do not have a square in the middle, so I am not totally sure, whether it is an exact square or not.

My Tile size is 512 and the Number of Threads is set to 16.

Update:

I now know, that it is enough to just calculate the NDVI several times to force the error. I just calculated six NDVI bands and two of them are corrupt. (calculate all six NDVI bands after each other, than save the product, close it and reload. The reload is important!)

Result:

I was “lucky” to habe the squares show up in the mid of my bands and your suggestion is right, the squares have a specific size of 512 x 512 px.

As this exactly matches the processing tile size, are we on a good way to find the reason?

Again: thanks a lot for your help.

Best regards,
Geo

then I recommend reducing the tile size to 256, for example, and checking the output again. Still, I wonder why this happens and I cannot reproduce it.

Your data gaps can provide a useful teaching moment. Before retiring, the lab where I worked did many short courses in ocean remote sensing (using the IDL-based NASA SeaDAS). One thing we learned from post-course follow-ups was that many participants struggled to keep the software working outside the course environment.

The first step in reporting problems is to describe the computer system and configuration details, so it is useful to have students learn how to collect this information (OS tools for version and hardware details, <snap_install_dir>/etc/snap.conf for the default_options, and JRE version).

Having that information, your students can look for a common hardware, OS, or configuration factor associated with the data gaps. At the very least, your students will be prepared to provide good problem reports in the future.

Hello,
I uploaded my data using the service wetransfer. Maybe you can reproduce the problem with my data:

https://we.tl/t-N55jzn0BU2

Best regards,
Geo

Yes, I have to keep this in mind. Especially these days. In the last years every class was given at the university in our PC labs. But due to Corona, all my students are working from home with their personal computers. So much more problems due to this situation…

there it is

can confirm this happens at random tiles.

Not a solution, but maybe for your course (until we have a response from the developers): You could use the Mask Manager to identify these areas based on the NDVI and work on with that.

I have followed the discussion. It could be that there is an issue, we will check.
It might take some time till we can have a look at this problem. Currently we are tackling some other issues already which have a high priority too.

I’ve created an issue for this. So we don’t forget it.
[SNAP-1382] Missing tiles in image view - JIRA (atlassian.net)

Thanks, Florian for the report and sorry for the inconvenience.

3 Likes