Update:
I deinstalled SNAP 8 and installed the older SNAP 7. Everything is okay in SNAP 7!
I used the same data, the same methods and the same calculations. No gaps in the data.
So it must be something with SNAP 8.
Best
Geo
thank you for your description. I just have to suggest these basic things because in 80% of cases, problems are solved by the more obvious things.
I tested with a Sentinel-2 image in SNAP 8 according to your steps and cannot reproduce it:
are the latest updates installed?
Hello ABraun,
I now downloaded and installed SNAP 8 (newest version) once again.
I tested it with a first scene and everything seemed to be fine.
But already on a second scene, the same problem occured once again.
I therefore repeated the same calculation several times on the same data and found, that the error seems to be random. Sometimes I have a gap - even already in the NDVI band.
Best regards,
Geo
ABraun,
in your example you used virtual bands. I did not use virtual bands. I therefore don`t know, whether the problem would also occur on virtual bands. Bur I think, it should also occur on your S2 data.
Thanks for your fast reply and help.
Best regards,
Geo
you’re right. I have tested it again, creating Band Maths with the “virtual” option disabled. I closed the product and opened it again, but still I am not able to reproduce it.
What kind of data are you using?
Also, which file type is the data?
I am currently using Landsat 8 Collection 2 Level 2 data. The data is converted to the *.dim format. But the problem also occured with Landsat 7 Level 1 data in my class. I have to mention, that the Landsat 7 data (with no changes) is used for the teaching since several years. I used the data last year and the year before. So I am totally shure, the data is okay! I did not do any changes to the data compared to the years before.
Now: I re installed SNAP 7: calculated the NDVI several times, calculated the mask several times. No Problems with SNAP 7.
No problem with SNAP7:
Can you maybe share the dataset so we can test this?
@marpet is there something new in the tiling/writing of the products?
@Geonow Are these gaps of a certain pixel size, for example 512x512 pixels?
I am wondering if it is related to this setting: Maybe it is worth testing what happens if you reduce the tile size in SNAP 8.
I will test your suggested settings in a minute.
First I want to show you, what I did in the last minutes.
I reinstalled SNAP 8 now and used the same scene shown for SNAP 7.
Again I calculated the NDVI four times and afterwards calculated the masks from the NDVIs.
I found two things of importance:
- the problem does not always occur. Only few calculations messed up.
- It must be a problem with data writing/reading? Because if you look at my screenshot, the NDVI8 is corrupt, but the mask calculated from the NDVI8 band is okay. It should at least suffer from the same NANs.
Hi,
I had a look at the size of one square.
The width is 520 px
The height is 572 px.
I currently do not have a square in the middle, so I am not totally sure, whether it is an exact square or not.
My Tile size is 512 and the Number of Threads is set to 16.
Update:
I now know, that it is enough to just calculate the NDVI several times to force the error. I just calculated six NDVI bands and two of them are corrupt. (calculate all six NDVI bands after each other, than save the product, close it and reload. The reload is important!)
Result:
I was “lucky” to habe the squares show up in the mid of my bands and your suggestion is right, the squares have a specific size of 512 x 512 px.
As this exactly matches the processing tile size, are we on a good way to find the reason?
Again: thanks a lot for your help.
Best regards,
Geo
then I recommend reducing the tile size to 256, for example, and checking the output again. Still, I wonder why this happens and I cannot reproduce it.
Your data gaps can provide a useful teaching moment. Before retiring, the lab where I worked did many short courses in ocean remote sensing (using the IDL-based NASA SeaDAS). One thing we learned from post-course follow-ups was that many participants struggled to keep the software working outside the course environment.
The first step in reporting problems is to describe the computer system and configuration details, so it is useful to have students learn how to collect this information (OS tools for version and hardware details, <snap_install_dir>/etc/snap.conf for the default_options, and JRE version).
Having that information, your students can look for a common hardware, OS, or configuration factor associated with the data gaps. At the very least, your students will be prepared to provide good problem reports in the future.
Hello,
I uploaded my data using the service wetransfer. Maybe you can reproduce the problem with my data:
Best regards,
Geo
Yes, I have to keep this in mind. Especially these days. In the last years every class was given at the university in our PC labs. But due to Corona, all my students are working from home with their personal computers. So much more problems due to this situation…
there it is
can confirm this happens at random tiles.
Not a solution, but maybe for your course (until we have a response from the developers): You could use the Mask Manager to identify these areas based on the NDVI and work on with that.
I have followed the discussion. It could be that there is an issue, we will check.
It might take some time till we can have a look at this problem. Currently we are tackling some other issues already which have a high priority too.
I’ve created an issue for this. So we don’t forget it.
[SNAP-1382] Missing tiles in image view - JIRA (atlassian.net)
Thanks, Florian for the report and sorry for the inconvenience.
3 Likes
Thanks for all your help!
@ABraun, this problem is not specific to mask-bands. It also appears for normal calculations. And my students need the Band Math Tool for change detection calculations and more.
@marpet thanks a lot. I will follow the issue. If this cant be solved in time, I will recommend using SNAP 7 for my students for their final project.
Best regards,
Geo
Thank you for reporting this, I raised the priority of the ticket to the highest level = “blocker”. Hopefully the root cause can be identified and we can issue a fix with the next module update.
I have also reproduced it:
And in log file there are a lot of JAI errors:
SEVERE [global]
java.lang.NullPointerException
SEVERE [global]
java.lang.RuntimeException: Waiting thread received a null tile.
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
at javax.media.jai.OpImage.getTile(Unknown Source)
at javax.media.jai.PlanarImage.getData(Unknown Source)
at org.esa.snap.core.image.VirtualBandOpImage.addDataToReferredRasterDataSymbols(VirtualBandOpImage.java:307)
at org.esa.snap.core.image.VirtualBandOpImage.computeTile(VirtualBandOpImage.java:235)
at com.sun.media.jai.util.SunTileScheduler.scheduleTile(Unknown Source)
at javax.media.jai.OpImage.getTile(Unknown Source)
at javax.media.jai.PlanarImage.getData(Unknown Source)
at com.bc.ceres.glevel.MultiLevelImage.getData(MultiLevelImage.java:64)
at org.esa.snap.core.dataio.ProductIO.writeTile(ProductIO.java:602)
at org.esa.snap.core.dataio.ProductIO.lambda$writeRasterDataFully$0(ProductIO.java:578)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[catch] at java.lang.Thread.run(Thread.java:748)
SEVERE [null]: Last record repeated again.
SEVERE [global]
java.lang.NullPointerException
SEVERE [null]: Last record repeated 3 more times.
SEVERE [org.esa.snap]: JAI error occurred: ‘Problem occurs when computing a tile by the owner.’ at com.sun.media.jai.util.SunTileScheduler@2ea4071c
Update:
When closing the product (and choosing to save it), the JAI error mentioned above in the log also appears in SNAP:
@geonow When I retired I planned to study the nature of problems reported in the NASA SeaDAS and ESA SNAP forums. Remote work has been difficult for people who were using lab systems configured and managed by staff and suddenly find themselves having to manage the tools on a personal system (often stuck with under-configured hardware due to ongoing shortages of RAM and mass storage). In our workshops, as the SeaDAS software we used didn’t run on Windows, devoting a couple afternoons to an introduction to the linux command-line proved helpful. Not only students, but other instructors, gave very positive feedback.