Problems with the Collocation Tool and georeference of Landsat 8

[pbrueck1]

Hello everybody,

i have Problems with the Collocation Tool and don’t know how to go on. Any help would be really apreciated!
I’ve made a water mask from a Landsat 8 Image of Cologne, wich i want to use for 40 different Landsat 5 and 8 Subset-Products of the same City. I’ve made the graph: ProductSet-Reader -> Collocate -> Write. In the ProductSet Reader i added the Product with the Watermask Band and one other random Product (image from 1985) (i had to use ProductSet Reader and add two Products, because when i use ‘read’ instead and only add the Watermask Product, i get the Error ‘At least two source Products have to be defined’ in the ‘Batch Processing’ and cannot choose a Master.
Then i saved this graph, and loaded it in Batch Processing. I added all my Products and
for Master i selected the Product with the Watermask Band. It run without an Error, and the Result were 40 images with their original names, but they all had the same bands (the Water mask as Master, and the ones from the image from 1985 as Slaves). So it seems, that the Collocation was only done with the two source images i uploaded at the beginning, but 40 times (The ammount of Products i the later uploaded in the Batch Processing. Where is the Problem here?
When i collocate the images individually it works for the Landsat 5 Images (but i would really like it if it would be possible with the Graph/ Batch processing, because i guess thats what it’s made for )

BUT when i want to do the same for Landsat 8 Products, i get the Error:

org.esa.snap.core.gpf.OperatorException: Source product ‘2020.02.15’ must be geo-coded.
at org.esa.snap.core.gpf.Operator.ensureSceneGeoCoding(Operator.java:506)
at org.esa.snap.collocation.CollocateOp.initialize(CollocateOp.java:284)
at org.esa.snap.core.gpf.internal.OperatorContext.initializeOperator(OperatorContext.java:486)
at org.esa.snap.core.gpf.internal.OperatorContext.getTargetProduct(OperatorContext.java:273)
at org.esa.snap.core.gpf.Operator.getTargetProduct(Operator.java:387)
at org.esa.snap.core.gpf.GPF.createProductNS(GPF.java:330)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:305)
at org.esa.snap.core.gpf.GPF.createProduct(GPF.java:284)
at org.esa.snap.collocation.visat.CollocationDialog.createTargetProduct(CollocationDialog.java:70)
[catch] at org.esa.snap.core.gpf.ui.SingleTargetProductDialog.onApply(SingleTargetProductDialog.java:131)
at org.esa.snap.ui.AbstractDialog.lambda$initUI$6(AbstractDialog.java:519)
at javax.swing.AbstractButton.fireActionPerformed(Unknown Source)
at javax.swing.AbstractButton$Handler.actionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.fireActionPerformed(Unknown Source)
at javax.swing.DefaultButtonModel.setPressed(Unknown Source)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(Unknown Source)
at java.awt.AWTEventMulticaster.mouseReleased(Unknown Source)
at java.awt.Component.processMouseEvent(Unknown Source)
at javax.swing.JComponent.processMouseEvent(Unknown Source)
at java.awt.Component.processEvent(Unknown Source)
at java.awt.Container.processEvent(Unknown Source)
at java.awt.Component.dispatchEventImpl(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.LightweightDispatcher.retargetMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.processMouseEvent(Unknown Source)
at java.awt.LightweightDispatcher.dispatchEvent(Unknown Source)
at java.awt.Container.dispatchEventImpl(Unknown Source)
at java.awt.Window.dispatchEventImpl(Unknown Source)
at java.awt.Component.dispatchEvent(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$500(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.awt.EventQueue$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:159)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)

I don’t understand why the Landsat 8 images should have no geo-code? When i go over the image with the cursor, i see geo-coordinates. Is it in anyway possible that it looses its geo-code while subsetting? Then i tested to Collocate with the original Landsat 8 images. This seems to work first, but when i want to open a band of the Collocation Product, i find the following error:

org.esa.snap.core.jexp.ParseException: Undefined symbol ‘flags_M.designated_fill_M’.
at org.esa.snap.core.jexp.impl.ParserImpl.reportError(ParserImpl.java:1036)
at org.esa.snap.core.jexp.impl.ParserImpl.parseCallOrRef(ParserImpl.java:722)
at org.esa.snap.core.jexp.impl.ParserImpl.parsePrimary(ParserImpl.java:678)
at org.esa.snap.core.jexp.impl.ParserImpl.parsePostfix(ParserImpl.java:637)
at org.esa.snap.core.jexp.impl.ParserImpl.parseUnary(ParserImpl.java:624)
at org.esa.snap.core.jexp.impl.ParserImpl.parseMul(ParserImpl.java:530)
at org.esa.snap.core.jexp.impl.ParserImpl.parseAdd(ParserImpl.java:487)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseAnd(ParserImpl.java:461)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseXOr(ParserImpl.java:434)
at org.esa.snap.core.jexp.impl.ParserImpl.parseBitwiseOr(ParserImpl.java:408)
at org.esa.snap.core.jexp.impl.ParserImpl.parseComparison(ParserImpl.java:319)
at org.esa.snap.core.jexp.impl.ParserImpl.parseLogicalAnd(ParserImpl.java:286)
at org.esa.snap.core.jexp.impl.ParserImpl.parseLogicalOr(ParserImpl.java:259)
at org.esa.snap.core.jexp.impl.ParserImpl.parseConditional(ParserImpl.java:230)
at org.esa.snap.core.jexp.impl.ParserImpl.parseAssign(ParserImpl.java:179)
at org.esa.snap.core.jexp.impl.ParserImpl.parseTerm(ParserImpl.java:168)
at org.esa.snap.core.jexp.impl.ParserImpl.parseImpl(ParserImpl.java:150)
at org.esa.snap.core.jexp.impl.ParserImpl.parse(ParserImpl.java:136)
at org.esa.snap.core.jexp.impl.ParserImpl.parse(ParserImpl.java:115)
at org.esa.snap.core.dataop.barithm.BandArithmetic.parseExpression(BandArithmetic.java:116)
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:106)
Caused: java.lang.IllegalArgumentException: expression: Undefined symbol ‘flags_M.designated_fill_M’.
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:108)
at org.esa.snap.core.image.VirtualBandOpImage.parseExpression(VirtualBandOpImage.java:94)
at org.esa.snap.core.datamodel.VirtualBand.createSourceImage(VirtualBand.java:259)
at org.esa.snap.core.datamodel.Mask$BandMathsType.createImage(Mask.java:251)
at org.esa.snap.core.datamodel.Mask.createSourceImage(Mask.java:128)
at org.esa.snap.core.datamodel.RasterDataNode.getSourceImage(RasterDataNode.java:2162)
at org.esa.snap.core.layer.MaskLayerType.createMultiLevelSource(MaskLayerType.java:81)
at org.esa.snap.core.layer.MaskLayerType.createMultiLevelSource(MaskLayerType.java:77)
at org.esa.snap.core.layer.MaskLayerType.createLayer(MaskLayerType.java:63)
at org.esa.snap.core.layer.MaskLayerType.createLayer(MaskLayerType.java:52)
at org.esa.snap.ui.product.ProductSceneImage.createMaskCollectionLayer(ProductSceneImage.java:338)
at org.esa.snap.ui.product.ProductSceneImage.getMaskCollectionLayer(ProductSceneImage.java:184)
at org.esa.snap.ui.product.ProductSceneView.getMaskCollectionLayer(ProductSceneView.java:1242)
at org.esa.snap.ui.product.ProductSceneView.setMaskOverlayEnabled(ProductSceneView.java:619)
at org.esa.snap.ui.product.ProductSceneView.(ProductSceneView.java:272)
[catch] at org.esa.snap.rcp.actions.window.OpenImageViewAction$1.done(OpenImageViewAction.java:223)
at javax.swing.SwingWorker$5.run(Unknown Source)
at javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.run(Unknown Source)
at sun.swing.AccumulativeRunnable.run(Unknown Source)
at javax.swing.SwingWorker$DoSubmitAccumulativeRunnable.actionPerformed(Unknown Source)
at javax.swing.Timer.fireActionPerformed(Unknown Source)
at javax.swing.Timer$DoPostEvent.run(Unknown Source)
at java.awt.event.InvocationEvent.dispatch(Unknown Source)
at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
at java.awt.EventQueue.access$500(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.awt.EventQueue$3.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
at java.awt.EventQueue.dispatchEvent(Unknown Source)
at org.netbeans.core.TimableEventQueue.dispatchEvent(TimableEventQueue.java:159)
at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
at java.awt.EventDispatchThread.run(Unknown Source)

So where is that Problem with the Landsat 8 images?

Any help would be really apreciated because i really don’t know how to go on!
Thanks in Advance
Paul

[ABraun]

Maybe screenshots help to viaualize your work so far, so we can think of ot a bit better.

[pbrueck1]

Ok sure. So this is the Watermask i’ve made from a Landsat 8 image.

It’s a product just containing the Watermask as a Band ‘Wassermaske’ (with values 0 = Land and 255=Water) and the mask itself.

Then i uploaded my 40 Subsets and built a graph, looking like this:

In the Product Set Reader, i now uploaded all of my Subsets. Then i saw the Error “Source Product ‘2020.02.15’ must be geo-coded”. This is a Landsat 8 Subset and there is the same Error for all Landsat 8 Subsets (Landsat 8 Subsets i just named after their date, Landsat 5 are always named, for example: 1985.01.06_subset_LT05) . So here is the first Problem, i don’t understand where the Problem with the geocodation could be. As i said, the Landsat 8 images show coordinates when going over the image with the cursor. However, to go on i deleted all Landsat 8 images, so i only had Landsat 5 left. Then the error disappeared.

In the Collocation Window i choose this settings:

Then i saved this graph and changed to the Batch Processing. Here i uploaded all my Landsat 5 Subsets again and in the collocation window i choose this settings:

So the result were 30 Products (the amount of Landsat 5 images) named like the Products before, all having the same content like this:

So the “collocation” worked for the landsat 5 images to the extent that they were all cropped together into one product to fit the water mask. And that 30 times.

As I described in the message above, I first uploaded in the ProductSet Reader only two products, the water mask and one of the landsat 5 scenes. I then saved this, uploaded it to Batch Processing, and inserted all products here. The result were 30 identical products, but which did not contain all products, but only the water mask and the other product I had loaded into the ProductSet Reader.

It seems that the ProductSet Reader determines which products are collocated at all, and the ones I upload in Batch Processing specify how often this should happen.

What I want, however, is to simply “add” the water mask to the Landsat products one at a time, and not write them all into one. What do I need to set up to make this work? And of course, what could be the problem with Landsat 8? Thanks for your help!!

[ABraun]

thank you, that helped to understand the issue.
Please check if you can replace the collocation operator by “create stack” in the radar coregistration menu. You have to select “Product geolocation” instead of “orbit” as initial offset method.

[pbrueck1]

Thanks for the quick Answer!
I tried this before, and had the Problem that i could not choose the master, it was just set. Now i did it again, and, i don’t know why, it automatically choose the Watermask. When clicking ‘Find optimal Master’ i saw this Error:

However, when applying this Graph in Batch Processing, the Result was simmilar to Collocation.
I have 30 products, all with the same content: in each, all 30 products cut to the water mask.

[ABraun]

have you exported the mask as BEAM DIMAP before selecting it in the coregistration module?
The product on top of the list is used as master.

It’s strange that the images get cropped to the mask.

[pbrueck1]

Yes. I’ve made the mask from a Product i’ve worked with. Then i made the mask to a band. Then i made a band subset containing only the mask band and saved this as .dim.

“It’s strange that the images get cropped to the mask.” I’m not sure if i explained it correctly. I want the images cropped to the extend of the mask image, to be able to use the mask on other images. Because for this purpose, the images have to have exactly the same extend right?
So this is what eather Collocation or Stack Tools do. This is not my problem. I don’t want all of my products to be written in one, and that 30 times. I just want to be able to be apply the mask to all of my Products, to be able to continue working with this for Emissivity calculation. And the main Problem is the issue with landsat 8 geo-referenciation. Because for the landsat 5 images, i could collocate/stack them to the mask extend individually, it would just be unnecessary work i guess. But with the Landsat 8 images, i can’t work at all.
I’m quite new to snap and remote sensing at all, maybe it’s a stupid mistake by me. But i did subsets for all of my images exactly the same way (zoomed in, right clicked ‘spatial subset from view’, and then ‘ok’). And while there is no issue with Landsat 5, all the landsat 8 images seem to have lost their geo-coordination…

[ABraun]

Now I’m even more confused.

Can we simplify it like this: You have several Landsat-8 and Landsat-5 products and want to remove all of each product’s pixels outside the mask?

[pbrueck1]

Yes, you could put it that way. What I really want is just to be able to apply the water mask to my various Landsat-8 and 5 products.

[ABraun]

So this is how I would approach it.

1. Download Landsat-5 and Landsat-8 data
2. Create the mask and save it as a separate one-band product as BEAM DIMAP (mask.dim)
3. Define a subset geometry using the digitizing tool
4. Retrieve its WKT (right-click > WKT from geometry)
5. Create a graph for the subsetting based on geographic coordinates and the WKT string (remove the line with the reference band from the XML because these are inconsistent beween L8 and L5)
6. Run the graph

This at least gives a number of subsets with the same extent which all have geocoding.
I also fail with automating the pairwise stacking of the mask with the Landsat bands - no matter how I do it: The result is always written into the same product because the dynamic adjustment of the output product based on the Lansdsat input per iteration is not working
I think the batch processing is not made for coregistration of multiple inputs with one reference product (maybe @marpet has an idea).

But this should at least help you to over come the first problem with the missing georeferencing.

[pbrueck1]

Ok, thank you really much, i will try it that way.

[pbrueck1]

I don’t get how to remove the reference band from the XML, and therefore i get this error:

When making the graph, in the subset window there was no possibility to delete it. Then i saved the graph and opened the xml file. Here i removed the line with the reference band and saved it. In the batch processing tool, i loaded the graph and also tried to delete the reference band here, but it was not posible.

[ABraun]

Don’t remove the entire line, just the entry between > and < (here: radiance_1)

[pbrueck1]

Ok i did this but still get the same error. When i load the edited graph in the Batch Processing, i still see the referance band:

[pbrueck1]

I now made subsets this way two times, one for Landsat-5 and one for -8 Products. So i could use the reference band which fits for them and this worked… But i still have the same Problem with Landsat 8 Data.

When i use the Collocation Tool to collocate the mask with one of the Landsat-8 Subsets, i still have the ‘must be geo-coded’ error:

When i collocate the master with one of the original Landsat 8 Products, it first seems to work, but when i want to open a band of the result Product, the following Error shows up:

[ABraun]

Have you downloaded Collection1 Level 1 products?

You can open them when you remove the valid pixel expression in the band properties.

[pbrueck1]

Yes i did. Valid pixel expression is empty, for Landsat 5 as for Landsat 8

[ABraun]

This doesn’t make sense to me. Are you using the latest version of SNAP?

[pbrueck1]

I tried with Snap 7 and 8, for both versions its the same.

[ABraun]

would you please try Collection 1 Level 2 (Surface reflectance) for one product as a test?