Handling sentinel data in python ( working with snappy)

Sorry, no idea. And I guess something like the follow you have tried already:

But you could also invoke system commands to unzip the data. Maybe this works better.
Did you have success to unzip the data on the command line?

If you are on linux, there are a couple ways to mount a .zip file as a read-only filesystem.

I’m using this code:

dir_name = ‘path_to_images’
extension = ‘.zip’
namfil1=‘S2A_MSIL1C_’
namfil2=‘S1A_IW_SLC_’
namfil3=‘S2A_OPER_’
for item in os.listdir(dir_name): # loop through items in dir
if item.endswith(extension) and item.startswith(namfil1) or item.startswith(namfil2)or item.startswith(namfil3): # check for “.zip” extension
file_name = dir_name + “/” + item
zip_ref = zipfile.ZipFile(file_name) # create zipfile object
zip_ref.extractall(dir_name) # extract file to dir
zip_ref.close() # close file

I can unzip files of 5gb, but in files Sentinel 2, in format S2A_OPER_PRD_MSIL1C_PDMC, I can’t unzip, using images 700Mb or 5Gb.

Any idea?

I’m on windows, the problem required using this os. :frowning:

Any other idea?

Try it without unzipping. If you want to open standard S1 or S2 data just do

product = ProductIO.readProduct(productPath)

A post was split to a new topic: Intersection between geoJson and product

Hi Marco,

I’m re-sampling a Sentinel 2 product using the suggested operator. What I realize is that by using the operator in result = snappy.GPF.createProduct(‘Resample’, parameters, source) we are filling with data the hard drive. I’m using the resampled imagery in creating subsets for more than 10 imagery set, after creating the subsets I don’t need anymore the full re-sampled imagery resulted from using the ‘Resample’ operator. What I realized is that the createProduct function is, by default, filling the hard drive in the folder: C:\Users\The Specialist\AppData\Local\Temp\snap-The Specialist with the re-sampled imagery. I integrated the suggested method in a function used in a loop that is creating the imagery subsets set.
If I run for several times the script the HDD comes full.
Can you support me in dealing with the issue ? I would like to free the space from the hard drive after the re-sampling operation + subset-ing each imagery.

Looking forward to your reply.

George

I resampled one S2 product, but I don’t see resampled images in my …\AppData\Local\Temp\… directory.

I know that data is cached by the S2 reader in the directory <USER_HOME>.snap\var\cache\s2tbx\l1c-reader. This could fill up your disk and the handling of this cache needs to be improved. There are already issues for this in our bug tracker (SIITBX-277, SIITBX-288).
In SNAP Desktop it is possible to change the interval how often the cache is deleted (Tools / Options / S2TBX). But this will probably not help you.
This cached data can be cleaned by a function call. I guess the following will work.

from snappy import jpy
S2CacheUtils = jpy.get_type('org.esa.s2tbx.dataio.cache.S2CacheUtils')

S2CacheUtils.deleteCache()

If you call deleteCache() after every processed product you should get rid of this problem. If the reader-cache is the problem and not something else. Probably it would be good if you first call on the source product dispose().

1 Like

I’ll make a try in the following days and I will come back with an answer :slight_smile:

Hi Marco,

After a few days short break I’ve come back and start dealing again with the issue.

What I figure out was that by reading the *.zip file corresponding to the S2 imagery, the readProduct function was creating data in the folder …\AppData\Local\Temp\snap-user and that data was not removed. The dispose() method called on the product obtained by using readProduct did the trick and now the hard drive is free of unwanted data. The code is looking something like this:


S2CacheUtils = jpy.get_type(‘org.esa.s2tbx.dataio.cache.S2CacheUtils’)

product = ProductIO.readProduct(file_path)
- create product subset
- do something with product subset
product.dispose()
S2CacheUtils.deleteCache()

In this way it is possible to remove the cache from subset-ing operation and remove the unnecessary data obtained after reading the product.

I would like to know if there is a difference in this case if I use product.closeIO() instead of product.dispose().

Thanks again for the tip!

George

1 Like

Good to hear that it now works for you.

The method dispose() calls closeIO().

closeIO closes only the input/output stream and dispose releases all resources which have been allocated.

In the end the implementations of both methods often do the same. I would suggest calling always the dispose method.

1 Like

snappy give me none output either i give zip file or unzip file. checked in two sentinel images. Instead when i open the manifest.safe in snap software it show result…

This happens either when no reader is found or when the file name is invalid. It is weird when you can open the file in SNAP Desktop, though. Can you tell me the value of ‘path1’?

Here is :
‘D:\SENTINEL\S1A_IW_GRDH_1SDV_20181008T131251_20181008T131316_024045_02A0A4_4743\manifest.safe’

Perhaps could it be a typo in the path? Normally it should be
‘D:\SENTINEL\S1A_IW_GRDH_1SDV_20181008T131251_20181008T131316_024045_02A0A4_4743.SAFE\manifest.safe’
When you unzip it, the product folder should be [yourProductName].SAFE and if you are opening it compressed, then the path should be:
‘D:\SENTINEL\S1A_IW_GRDH_1SDV_20181008T131251_20181008T131316_024045_02A0A4_4743.zip’

try it too. but same problem. give path with distinct ways :confused:

Pity, the missing ‘.SAFE’ really looked like the problem. Anyway, if the .SAFE is missing, it won’t open.

Do you maybe have multiple installations of SNAP on your computer?

Can you open other S1 products with snappy? Or any other products for that matter?

yes i opened .geotif with snappy ,no error detected while reading geotif

Try this:

reader = ProductIO.getProductReader(‘SENTINEL-1’)
product = reader.readProductNodes(path1, None)

Is the reader None or is it an object?

You should use the path with double back-slash like ‘\\’, e.g. ‘x:\\your_dir\\imag_name.zip’

To correctly read the product you must have something like:

from snappy import ProductIO

product = ProductIO.readProduct( ‘x:\\your_dir\\imag_name.zip’)
band_names = product.getBandNames()
print (list(band_names))

If you have the correct python setup for snappy (python 3.4 !!! + snappy) then the above scratch should work.