Failed to import geometry with particular shapefiles when using EODAG and xml/config files

craigmpearce1 · February 11, 2022, 8:22am

Describe the bug
I run the code provided below (which is derived from the code provided by the website given below) with the attached config file in Pycharm on Linux OS and everything works fine when I use the shapefile provided by the website (iho.shp) or a similar shapefile I made (Trieste_test.shp). But when I try to use a different shapefile, I get an error which suggests that the shapefile does not exist when in fact it does. Can you tell me if this is something I am doing wrong or if there is a bug in the source code? This issue was originally reported on the EODAG Github website (also provided below) and they recommended reporting the issue here

Attached is a txt copy of the .yml config file and zip folders containing the shapefiles and the xml files

eodag.txt (5.1 KB)
xmls.zip (1.5 KB)
shps.zip (3.9 MB)

Thank you in advance

Kind regards

Craig Pearce

https://eodag.readthedocs.io/en/stable/notebooks/tutos/tuto_ship_detection.html

Code To Reproduce
import os
from eodag.api.core import EODataAccessGateway
from eodag import setup_logging

Set working directory

base_wd = “/home/craig/PycharmProjects/S1_image_processing/”
os.chdir(base_wd)

Make output verbose

setup_logging(verbose=3)

Create the workspace folder.

workspace = ‘eodag_workspace_shipdetection’
if not os.path.isdir(workspace):
os.mkdir(workspace)

Add absolute path to SNAP bin folder to make the gpt command available

os.environ[“PATH”] = “/home/craig/snap/bin” + “:” + os.environ[“PATH”]

Start EODAG using config file

dag = EODataAccessGateway(os.path.join(workspace, ‘eodag.yml’))

Set product type and extent

product_type = ‘S1_SAR_GRD’
extent = {
‘lonmin’: 13.054504,
‘lonmax’: 13.808441,
‘latmin’: 45.475540,
‘latmax’: 45.762733
}

Set dates and get list of products

products, estimated_nbr_of_results = dag.search(
productType=product_type,
start=‘2021-05-01’,
end=‘2021-05-03’,
geom=extent
)
products

product = products[1]

product_path = product.download()
product_path

Set up xml files

Set up then run gpt

Set the environment - variable LD_LIBRARY_PATH to the current directory

os.environ[‘LD_LIBRARY_PATH’] = ‘.’

Set names for vector file

vector_file = os.path.join(workspace, ‘ne_10m_ocean’, ‘ne_10m_ocean.shp’)
vector_file_short = ‘ne_10m_ocean’

Run gpt

!gpt ‘eodag_workspace_shipdetection/Subset.xml’ -Pvectorfile={vector_file}

!gpt {graph_process} -Pinputproduct={os.path.join(workspace, ‘S1A_IW_GRDH_1SDV_20170501T165831_9CD1_Orb.dim’)} -Pvectorfileshort={vector_file_short} -Poutputproduct1={os.path.join(workspace, ‘S1A_IW_GRDH_1SDV_20170501T165831_9CD1_processed’)} -Poutputproduct2={os.path.join(workspace, ‘S1A_IW_GRDH_1SDV_20170501T165831_9CD1_processed’)}

Test whether shapefile exists

os.path.isfile(’/home/craig/PycharmProjects/S1_image_processing/eodag_workspace_shipdetection/ne_10m_ocean/ne_10m_ocean.shp’)

Output

Backend TkAgg is interactive backend. Turning interactive mode on.
2022-01-31 13:11:39,761-15s eodag.config                     [INFO    ] (config           ) Loading user configuration from: /home/craig/PycharmProjects/S1_image_processing/eodag_workspace_shipdetection/eodag.yml
2022-01-31 13:11:39,827-15s eodag.core                       [DEBUG   ] (core             ) Opening product types index in /home/craig/.config/eodag/.index
2022-01-31 13:11:39,832-15s eodag.core                       [INFO    ] (core             ) Locations configuration loaded from /home/craig/.config/eodag/locations.yml
2022-01-31 13:11:40,333-15s eodag.core                       [INFO    ] (core             ) Searching product type 'S1_SAR_GRD' on provider: peps
2022-01-31 13:11:40,333-15s eodag.core                       [DEBUG   ] (core             ) Using plugin class for search: QueryStringSearch
2022-01-31 13:11:40,333-15s eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Mapping eodag product type to provider product type
2022-01-31 13:11:40,333-15s eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting provider product type definition parameters for S1_SAR_GRD
2022-01-31 13:11:40,333-15s eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Getting provider product type definition parameters for S1_SAR_GRD
2022-01-31 13:11:40,333-15s eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Building the query string that will be used for search
2022-01-31 13:11:40,333-15s eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Retrieving queryable metadata from metadata_mapping
2022-01-31 13:11:40,334-15s eodag.plugins.search.qssearch    [INFO    ] (qssearch         ) Sending count request: https://peps.cnes.fr/resto/api/collections/S1/search.json?startDate=2017-05-01&completionDate=2017-05-02&geometry=POLYGON ((13.0545 45.4755, 13.0545 45.7627, 13.8084 45.7627, 13.8084 45.4755, 13.0545 45.4755))&productType=GRD&maxRecords=1&page=1
2022-01-31 13:11:41,471-15s eodag.plugins.search.qssearch    [INFO    ] (qssearch         ) Sending search request: https://peps.cnes.fr/resto/api/collections/S1/search.json?startDate=2017-05-01&completionDate=2017-05-02&geometry=POLYGON ((13.0545 45.4755, 13.0545 45.7627, 13.8084 45.7627, 13.8084 45.4755, 13.0545 45.4755))&productType=GRD&maxRecords=20&page=1
2022-01-31 13:11:42,331-15s eodag.plugins.search.qssearch    [DEBUG   ] (qssearch         ) Adapting 2 plugin results to eodag product representation
2022-01-31 13:11:42,361-15s eodag.core                       [INFO    ] (core             ) Found 2 result(s) on provider 'peps'
0.00B [00:00, ?B/s]
S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1: 0.00B [00:00, ?B/s]2022-01-31 13:11:42,363-15s eodag.plugins.download.base      [INFO    ] (base             ) Download url: https://peps.cnes.fr/resto/collections/S1/46d48388-c2c0-5fdb-b3c9-aadafeeedf76/download
2022-01-31 13:11:42,363-15s eodag.plugins.download.base      [INFO    ] (base             ) Product already downloaded: /home/craig/Downloads/S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1.zip
S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1: 0file [00:00, ?file/s]2022-01-31 13:11:42,363-15s eodag.plugins.download.base      [INFO    ] (base             ) Extraction cancelled, destination directory already exists and is not empty: /home/craig/Downloads/S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1
S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1:   0%|          | 0/1 [00:00<?, ?file/s]
S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1: 100%|██████████| 1/1 [00:00<00:00, 4777.11file/s] 
2022-01-31 13:11:42,364-15s eodag.api.product                [DEBUG   ] (_product         ) Product location updated from 'https://peps.cnes.fr/resto/collections/S1/46d48388-c2c0-5fdb-b3c9-aadafeeedf76/download' to 'file:///home/craig/Downloads/S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1/S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1.SAFE'
2022-01-31 13:11:42,364-15s eodag.api.product                [INFO    ] (_product         ) Remote location of the product is still available through its 'remote_location' property: https://peps.cnes.fr/resto/collections/S1/46d48388-c2c0-5fdb-b3c9-aadafeeedf76/download
S1A_IW_GRDH_1SDV_20170501T165831_20170501T165856_016391_01B235_1399
S1A_IW_GRDH_1SDV_20170501T165806_20170501T165831_016391_01B235_9CD1
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.3.2 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
....10%....20%....30%....40%....50%....60%....70%....80%....90% done.
INFO: org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterIO: Initializing external tool adapters
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Incompatible GDAL 3.3.2 found on system. Internal GDAL 3.0.0 from distribution will be used.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
INFO: org.esa.snap.core.util.EngineVersionCheckActivator: Please check regularly for new updates for the best SNAP experience.
INFO: org.esa.s2tbx.dataio.gdal.GDALVersion: Internal GDAL 3.0.0 set to be used by SNAP.
Executing processing graph
INFO: org.hsqldb.persist.Logger: dataFileCache open start
expression: Undefined symbol 'ne_10m_ocean'. due to Undefined symbol 'ne_10m_ocean'.
 done.
Error: [NodeId: Land-Sea-Mask] expression: Undefined symbol 'ne_10m_ocean'. due to Undefined symbol 'ne_10m_ocean'.
Out[2]: True

Environment:

Python version: 3.8.10 (default, Nov 26 2021, 20:14:08) \n[GCC 9.3.0]
EODAG version: version 2.3.4
PyCharm 2021.3.2 (Community Edition)
Build #PC-213.6777.52, built on January 28, 2022
Runtime version: 11.0.13+7-b1751.25 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Linux 5.13.0-27-generic
GC: G1 Young Generation, G1 Old Generation
Memory: 2048M
Cores: 12
Non-Bundled Plugins:
R4Intellij (213.5744.6)
Current Desktop: ubuntu:GNOME

marpet · February 11, 2022, 10:14am

Somehow the shapefile is strange. I can’t import into a product.
Even on a global product it fails and says the geometry is outside of the product bounds.
If the shapefile is not imported the referenced mask '‘ne_10m_ocean’ is not available.
In QGIS it can be imported but a message is shown that the layer has now CRS:

But even after setting the CRS manually to WGS84 and exporting it as a new shapefile it is not working.

@ABraun Do you have an idea?

ABraun · February 11, 2022, 10:33am

I have loaded it into QGIS and it looked fine. But I also couldn’t load it into SNAP. So I applied

Fix Geometries
Multipart to singlepart
in QGIS to create a new dataset.

I don’t know which of both steps fixed the problem, but the created polygon (now two features instead of one) works in SNAP: ne_10m_ocean_fix_singlepart.zip (3.0 MB)

craigmpearce1 · February 11, 2022, 11:18am

Hi ABraun,

This is just to say thank you for looking into this for me. With the code above, and with a separate script that uses similar xml files, the code runs successfully with the shapefile you sent

Kind regards

Craig

gnwiii · February 12, 2022, 2:36pm

Many SNAP versions ago I used gpt with the same .prj files as Natural Earth Coastline/Ocean shapefiles in a batch production system. The ne_10m_coastline.prj file is identical to the ne_10_ocean.prj referenced in this thread and to ne_10m_ocean_fix_singlepart.prj from ne_10m_ocean_fix_singlepart.zip. I do recall some having issues with multiple features when using “point in polygon” tools. Fix Geometries seems to be a necessary step even if not the complete solution.