Hello guys!
I want to use batch processing in windows from the gpt command line to merge two different products. I arrange everything, but when I try to merge one product that is .dim and one product that its .SAFE, simply it won’t work. I don’t now why, it won’t give me any errors or something like that.
When there are two products that are in .SAFE format everything works just fine.
Below I’ll describe and write the whole process.
THIS IS THE WINDOWS BATCH
::@echo off
:: enable delayed expansion - used to resolve variable in loop
:: variable has to be used with '!' instead of '%'
setlocal ENABLEDELAYEDEXPANSION
::::::::::::::::::::::::::::::::::::::::::::
:: User Configuration
::::::::::::::::::::::::::::::::::::::::::::
:: adapt this path to your needs
set gptPath="C:\Program Files\snap\bin\gpt.exe"
::::::::::::::::::::::::::::::::::::::::::::
:: Command line handling
::::::::::::::::::::::::::::::::::::::::::::
:: first parameter is a path to the graph xml
set graphXmlPath=%1
:: use third parameter for path to source products
set sourceDirectory=%2
:: if sourceDirectory ends with '\' remove it
if %sourceDirectory:~-1%==\ set sourceDirectory=%sourceDirectory:~0,-1%
:: use third parameter for path to source products
set sourceDirectory2=%3
:: if sourceDirectory ends with '\' remove it
if %sourceDirectory2:~-1%==\ set sourceDirectory=%sourceDirectory2:~0,-1%
:: use third parameter for path to target products
set targetDirectory=%4
:: if targetDirectory ends with '\' remove it
if %targetDirectory:~-1%==\ set targetDirectory=%targetDirectory:~0,-1%
set targetFilePrefix=L1C_IDEP_
:: Create the target directory
::md %targetDirectory%
::::::::::::::::::::::::::::::::::::::::::::
:: Main processing
::::::::::::::::::::::::::::::::::::::::::::
:: double '%' in batch file and only a single '%' on command line
:: '/D' is for directories like Sentinel data. Remove '/D' when you open files.
for /D /R %sourceDirectory% %%F in (S2*.SAFE) do (
for /D /R %sourceDirectory2% %%N in (*.dim) do (
echo
:: '~fF' means abolute path of 'F'
set sourceFile=%%~fF
set sourceFile2=%%~fN
echo sourceFile2=%sourceDirectory2%\%%~fN
echo Processing !sourceFile!
:: '~nF' means filename without extension of 'F'
set targetFile=%targetDirectory%\%targetFilePrefix%%%~nF.dim
set procCmd=%gptPath% %graphXmlPath% -e -t "!targetFile!" "!sourceFile!" "!sourceFile2!"
call !procCmd!
))
HERE IS THE .XML GRAPH FOR MERGING BANDS FROM DIFFERENT PRODUCTS AND ALSO RESAMPLE THE NEW PRODUCT IN THE END
Hi @abruescas. I tried with this approach because I had 2 outputs from previous processing, one in .dim (resulted from IDEPIX operator) and one as a .SAFE (resulted from SEN2COR280 operator).
I found a intermediate solution that it will be posted as soon as I have all the final results, it seems that it works for now. If it not, I will try the approach that you proposed it, that with 2 .dim files. Thank you for your kind suggest!
Hello again @abruescas. I return with news regarding my issue. Now if I want to try this processing chain with to .dim files the batch won’t start. It didn’t pops any error or something like that. On SAFE file it just works fine, also I removed the steps that are especially need files in SAFE format, like SEN2COR for example. You have any ideas at what can cause this? I am out of ideas.
you start with only two products and using the command line, something like this (this example is just to process S3-OLCI with C2RCC):
./gpt /Users/name/Desktop/SNAP_Forum/xml/c2rcc_S3.xml -Sinput=/Users/name/Desktop/IOPS/S3B_OL_1_EFR____20200728T091115_20200728T091415_20200729T134511_0179_041_321_2160_MAR_O_NT_002.SEN3 -t /Users/name/Desktop/IOPS/S3B_OL_1_EFR____20200728T091115_20200728T091415_20200729T134511_0179_041_321_2160_MAR_O_NT_002_c2rcc.dim
The pattern should be similar, but it will change for the merge node. Try first with only two products in .dim. Once you have this running, we continue.
By the way, I am using mac, for Windows you will have to add “name_of_product” and probably the path for the gpt too.
I tried to use one atmospheric corrected .dim file for a resample test graph, using a modified batch file (as the one described in the beginning of the post). Neither any result. If the same batch file is modified to read .data files (which I know that is wrong) the process begins, but after a couple of seconds will pop an error with the fact that “No product reader is found” (which is obvious). I am helpless here, I really don’t know what It can be.
If the batch processing is not working, we need to identify the problem fisrt. It could be the products yourare using or the writing of the xml, the bash file…Many things. Start from the beginning. Please try the merge function within the command line and then we will see. I send you here an example with a S3OLCI. I apply Idepix.Olci to the original S3OLCI, then I apply c2rcc.olci again to the original scene, and later I merge both products. My order in Windows would look like this (Command line):
Hi again. From the command line with an approach that you suggested (an XML from your model), everything went smooth and i got the desired result, a product formed by the 2 input products. I really want to see the problem but I can’t see it. I thought that maybe if the inputs were multiple dims files in one folder that will be the issue, but I tested it with just one input per folder. Also, I really don’t think the batch is the problem, because it will work with any file format (as I described above, even with the .data format, which is complementary to the dim data).
Just in case I upload the windows batch file and the .XML file.
I would suggest to set your directories in every parameter (set graphXmlPath="C:\user\my_xml_path*.xml). I send attached one .bat file that has worked properly:
What I usually do is to copy my command line in a text file and change the input and output names of each row. If you save it as .bat, it should also run properly. Ex. in .sh (for mac) below. It s more “manual” but it really does the work.
Good morning @abruescas. First of all I want to thank you and @marpet for your time spent to answear my questions. Your last solution, that with each row passed manually did the job for two .dim files, my solution worked for 2 SAFE files. The problem with .SAFE files it is that it got the IDEPIX process that purely chokes the processor (on a I7 8600 with 3.6 GHz and 64 GB RAM unit). That is happening when I write the final product, even if the first product it’s written it will jam at 10% (after 2 days it was still on 10% with no others processes running). I think this issue is from the fact that the IDEPIX process is running virtually (or somehow this is what I think because with the .dim file of IDEPIX and .dim file of the atmospherically corrected product everything runs smoothly, with barely 20 GB of RAM and something like 30% of the processor occupied ).
I hope that my explanation was clear for you. I will post the batch for use with the merge operator, to help others who have the same questions as I have when I began this step.
Yes, Idepix has some issues. Especially the performance is not good for S2.
We are working on this.
But maybe you issue is already solved. Do you still use the S2 Idepix v8.0.0?
Current is v8.0.2. In version 8.0.1, we fixed an issue which caused Idepix to freeze. Release S2MSI v8.0.1 update · bcdev/snap-idepix (github.com)
Greetings Marco,
I use IDEPIX v8.0.2, but the freezing probem is only when i got the IDEPIX operator as an intermediat process in a .xml graph. I think there is the problem, somehow. If the IDEPIX operator is last in the process (I save the product with pixel_classif_flags band) everything works properly, a little slower but it gets the job done. I really don’t know why it does that, but I get my job done in a way.
Thank you very much for your kind answear.
