Writing Python plugin

Hi Oana,
Thank you for the useful tips! Unfortunately not everything seems to be working.

Q2. If I put any of those elements under <operator> I get “Unexpected Exception: Could not initialize class org.esa.snap.python.gpf.PyOperatorSpi” when starting SNAP and cannot open the operator.

Q3. I cannot find my operators under Add -> Tools. Is it because operatorClass is org.esa.snap.python.gpf.PyOperator and not org.esa.snap.core.gpf.operators.tooladapter.ToolAdapterOp? Is there a way to add org.esa.snap.python.gpf.PyOperator to graphs?

Q5. java.ioFile works fine but if I use $USERPROFILE in the I get an “Unexpected Exception”. However this is not a critical issue for me.

Q8. <autoWriteSuppressed>false</autoWriteSuppressed> works (i.e. I do not get an exception when starting SNAP) but in the operator dialog there is still Target Product area on the I/O tab. Is there a way to remove the Target Product selection from this tab? Alternatively, can I somehow limit the output format of the Target Products (e.g. just to NetCDF).

Thank you!
Rado

Hello Rado,
You are right, there are several differences for python.
Do you have access to the following guide:
https://senbox.atlassian.net/wiki/spaces/SNAP/pages/42041346/How+to+write+a+processor+in+Python
?
If not, you can find attached the PDF export of this page.SNAP-HowtowriteaprocessorinPython-190819-1518.pdf (75.4 KB)
Here you will also find how to make the plugin visible in graph builder (by adding a org.esa.snap.python.gpf.PyOperatorSpi in META-INF/services).
Best regards,
Oana

Hi Oana,

Thank you for the links. I have access to the guide but still was not sure what is the purpose of org.esa.snap.python.gpf.PyOperatorSpi. However, I have the the directory and file structure as described in the guide, including org.esa.snap.python.gpf.PyOperatorSpi and still the operators do not appear in the GPF graph builder. Does the content of PyOperatorSpi look correct?

There is no mention of <progressPattern>, <errorPattern> or <autoWriteSuppressed> in the Python guide. Does this mean that those parameters do not work with Python processors?

Best regards
Rado

Hi Rado,
PyOperator uses DefaultOperatorDescriptor, which contains less fields than ToolAdapterOperatorDescriptor.
However, among the 3 fields you mentioned, autoWriteSuppressed is available in both types of descriptors (so it’s available for PyOperator)
Regarding your org.esa.snap.python.gpf.PyOperatorSpi file, it looks OK…

Best regards,
Oana

Hi Oana,
Does this mean that there is no way to display progress and errors to the users, in a similar mechanism to <progressPattern>, <errorPattern>? And what does <autoWriteSuppressed> do? I do not see any difference when this property is set to False or True in the behavior of the operator? Finally, would you know where I could look for a reason for why the operators do not appear in the graph builder even though PyOperatorSpi is fine? Is there any log where a relevant error could be found?
Thank you!
Rado

Hello Rado,
I build the nbm for your plugin, I installed it in SNAP, there was no error in the log.
I see the operators in SNAP menu. Also, they are available in gpt (open a cmd window and type “gpt -h”), but not exaclty under the names declared in org.esa.snap.python.gpf.PyOperatorSpi.

So, even if they are not available in Graph Builder (for which I asked further details to Marco), you can use them inside a graph and execute the graph in gpt.
Have you tried it already?
Best regards,
Oana

Hi Oana,
Thank you for testing this! I did not try looking for the operators in the command line interface of GPT since I did not find them in the in the GUI. I will do some testing through the command line but still hope that there is a way to make them appear in the GUI.
Regards
Rado

Hi Rado,
Maybe @obarrilero or @marpet can shed some light on whether/how external python operators can appear in Graph Builder.
Best regards,
Oana

Yes, I can. It is a known issue of the GraphBuilder.
https://senbox.atlassian.net/browse/SNAP-277
So, for the time being they don’t show up in the GraphBuilder.

Thanks for the clarification @marpet. Would you or @obarrilero have any tips regarding other issues listed at the top of this thread?

I just checked which questions were not yet answered.

Progress reporting is currently not supported, maybe there is some default progress indicator, but I’m not sure. Validation messages are shown automatically to the user.
The parameters Oana mentioned do only work if you use the standard tools adapter, but nor if you only implement a python operator.

Load the class java.lang.Runtime via jpy.get_type()
and then call:
runtime = Runtime.getRuntime()
total = runtime.totalMemory()
free = runtime.freeMemory()
max = runtime.maxMemory()
numProc = runtime.availableProcessors()

See also: https://docs.oracle.com/javase/8/docs/api/java/lang/Runtime.html

This is currently not possible. But you can set a value set of names, from which the user can select from.

As Oana already said, some validation is already done. If you want to throw your own exception you can throw a RuntimeError.

If you like to look at an example. Check this repository: https://github.com/senbox-org/snap-rut

Now, I go on holiday. See you in three weeks :beach_umbrella:

Hi again,

In the end we decided to switch to ToolAdapterOp approach. This is mainly because in PyOperator there seems to be no way to report progress to the user (as confirmed by @marpet) and this is critical functionality.

I am wandering now how to set target product in ToolAdapterOp. All the adapters in https://github.com/senbox-org/s2tbx/tree/master/s2tbx-sta-adapters are setting the output as one of the parameters. I tried the following in descriptor.xml but still there is no output in the I/O Parametrs tab of the operator dialog:
<targetPropertyDescriptors>
<org.esa.snap.core.gpf.descriptor.DefaultTargetPropertyDescriptor>
<name>Output</name>
</org.esa.snap.core.gpf.descriptor.DefaultTargetPropertyDescriptor>
</targetPropertyDescriptors>

Is there an example or instructions somewhere on which tags to use?

Thanks
Rado

Hi,
@oana_hogoiu, @marpet, @obarrilero could you please assist me with the question from the previous post?

In the meantime I got one more question. Using the <rasterDataNodeClass>org.esa.snap.core.datamodel.Band</rasterDataNodeClass> tag for a parameter does not link it to any source products of the operator. When used with a Python operator it would link the parameter to the first source product. Is there a way to automatically generate a drop down list of available bands in ToolAdapterOp?

Regards
Rado

Hello Rado,
Regarding the output parameter, the only way to do it is using a parameter having its name “targetProductFile”.
This is a default parameter:

Also, in ToolAdapterConstants you can see a constant with this value, which is after used in finding the output parameter (by its name).

As for the second question, from the code that I see inside ParameterDescriptorFactory (lines 195-208), it should detect the bands of the first source product and fill the value set for that parameter with the list of bands. You can try it with org.esa.snap.core.datamodel.Band.class (I see it used like this in the annotated operators)

Best regards,
Oana

Hi Oana,
Thank you for the tip. Indeed naming a parameter as targetProductFile and setting <isHandlingOutputName> to false results in Target Product appearing on the I/O Parameters tab. But it also raises other issues.

The output parameter also appears in the Processing Parameters tab. So where should it be set and how do the two outputs relate? Target Product is always set to be saved in temp directory and in DIM format (is there a way to change this dynamically) while in the parameters tab I can control both the output path and the file format. Is there documentation which explains the behavior?

Related to the previous question regarding <rasterDataNodeClass>, do I have to specify the source product explicitly? I noticed that while previously (in Python Operator) I could write something like
<valueSet>Sentinel-2,Sentinel-3</valueSet>
in ToolAdapterOp this has to be written as
<valueSet>
<string>Sentinel-2</string>
<string>Sentinel-3</string>
</valueSet>
So I am wondering if there is a similar change in syntax for raster data node.

Regards
Rado

Hi again,
Just a bit more information on setting rasterDataNodeClass.

When I set it to org.esa.snap.core.datamodel.Band.class, the following error is reported when SNAP is starting the and operator dialog does not load:

When I set it to org.esa.snap.core.datamodel.Band, the operator dialog loads but there is no connection between the source products and the band parameter (i.e. the drop down list is always empty).

Regards
Rado

Hello Rado,
Using rasterDataNodeClass, the value set is dynamically filled.
If you want to set fixed values in the xml descriptor, you can specify the values as in your example (string array of values).
The handling of the output parameter should be done in an “after/post” velocity template.
Regarding the error “No such field MemoryTemplate.file” maybe you have an error in your template? Are you using this object in your template? (there is no such field indeed…)

Best regards,
Oana

Hi Oana,

The error appears when I change <rasterDataNodeClass>org.esa.snap.core.datamodel.Band</rasterDataNodeClass> to <rasterDataNodeClass>org.esa.snap.core.datamodel.Band.class</rasterDataNodeClass>. I made no other changes to descriptor.xml or the VM template.

When using <rasterDataNodeClass>org.esa.snap.core.datamodel.Band</rasterDataNodeClass> no error appears but the value set of the parameter is not filled dynamically with the bands of the first source product. It remains empty.

Regards
Rado

Can you try debugging through ParameterDescriptorFactory lines 190-208 to see if the rasterDataNodeClass is considered and correctly retrieved?
(perform the debug after installing your plugin in SNAP, and open the UI for your plugin from SNAP menu)

Hi,
Unfortunately I have no experience in debugging with IntelliJ and we have little time to get the plugin out. So for now we will leave the band parameters as strings.

We are now trying to package and distribute the plugins and are basing ourselves very closely on the structure of the OTB STA [1]. Therefore, I created an s2tbx-senet-adapters-kit [2] which has a pom.xml listing all the other adapters as dependencies and packaging set to “nbm”. I was hoping that this would create an .nbm file which contains all of the adapters and which would allow the plugin to be installed in SNAP through Tools > Plugins > Downloaded > Add Plugins… However when I select s2tbx-senet-adapters-kit.nbm nothing happens, no information is even displayed in the right panel of the Plugins dialog. When I select an .nbm for another STA (e.g. energy-fluxes [3]), it displays some information of the right panel but when I click Install it get to the state where it prints “unpacking org.esa.snap.core.gpf.operators.tooladapters.energy-fluxes” but than I get the dialog shown below:
image.

Could you please check if there is anything wrong in my pom.xml files or in the directory structure. The only major difference (when it comes to pom.xml and directory structure) between this plugin and the OTB one is that the manifest.mf of s2tbx-senet-adapters-kit is located in a different directory because a change in version 3.0.0 of Maven JAR Plugin [4] stopped the plugin from compiling if I used useDefaultManifestFile tag as OTB does.

Thanks

[1] https://github.com/senbox-org/s2tbx/tree/master/s2tbx-sta-adapters/orfeotoolbox
[2] https://github.com/DHI-GRAS/sen-et-snap-sta/tree/master/s2tbx-senet-adapters-kit
[3] https://github.com/DHI-GRAS/sen-et-snap-sta/tree/master/energy-fluxes
[4] https://maven.apache.org/plugins/maven-jar-plugin/