I would be very interested in answers to these questions as well!
Many of the parameters are for the Java Virtual Machine. There are many web sites devoted to “Java Performance Tuning”, but the official site is a good place to start. Much of this information was published years ago so doesn’t cover the complexities of current multi-core NUMA platforms. There have been improvements in the garbage collection (GC), so there is more recent guide devoted to GC tuning
SNAP users with performance concerns will find it worth spending a few hours viewing this introductory tutorial helpful and then
talk by author of Java Performance Tuning web sit.
Large organizations devote many person-years to tuning performance of critical applications where a gain in performance means they can save big money on the hardware. There are Java Performance Tuning with Mission Control and Flight Recorder and the like courses for people doing this sort of work.
Applications like SNAP are used in many different ways so it is impossible to have simple recipes that work for everyone. I suspect the vast majority of Java applications are used for databases and transaction processing workloads, so much of the advice you can find on the internet may not apply to SNAP workloads.
SNAP users who have concerns over performance should start by learning a bit about how Java developers approach performance and how to use the instrumentation available from their OS to provide objective measurements and identify the bottlenecks that are responsible for performance problems. SNAP is a very complex system and is sure to have some bugs (memory leaks) so you want to be able to properly identify and report bugs to the developers. javaperformancetuning.com collects performance tweaks for Java, but to use it you need to know much more than just “my calculations take too long”.
Note that Java 8 won;t get updates after Jan. 2019, and Java 9 has many improvements in performance monitoring.
Thanks, gnwiii. This is definitely very useful information about Java performance tuning. Still I asked some very specific questions about SNAP and I didn’t get the answers:
- Is setting the flags in calling the gpt overriding the contents of gpt.vmoptions file ?
- The -c flag. What is the recommended option for tile cache size? How it is meant to be used respect to the available system memory? As close as possible to the -Xmx setting or something different?
- Can someone explain the tile caching mechanism in SNAP (how it is implemented), if possible then with drawings?
- Is it safe to keep the -x option always switched on? Or it can have some negative effects?
- What does setting the “-XX:+AggressiveOpts” parameter actually do?
If someone could answer those questions I would be very grateful.
First, your questions are useful because many others will be interested in the answers.
Many of your questions (the possible exception being #3) are in the category of “excercises left for the reader” or the parable “give a person a fish and that person is fed for a day, but teach a person to fish and that person has food for life”. In other words, the answers are less useful than knowing how to work then out for yourself.
Some insight into the gtp command can be gained by just looking at the gpt script. Here are some examples for a Unix (linux or macOS) installation:
$ file gpt gpt: POSIX shell script, ASCII text executable, with very long lines $ wc -l gpt 441 gpt
The best documentation is usually the source code, but on Windows, we only have
gpt.exe. If you are on Windows, look in the snap-installer unix bin directory. At this stage, it is useful to know a bit of POSIX shell scripting language. In practice, the majority of Unix systems use the
bash implementation. There are many introductory tutorials for
bash (including languages other than English). I recommend Linux Command. Windows 10 users can activate Windows Subsystem for Linux (WSL) and install any of the available (and free!) Linux distros from the “Store” to get
bash in Windows 10.
Once you feel comfortable reading simple shell scripts, you can examine the rather complicated
gpt script. You should discover a shell function called
read_vmoptions and also the following “very long line”:
$INSTALL4J_JAVA_PREFIX exec "$app_java_home/bin/java" [...] $INSTALL4J_ADD_VM_PARAMS [...] "$@"
If you are bit familiar with shell scripts, you can guess the answer to the first question, and devise experiments to check that your guess is correct. In practice,
gpt has become very chatty, so it won’t hurt to have it print a bit of extra detail to record values of options you are tweaking.
Question 2. I suspect the appropriate choice for
<cache size> depends on your workload and hardware environment, so the answer requires using JRE and host performance monitoring tools and experimentation.
Question 4. If you watched the tutorials, you should have some insights into how Java manages memory. It likes to have ample memory, so for those of us who don’t have ample budgets, I suspect this option gives Java a bit of help for cases where the “proper” fix might be to purchase more RAM. There are many other potential causes for “Memory problems”, so you should also consider Java performance monitoring tools.
Question 5. There is an explanation of what it is meant to do in various online sources, but also have a look at Java Bug Database to find what it has actually done in the past (I just check and found 22 bugs for
AggressiveOpts). Many compilers have similar options. Such options give performance improvements, but can also introduce (usually minor) changes in the results that make it harder to compare output on different hardware. One of the key applications I use has consistently given bit-for-bit identical results across Windows and Unix platforms for a test data set over a period approcahing 10 years. The result differ slightly when using aggressive optimizations and the performance gains small.
Hi @kaupovoormansik and thanks @gnwiii for the exhaustive explanations.
Some more comments from me (one of the developers of SNAP).
The options intended to be used for Java VM should go into vmoptions file. For example the Xms and Xmx options can’t be set on the command line. The specific gpt options like c and q must be given on the command line and can not be specified the vmoptions file.
But there are other ways to set tile cache size and the parallelisation.
In the etc folder of the SNAP installation directory you find the snap.properties file. Inside you finde sever properties to configure the behaviour of SNAP. The properties can also be specified in the
<userdir>/.snap/etc/snap.properties files. The properties int the userdir override the system properties in the installation directory. The same properties can also be specified on the command line when calling gpt, e.g.
A list of properties can be found in our wiki.
From my experience you can increase the cache size to a value of 75% of the max memory (Xmx) value. Depending on the actual processing the value might vary. In some cases it can be increased up to 90% and sometime a value of 60% is already the limit.
The caching mechanism is pretty simple. If the cache size reaches the limit, the oldest tiles are removed. I think, till a fill level of 70% is reached (not sure about this). That’s roughly how it works. The implementation is not ours. We are using JAI.
If you have enough memory to keep a whole line of tiles it will not have negative effects. But this might change from product type to product type. Some are bigger than others.
This only influence how the VM will try to optimise the compiler output. Several parameters of this type are described on this page. But I usually don’t care much about these very special settings.
I am trying to generate a DEM using SLC but I am facing some performance issues with gpt before getting a result. So far, I am trying to run this graph (in GPT) for the second sub-swath in one of the polarizatiosn
I have been reading on the forum about the different ways to set the configuration of GPT properly but still facing issues. I am using a Linux machine with 15 GB RAM
Here is my gpt.vmoptions
I have computed the VM parameters on the snap-conf-optimiser and paste them in the gpt.vmoptions to update the settings of GPT. The process starts but at some point it runs out of memory
Any idea on how to proceed?
Is this still an issue?
To which value have you set the tile cache? (the c option of gpt)
Could be that 15GB is not sufficient for this type of processing.
You could try to split your graph into three. 2 for applying the orbit files and the third for doing the remaining part.
We need to review and optimise the memory-usage and performance of S1TBX operators as they seem to be causing the most problems.
When setting the snap.jai.tileCacheSize in snap.properties,
gpt --diag shows 0 B for most of the settings. I’ve tried multiply different combinations…
eg. Setting snap.jai.tileCacheSize=20000 (which would be 20GB) shows:
Is that just a bug from GPT displaying the environment variables and the processing is done using 20GB cache size, or is the tileCacheSize disabled by error?
(Same thing happens when using SNAP GUI Performance tuning for setting the variables.)
When trying different values sometimes, gpt --diag shows eg. 1.5GB for snap.jai.tileCacheSize = 22000. I can’t see any pattern in the way the tileCacheSize is set.
Unfortunately, you are right, this is a bug.
Due to a numerical overflow, the cache size is set to zero when using values higher than 2000. Due to multiple overflows for a value of 22000 you got a cache of 1.5GB. This will be fixed with a module update in the coming weeks.
You can work around this by using the -c option from the command line. This way the cache is correctly set.
Thanks good to know ;).
Thanks for the information about setting the Java parameters.
You mention that -J-Xmx16G will not overwrite the -Xmx8G form the gpt.vmoptions file. Is this really impossible ? My problem is that I’ve installed snap on a cluster with 2 types of machines (one with a larger RAM than the other). So I would like to launch on each machine with the optimal parameters, but I don’t see how if the -J-Xmx is not taken into account. Is it maybe possible to specify the gpt.vmoptions file to use with my gpt depending on the machine? Or should I try to install snap twice (but it has a single front end)?
Would you please to give your opinion, I’m a bit confused, the machine was 16 GB RAM and 1 TB HHD, the creating coherence graph of two SLC images with multilook it took a few seconds to one or two minutes,
Now the machine 32 GB RAM and 2 TB SSD,
The same process takes more than 10 minutes, The SNAP is reinstalled and this is the snap.properties from snap.
and the following is the SNAP properties from SNAP\etc
and this is the gpt.vmoptions,
You see me puzzeld, too.
One thing I recently noticed is setting the properties
can slow down the processing.
Delete them in the snap.properties and try again.
Thanks a lot for this very precise technical note, the difference of time to the same graph below, is, (19 minutes) after deleting those two lines, and (29 minutes) within two lines.
But, is this duration of time (19 min.) is logical with 32 GB RAM and 2 TB SSD? (using gpt)
I think in your case, it will be useful to increase the snap.jai.tilecache perhaps you could try 12000 or 15000 (if you are using gpt, then you can use the -c parameter).
Okay, that’s good. But you say processing finished within 2 minutes.
This means it is still much slower.
Are you sure about the timings for your previous memory configuration?
Somehow I doubt that this graph would run such fast, from looking at what users reported.
The 19 minutes seem to be reasonable to me considering the processing graph. It should take almost the same time with the configurations you have mentioned.
And increasing the tile cache might help, as Omar suggested.
Now I really confused with time, but most probably were a few minutes.
Hi Marco and Omar @obarrilero
After deleting those two lines, and increase the cachesize to (15000) the same graph duration time is only (03 min.) .
But Omar, I’m not quite sure if got properly this point, would you please to clarify it!
@obarrilero shouldn’t the configuration optimiser suggest that automatically?