I’m trying to run my (Sentinel 1) processing chain on a big cluster which is shared with many users.
It seems to me that disk I/O is a bottleneck, causing it to run magnitudes slower than on my local machine with only 4 cores.
On our cluster a single node has 28 cores and about 62GB of available memory.
GPT operators are called from a bash script to execute xml processing graphs.
Options I’m using so far are in gpt.vmoptions
-Xverify:none -Xmx58g -Xms2048m -XX:+AggressiveOpts -Djava.io.tmpdir=/gpfs/scratch/gpt_temp
and in the gpt call:
$GPT -c 2G -q 28 -x p0.xml ...
Are there any further measures I could take to reduce disk I/O?
Or should I ramp up the cache size and reduce the number of cores?