Hi,
I have a fairly big workstation available for my work (24 threads 48 GB)
For some large scale classifications I am running in computer power limitations by lack of time (or patience?). I have tried to optimise a lot but I feel like I would need some more. Of course the bandwidth of Cloud computer time, their tireless power and the price seem attractive.
I am looking into classical commercial opportunities such as Amazon or Google. I am almost convinced that I should go for it instead of having my own blades. However, I don’t completely know how these VM work and if it is so handy to set up a complex software with the proper dependencies such as SNAP.
I would like to avoid doing mistakes when setting it up, anyone with experience configuring SNAP on a commercial cloud system with a Linux VM? Advices on what to do, what not to do?
At my work we have quite a number of 24 core machines (two Intel E5-2630v3) processors, but a single disk for data. These perform very well on some numerical models, but for batch processing with NASA OCSSW and ESA GPT many of the jobs take much longer than on a garden variety core i5 system. Many factors may be involved: a) as I recall, the E5-2630v3 processor has one floating point unit for 3 cores, b) the single disk is a bottleneck, c) it is difficult to ensure that cache memory is used effectively, and d) the kernel scheduler (for Ubuntu 14.04) may not be indeal for this combination of workload and processor. Now we are promised mitigation for Meltdown and Spectre that may affect performance.
Since it can be tricky to get good performance from multi-core systems, it is worth spending a bit of time looking for “easy” fixes such a as upgrading the disk storage for better I/O thruput and doing some experiments with process affinity.
Thre are other advantages of cloud systems: someone else is responsible for backups and maintenance of hardware and software.
If your projects include working with Sentinel data you should definitely check RUS-Copernicus (https://rus-copernicus.eu/portal/). They provide VMs with all the necessary software you need and different levels to meet your needs
Thanks a million for your advices
I am very confused by the standard cloud offers I will look into this.
My machine at home is fairly decent dual 6 core xeon with 98gb and plenty of fast discs but I am just thinking into go further into the sentinel processing and that could use up all I have to work now
Thanks
