I'm having trouble getting sen2cor to run in an hpc environment. I basically loop over ~170 files, each time calling sen2cor with sbatch. This all works until multiple files get associated with the same node. E.g., for simplicity I generally just book 1 core per job, so a particular node might get selected multiple times to process different scenes (memory is not the issue).
So, my log files look somewhat like this:
Sentinel-2 Level 2A Prototype Processor (Sen2Cor), 2.2.1, created: 2016.04.29 started ...
Traceback (most recent call last):
File "/home/rhagensi/opt/anaconda2/bin/L2A_Process", line 11, in
load_entry_point('sen2cor==2.2.1', 'console_scripts', 'L2A_Process')()
File "/home/rhagensi/opt/anaconda2/lib/python2.7/site-packages/sen2cor-2.2.1-py2.7.egg/sen2cor/L2A_Process.py", line 221, in main
result = config.readPreferences()
File "/home/rhagensi/opt/anaconda2/lib/python2.7/site-packages/sen2cor-2.2.1-py2.7.egg/sen2cor/L2A_Config.py", line 3258, in readPreferences
File "/home/rhagensi/opt/anaconda2/lib/python2.7/site-packages/sen2cor-2.2.1-py2.7.egg/sen2cor/L2A_XmlParser.py", line 161, in export
objectify.deannotate(self.root, xsinil=True, cleanup_namespaces=True)
File "src/lxml/lxml.objectify.pyx", line 1731, in lxml.objectify.deannotate (src/lxml/lxml.objectify.c:24548)
File "src/lxml/cleanup.pxi", line 49, in lxml.etree.strip_attributes (src/lxml/lxml.etree.c:150273)
File "src/lxml/apihelpers.pxi", line 63, in lxml.etree._rootNodeOrRaise (src/lxml/lxml.etree.c:15695)
TypeError: Invalid input object: NoneType
Did anyone run into similar troubles already? It's hard for me to judge whether this is an issue with sen2cor or just SLURM being set up unfavorably.
Also, just to reiterate: I do not aim to parallelize sen2cor processing via the config-file. For me the ideal solution would be to process many different scenes yet each on one core. And everything works fine except when processing multiple scenes on the same node. Any ideas?
Thanks in advance,