Proposition of a step by step tuto to install Sen2cor on Ubuntu VM 16.10


#1

Hi everybody

I would like to share with you my experience about installation of production server ubuntu 16.10 with sen2cor 2.3.0 and Snap
I will share all my command line.
With this way, we obtain a functionnal Production server for L2A sentinel products

In this tuto, my username is “geouser”, you have to replace paths with your own…
You have to make a clear and fresh installation of Ubuntu Server 16.10 and you can follow the steps hereafter

Installation of SSH (for remote access to the server)
sudo apt-get install ssh

Download & Install Anconda

mkdir /home/geouser/ANACONDA
cd /home/geouser/ANACONDA
wget https://repo.continuum.io/archive/Anaconda2-4.2.0-Linux-x86_64.sh
bash /home/geouser/ANACONDA/Anaconda2-4.2.0-Linux-x86_64.sh
Follow the differents steps, note differents paths

Test anaconda
in a terminal enter:
python
it will open an interpreter with anaconda mentionned
now check the path of python
which python
my own path is for exemple (note this path for later)
/home/geouser/anaconda2/bin/python

SNAP

mkdir /home/geouser/SNAP
cd /home/geouser/SNAP
wget http://step.esa.int/downloads/5.0/installers/esa-snap_all_unix_5_0.sh
sudo sh esa-snap_all_unix_5_0.sh -c

when snap ask about configuration of python, do it and enter the path of python you have noted during anaconda install(in my case: /home/geouser/anaconda2/bin/python)

SEN2COR

mkdir /home/geouser/SEN2COR
cd /home/geouser/SEN2COR
cd /home/geouser/SEN2COR
wget http://step.esa.int/thirdparties/sen2cor/2.3.1/sen2cor-2.3.1.tar.gz
tar xvzf sen2cor-2.3.1.tar.gz
cd sen2cor-2.3.1
Becarefull with the next step: if there is an error you have probably a problem with your python path…
python setup.py install

Next you have to define environement variables:
sudo nano /etc/bash.bashrc

add the following lines at the end of the doc , save and quit

export SEN2COR_HOME=/home/geouser/sen2cor
export SEN2COR_BIN=/home/geouser/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor
export GDAL_DATA=/home/geouser/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor/cfg/gdal_data

Now you can check sen2cor with this command line:
L2A_Process

I hope this few lines help anyone.
Cheers


Sen2cor fails on my linux system
#2

I have forgotten something:
at the end of installation, you have to reboot
and now you can try sen2cor:

Launch sen2cor
You have to edit the “L2A_GIPP.xml” before (save a copy of this file beofre editing)

Sudo nano /home/geouser/sen2cor/cfg/L2A_GIPP.xml

For the test of sen2cor we will not set all the variables, but just three of them:

Change parameters for number of processors
Becarefull if you have a 6 cores VM you have to assign 24 GB of RAM (1 core =4GB), for security i put 6 cores and 30 Gb of RAM. In the xml you just have to set the number of cores.

In Atmospheric correction, Look Up table, you have to check following paramters:

  • Aerosol type: First time select MARITIME or RURAL, and when you are sure sen2cor is functionnal you can try with AUTO parameter
  • Mid_Latitude: Never set this parameter on AUTO (if you do that you will have an error like) :
    [Errno 2] No such file or directory: ‘/home/geouser/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.0-py2.7.egg/sen2cor/lib/20_60/h99000_wv02_rura.atm’
    I hope this bug will be correct in the next release of sen2cor.

Now you can save the L2A_GIPP.xml and quit
Import a S2AL1C tile in geouser directory (or another local directory)
You can try with a non-local emplacement but becarefull with file perimission on Linux…

And you can laucnh a commandline like that:

L2A_Process --resolution=10 --GIP_L2A /home/geouser/sen2cor/cfg/L2A_GIPP.xml /home/geouser/S2A_MSIL1C_20170102T231852_N0204_R001_T58KDC_20170102T231846/S2A_MSIL1C_20170102T231852_N0204_R001_T58KDC_20170102T231846.SAFE

If the process finish successfully youhave a new directory (in the same of your S2AL1C-----.SAFE) with L2A product

Tadaaa!!!
:slight_smile:
Cheers


#3

Great work! :+1:
cheers


#4

Exactly.

You don’t need SNAP to run sen2cor, though.
You can simply add the anacoda bin directory to your path like that:
export PATH=/home/geouser/anaconda2/bin:$PATH
After that you can call it directly from the same terminal session.


#5

Hi,

How to be sure for the different paths ?

Could you explain how to know the different paths below and precisely what they correspond ?

/home/geouser/anaconda2/bin/python (“python” works but “which python” does not work…)

export SEN2COR_HOME=/home/geouser/sen2cor (is it the folder that contains the installation file?)
export SEN2COR_BIN=/home/geouser/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor (??)
export GDAL_DATA=/home/geouser/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor/cfg/gdal_data (??)

–> i understand geouser is your username but I never found the anaconda2 folder by navigating in ubuntu…
I asked that because I received the following error message after lauching

L2A_Process --resolution=10 --GIP_L2A /home/administrateur/en2cor/cfg/L2A_GIPP.xml /media/administrateur/MyPassport/POSTDOC/Lena_AWI/sentinel2/S2A_OPER_PRD_MSIL1C_PDMC_20160607T174428_R004_V20160607T035734_20160607T035734.SAFE
Traceback (most recent call last):
File “/root/anaconda2/bin/L2A_Process”, line 11, in
load_entry_point(‘sen2cor==2.3.1’, ‘console_scripts’, ‘L2A_Process’)()
File “/root/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor/L2A_Process.py”, line 166, in main
config = L2A_Config(None, directory)
File “/root/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor/L2A_Config.py”, line 53, in init
os.mkdir(self._configDir)
OSError: [Errno 2] No such file or directory: ‘/home/administrateur/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor/cfg’

Thanks a lot


#6

Hi S2Ger

Can you verify anaconda installation ?
I think it’s probably an error during installation of python…

First time you have to verify your installation of anaconda2:
do the installation with right permission in a repertory like :
/home/administrateur/ANACONDA
and your python path should be /home/administrateur/ANACONDA/bin/python

which python should work… What is your OS ?
Ubuntu 16.04 is great (beacause there is long term support)

I think the installation of sen2cor was not complete,
when you did
python setup.py install
Was there an error message ?

differents path allow you to call sen2cor anywhere in you terminal, and call right library

export SEN2COR_HOME=/home/geouser/sen2cor
it point to sen2cor installation repertory
in your case i think something like
/home/administrateur/en2cor/

export SEN2COR_BIN=/home/geouser/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor
in your case i think something like
/root/anaconda2/lib/…

export GDAL_DATA=/home/geouser/anaconda2/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor/cfg/gdal_data
idem
/root/anaconda2/lib/…

I am not sur it’s a good practice to choose root to instal anaconda2 (but i am not sure)

I had already this kind of error, you are very close to success !!!
Keep me posted


#7

Hi,
Thank you very much for your tutorial.
I have followed the steps and got an error on the sen2cor installation (on Ubuntu 16.04 64bits). I verified the anaconda installation, installed snap and set the configuration of the python path. Here I copy the error while installing sen2cor:

Installed /usr/local/lib/python2.7/dist-packages/sen2cor-2.3.1-py2.7.egg
Processing dependencies for sen2cor==2.3.1
Finished processing dependencies for sen2cor==2.3.1
Copying OpenJPEG library to /usr/lib/libopenjp2.so ...
Traceback (most recent call last):
  File "setup.py", line 215, in <module>
    copyfile(libOpj2Source, libOpj2Target)
  File "/usr/lib/python2.7/shutil.py", line 82, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/usr/lib/python2.7/site-packages/sen2cor-2.3.1-py2.7.egg/sen2cor/build/lib.linux-x86_64-2.7/libopenjp2.so'

Thanks in advance for any advice to solve it!
Natalia

Thanks in advance for any advice to solve it!
Natalia


#8

Hi Natalia
I’m not sure, but i think i 'have already seen this error. I don’t know if it isn’t a know issue, maybe you can search this error in another part of the step forum…

This error append when you launch
python setup.py install
during sen2cor 2.3.1 installation ?

i am not an advanced user of sen2cor, i have just shared my installation experience on the forum, but in your case, i think you can try another time with a fresh install of ubuntu 16.04. Try without installing snap (just skip the procedure).

I will ask around me, i’m sure we have already seen this error…
See you
Léo


#9

Thanks Léo for the response!
I’ve seen that it’s a common error… I followed several forum instructions but none has worked for me.
Yes, the error is when I launch
python setup.py install

I have an almost fresh ubuntu 16.04 installation, however I had already installed snap… I tried re-installing snap to set anaconda in the python configuration, but the error in sen2cor installation is still there.

Best,
Natalia


#10

Did you use a Virtual Machine or a physical one ?


#11

Hi all,

Is there any chance this installation procedure has been written in a Python script?

Thanks!


#12

If Anaconda is installed correctly, setup.py is doing exactly that, by calling python setup.py install


#13

Hello,
I’m beginner in Python/Ubunto scripting.
I would use sen2cor for a lot of S2 images.

Can you help me to write an iterative processing script ?
thank you


#14

this would be quite easy

  1. collect all of your S2 scenes in one directory (ordered by sub-folders, the ones which end with .SAFE)
  2. create a bat file calling the scenes
  3. execute the bat file

You just create a txt file that looks like this:

L2A_process --resolution 60 S2A_MSIL1C_20170512T110651_N0205_R137_T33WVR_20170512T110654.SAFE
L2A_process --resolution 60 S2B_MSIL1C_20171019T102019_N0205_R065_T32UPU_20171019T102055 .SAFE
L2A_process --resolution 60 S2A_MSIL2A_20171017T103021_N0205_R108_T32UNV_20171017T103024 .SAFE
L2A_process --resolution 60 S2A_MSIL2A_20171014T102021_N0205_R065_T32UPV_20171014T102235 .SAFE

Then you save it as script.bat and execute this file. This calls one process after the other.


#15

Hi, How can I execute this file?
I guess it looks like:

L2A_Process --resolution=10 --GIP_L2A /home/user/sen2cor/cfg/L2A_GIPP.xml /home/ga53lom/S2Cor/script.bat

Thanks a lot!


#16

Hi, How can I execute this file?
I guess it looks like:

L2A_Process --resolution=10 --GIP_L2A /home/user/sen2cor/cfg/L2A_GIPP.xml /home/ga53lom/S2Cor/script.bat

Thanks a lot!


#17

first you create this file and then you can just call it from the command line.
Navigate to the folder where it is stored and open your command shell there and type: script.bat and hit Return.
Your shell then executes line by line.


#18

I also need
chmod +x script.bat
before run it

Thanks so much for answering my so ‘stupid’ question.


#19

it’s not stupid. Moreover, classic bat files are for Windows (my bad) but I’m glad you already found a way to execute them on Ubuntu.


#20

Actually I did not try file.bat, but file.sh. It works for me.

Thanks a lot for this nice blog!