Sen2Cor 2.4.0 with Docker


#1

EDIT

I made this docker version because I was not able to run the stand-alone version of Sen2Cor in my computer. If you’re running into the same problem you may want to check the post marked as solution.


ORIGINAL POST

Dear all,

After trying the Sen2Cor stand-alone installation on Ubuntu 14.04 without success and not willing to dig into anaconda installation, I decided to wrap this stand-alone version in a docker container. I’m sharing it here in case someone is interested.

To work around annoying user and group ids problems, you’ll have to build the docker image yourself. Create an empty directory and copy the following contents in a file named Dockerfile:

FROM ubuntu:xenial

# Set sen2cor version string to be used below
ENV SEN2COR_VERSION 2.4.0

# Install needed packages
RUN apt-get update && apt-get install -y \
      wget \
      file \
    && rm -rf /var/lib/apt/lists/*

# Create a default user named appuser with default UID. If you want to have
# the processed producs with your user and grounp ids (recommended), build the
# image using the options:
# --build-arg APPUSER_UID=$(id -u) --build-arg APPUSER_GROUP=$(id -g)
ARG APPUSER_UID
ARG APPUSER_GROUP
ENV HOME /home/appuser
RUN groupadd ${APPUSER_GROUP:+-g} ${APPUSER_GROUP} appuser \
    && useradd ${APPUSER_UID:+-u} ${APPUSER_UID} -g appuser appuser \
    && mkdir -p $HOME \
    && chown -R appuser $HOME
WORKDIR $HOME
USER appuser

# Download and install sen2cor into /home/appuser
RUN wget http://step.esa.int/thirdparties/sen2cor/${SEN2COR_VERSION}/Sen2Cor-${SEN2COR_VERSION}-Linux64.run \
  && chmod +x Sen2Cor-${SEN2COR_VERSION}-Linux64.run \
  && ./Sen2Cor-${SEN2COR_VERSION}-Linux64.run \
  && rm Sen2Cor-${SEN2COR_VERSION}-Linux64.run

# Configure the container to run as a L2A_Process executable
ENTRYPOINT ["/home/appuser/Sen2Cor-2.4.0-Linux64/bin/L2A_Process"]
CMD ["--help"]

This Dockerfile downloads and installs Sen2Cor in the /home/appuser directory inside the container, adds an user named appuser, and finally creates an entry-point that allow us to use the container as if it was the L2A_Process executable. It also offers the possibility to inform the user and group ids for the appuser with --build-arg options.

You can build the docker image tagged l2a_process:2.4.0 by running the following command on the directory with the Dockerfile:

$ docker build -t l2a_process:2.4.0 --build-arg APPUSER_UID=$(id -u) --build-arg APPUSER_GROUP=$(id -g) .

By informing your ids in the build process, any files created in the container by appuser in a shared directory between your computer and the container (for example, your 1C products directory) will have your current user as owner. This way you don’t need to change the owner of the generated 2A products.

You can test the container using docker run --rm l2a_process:2.4.0, that will show the L2A_Process.py help message:

$ docker run --rm l2a_process:2.4.0 
usage: L2A_Process.py [-h] [--resolution {10,20,60}] [--sc_only] [--cr_only]
                      [--refresh] [--GIP_L2A GIP_L2A]
                      [--GIP_L2A_SC GIP_L2A_SC] [--GIP_L2A_AC GIP_L2A_AC]
                      directory

Sentinel-2 Level 2A Processor (Sen2Cor). Version: 2.4.0, created: 2017.06.05,
supporting Level-1C product version: 14.

positional arguments:
  directory             Directory where the Level-1C input files are located

optional arguments:
  -h, --help            show this help message and exit
  --resolution {10,20,60}
                        Target resolution, can be 10, 20 or 60m. If omitted,
                        all resolutions will be processed
  --sc_only             Performs only the scene classification at 60 or 20m
                        resolution
  --cr_only             Performs only the creation of the L2A product tree, no
                        processing
  --refresh             Performs a refresh of the persistent configuration
                        before start
  --GIP_L2A GIP_L2A     Select the user GIPP
  --GIP_L2A_SC GIP_L2A_SC
                        Select the scene classification GIPP
  --GIP_L2A_AC GIP_L2A_AC
                        Select the atmospheric correction GIPP

To use L2A_Product you’ll need to have the 1C products available inside the container. For this, you can mount the directory with the products in the container using --mount option, and then use the path to the product directory inside the container. For example, if you are inside a directory with the 1C products, you can run process one of the products with:

$ docker run --rm --mount type=bind,source="$(pwd)",target=/home/appuser/products l2a_process:2.4.0 products/S2A_MSIL1C_<...>.SAFE

This will mount your products directory on /home/appuser/products inside the container. Then the product can be selected by informing its directory after the image tag name. You can also pass any L2A_Process.py options after the image tag name and before the product directory (as if you were using the L2A_Process.py script directly).

I hope this will be useful to someone.


#2

Hello Mellon,

Can you give some details on the errors from Sen2Cor standalone installers ?.
I see your docker uses same install steps and assume it is working.
Please also share information such as which version of OS, output of
lsb_release -a or uname -a
And your installation steps, if different from install steps in docker, please include that too…


#3

Hello rashadkm,

Sorry I didn’t expressed myself properly. I think the installation worked properly, but when trying to run L2A_Process I got errors like this:

Traceback (most recent call last):
  File "/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Process.py", line 6, in <module>
    from numpy import *
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 153, in <module>
    from . import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 6, in <module>
    from . import multiarray
ImportError: Error relocating /usr/lib/python2.7/dist-packages/numpy/core/multiarray.so: __strcat_chk: symbol not found

I got the same result with and without sourcing L2A_Bashrc, by running it with from Sen2Cor-2.4.0-Linux64/bin or other directory, and by using ./L2A_Process or bash L2A_Process. I think the problem is because L2A_Process is using my system libraries, instead of the ones packed with Sen2Cor, and they are outdated:

$ python -c "import numpy; print numpy.__version__"
1.8.2

Funny fact, even the print version command below fails after I source L2A_Bashrc:

$ cd ~/devel/src/SNAP/Sen2Cor-2.4.0-Linux64
$ python -c "import numpy; print numpy.__version__"
1.8.2
$ source L2A_Bashrc
$ python -c "import numpy; print numpy.__version__"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 153, in <module>
    from . import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 6, in <module>
    from . import multiarray
ImportError: Error relocating /usr/lib/python2.7/dist-packages/numpy/core/multiarray.so: __strcat_chk: symbol not found

At the end I didn’t want to mess with my system nor its environment variables, so I went for the docker solution.

I followed the same installation steps for both docker and my computer (basically extract, source and use).

Here it is the information about my OS:

$ lsb_release -a
LSB Version:	core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:core-4.1-amd64:core-4.1-noarch:security-4.0-amd64:security-4.0-noarch:security-4.1-amd64:security-4.1-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.5 LTS
Release:	14.04
Codename:	trusty

#4

If you extract and simply run ./bin/L2A_Process, do you have same error?
sourcing of L2A_bashrc is done in this startup script (./bin/L2A_Process)

when you do ‘which python’ after sourcing L2A_Bashrc what does it say?

If it points to correct python from Sen2Cor Package, can you try to set PYTHONPATH
export PYTHONPATH=/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/lib/python2.7:/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/lib/python2.7/site-packages/
and see if this works.

In anycase, there is something to be fixed on packaging side, going for docker is a temporary solution and I don’t recommend every user to use docker.


#5

Yes, same error.

The one packed with Sen2Cor.

No, it doesn’t. I get the same error:

╭─02:59:28 ~$ 
╰─➤ cd ~/devel/src/SNAP/Sen2Cor-2.4.0-Linux64
╭─02:59:38 ~/devel/src/SNAP/Sen2Cor-2.4.0-Linux64$ 
╰─➤ source L2A_Bashrc
╭─03:00:05 ~/devel/src/SNAP/Sen2Cor-2.4.0-Linux64$ 
╰─➤ which python
/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/bin/python
╭─03:00:09 ~/devel/src/SNAP/Sen2Cor-2.4.0-Linux64$ 
╰─➤ export PYTHONPATH=/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/lib/python2.7:/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/lib/python2.7/site-packages/
╭─03:00:20 ~/devel/src/SNAP/Sen2Cor-2.4.0-Linux64$ 
╰─➤ echo $PYTHONPATH
/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/lib/python2.7:/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/lib/python2.7/site-packages/
╭─03:00:25 ~/devel/src/SNAP/Sen2Cor-2.4.0-Linux64$ 
╰─➤ ./bin/L2A_Process
Traceback (most recent call last):
  File "/home/emendes/devel/src/SNAP/Sen2Cor-2.4.0-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Process.py", line 6, in <module>
    from numpy import *
  File "/usr/lib/python2.7/dist-packages/numpy/__init__.py", line 153, in <module>
    from . import add_newdocs
  File "/usr/lib/python2.7/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/lib/python2.7/dist-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/lib/python2.7/dist-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/lib/python2.7/dist-packages/numpy/core/__init__.py", line 6, in <module>
    from . import multiarray
ImportError: Error relocating /usr/lib/python2.7/dist-packages/numpy/core/multiarray.so: __strcat_chk: symbol not found

I agree with you, although for me that’s the way I made it work. Maybe it may be the solution for someone else before the installer is fixed. :slight_smile:


#6

Thanks for sharing these information.

you have to add export PYTHONPATH in bin/L2A_Process script after the “L2A_Bashrc” call and before calling python2.7
$OUT_DIR/bin/python2.7 $SEN2COR_BIN/L2A_Process.py

then start a new terminal, and only run ./bin/L2A_Process. (bashrc is called by process script)
thanks for sharing dockerfile. In the mean time, I will try to find what is going wrong and will let you know.


#7

I got the same errors after changing the python path inside L2A_Process, but I think I found what is the problem.

When you launch the python interpreter it automatically imports the site module that is responsible to build the path where the other modules will be searched (from PYTHONPATH and from other “obscure” places… Check sys.path variable). This can be suppressed by launching the interpreter with the -s option.

I confirm that by adding -s after $OUT/bin/python2.7 inside L2A_Process I managed to start the process of one 1C product. I’ll report back when it finishes.

UPDATE

Everything is working fine now. :slight_smile:


#8

Thanks for your patience in going through testing. I really appreciate that.
I guess now, it is safe to use -s flag in bin/L2A_Process scripts. I will check with @umwilm and sebasiten about this.
You can close this thread as solved.