Sen2Cor 2.11.0 linking to musl is causing issues on modern Linux distros

lnicola · January 27, 2023, 8:51am

This happens during installation, but the previous versions used to work:

Creating directory Sen2Cor-02.11.00-Linux64
Verifying archive integrity... All good.
Uncompressing SEN2COR 02.11.00  100%  
Configuring...
./pkgsetup: line 46: 751283 Segmentation fault      (core dumped) "$INTERPRETER" "$PATCHELF_EXE" "--set-rpath" "$OUT_DIR/lib" "$lib_file" > /dev/null 2>&1
./pkgsetup: line 46: 751361 Segmentation fault      (core dumped) "$INTERPRETER" "$PATCHELF_EXE" "--set-rpath" "$OUT_DIR/lib" "$lib_file" > /dev/null 2>&1
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
file: Relink `~/Downloads/Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
find: ‘file’ terminated by signal 11
copying ~/Downloads/Sen2Cor-02.11.00-Linux64/lib/python2.7/site-packages/sen2cor/cfg/L2A_GIPP.xml to ~/sen2cor/2.11/cfg/L2A_GIPP.xml!
Congratulations, Installation successful...

On one of the systems I tried, this might be only a cosmetic issue and at least the binary can display the usage instructions. But on others, it doesn’t even run:

$ Sen2Cor-02.11.00-Linux64/bin/L2A_Process
Sen2Cor-02.11.00-Linux64/bin/L2A_Process: line 5: Sen2Cor-02.11.00-Linux64/bin/python2.7: No such file or directory

$ Sen2Cor-02.11.00-Linux64/bin/python2.7
exec: Failed to execute process 'Sen2Cor-02.11.00-Linux64/bin/python2.7': The file exists and is executable. Check the interpreter or linker?

$ lddtree Sen2Cor-02.11.00-Linux64/bin/python2.7
Sen2Cor-02.11.00-Linux64/bin/python2.7 (interpreter => /lib/ld-musl-x86_64.so.1)
    libpython2.7.so.1.0 => Sen2Cor-02.11.00-Linux64/lib/libpython2.7.so.1.0
    libc.musl-x86_64.so.1 => Sen2Cor-02.11.00-Linux64/lib/libc.musl-x86_64.so.1

Sen2Cor 2.10.01 works fine on the same machine:

$ Sen2Cor-02.10.01-Linux64/bin/L2A_Process
usage: L2A_Process.py [-h] [--mode MODE] [--resolution {10,20,60}]
                      [--datastrip DATASTRIP] [--tile TILE]
                      [--output_dir OUTPUT_DIR] [--work_dir WORK_DIR]
                      [--img_database_dir IMG_DATABASE_DIR]
                      [--res_database_dir RES_DATABASE_DIR]
                      [--processing_centre PROCESSING_CENTRE]
                      [--archiving_centre ARCHIVING_CENTRE]
                      [--processing_baseline PROCESSING_BASELINE] [--raw]
                      [--tif] [--sc_only] [--sc_classic] [--sc_cog]
                      [--cr_only] [--debug] [--GIP_L2A GIP_L2A]
                      [--GIP_L2A_SC GIP_L2A_SC] [--GIP_L2A_AC GIP_L2A_AC]
                      [--GIP_L2A_PB GIP_L2A_PB]
                      input_dir
L2A_Process.py: error: too few arguments

$ lddtree Sen2Cor-02.10.01-Linux64/bin/python2.7
Sen2Cor-02.10.01-Linux64/bin/python2.7 (interpreter => Sen2Cor-02.10.01-Linux64/lib/ld-musl-x86_64.so.1)
    libpython2.7.so.1.0 => Sen2Cor-02.10.01-Linux64/lib/libpython2.7.so.1.0
    libc.musl-x86_64.so.1 => Sen2Cor-02.10.01-Linux64/lib/libc.musl-x86_64.so.1

Tested on Ubuntu 22.04 and Arch Linux. Unfortunately, I can’t spend much more time on this right now.

oana_hogoiu · January 27, 2023, 9:36am

@Amalia-Castro-Gomez Can you please inform the Sen2Cor team? Thank you.

Amalia-Castro-Gomez · February 2, 2023, 3:32pm

Dear both, thanks for your patience, I asked the Sen2Cor team and as soon as they reply, I will let you know.

not1987 · February 7, 2023, 9:53pm

In a debian-based system I solved this issue by installing the “file” command.
Hope this might be helpful.
Antonio

lnicola · February 8, 2023, 6:37am

That doesn’t sound quite right.

If file is missing, the installer shows a wall of find: ‘file’: No such file or directory messages. While they appear similar, that’s a different and unrelated error. In my case, file exists, but crashes because it picks up a MUSL-based libz from Sen2Cor which is not compatible with the file command on my system. I can reproduce it by setting LD_LIBRARY_PATH to the Sen2Cor installation:

$ LD_LIBRARY_PATH=Sen2Cor-02.11.00-Linux64/lib file
file: Relink `Sen2Cor-02.11.00-Linux64/lib/libz.so.1' with `/usr/lib/libc.so.6' for IFUNC symbol `memchr'
Segmentation fault (core dumped)

So one of the problems here is that the installer is using Sen2Cor’s lib with programs from my system. And indeed, comparing the setup scripts from the two versions (run the installer with --noexec --keep to get it), 2.11.0 now sets it:

--- Sen2Cor-02.10.01-Linux64/pkgsetup	2021-12-16 12:31:25.000000000 +0200
+++ Sen2Cor-02.11.00-Linux64/pkgsetup	2022-11-24 10:40:39.000000000 +0200
@@ -20,12 +20,13 @@
 
 DIRNAME_0=$(dirname "$0")
 cd  "$DIRNAME_0"
-# NOTE: $OUT_DIR is also used in make_sylinks script (see below)
+# NOTE: $OUT_DIR is also used in make_symlinks script (see below)
 # Avoid any pre-mature optimization on variable names here.
 
 OUT_DIR=$(pwd)
 
 PATCHELF_EXE="$OUT_DIR/lib/patchelf"
+INTERPRETER="$OUT_DIR/lib/ld-musl-x86_64.so.1"
 
 # No no interference with LD_LIBRARY_PATH
 LD_LIBRARY_PATH=
@@ -33,36 +34,38 @@
 echo "Configuring..."
 
 # -maxdepth 1 is used to exclude files from lib/python/*/*/.so
-LIB_FILES=$(find "$OUT_DIR/bin" "$OUT_DIR/lib" -type f -not -name "patchelf" -not -name "ld-musl*"  -exec file {} \; | grep -i elf|cut -f1 -d':')
+LIB_FILES=$(find "$OUT_DIR/bin" "$OUT_DIR/lib" -type f -not -name "patchelf" -not -name "ld-musl*" -exec file {} \; | grep -i elf|cut -f1 -d':')
+
+export LD_LIBRARY_PATH="$OUT_DIR/lib"
+
+#echo "Creating symbolic links..."
+. ./make_symlinks
+ln -sf "$OUT_DIR/lib/ld-musl-x86_64.so.1" "$OUT_DIR/lib/libc.musl-x86_64.so.1"
 
 # run patchelf
 for lib_file in $LIB_FILES; do
     if [ -f "$lib_file" ] ; then
 	#echo "adding rpath to $lib_file"
-        "$PATCHELF_EXE" "--set-rpath" "$OUT_DIR/lib" "$lib_file" > /dev/null 2>&1
+        "$INTERPRETER" "$PATCHELF_EXE" "--set-rpath" "$OUT_DIR/lib" "$lib_file" > /dev/null 2>&1
 	chmod u+x "$lib_file"
     fi
 done
 
 chmod 755 "$OUT_DIR/lib/ld-musl-x86_64.so.1"
 
-BIN_FILES=$(find "$OUT_DIR/bin" -type f -exec file {} \; | grep -i elf|cut -f1 -d':')
+BIN_FILES=$(find "$OUT_DIR/bin" -type f -not -name "patchelf" -exec file {} \; | grep -i elf|cut -f1 -d':')
 for bin_file in $BIN_FILES; do
     if [ -f "$bin_file" ] ; then
-	"$PATCHELF_EXE" --set-interpreter "$OUT_DIR/lib/ld-musl-x86_64.so.1" "$bin_file"  > /dev/null 2>&1
-	chmod u+x "$lib_file"
+	"$INTERPRETER" "$PATCHELF_EXE" --set-interpreter "$INTERPRETER" "$bin_file"  > /dev/null 2>&1
+	chmod u+x "$bin_file"
     fi
 done
 
 #"$PATCHELF_EXE" --set-interpreter "$OUT_DIR/lib/ld-musl-x86_64.so.1" "$OUT_DIR/bin/python2.7"
+export LD_LIBRARY_PATH=
 
-#echo "Creating symbolic links..."
-. ./make_symlinks
 rm -f  "$OUT_DIR/make_symlinks"
 
-ln -sf "$OUT_DIR/lib/ld-musl-x86_64.so.1" "$OUT_DIR/lib/libc.musl-x86_64.so.1"
-
-
 #cfghome = os.environ['XDG_CONFIG_HOME']
 #except:
 #cfghome = os.environ['HOME']
@@ -72,9 +75,9 @@
 #     L2A_Bashrc += 'export LC_ALL=en_US.UTF-8\n'
 #     L2A_Bashrc += 'export LANG=en_US.UTF-8\n'
 
-SEN2COR_HOME=$HOME/sen2cor/2.10
+SEN2COR_HOME=$HOME/sen2cor/2.11
 if [ -z "\$XDG_CONFIG_HOME" ]; then
-    SEN2COR_HOME=$XDG_CONFIG_HOME/sen2cor/2.10
+    SEN2COR_HOME=$XDG_CONFIG_HOME/sen2cor/2.11
 fi;
 
 SEN2COR_BIN=$OUT_DIR/lib/python2.7/site-packages/sen2cor
@@ -132,7 +135,7 @@
     cp "$SEN2COR_BIN/cfg/L2A_GIPP.xml" "$SEN2COR_HOME/cfg/L2A_GIPP.xml"
     echo "copying $SEN2COR_BIN/cfg/L2A_GIPP.xml to $SEN2COR_CONFIG_FILE!"
 else
-    echo "File '$SEN2COR_CONFIG_FILE' already exists. So we don't change it!"
+    echo "File '$SEN2COR_CONFIG_FILE' already exists. So we don't change it! Please make sure its structure is in line with version installed."
 fi
 
 printf %s\\n "Congratulations, Installation successful..."
@@ -140,6 +143,7 @@
 printf %s\\n "Default configuration file is '$SEN2COR_CONFIG_FILE'"
 
 rm -f  "$OUT_DIR/lib/patchelf"
+#rm -f  "$OUT_DIR/bin/patchelf"
 
 rm -f  "$OUT_DIR/pkgsetup"

Note how BIN_FILES is computed after setting LD_LIBRARY_PATH="$OUT_DIR/lib".

As for Sen2Cor failing to start, it has the same root cause: patchelf doesn’t run during the installation, so the interpreter doesn’t get set. It works if I do it manually:

$ LD_LIBRARY_PATH=$PWD/lib lib/ld-musl-x86_64.so.1 bin/patchelf --set-interpreter $PWD/lib/ld-musl-x86_64.so.1 python2.7
warning: working around a Linux kernel bug by creating a hole of 2097152 bytes in ‘bin/python2.7’
$ source L2A_Bashrc 
$ bin/L2A_Process 
usage: L2A_Process.py [-h] [--mode MODE] [--resolution {10,20,60}]
                      [--datastrip DATASTRIP] [--tile TILE]
                      [--output_dir OUTPUT_DIR] [--work_dir WORK_DIR]
                      [--img_database_dir IMG_DATABASE_DIR]
                      [--res_database_dir RES_DATABASE_DIR]
                      [--processing_centre PROCESSING_CENTRE]
                      [--archiving_centre ARCHIVING_CENTRE]
                      [--processing_baseline PROCESSING_BASELINE] [--raw]
                      [--tif] [--sc_only] [--sc_classic] [--sc_cog]
                      [--cr_only] [--debug] [--GIP_L2A GIP_L2A]
                      [--GIP_L2A_SC GIP_L2A_SC] [--GIP_L2A_AC GIP_L2A_AC]
                      [--GIP_L2A_PB GIP_L2A_PB]
                      input_dir
L2A_Process.py: error: too few arguments

crecondo · March 2, 2023, 1:03am

Is it possible to use Sen2Cor 2.11.0 with the SNAP version 2.8?

Thanks,

Carmen Recondo

oana_hogoiu · March 2, 2023, 7:30am

Hello @crecondo ,
No. It is always recommended to use the latest SNAP version, 9.0.4 currently.
The 9.0.0 installer can be found here and then the updates can be installed directly from the software.

crecondo · March 2, 2023, 4:59pm

The problem is that my PC do not have enough resources.

And a Sen2Cor version higher than version 2.8.0(280) that it works with SNAP version 2.8? My interest is to use a 30 m DEM.

Thank you

lnicola · March 2, 2023, 5:05pm

What is SNAP 2.8? Do you really mean the version from 2016?

crecondo · March 2, 2023, 5:34pm

Sorrry, it is the version 8.0.0

lnicola · March 2, 2023, 5:45pm

I don’t have an example on hand, but you can actually run Sen2Cor outside of SNAP. Basically, You run L2A_Process and pass it the path to the input product. It even has a user manual which you can take a look at.

crecondo · March 3, 2023, 4:07am

Thank you, I will try it.

oana_hogoiu · March 3, 2023, 11:23am

You can find Sen2Cor 2.11 standalone tool (and the user manual) at Sen2Cor 2.11

crecondo · March 29, 2023, 11:52pm

Hi @Inicola,

I can find in the maual how run L2A_Process and pass it the path to the input product…

Can you help me?

crecondo · March 29, 2023, 11:54pm

Is it not possible install Sen2Cor 2.11 and create a SNAP toolbox with it?

Sen2cor_dev_team · April 19, 2023, 2:55pm

As mentioned in point 7) of Sen2Cor FAQ, a new updated Sen2Cor package 02.11.00 for Linux will be available soon. This package solves the installation issue on modern Linux distributions like Ubuntu 22.04.

Cheers

@Sen2cor_dev_team

Sen2cor_dev_team · April 28, 2023, 10:06am

Dear user,

The updated Sen2Cor package as been released on 27/04/2023 and is available at Sen2Cor v2.11 – STEP , as indicated in the related news. This package solves the installation issue.

Cheers,
@Sen2cor_dev_team