Workaround for sen2cor bug at resolution 10 m

sen2cor seems fundamentally broken at a resolution of 10 m, but there is a workaround:

Background Information

When started as L2A_Process --resolution 10 sen2cor_L1C_test the process performs the preliminary calculations for resolution 60 m and resolution 20 m and then crashes with a DistutilsFileError, because it could not copy a file over another file that was still in use.

The obvious error is the DistutilsFileError: the _copy_file_contents() method in file_util.py checks if the copy destination is an existing file, and if so, it attempts to delete that file first (according to the docs, you can silently overwrite an existing file under Linux, while this results in an error under Windows). That delete operation fails if the target is still in use – which is the case if you try to copy a file to itself.

I didn’t delve into the code to find out why the L2A_Process wants to copy a file to itself (the root error), but a basic workaround is to skip the copy operation if source and destination are one and the same file:

Workaround

In file L2A_Tables.py at lines 1380++, replace

def postprocess(self):
    dummy, basename = os.path.split(self.config.L2A_TILE_MTD_XML)
    fnAux = basename.replace('_MTD_', '_GIP_')
    # copy configuration to AUX dir:
    target = os.path.join(self._L2A_AuxDataDir,fnAux)
    copy_file(self.config.configFn, target)
    return

with

def postprocess(self):
    dummy, basename = os.path.split(self.config.L2A_TILE_MTD_XML)
    fnAux = basename.replace('_MTD_', '_GIP_')
    # copy configuration to AUX dir:
    target = os.path.join(self._L2A_AuxDataDir,fnAux)
    # begin new stuff (CSe, 2016-01-25)
    if( os.path.normpath( target ) == os.path.normpath( self.config.configFn ) ):
        sys.stdout.write('Skipped auto-referential copy of "' + target + '"\n')
        return
    if( os.path.exists(target) ):
        sys.stdout.write('Trying to copy\n"%s" to\n"%s"\n..., but target already exists!\n' % (self.config.configFn, target))
        try:
            os.unlink(target)
        except os.error, (errno, errstr):
            sys.stdout.write('Could not delete blocking target file "%s" before copy operation: %s\n' % (target, errstr))
            sys.stdout.write('Please delete the file manually.\n' )
            raw_input('\nIs the file deleted? [y/n]: ')
    # end new stuff (CSe, 2016-01-25)
    copy_file(self.config.configFn, target)
    return

Please note that Python is picky about indentation: make sure that the entire method body is indented four more spaces than the def postprocess(self): line, that the if()s are indented as far as the copy_file() call, that the parts within the if scopes are indented four more spaces, that the parts in the try and except scopes are indented yet another four spaces, and that you are indeed using spaces instead of tabs.

This first checks if copy source and copy destination are identical and if so, it skips the operation. If they are different files it checks if the target already exists, and if so, it tries to delete the file. If that fails, it waits for the user to manually delete the file before performing the actual copy operation – but that’s probably overkill, and I don’t know how it work in the SNAP GUI environment instead of a command line interface …

I can’t promise that I’ll find the time today to create a patch for the GitHub repository; feel free to patch and test it yourselves.

7 Likes

Dear ChristianSeverin,

thanks for your help. I did it what you wrote but the process doesn’t work. I changed description in file L2A_Tables.py (picture below) and i wanted begin the atmospheric correction process (path to the file C:\Anaconda2\Lib\site-packages\sen2cor-2.0.6-py2.7.egg\sen2cor) .

After when i wrote command L2A_Process D:\S2A_OPER_PRD_MSIL1C_PDMC_20160104T181239_R079_V20160104T100050_20160104T100050.SAFE --resolution 10 I received this error.

Before the change, process was working but on the end of first granule processing i receive error that “The process cannot access the file because it is being used by another process”

Where I make a mistake

Best Regards
Wojtek (Poland)

You seem to be using Notepad++, right? What do you see when you click on the ¶ button, “Show All Characters”?
In my case, this kind of error occurred when I used tabs instead of spaces.

Hello,
Yes you’re right i’m using notepad++. When im check ¶ button, "Show All Characters I see that: (picture below)

Best regards

Ah, I see:
the entire method body of postprocess() needs to be indented relative to the def postprocess(self) line. Look at the methods above and below for reference.

My bad, I overlooked that detail in your first post.

Hello,
thanks for your answer. It’s working now. I will see later what is result. I have slowly computer and process can last long. I’ll inform you about results.

Best regards

thanks a lot Christian Severin!!!

Thanks a lot ChristianSeverin!

The workaround worked perfectly for me, even though it required a bit of fiddling around to get the indents right for python.

Thanks again!

Val

Thanks a lot for the workaround Christian!

One question: I miss the geographic projection in the output files when I open the granules individually. Any way to solve that?

Sorry, unnic, no idea.
It’s probably best if you create a separate topic for that, so that your question will appear in the main list instead of buried at the bottom of another thread…

Thanks a lot @ChristianSeverin, finally I obtained Sentinel L2A 10m resolution! :smile: -)

Your workaround works for me too. Thanks for your time and effort to solve this.

thanks a lot… It seems to work for me :smile:

@ChristianSeverin

can you tell why the 10 meter resolution classification is not included in the output? Maybe you have a workaround for this as well or you can point me in the right direction? If I’m able to solve it I’d share it here as well.

(Sorry for the delay)

When you say that

the 10 meter resolution classification is not included in the output

do you find the other resolutions in your output folder?
Under <result_folder>/GRANULE/<granule_name>/IMG_DATA/ I have two preview images for the 20 m and 60 m resolutions, and three folders named R10m, R20m and R60m. What do you find in these folders?

(And again: perhaps it is better to open a new thread for this, with a detailed description starting from zero.)

Hi Christian

I am having a bit of an issue after doing the relevant changes, managed to do the update but failing at the “def testDb(safe):” section, please see below, hoping someone would be able to assist.

Thank you

Seems to be a problem with identation - python is really sensible when it comes to that. Make sure that you follow exactly the instructions by @ChristianSeverin:

If you compare your screenshot with the different other screenshots in this thread, you’ll see that the return after copy_file(self.config.configFn, target) is not indented the required 4(?) spaces.

Thank you for the response but if I include the spaces I get the below therefor the reason why I left out the spaces?

Just to tell everyone that this method also works for me. Thank you ChristianSeverin!