Sen2cor supporting processing baseline 05.11

TomAugspurger · July 23, 2024, 9:57pm

ESA started publishing L1C assets with product baseline of 05.11 today: Deployment of Sentinel-2 Processing Baseline in version 05.11 on 23 July - Sentinel Online

With an older version of sen2cor (I’ll edit to add the exact version later), sen2cor fails to process those scenes with the message Product metadata file cannot be read.

I’ve tried to use sen2cor 2.11.0 (which I think is the newest version) and it fails with this output:

wrong identifier for xml structure.
unspecific error in metadata
wrong identifier for xml structure.
error in copying metadata files to target product
Sen2Cor. Version: 02.11.00, created: 2022.10.20, supporting Level-1C product version 14.2 - 14.9 started ...
Product version: 15.0
Operation mode: TOOLBOX
Processing baseline: 05.11
Progress[%]:  0.00 : Generating datastrip metadata
unspecific error in metadata
Incorrect datastrip L1C xml format
Traceback (most recent call last):
  File "/scripts/sen2cor", line 98, in <module>
    main()
  File "/scripts/sen2cor", line 82, in main
    l2a_item = pc_onboarding.sentinel2.process_l2a_item(
  File "/scripts/pc_onboarding/sentinel2/_s2.py", line 1726, in process_l2a_item
    l2a_item = do_l2a(
  File "/scripts/pc_onboarding/sentinel2/_s2.py", line 1384, in do_l2a
    l2a_jp2000_path = run_sen2cor(
  File "/scripts/pc_onboarding/sentinel2/_s2.py", line 1138, in run_sen2cor
    run_subprocess(args, env=env)
  File "/scripts/pc_onboarding/sentinel2/_s2.py", line 1223, in run_subprocess
    raise err
pc_onboarding.sentinel2._s2.CalledProcessErrorWithOutput: CalledProcessError command=['/Sen2Cor/Sen2Cor-02.11.00-Linux64/bin/python2.7', '-s', '/Sen2Cor/Sen2Cor-02.11.00-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Process.py', '/data/l1c/S2A_MSIL1C_20240723T121641_N0511_R123_T24FVF_20240723T170119.SAFE/', '--output_dir', '/data/l2a/jp2000/', '--processing_baseline', '05.11', '--processing_centre', 'MSFT', '--debug', '--GIP_L2A', '/root/sen2cor/2.11/cfg/L2A_GIPP.xml'] returncode=1 stdout=wrong identifier for xml structure.
unspecific error in metadata
wrong identifier for xml structure.
error in copying metadata files to target product
Sen2Cor. Version: 02.11.00, created: 2022.10.20, supporting Level-1C product version 14.2 - 14.9 started ...
Product version: 15.0
Operation mode: TOOLBOX
Processing baseline: 05.11
Progress[%]:  0.00 : Generating datastrip metadata
unspecific error in metadata
Incorrect datastrip L1C xml format
 stderr=Traceback (most recent call last):
  File "/Sen2Cor/Sen2Cor-02.11.00-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Process.py", line 734, in <module>
    sys.exit(main())
  File "/Sen2Cor/Sen2Cor-02.11.00-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Process.py", line 679, in main
    L2A_TILES = config.updateTiles()
  File "/Sen2Cor/Sen2Cor-02.11.00-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Config.py", line 4015, in updateTiles
    if self.create_L2A_Tile():
  File "/Sen2Cor/Sen2Cor-02.11.00-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Config.py", line 4040, in create_L2A_Tile
    self.getEntriesFromDatastrip()
  File "/Sen2Cor/Sen2Cor-02.11.00-Linux64/lib/python2.7/site-packages/sen2cor/L2A_Config.py", line 3968, in getEntriesFromDatastrip
    self.spacecraftName = di.SPACECRAFT_NAME.text
AttributeError: 'bool' object has no attribute 'SPACECRAFT_NAME'

Is there a list of sen2cor versions that should be compatible with the new processing baseline?

vicencpala · July 24, 2024, 7:46am

Same experience with the new baseline downloaded products. There is no way to generate a L2A product from baseline 05.11 with the last sen2cor 2.11

Sen2cor_dev_team · July 25, 2024, 3:06pm

Dear @TomAugspurger , @vicencpala ,

Thank you for your messages,

The new Processing Baseline 5.11 (and Product Version 15.0) will be supported by a forthcoming updated version of Sen2Cor. The release of this new version of Sen2Cor will be announced via the official channels and blogs.

Sen2Cor 2.11 supports Product Version 14.2-14.9. As Processing Baseline 5.11 introduces a new Product Version (15.0), the current version of Sen2Cor 2.11 is not able to support it.

We will update this thread to share the pages/links relevant to the release of this new version of Sen2Cor.

Cheers,
@Sen2cor_dev_team

cpaulik · July 26, 2024, 7:45am

Just to say that this is ridiculous. There are people that have actual business dependent on the ability to process Sentinel 2 data with sen2cor.

Given past experiences my expectations were not high but this might be the worst case yet of not caring at all about your users.

carlo · July 26, 2024, 11:26am

I want to point out that SNAP 10 is currently unable to open Baseline 5.11 (and Product Version 15.0) products See screenshot. TThis issue is blocking for every activity that uses the snap engine

diana_harosa · July 26, 2024, 1:42pm

Jira ticket SNAP-3756 created

roy.ben.hayun · July 29, 2024, 1:59pm

see manual workaround in my post at forum.dataspace.copernicus.eu “sen2cor-cant-process-sentinel-2-product-fails-with-no-attribute-spacecraft-name” /961/5
(can’t post links in this forum…)

can manually edit the 3 XML files and then run L2A_product.bat

yuanzhangzy · July 30, 2024, 6:37am

I’m having the same problem as you.

diana_harosa · July 30, 2024, 3:04pm

A module update (10.0.2) is available for the Optical Toolbox module. Please update your SNAP.

egremmer · July 31, 2024, 8:18am

It has been a week since the release of the new processing baseline. Is there any news regarding when we can expect a new version that supports PB 5.11?

roy.ben.hayun · August 5, 2024, 2:44pm

EO S2A_MSIL1C_20240802T081611_N0511_R121_T36RXV_20240802T101946 from 2nd August still same issue and Sen2Cor v2.11 is still the latest version supporting PSD 14.9 only.
Any update or fix?

hsiao-wei · August 7, 2024, 6:46am

I have the same issue. If possible, could you please let me know when it is expected to be resolved?

roy.ben.hayun · August 13, 2024, 7:42pm

tried now S2A_MSIL1C_20240812T081611_N0511_R121_T36RXV_20240812T105014
still not working.

I’m using a manual hack meanwhile… still no fix for this?
what do people do all over the community?

egremmer · August 15, 2024, 7:26am

@roy.ben.hayun I’m using a manual hack as well.

@Sen2cor_dev_team it would be great if there would be at least some communication about when we can expect a compatible version of Sen2Cor. It has been over 3 weeks, and no updates at all.

UrielMendoza · August 20, 2024, 6:25pm

Hello
Could you share the manual trick? I also have a stopped process and we need to activate it again.

Thank you

egremmer · August 23, 2024, 9:48am

Sorry for the late reply @UrielMendoza.
The manual hack we implemented is to extend the xml check in L2A_Config.py from productVersion 14 to include 15. Note that this will raise a plethora of errors. However, in our case, the output seems largely unaffected.

@Sen2cor_dev_team it has been a month. This is getting ridiculous. Any updates?

bschumac · September 20, 2024, 7:02am

@Sen2cor_dev_team could you please provide an update when you plan to release a new version of sen2cor? This is really a big problem for all institutions who run sen2cor for specific applications operationally.
Unfortunately we cannot depend on a hack on our side because we need a stable running sen2cor for our processing chain.

diana_harosa · September 20, 2024, 10:11am

Dear @bschumac,

A new sen2cor version will be available early next week.

seb · October 3, 2024, 2:52pm

Dear @diana_harosa @Sen2cor_dev_team : is there any update on that?

As pointed out by @carlo in this post, this issue is impeding basic opening of Sentinel-2 products with the SNAP engine, and thus seriously affecting our processing chain since late July.

diana_harosa · October 3, 2024, 3:13pm

Dear @seb
Sen2cor 2.12 is already available: