GRD assets available in Google Earth Engine

simonf · February 17, 2016, 4:19pm

Hello everyone,

As some of you know, I work on the Google Earth Engine team.

Last week, we announced that S1 GRD scenes are now available to all Earth Engine users. See the dataset description and documentation pages. The assets are preprocessed in the S1 Toolbox - they go through thermal noise removal, calibration and terrain correction.

Big thanks to the Copernicus program and the Toolbox developers for making this possible, and to forum members for answering our questions.

You can apply to be an Earth Engine user here.

Simon

css · February 18, 2016, 4:24pm

Are you planning to upload all the S1 GRD scenes made available by ESA, or are you going to limit them geographically/temporarily?
Will you upload the new scenes as they become available or in batches?

simonf · February 18, 2016, 4:53pm

We are going to have the all existing GRD scenes (as of today we have 143K out of about 180K total scenes already available, with the rest coming in shortly), and we are ingesting all the new scenes daily.

css · February 18, 2016, 5:10pm

Reading the documentation page it is not clear to me which S1TBX operations you applied in the first pre-processing step (Thermal noise removal). Did you apply the operator called ‘S-1 Thermal Noise Removal’, the one called ‘S-1 GRD Border Noise Removal’ or both?

The following sentence from the documentation combines information from both operators:
Thermal noise removal uses the noise vectors to remove dark strips near scene edges with invalid data (this operation cannot be applied to some earlier images).

simonf · February 18, 2016, 5:23pm

In the graph builder and the graph XML file, this operation is called just ‘Thermal Noise Removal’. It corresponds to ‘S-1 Thermal Noise Removal’ in the main UI.

valgur · February 18, 2016, 5:32pm

For the curious, some stats about the collection and a metadata example I managed to gather from twiddling with the Earth Engine Code Editor:

var collection = ee.ImageCollection('COPERNICUS/S1_GRD');

var count = collection.size();
print('Count: ', count);

var byInstrumentMode = collection.aggregate_histogram('instrumentMode');
print('By mode:', byInstrumentMode);

var dates = ee.List(collection.get('date_range'));
var dateRange = ee.DateRange(dates.get(0), dates.get(1));
print('Date range: ', dateRange);

print(collection.filter(ee.Filter.eq('instrumentMode', 'IW')).first());

yields

Count: 101447
Date range: DateRange [2014-10-03 00:00:00, 2016-02-11 00:00:00]
By mode:
	EW: 7472
	IW: 91199
	SM: 2776
Image COPERNICUS/S1_GRD/S1A_IW_GRDH_1SDH_20141006T074304_20141006T074324_002706_00306E_07A2 (3 bands)
	type: Image
	id: COPERNICUS/S1_GRD/S1A_IW_GRDH_1SDH_20141006T074304_20141006T074324_002706_00306E_07A2
	version: 1455300028581000
	bands: List (3 elements)
		0: "HH", float, EPSG:32627, 28010x18118 px
		1: "HV", float, EPSG:32627, 28010x18118 px
		2: "angle", float, EPSG:4326, 21x8 px
	properties: Object (59 properties)
		GRD_Post_Processing_facility_country: Italy
		GRD_Post_Processing_facility_name: ESRIN headquarters
		GRD_Post_Processing_facility_organisation: ESRIN
		GRD_Post_Processing_facility_site: Rome
		GRD_Post_Processing_software_name: Sentinel-1 IPF
		GRD_Post_Processing_software_version: 002.36
		GRD_Post_Processing_start: 1412595469012
		GRD_Post_Processing_stop: 1412596119000
		HH_log_bias: 65535
		HH_log_gain: 2337.2962622377704
		HH_original_max: 6.90268611907959
		HH_original_min: -118.93456268310547
		HV_log_bias: 65535
		HV_log_gain: 1999.1666597038397
		HV_original_max: -6.533178329467773
		HV_original_min: -121.19938659667969
		SLC_Processing_facility_country: Italy
		SLC_Processing_facility_name: ESRIN headquarters
		SLC_Processing_facility_organisation: ESRIN
		SLC_Processing_facility_site: Rome
		SLC_Processing_software_name: Sentinel-1 IPF
		SLC_Processing_software_version: 002.36
		SLC_Processing_start: 1412595575000
		SLC_Processing_stop: 1412595966000
		cycleNumber: 17
		familyName: SENTINEL-1
		instrument: Synthetic Aperture Radar
		instrumentConfigurationID: 3
		instrumentMode: IW
		instrumentSwath: IW
		missionDataTakeID: 12398
		nssdcIdentifier: 0000-000A
		orbitNumber_start: 2706
		orbitNumber_stop: 2706
		orbitProperties_ascendingNodeTime: 1412579396429
		orbitProperties_pass: DESCENDING
		phaseIdentifier: 1
		platform_number: A
		productClass: S
		productClassDescription: SAR Standard L1 Product
		productComposition: Slice
		productTimelinessCategory: Fast-24h
		productType: GRD
		relativeOrbitNumber_start: 11
		relativeOrbitNumber_stop: 11
		resolution: H
		resolution_meters: 10
		segmentStartTime: 1412581280684
		sliceNumber: 5
		sliceProductFlag: true
		startTimeANX: 1987720
		stopTimeANX: 2007695
		system:footprint: LinearRing, 20 vertices
		system:index: S1A_IW_GRDH_1SDH_20141006T074304_20141006T074324_002706_00306E_07A2
		system:time_end: 1412581404124
		system:time_start: 1412581384149
		toolbox_version: 2.0
		totalSlices: 5
		transmitterReceiverPolarisation: ["HH","HV"]

simonf · February 18, 2016, 5:39pm

Thank you, valgur@. To clarify the asset counts, the user-visible collection currently has 101K assets. This collection is updated once a week. The underlying raw collection is updated daily, and has 143K assets.

edwardpmorris · February 24, 2016, 11:13am

Hi simonf and everyone, great news that the processed S1 GRD collection is available at full ‘spatial resolution’ via GEE; this potentially saves everyone time and hassle, considering the sometimes slow scihub API.

Processing this collection with GEE is awesome! Still sometimes there are a few things ‘best’ done offline, please could you outline how to access this dataset directly via gsutil and the Google Cloud Platform as can be done for the Landsat archive.

From a quick look in https://console.cloud.google.com/storage/browser/earthengine-public/ I did not see the Copernicus collection listed.

cheers Ed.

simonf · February 25, 2016, 2:18pm

Ed - unfortunately, we are not currently planning to make the raw assets available via cloud.google.com.

css · February 25, 2016, 5:03pm

simonf

I have looked at a few S1 images on GEE and they all seem to have been capped at 0db (sigma 0).
Is this the result of the clamping of the pixel values to the 1st and 99th percentile, as described in the documentation?
Many resolution cells genuinely have high scattering values, they are not anomalous.

simonf · February 25, 2016, 9:34pm

css@: Yes, it’s due to clamping. The word “anomalous” is not best choice there, I should fix it. The reasoning is that since values >0 dB are typically returns from man-made structures, clamping them will lose less valuable information, as compared to not using clamping and having large tails of very high and very low values - which would have reduced the amount of information available for biomass or water detection. This reasoning may be flawed, and we are open to reprocessing the collection in a different way if there are obvious issues with it.

glemoine · February 26, 2016, 12:08am

You can find values > 0 dB, e.g. over mountainous areas. If frames are scaled on the 1% and 99% quantiles, it matters how many pixels in a scene are in these extreme quantiles. So, if you have scenes that include some urban areas, the quantiles > 0 dB tend to be small, and everything gets clamped. Over extended mountainous areas, where large quantiles can be > 0 dB, positive dB values are retained. So 0 dB is not some absolute threshold.

erehm · March 5, 2016, 4:29pm

Is there a plan to update the COPERNICUS/S1_GRD collection more rapidly (e.g., daily)?

(And same question for the MODIS Daily L2G 250 m MODIS/MOD09GQ…)

I’m developing a EE tool to help us pilot our underwater gliders near the ice edge in Baffin Bay. The latest S1 and MODIS 250 m data is what we need. Right now we use PolarView and other tools, select the swaths manually and import into Google Earth. Uggh…

simonf · March 7, 2016, 2:17am

erehm@: All periodic collections are actually updated daily. For technical reasons, some of the datasets referencing those collections are updated weekly, but you can always access the underlying raw collection to see the most recent data.

For Sentinel GRD, use this:

function db(image) {
  return ee.call("S1.dB", image);
}

var rawCollection = ee.ImageCollection('COPERNICUS/S1_GRD_INT');
var collection = rawCollection.map(db);

At this point the variable ‘collection’ can be used exactly in the same way as ee.ImageCollection(‘COPERNICUS/S1_GRD’).

The MODIS products don’t have this separation - all MODIS datasets are updated daily.

erehm · March 7, 2016, 10:15pm

simonf@: Thanks. That worked.

Regarding MODIS, I’m not getting anything newer than 2016-03-03 for these two daily MYD09 data sets. (Also, what are the MOD09 variants?)

var modisName = ‘MODIS/MYD09GQ’; // 250 m bands
//var modisName = ‘MODIS/MYD09GA’; // 500 m, 1 km
print(modisName)
var startDate = ‘2016-03-01’;
var endDate = ‘2016-03-07’;
var cModis = ee.ImageCollection(modisName).filterDate(startDate, endDate).sort(‘system:time_start’, false);
print(cModis);

simonf · March 7, 2016, 11:08pm

We are usually about a day behind LPDAAC, and the last mosaic they have is for Mar 4, added today: http://e4ftl01.cr.usgs.gov/MOLA/MYD09GQ.005/

MOD09 are the Terra data, MYD09 are the Aqua data.

Fernerkundung · March 16, 2016, 3:34pm

Thanks for making the GRD dataset available in EE and also openly sharing information about the creation process @simonf

I am trying to recreate your pre-processing steps to process some data locally and develop an algorithm that I then port over to EE. From the documentation I am not 100% clear which pre-processing steps are in which order (especially with noise removal).

Are you able to share the graph you are using/used for pre-processing the EE data?

simonf · March 16, 2016, 8:53pm

@Fernerkundung: Can you point out where our docs are unclear about the preprocessing steps? I’d like to fix that. Currently they say:

Each scene was pre-processed with Sentinel-1 Toolbox using the following steps:

Thermal noise removal

Radiometric calibration

Terrain correction using SRTM 30 or ASTER DEM for areas greater than ±60° latitude, where SRTM is not available.

The final terrain corrected values are converted to decibels via log scaling (10*log10(x)) and quantized to 16-bits.

mengdahl · March 17, 2016, 6:07am

Thermal noise has not been removed from S-1 GRD products.

simonf · March 17, 2016, 6:15am

Thank you, I’ll check.