Opened 5 years ago

Last modified 2 years ago

#1922 assigned defect

netcdf preserve file metadata — at Version 2

Reported by: Dimitar Misev Owned by: Bang Pham Huu
Priority: major Milestone: 11.0
Component: wcst_import Version: development
Keywords: Cc: Vlad Merticariu
Complexity: Medium

Description (last modified by Dimitar Misev)

When importing netcdf files the metadata should be preserved precisely, and not converted to string. For example in eobstest

	short tg(time, latitude, longitude) ;
		tg:long_name = "mean temperature" ;
		tg:units = "Celsius" ;
		tg:standard_name = "air_temperature" ;
		tg:_FillValue = -9999s ;
		tg:scale_factor = 0.01f ;

_FillValue and scale_factor should be kept as short and float values, and should not be converted to string.

Furthermore, some metadata is not preserved at all (check unix, E, N)?

Input file metadata (in wcps_irregular_time_nc):

netcdf irr_time {
dimensions:
	unix = 15 ;
	N = 1 ;
	E = 1 ;
variables:
	short band_1(unix, N, E) ;
		band_1:valid_min = -32768s ;
		band_1:valid_max = 32767s ;
		band_1:units = "10^0" ;
	short band_2(unix, N, E) ;
		band_2:valid_min = -32768s ;
		band_2:valid_max = 32767s ;
		band_2:units = "10^0" ;
	short band_3(unix, N, E) ;
		band_3:valid_min = -32768s ;
		band_3:valid_max = 32767s ;
		band_3:units = "10^0" ;
	short band_4(unix, N, E) ;
		band_4:valid_min = -32768s ;
		band_4:valid_max = 32767s ;
		band_4:units = "10^0" ;
	short band_5(unix, N, E) ;
		band_5:valid_min = -32768s ;
		band_5:valid_max = 32767s ;
		band_5:units = "10^0" ;
	short band_6(unix, N, E) ;
		band_6:valid_min = -32768s ;
		band_6:valid_max = 32767s ;
		band_6:units = "10^0" ;
	short band_7(unix, N, E) ;
		band_7:valid_min = -32768s ;
		band_7:valid_max = 32767s ;
		band_7:units = "10^0" ;
	double E(E) ;
		E:axis = "X" ;
		E:standard_name = "longitude" ;
		E:units = "m" ;
	double N(N) ;
		N:axis = "Y" ;
		N:standard_name = "latitude" ;
		N:units = "m" ;
	double unix(unix) ;
		unix:axis = "T" ;
		unix:standard_name = "unix" ;
		unix:units = "d" ;

// global attributes:
		:Conventions = "CF-1.6, ACDD-1.3" ;
		:date_created = "2016-04-12T11:11:42.114427" ;
		:history = "NetCDF-CF file created by datacube version \'1.0.2\' at 20160412." ;
		:product_version = "0.0.0" ;
		:source = "This data is a reprojection and retile of Landsat surface reflectance scene data." ;
		:summary = "These files are experimental, short lived, and the format will change." ;
		:title = "Experimental Data files From the Australian Geoscience Data Cube - DO NOT USE" ;
}

output file metadata:

netcdf \153-irr_cube_3D_time_irregular {
dimensions:
	unix = 15 ;
	N = 1 ;
	E = 1 ;
variables:
	short band_1(unix, N, E) ;
		band_1:valid_min = -32768s ;
		band_1:valid_max = 32767s ;
		band_1:missing_value = -999s ;
		band_1:_FillValue = -999s ;
		band_1:description = "Nadir BRDF Adjusted Reflectance 0.43-0.45 microns (Coastal Aerosol)" ;
		band_1:product_version = "0.0.0" ;
		band_1:test_empty_attribute = "" ;
		band_1:units = "10^0" ;
	short band_2(unix, N, E) ;
		band_2:valid_min = -32768s ;
		band_2:valid_max = 32767s ;
		band_2:missing_value = -999s ;
		band_2:_FillValue = -999s ;
		band_2:units = "10^0" ;
	short band_3(unix, N, E) ;
		band_3:valid_min = -32768s ;
		band_3:valid_max = 32767s ;
		band_3:missing_value = -999s ;
		band_3:_FillValue = -999s ;
		band_3:units = "10^0" ;
	short band_4(unix, N, E) ;
		band_4:valid_min = -32768s ;
		band_4:valid_max = 32767s ;
		band_4:missing_value = -999s ;
		band_4:_FillValue = -999s ;
		band_4:units = "10^0" ;
	short band_5(unix, N, E) ;
		band_5:valid_min = -32768s ;
		band_5:valid_max = 32767s ;
		band_5:missing_value = -999s ;
		band_5:_FillValue = -999s ;
		band_5:units = "10^0" ;
	short band_6(unix, N, E) ;
		band_6:valid_min = -32768s ;
		band_6:valid_max = 32767s ;
		band_6:missing_value = -999s ;
		band_6:_FillValue = -999s ;
		band_6:units = "10^0" ;
	short band_7(unix, N, E) ;
		band_7:valid_min = -32768s ;
		band_7:valid_max = 32767s ;
		band_7:missing_value = -999s ;
		band_7:_FillValue = -999s ;
		band_7:Conventions = "CF-1.6, ACDD-1.3" ;
		band_7:date_created = "2016-04-12T11:11:42.114427" ;
		band_7:units = "10^0" ;
	double E(E) ;
	double N(N) ;
	double unix(unix) ;
		unix:directPositions = "-1000012.5" ;
		unix:max = "1387331687.45" ;
		unix:min = "1370137750.3699999" ;

// global attributes:
		:Conventions = "CF-1.6, ACDD-1.3" ;
		:date_created = "2016-04-12T11:11:42.114427" ;
		:history = "NetCDF-CF file created by datacube version \'1.0.2\' at 20160412." ;
		:product_version = "0.0.0" ;
		:source = "This data is a reprojection and retile of Landsat surface reflectance scene data." ;
		:summary = "These files are experimental, short lived, and the format will change." ;
		:test_empty_attribute = "" ;
		:title = "Experimental Data files From the Australian Geoscience Data Cube - DO NOT USE" ;

Change History (2)

comment:1 by Dimitar Misev, 5 years ago

Description: modified (diff)

comment:2 by Dimitar Misev, 5 years ago

Description: modified (diff)
Note: See TracTickets for help on using tickets.