Marine cdm_reader_mapper toolbox documentation

The cdm_reader_mapper toolbox is a python3 tool designed for

• reading original marine-meteorological data files compliant with a user specified data model (Data Models) into a Marine Data Format (MDF) file.

• mapping observed meteorological variables and its associated metadata from a data model (Data Models) to the C3S CDS Common Data Model (CDM) format or imodel as called in this tool.

• to detect and flag or remove duplicated observations

It was developed with the initial idea of reading data from the International Comprehensive Ocean-Atmosphere Data Set (ICOADS) stored in the International Maritime Meteorological Archive (IMMA) data format. In the meanwhile, it can read data C-RAID Copernicus in situ project too.

The tool has been further enhanced to account for any marine meteorological data format, provided that this data meets the following specifications:

• Data is stored in a human-readable manner: ASCII.

• Data is stored in a Network Common Data Format: NetCDF.

• Data is organized in single line reports (e.g. rows of observations separated by a delimiter like .csv).

• Reports have a coherent internal structure that can be modelized.

• Reports are fixed width or field delimited types.

• Reports can be organized in sections, in which case each section can be of different types (fixed width of delimited).

Several data models have been added to the tool including both the IMMA and the C-RAID schema.

Note

Data from other data models than those already available can be read, providing that this data meets the basic specifications listed above. A data model can be built externally and fed into the tool.

The read_mdf function uses the information provided in a data model to read meteorological data into a so-called cdm_reader_mapper.DataBundle object. This object includes several main attributes:

• data: meteorogical data stored as a pandas.DataFrame, with the column names and data types set according to each data element’s description specified in the data model or schema.

• mask: mask of data that validates data elements against the schema provided.

The reader allows for basic transformations of the data. This feature includes basic numeric data decoding (base36, signed_overpunch) and numeric data conversion (scale and offset).

In addition, the cdm_reader_mapper.DataBundle object has several main method functions:

• DataBundle.map_model(): map observed variables and its associated metadata from a data model or models combination to the standardized C3S CDS Common Data Model (CDM) format.

• DataBundle.duplicate_check(): detect duplicated observations

• DataBundle.flag_duplicates(): flag detected duplicated observations

• DataBundle.remove_duplicates(): remove detected duplicated observations

• DataBundle.write(): save both observational MDF files as a coma-separated list and observational standardized CDM tables as pipe-seperated lists

UserAPI

• API reference
  • Read data from disk
  • DataBundle
  • Useful functions
  • DupDetect

All Modules

• cdm_reader_mapper

GitHub repository

• glamod/CDM-Reader-Mapper-Toolbox

About

Version:

Citation:

License: