cdm_reader_mapper.mdf_reader package¶

Common Data Model (CDM) MDF reader package.

Subpackages¶

Submodules¶

cdm_reader_mapper.mdf_reader.properties module¶

Common Data Model (CDM) reader properties.

cdm_reader_mapper.mdf_reader.reader module¶

Common Data Model (CDM) MDF reader.

cdm_reader_mapper.mdf_reader.reader.read_data(data_file, mask_file=None, info_file=None, data_format='parquet', imodel=None, col_subset=None, encoding=None, delimiter=None, **kwargs)[source]¶

Read MDF data which is already on a pre-defined data model.

Parameters:

data_file (str) – The data file (including path) to be read.
mask_file (str, optional) – The validation file (including path) to be read.
info_file (str, optional) – The information file (including path) to be read.
data_format ({"csv", "parquet", "feather"}, default: "parquet") – Format of input data file(s).
imodel (str, optional) – Name of internally available input data model, e.g. icoads_r300_d704.
col_subset (str, tuple or list, optional) – Specify the section or sections of the file to write.
- For multiple sections of the tables: e.g col_subset = [columns0,…,columnsN]
- For a single section: e.g. list type object col_subset = [columns]
Column labels could be both string or tuple.
encoding (str, optional) – The encoding of the input file. Overrides the value in the imodel schema file.
delimiter (str, optional) – The delimiter used in the input file. Overrides the value in the imodel schema file.
**kwargs (Any) – Key-word arguments that will be passed to read fuunction.

Return type:

DataBundle

Returns:

cdm_reader_mapper.DataBundle – DataBundle containing MDF data.

cdm_reader_mapper.mdf_reader.writer module¶

Common Data Model (CDM) MDF writer.

cdm_reader_mapper.mdf_reader.writer.write_data(data, mask=None, data_format='parquet', dtypes=None, parse_dates=False, encoding='utf-8', out_dir='.', prefix=None, suffix=None, extension=None, filename=None, separator='_', col_subset=None, delimiter=',', **kwargs)[source]¶

Write pandas.DataFrame to MDF file on file system.

Parameters:

data (pandas.DataFrame or Iterable[pd.DataFrame]) – Data to export.
mask (pandas.DataFrame or Iterable[pd.DataFrame], optional) – Validation mask to export.
data_format ({"csv", "parquet", "feather"}, default: "parquet") – Format of output data file(s).
dtypes (dict, optional) – Dictionary of data types on data. Dump dtypes and parse_dates to json information file.
parse_dates (list | bool, default: False) – Information of how to parse dates in data. Dump dtypes and parse_dates to json information file. For more information see pandas.read_csv().
encoding (str, default: "utf-8") – A string representing the encoding to use in the output file, defaults to utf-8.
out_dir (str, default: ".") – Path to the output directory.
prefix (str, optional) – Prefix of file name structure: <prefix>-data-*<suffix>.<extension>.
suffix (str, optional) – Suffix of file name structure: <prefix>-data-*<suffix>.<extension>.
extension (str, optional) – Extension of file name structure: <prefix>-data-*<suffix>.<extension>. By default, extension depends on data_format.
filename (str or dict, optional) – Name of the output file name(s). List one filename for both data and mask ({“data”:<filenameD>, “mask”:<filenameM>}). By default, automatically create file name from table name, prefix and suffix.
separator (str, optional) – Separator to join the file name pattern components (default “_”).
col_subset (str, tuple or list, optional) – Specify the section or sections of the file to write.
- For multiple sections of the tables: e.g col_subset = [columns0,…,columnsN]
- For a single section: e.g. list type object col_subset = [columns]
Column labels could be both string or tuple.
delimiter (str, default: ",") – Character or regex pattern to treat as the delimiter while reading with df.to_csv.
**kwargs (Any) – Additional keyword-arguments passed to to_csv when data_format is ‘csv’.

Raises:

ValueError – If data_foramt is not one of ‘csv’, ‘parquet’ or ‘feather’. If type of data and type of mask do not match.