{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Mapping data from ICOADS deck 704 to the Common Data Model (CDM)\n", "\n", "Here we extract supplemental metadata from [ICOADSv3.0](https://icoads.noaa.gov/r3.html) stored in the [IMMA version 1](https://icoads.noaa.gov/e-doc/imma/R3.0-imma1.pdf) format. \n", "We will then map this data (including the supplemental data) to the Common Data Model (CDM) format defined in the [CDM Documentation](https://github.com/glamod/common_data_model/blob/master/cdm_latest.pdf)..\n", "\n", "The supplementary data are mapped to the CDM using the [tables](https://github.com/glamod/cdm_reader_mapper/tree/main/cdm_reader_mapper/cdm_mapper/tables/icoads/r300/d704) and [codes](https://github.com/glamod/cdm_reader_mapper/tree/main/cdm_reader_mapper/cdm_mapper/codes/icoads/r300/d704) specific to deck 704. The generic ICOADS [tables](https://github.com/glamod/cdm_reader_mapper/tree/main/cdm_reader_mapper/cdm_mapper/tables/icoads) are used to map the common ICOADS data components." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We are analysing deck: `704`, the [US Marine Meteorological Journals Collection](https://icoads.noaa.gov/usmmj.html)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "from __future__ import annotations\n", "\n", "import pandas as pd\n", "\n", "from cdm_reader_mapper import read_mdf, test_data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We first read the supplemental data information from the `c99` imma format for a subset of the data (e.g. 1878/10). For this we need to use the `\"icoads_r300_d704\"` schema. The convention for schema names is: `\"format_version_deck\"`\n", "\n", "* format/data model: \"icoads\"\n", "* version/release: \"r300\" (release 3.0.0)\n", "* deck: \"d704\"\n", "\n", "In this notebook we load the icoads r3.0.0 deck 704 test file to use as an example." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "WARNING:root:Unknown column_type 'object' for column '('c8', 'PUID')'\n", "WARNING:root:Unknown column_type 'object' for column '('c95', 'ARCR')'\n", "WARNING:root:Unknown column_type 'object' for column '('c96', 'ARCI')'\n", "WARNING:root:Unknown column_type 'object' for column '('c97', 'ARCE')'\n", "WARNING:root:Unknown column_type 'object' for column '('c99_sentinel', 'BLK')'\n", "C:\\Users\\llierham\\mobaxterm\\github_ll\\cdm_reader_mapper\\src\\cdm_reader_mapper\\mdf_reader\\utils\\validators.py:240: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n", " to_bool = data[validated_columns].applymap(convert_str_boolean)\n", "C:\\Users\\llierham\\mobaxterm\\github_ll\\cdm_reader_mapper\\src\\cdm_reader_mapper\\mdf_reader\\utils\\validators.py:241: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n", " false_mask = to_bool.applymap(_is_false)\n", "C:\\Users\\llierham\\mobaxterm\\github_ll\\cdm_reader_mapper\\src\\cdm_reader_mapper\\mdf_reader\\utils\\validators.py:242: FutureWarning: DataFrame.applymap has been deprecated. Use DataFrame.map instead.\n", " true_mask = to_bool.applymap(_is_true)\n" ] } ], "source": [ "schema = \"icoads_r300_d704\"\n", "\n", "data_file_path = test_data.test_icoads_r300_d704[\"source\"] # Load the example file from the cdm_reader_mapper test data\n", "data_bundle = read_mdf(data_file_path, imodel=schema)\n", "data_raw = data_bundle.data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The data from the c99 column for this deck is separated in the following sub sections:\n", "- c99_sentinal\n", "- c99_journal\n", "- c99_voyage\n", "- c99_daily\n", "- c99_data4\n", "- c99_data5" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
ATTIATTLBLK
0990None
1990None
2990None
3990None
4990None
\n", "
" ], "text/plain": [ " ATTI ATTL BLK\n", "0 99 0 None\n", "1 99 0 None\n", "2 99 0 None\n", "3 99 0 None\n", "4 99 0 None" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw.c99_sentinel.head()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sentinelreel_nojournal_noframe_noship_namejournal_edrigship_materialvessel_typevessel_lengthvessel_beamcommandercountryscrew_paddlehold_depthtonnagebaro_typebaro_heightbaro_cdatebaro_locbaro_unitsbaro_corthermo_mountSST_I
0100200180003Panay78011118737S.P.Bray,Jr013231190214NoneBulkhead of cabin1- .1022None
1100200180003Panay78011118737S.P.Bray,Jr013231190214NoneBulkhead of cabin1- .1022None
2100200180003Panay78011118737S.P.Bray,Jr013231190214NoneBulkhead of cabin1- .1022None
3100200180003Panay78011118737S.P.Bray,Jr013231190214NoneBulkhead of cabin1- .1022None
4100200180003Panay78011118737S.P.Bray,Jr013231190214NoneBulkhead of cabin1- .1022None
\n", "
" ], "text/plain": [ " sentinel reel_no journal_no frame_no ship_name journal_ed rig ship_material \\\n", "0 1 002 0018 0003 Panay 78 01 1 \n", "1 1 002 0018 0003 Panay 78 01 1 \n", "2 1 002 0018 0003 Panay 78 01 1 \n", "3 1 002 0018 0003 Panay 78 01 1 \n", "4 1 002 0018 0003 Panay 78 01 1 \n", "\n", " vessel_type vessel_length vessel_beam commander country screw_paddle \\\n", "0 1 187 37 S.P.Bray,Jr 01 3 \n", "1 1 187 37 S.P.Bray,Jr 01 3 \n", "2 1 187 37 S.P.Bray,Jr 01 3 \n", "3 1 187 37 S.P.Bray,Jr 01 3 \n", "4 1 187 37 S.P.Bray,Jr 01 3 \n", "\n", " hold_depth tonnage baro_type baro_height baro_cdate baro_loc \\\n", "0 23 1190 2 14 None Bulkhead of cabin \n", "1 23 1190 2 14 None Bulkhead of cabin \n", "2 23 1190 2 14 None Bulkhead of cabin \n", "3 23 1190 2 14 None Bulkhead of cabin \n", "4 23 1190 2 14 None Bulkhead of cabin \n", "\n", " baro_units baro_cor thermo_mount SST_I \n", "0 1 - .102 2 None \n", "1 1 - .102 2 None \n", "2 1 - .102 2 None \n", "3 1 - .102 2 None \n", "4 1 - .102 2 None " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "pd.options.display.max_columns = None\n", "data_raw.c99_journal.head()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sentinelreel_nojournal_noframe_startfrom_cityto_city
0200200180014BostonRio de Janeiro
1200200180014BostonRio de Janeiro
2200200180014BostonRio de Janeiro
3200200180014BostonRio de Janeiro
4200200180014BostonRio de Janeiro
\n", "
" ], "text/plain": [ " sentinel reel_no journal_no frame_start from_city to_city\n", "0 2 002 0018 0014 Boston Rio de Janeiro\n", "1 2 002 0018 0014 Boston Rio de Janeiro\n", "2 2 002 0018 0014 Boston Rio de Janeiro\n", "3 2 002 0018 0014 Boston Rio de Janeiro\n", "4 2 002 0018 0014 Boston Rio de Janeiro" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw.c99_voyage.head()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sentinelreel_nojournal_noframe_startframeyearmonthdaydistancelat_deg_anlat_min_anlat_hemis_anlon_deg_anlon_min_anlon_hemis_anlat_deg_onlat_min_onlat_hemis_onlon_deg_oflon_min_oflon_hemis_ofcurrent_speedcurrent_direction
0300200180014001518781020NaN<NA><NA>None<NA><NA>None4220N6630W0.1E
1300200180014001518781020NaN<NA><NA>None<NA><NA>None4220N6630W0.1E
2300200180014001518781020NaN<NA><NA>None<NA><NA>None4220N6630W0.1E
3300200180014001518781020NaN<NA><NA>None<NA><NA>None4220N6630W0.1E
4300200180014001518781020NaN<NA><NA>None<NA><NA>None4220N6630W0.1E
\n", "
" ], "text/plain": [ " sentinel reel_no journal_no frame_start frame year month day distance \\\n", "0 3 002 0018 0014 0015 1878 10 20 NaN \n", "1 3 002 0018 0014 0015 1878 10 20 NaN \n", "2 3 002 0018 0014 0015 1878 10 20 NaN \n", "3 3 002 0018 0014 0015 1878 10 20 NaN \n", "4 3 002 0018 0014 0015 1878 10 20 NaN \n", "\n", " lat_deg_an lat_min_an lat_hemis_an lon_deg_an lon_min_an lon_hemis_an \\\n", "0 None None \n", "1 None None \n", "2 None None \n", "3 None None \n", "4 None None \n", "\n", " lat_deg_on lat_min_on lat_hemis_on lon_deg_of lon_min_of lon_hemis_of \\\n", "0 42 20 N 66 30 W \n", "1 42 20 N 66 30 W \n", "2 42 20 N 66 30 W \n", "3 42 20 N 66 30 W \n", "4 42 20 N 66 30 W \n", "\n", " current_speed current_direction \n", "0 0.1 E \n", "1 0.1 E \n", "2 0.1 E \n", "3 0.1 E \n", "4 0.1 E " ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw.c99_daily.head()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sentinelreel_nojournal_noframe_startframeyearmonthdaytime_indhourship_speedcompass_indship_course_compasscompass_correctionship_course_truewind_dir_magwind_dir_truewind_forcebarometertemp_indattached_thermometerair_temperaturewet_bulb_temperaturesea_temperaturepresent_weathercloudssky_clearsea_state
0400200180014001518781020128.5NoneEXS<NA>NoneWSWNone06296015.8NaNNaNNaNBOCCU5R
1400200180014001518781020148.5NoneEXS<NA>NoneWSWNone06296015.6NaNNaNNaNBOCSC3R
2400200180014001518781020168.5NoneEXS<NA>NoneWNone06296215.64.8NaN5.2OCGSC0R
3400200180014001518781020188.0NoneEXS<NA>NoneWNone06296415.64.8NaN5.2CGSC0R
44002001800140015187810201108.5NoneEXS<NA>NoneWNone06296915.74.8NaN5.0BCSC2L
\n", "
" ], "text/plain": [ " sentinel reel_no journal_no frame_start frame year month day time_ind \\\n", "0 4 002 0018 0014 0015 1878 10 20 1 \n", "1 4 002 0018 0014 0015 1878 10 20 1 \n", "2 4 002 0018 0014 0015 1878 10 20 1 \n", "3 4 002 0018 0014 0015 1878 10 20 1 \n", "4 4 002 0018 0014 0015 1878 10 20 1 \n", "\n", " hour ship_speed compass_ind ship_course_compass compass_correction \\\n", "0 2 8.5 None EXS \n", "1 4 8.5 None EXS \n", "2 6 8.5 None EXS \n", "3 8 8.0 None EXS \n", "4 10 8.5 None EXS \n", "\n", " ship_course_true wind_dir_mag wind_dir_true wind_force barometer temp_ind \\\n", "0 None WSW None 06 2960 1 \n", "1 None WSW None 06 2960 1 \n", "2 None W None 06 2962 1 \n", "3 None W None 06 2964 1 \n", "4 None W None 06 2969 1 \n", "\n", " attached_thermometer air_temperature wet_bulb_temperature \\\n", "0 5.8 NaN NaN \n", "1 5.6 NaN NaN \n", "2 5.6 4.8 NaN \n", "3 5.6 4.8 NaN \n", "4 5.7 4.8 NaN \n", "\n", " sea_temperature present_weather clouds sky_clear sea_state \n", "0 NaN BOC CU 5 R \n", "1 NaN BOC SC 3 R \n", "2 5.2 OCG SC 0 R \n", "3 5.2 CG SC 0 R \n", "4 5.0 BC SC 2 L " ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw.c99_data4.head()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
sentinelreel_nojournal_noframe_startframeyearmonthdaytime_indhourship_speedcompass_indship_course_compassblankship_course_truewind_dir_magwind_dir_truewind_forcebarometertemp_indattached_thermometerair_temperaturewet_bulb_temperaturesea_temperaturepresent_weathercloudssky_clearsea_statecompass_correction_indcompass_correctioncompass_correction_dir
0NoneNoneNoneNoneNone<NA><NA><NA>None<NA>NaNNoneNoneNoneNoneNoneNoneNoneNoneNoneNaNNaNNaNNaNNoneNone<NA>NoneNoneNaNNone
1NoneNoneNoneNoneNone<NA><NA><NA>None<NA>NaNNoneNoneNoneNoneNoneNoneNoneNoneNoneNaNNaNNaNNaNNoneNone<NA>NoneNoneNaNNone
2NoneNoneNoneNoneNone<NA><NA><NA>None<NA>NaNNoneNoneNoneNoneNoneNoneNoneNoneNoneNaNNaNNaNNaNNoneNone<NA>NoneNoneNaNNone
3NoneNoneNoneNoneNone<NA><NA><NA>None<NA>NaNNoneNoneNoneNoneNoneNoneNoneNoneNoneNaNNaNNaNNaNNoneNone<NA>NoneNoneNaNNone
4NoneNoneNoneNoneNone<NA><NA><NA>None<NA>NaNNoneNoneNoneNoneNoneNoneNoneNoneNoneNaNNaNNaNNaNNoneNone<NA>NoneNoneNaNNone
\n", "
" ], "text/plain": [ " sentinel reel_no journal_no frame_start frame year month day time_ind \\\n", "0 None None None None None None \n", "1 None None None None None None \n", "2 None None None None None None \n", "3 None None None None None None \n", "4 None None None None None None \n", "\n", " hour ship_speed compass_ind ship_course_compass blank ship_course_true \\\n", "0 NaN None None None None \n", "1 NaN None None None None \n", "2 NaN None None None None \n", "3 NaN None None None None \n", "4 NaN None None None None \n", "\n", " wind_dir_mag wind_dir_true wind_force barometer temp_ind \\\n", "0 None None None None None \n", "1 None None None None None \n", "2 None None None None None \n", "3 None None None None None \n", "4 None None None None None \n", "\n", " attached_thermometer air_temperature wet_bulb_temperature \\\n", "0 NaN NaN NaN \n", "1 NaN NaN NaN \n", "2 NaN NaN NaN \n", "3 NaN NaN NaN \n", "4 NaN NaN NaN \n", "\n", " sea_temperature present_weather clouds sky_clear sea_state \\\n", "0 NaN None None None \n", "1 NaN None None None \n", "2 NaN None None None \n", "3 NaN None None None \n", "4 NaN None None None \n", "\n", " compass_correction_ind compass_correction compass_correction_dir \n", "0 None NaN None \n", "1 None NaN None \n", "2 None NaN None \n", "3 None NaN None \n", "4 None NaN None " ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw.c99_data5.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we have separated the c99 data into the different sections, we see that this deck is composed of two types of data, which are the same:\n", " \n", " - c99_data4\n", " - c99_data5\n", " \n", "Both sections have the same name in variables. To map the correct section into the CDM it is necessary to impose a filter on the sections composed only of NaN data. \n", "The problem is that we dont know which years in the time series will have a section c99_data4 and which will have a c99_data5\n", "\n", "> Note that this solution of excluding one section, will only work for decks from which sections are exclusive: Among the sections listed in the block, only one of them appears in every report.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now use the `\"icoads_r300_d704\"` model to map the raw data to the Common Data Model [glamod/common_data_model](https://www.github.com/glamod/common_data_model). The method function `map_model` contains all the functions for the model to convert variables to the correct units and/or specification following the [CDM Documentation](https://github.com/glamod/common_data_model/blob/master/cdm_latest.pdf).\n", "\n", "To run the data model we need three things:\n", "\n", "- raw data (the data we just read above)\n", "- attributes of the raw data (sections and column names)\n", "- the name of the model" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "2026-05-07 08:31:39,537 - root - INFO - Initialized basic logging configuration successfully\n", "C:\\Users\\llierham\\mobaxterm\\github_ll\\cdm_reader_mapper\\src\\cdm_reader_mapper\\cdm_mapper\\mapper.py:75: FutureWarning: 'any' with datetime64 dtypes is deprecated and will raise in a future version. Use (obj != pd.Timestamp(0)).any() instead.\n", " list_cols = [col for col in df.columns if df[col].apply(lambda x: isinstance(x, list)).any()]\n", "C:\\Users\\llierham\\mobaxterm\\github_ll\\cdm_reader_mapper\\src\\cdm_reader_mapper\\cdm_mapper\\mapper.py:75: FutureWarning: 'any' with datetime64 dtypes is deprecated and will raise in a future version. Use (obj != pd.Timestamp(0)).any() instead.\n", " list_cols = [col for col in df.columns if df[col].apply(lambda x: isinstance(x, list)).any()]\n" ] } ], "source": [ "cdm_tables = data_bundle.map_model()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, have we succeeded in writing some of the data to the CDM format?\n", "\n", "We were looking to write the following data \n", "\n", "## Header section\n", "\n", " - Platform type and sub type\n", " - primary station id: original ship names\n", " - Longitude and Latitudes: converted from Degrees Minutes and Hemisphere to Decimal degrees\n", " - Location accuracy\n", " \n", " \n", "## Observations tables\n", "\n", "- `Observations-at`: latitude, longitude and location precision\n", "- `Observations-dpt`: latitude, longitude and location precision\n", "- `Observations-slp`: latitude, longitude and location precision\n", " - z_coordinate_type: Barometer height in feet converted to m.\n", " - original units: written in the CDM code format\n", "\n", "- `Observations-sst`: latitude, longitude and location precision\n", "- `Observations-wbt`: latitude, longitude and location precision\n", "- `Observations-wd`: latitude, longitude and location precision\n", "- `Observations-ws`: latitude, longitude and location precision\n" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
report_idregionsub_regionapplication_areaobserving_programmereport_typestation_namestation_typeplatform_typeplatform_sub_typeprimary_station_idstation_record_numberprimary_station_id_schemelongitudelatitudelocation_accuracylocation_methodlocation_qualitycrsstation_speedstation_coursestation_headingheight_of_station_above_local_groundheight_of_station_above_sea_levelheight_of_station_above_sea_level_accuracysea_level_datumreport_meaning_of_timestampreport_timestampreport_durationreport_time_accuracyreport_time_qualityreport_time_referenceprofile_idevents_at_stationreport_qualityduplicate_statusduplicatesrecord_timestamphistoryprocessing_levelprocessing_codessource_idsource_record_id
0ICOADS-300-020N16<NA><NA>[1, 7, 10, 11][5, 7, 56]0Panay2226Panay18-68.41000442.28<NA><NA>004.1155290.0<NA>0.00.0<NA><NA>21878-10-20 06:00:00113600.02<NA><NA><NA>04<NA>2026-05-07 06:31:39.602716+00:002026-05-07 06:31:39. Initial conversion from I...<NA><NA>ICOADS-3-0-0T-125-704-1878-10020N16
1ICOADS-300-020N1P<NA><NA>[1, 7, 10, 11][5, 7, 56]0Panay2226Panay18-68.02999942.31<NA><NA>004.1155290.0<NA>0.00.0<NA><NA>21878-10-20 08:00:00113600.02<NA><NA><NA>04<NA>2026-05-07 06:31:39.602716+00:002026-05-07 06:31:39. Initial conversion from I...<NA><NA>ICOADS-3-0-0T-125-704-1878-10020N1P
2ICOADS-300-020N25<NA><NA>[1, 7, 10, 11][5, 7, 56]0Panay2226Panay18-67.63999942.33<NA><NA>004.1155290.0<NA>0.00.0<NA><NA>21878-10-20 10:00:00113600.02<NA><NA><NA>04<NA>2026-05-07 06:31:39.602716+00:002026-05-07 06:31:39. Initial conversion from I...<NA><NA>ICOADS-3-0-0T-125-704-1878-10020N25
3ICOADS-300-020N2Q<NA><NA>[1, 7, 10, 11][5, 7, 56]0Panay2226Panay18-67.29000142.35<NA><NA>004.1155290.0<NA>0.00.0<NA><NA>21878-10-20 12:00:00113600.02<NA><NA><NA>04<NA>2026-05-07 06:31:39.602716+00:002026-05-07 06:31:39. Initial conversion from I...<NA><NA>ICOADS-3-0-0T-125-704-1878-10020N2Q
4ICOADS-300-020N3A<NA><NA>[1, 7, 10, 11][5, 7, 56]0Panay2226Panay18-66.90000242.37<NA><NA>004.1155290.0<NA>0.00.0<NA><NA>21878-10-20 14:00:00113600.02<NA><NA><NA>04<NA>2026-05-07 06:31:39.602716+00:002026-05-07 06:31:39. Initial conversion from I...<NA><NA>ICOADS-3-0-0T-125-704-1878-10020N3A
\n", "
" ], "text/plain": [ " report_id region sub_region application_area observing_programme \\\n", "0 ICOADS-300-020N16 [1, 7, 10, 11] [5, 7, 56] \n", "1 ICOADS-300-020N1P [1, 7, 10, 11] [5, 7, 56] \n", "2 ICOADS-300-020N25 [1, 7, 10, 11] [5, 7, 56] \n", "3 ICOADS-300-020N2Q [1, 7, 10, 11] [5, 7, 56] \n", "4 ICOADS-300-020N3A [1, 7, 10, 11] [5, 7, 56] \n", "\n", " report_type station_name station_type platform_type platform_sub_type \\\n", "0 0 Panay 2 2 26 \n", "1 0 Panay 2 2 26 \n", "2 0 Panay 2 2 26 \n", "3 0 Panay 2 2 26 \n", "4 0 Panay 2 2 26 \n", "\n", " primary_station_id station_record_number primary_station_id_scheme \\\n", "0 Panay 1 8 \n", "1 Panay 1 8 \n", "2 Panay 1 8 \n", "3 Panay 1 8 \n", "4 Panay 1 8 \n", "\n", " longitude latitude location_accuracy location_method location_quality \\\n", "0 -68.410004 42.28 0 \n", "1 -68.029999 42.31 0 \n", "2 -67.639999 42.33 0 \n", "3 -67.290001 42.35 0 \n", "4 -66.900002 42.37 0 \n", "\n", " crs station_speed station_course station_heading \\\n", "0 0 4.11552 90.0 \n", "1 0 4.11552 90.0 \n", "2 0 4.11552 90.0 \n", "3 0 4.11552 90.0 \n", "4 0 4.11552 90.0 \n", "\n", " height_of_station_above_local_ground height_of_station_above_sea_level \\\n", "0 0.0 0.0 \n", "1 0.0 0.0 \n", "2 0.0 0.0 \n", "3 0.0 0.0 \n", "4 0.0 0.0 \n", "\n", " height_of_station_above_sea_level_accuracy sea_level_datum \\\n", "0 \n", "1 \n", "2 \n", "3 \n", "4 \n", "\n", " report_meaning_of_timestamp report_timestamp report_duration \\\n", "0 2 1878-10-20 06:00:00 11 \n", "1 2 1878-10-20 08:00:00 11 \n", "2 2 1878-10-20 10:00:00 11 \n", "3 2 1878-10-20 12:00:00 11 \n", "4 2 1878-10-20 14:00:00 11 \n", "\n", " report_time_accuracy report_time_quality report_time_reference \\\n", "0 3600.0 2 \n", "1 3600.0 2 \n", "2 3600.0 2 \n", "3 3600.0 2 \n", "4 3600.0 2 \n", "\n", " profile_id events_at_station report_quality duplicate_status duplicates \\\n", "0 0 4 \n", "1 0 4 \n", "2 0 4 \n", "3 0 4 \n", "4 0 4 \n", "\n", " record_timestamp \\\n", "0 2026-05-07 06:31:39.602716+00:00 \n", "1 2026-05-07 06:31:39.602716+00:00 \n", "2 2026-05-07 06:31:39.602716+00:00 \n", "3 2026-05-07 06:31:39.602716+00:00 \n", "4 2026-05-07 06:31:39.602716+00:00 \n", "\n", " history processing_level \\\n", "0 2026-05-07 06:31:39. Initial conversion from I... \n", "1 2026-05-07 06:31:39. Initial conversion from I... \n", "2 2026-05-07 06:31:39. Initial conversion from I... \n", "3 2026-05-07 06:31:39. Initial conversion from I... \n", "4 2026-05-07 06:31:39. Initial conversion from I... \n", "\n", " processing_codes source_id source_record_id \n", "0 ICOADS-3-0-0T-125-704-1878-10 020N16 \n", "1 ICOADS-3-0-0T-125-704-1878-10 020N1P \n", "2 ICOADS-3-0-0T-125-704-1878-10 020N25 \n", "3 ICOADS-3-0-0T-125-704-1878-10 020N2Q \n", "4 ICOADS-3-0-0T-125-704-1878-10 020N3A " ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data = cdm_tables[\"header\"]\n", "data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now show an example of Lat and Lon" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(0 42.28\n", " 1 42.31\n", " 2 42.33\n", " 3 42.35\n", " 4 42.37\n", " Name: latitude, dtype: Float64,\n", " 0 -68.410004\n", " 1 -68.029999\n", " 2 -67.639999\n", " 3 -67.290001\n", " 4 -66.900002\n", " Name: longitude, dtype: Float64)" ] }, "execution_count": 11, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data.latitude.head(), data.longitude.head()" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
lat_deg_onlat_min_onlat_hemis_onlon_deg_oflon_min_oflon_hemis_of
04220N6630W
14220N6630W
24220N6630W
34220N6630W
44220N6630W
\n", "
" ], "text/plain": [ " lat_deg_on lat_min_on lat_hemis_on lon_deg_of lon_min_of lon_hemis_of\n", "0 42 20 N 66 30 W\n", "1 42 20 N 66 30 W\n", "2 42 20 N 66 30 W\n", "3 42 20 N 66 30 W\n", "4 42 20 N 66 30 W" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw.c99_daily[\n", " [\n", " \"lat_deg_on\",\n", " \"lat_min_on\",\n", " \"lat_hemis_on\",\n", " \"lon_deg_of\",\n", " \"lon_min_of\",\n", " \"lon_hemis_of\",\n", " ]\n", "].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This has been successfully converted to Decimal degrees with the right (-) for each hemisphere. \n", "\n", "\n", "Now for the SLP we have other information:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
baro_typebaro_heightbaro_units
02141
12141
22141
32141
42141
\n", "
" ], "text/plain": [ " baro_type baro_height baro_units\n", "0 2 14 1\n", "1 2 14 1\n", "2 2 14 1\n", "3 2 14 1\n", "4 2 14 1" ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_raw.c99_journal[[\"baro_type\", \"baro_height\", \"baro_units\"]].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Baro type original code table\n", "\n", "```\n", "{\n", "\t\"1\":\"aneroid\",\n", "\t\"2\":\"mercurial\"\n", "}\n", "```\n", "Baro units original code table. It has been left like this:\n", "\n", "```\n", "{\n", "\t\"1\":\"inches\",\n", "\t\"2\":\"millimeters\",\n", "\t\"3\":\"millibars\",\n", "\t\"4\":\"unable to determine\",\n", "\t\"5\":\"Paris inches\"\n", "}\n", "```\n", "\n", "Our CDM table will be\n", "```\n", "{\n", " \"1\":1001,\n", " \"2\":1002,\n", " \"3\":1003,\n", " \"4\":9999,\n", " \"5\":1005\n", "}\n", "```\n", "\n", "9999 will be the `\"fill_value\": 9999` that indicates to the CDM-mapper that these are NaN values.\n" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
observation_idreport_iddata_policy_licencedate_timedate_time_meaningobservation_durationlongitudelatitudecrsz_coordinatez_coordinate_typeobservation_height_above_station_surfaceobserved_variablesecondary_variableobservation_valuevalue_significancesecondary_valueunitscode_tableconversion_flaglocation_methodlocation_precisionz_coordinate_methodbbox_min_longitudebbox_max_longitudebbox_min_latitudebbox_max_latitudespatial_representativenessquality_flagnumerical_precisionsensor_idsensor_automation_statusexposure_of_sensororiginal_precisionoriginal_unitsoriginal_code_tableoriginal_valueconversion_methodprocessing_codeprocessing_leveladjustment_idtraceabilityadvanced_qcadvanced_uncertaintyadvanced_homogenisationsource_id
0ICOADS-300-020N16-SLPICOADS-300-020N1601878-10-20 06:00:0028-68.41000442.2804.2704.2758<NA>99610.02<NA>32<NA>0<NA><NA><NA><NA><NA><NA><NA>32<NA><NA>53<NA>1001<NA>996.17<NA>3<NA>2000ICOADS-3-0-0T-125-704-1878-10
1ICOADS-300-020N1P-SLPICOADS-300-020N1P01878-10-20 08:00:0028-68.02999942.3104.2704.2758<NA>99630.02<NA>32<NA>0<NA><NA><NA><NA><NA><NA><NA>32<NA><NA>53<NA>1001<NA>996.37<NA>3<NA>2000ICOADS-3-0-0T-125-704-1878-10
2ICOADS-300-020N25-SLPICOADS-300-020N2501878-10-20 10:00:0028-67.63999942.3304.2704.2758<NA>99690.02<NA>32<NA>0<NA><NA><NA><NA><NA><NA><NA>32<NA><NA>53<NA>1001<NA>996.97<NA>3<NA>2000ICOADS-3-0-0T-125-704-1878-10
3ICOADS-300-020N2Q-SLPICOADS-300-020N2Q01878-10-20 12:00:0028-67.29000142.3504.2704.2758<NA>99760.02<NA>32<NA>0<NA><NA><NA><NA><NA><NA><NA>32<NA><NA>53<NA>1001<NA>997.67<NA>3<NA>2000ICOADS-3-0-0T-125-704-1878-10
4ICOADS-300-020N3A-SLPICOADS-300-020N3A01878-10-20 14:00:0028-66.90000242.3704.2704.2758<NA>99920.02<NA>32<NA>0<NA><NA><NA><NA><NA><NA><NA>32<NA><NA>53<NA>1001<NA>999.27<NA>3<NA>2000ICOADS-3-0-0T-125-704-1878-10
\n", "
" ], "text/plain": [ " observation_id report_id data_policy_licence \\\n", "0 ICOADS-300-020N16-SLP ICOADS-300-020N16 0 \n", "1 ICOADS-300-020N1P-SLP ICOADS-300-020N1P 0 \n", "2 ICOADS-300-020N25-SLP ICOADS-300-020N25 0 \n", "3 ICOADS-300-020N2Q-SLP ICOADS-300-020N2Q 0 \n", "4 ICOADS-300-020N3A-SLP ICOADS-300-020N3A 0 \n", "\n", " date_time date_time_meaning observation_duration longitude \\\n", "0 1878-10-20 06:00:00 2 8 -68.410004 \n", "1 1878-10-20 08:00:00 2 8 -68.029999 \n", "2 1878-10-20 10:00:00 2 8 -67.639999 \n", "3 1878-10-20 12:00:00 2 8 -67.290001 \n", "4 1878-10-20 14:00:00 2 8 -66.900002 \n", "\n", " latitude crs z_coordinate z_coordinate_type \\\n", "0 42.28 0 4.27 0 \n", "1 42.31 0 4.27 0 \n", "2 42.33 0 4.27 0 \n", "3 42.35 0 4.27 0 \n", "4 42.37 0 4.27 0 \n", "\n", " observation_height_above_station_surface observed_variable \\\n", "0 4.27 58 \n", "1 4.27 58 \n", "2 4.27 58 \n", "3 4.27 58 \n", "4 4.27 58 \n", "\n", " secondary_variable observation_value value_significance secondary_value \\\n", "0 99610.0 2 \n", "1 99630.0 2 \n", "2 99690.0 2 \n", "3 99760.0 2 \n", "4 99920.0 2 \n", "\n", " units code_table conversion_flag location_method location_precision \\\n", "0 32 0 \n", "1 32 0 \n", "2 32 0 \n", "3 32 0 \n", "4 32 0 \n", "\n", " z_coordinate_method bbox_min_longitude bbox_max_longitude \\\n", "0 \n", "1 \n", "2 \n", "3 \n", "4 \n", "\n", " bbox_min_latitude bbox_max_latitude spatial_representativeness \\\n", "0 3 \n", "1 3 \n", "2 3 \n", "3 3 \n", "4 3 \n", "\n", " quality_flag numerical_precision sensor_id sensor_automation_status \\\n", "0 2 5 \n", "1 2 5 \n", "2 2 5 \n", "3 2 5 \n", "4 2 5 \n", "\n", " exposure_of_sensor original_precision original_units \\\n", "0 3 1001 \n", "1 3 1001 \n", "2 3 1001 \n", "3 3 1001 \n", "4 3 1001 \n", "\n", " original_code_table original_value conversion_method processing_code \\\n", "0 996.1 7 \n", "1 996.3 7 \n", "2 996.9 7 \n", "3 997.6 7 \n", "4 999.2 7 \n", "\n", " processing_level adjustment_id traceability advanced_qc \\\n", "0 3 2 0 \n", "1 3 2 0 \n", "2 3 2 0 \n", "3 3 2 0 \n", "4 3 2 0 \n", "\n", " advanced_uncertainty advanced_homogenisation \\\n", "0 0 0 \n", "1 0 0 \n", "2 0 0 \n", "3 0 0 \n", "4 0 0 \n", "\n", " source_id \n", "0 ICOADS-3-0-0T-125-704-1878-10 \n", "1 ICOADS-3-0-0T-125-704-1878-10 \n", "2 ICOADS-3-0-0T-125-704-1878-10 \n", "3 ICOADS-3-0-0T-125-704-1878-10 \n", "4 ICOADS-3-0-0T-125-704-1878-10 " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "data_obs = cdm_tables[\"observations-slp\"]\n", "data_obs.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.2" } }, "nbformat": 4, "nbformat_minor": 4 }