Outines the process of making an MTH5 file, generating a processing config, and runs the aurora processor¶
[1]:
# Required imports for theh program.
from pathlib import Path
import sys
import pandas as pd
from mth5.clients.make_mth5 import MakeMTH5
from mth5 import mth5, timeseries
from mt_metadata.utils.mttime import get_now_utc, MTime
from aurora.config.config_creator import ConfigCreator
from aurora.pipelines.process_mth5 import process_mth5_run
2021-11-05 15:25:44,710 [line 135] mth5.setup_logger - INFO: Logging file can be found /Users/tronan/Desktop/Projects/mth5/logs/mth5_debug.log
Builds MTH5 File¶
[2]:
# Set path so MTH5 file builds to current working directory.
default_path = Path().cwd()
[3]:
# Initialize the Make MTH5 code.
m = MakeMTH5(mth5_version='0.1.0')
m.client = "IRIS"
[4]:
# Generate data frame of FDSN Network, Station, Location, Channel, Startime, Endtime codes of interest
ZUCAS04LQ1 = ['ZU', 'CAS04', '', 'LQE', '2020-06-02T19:00:00', '2020-07-13T19:00:00']
ZUCAS04LQ2 = ['ZU', 'CAS04', '', 'LQN', '2020-06-02T19:00:00', '2020-07-13T19:00:00']
ZUCAS04BF1 = ['ZU', 'CAS04', '', 'LFE', '2020-06-02T19:00:00', '2020-07-13T19:00:00']
ZUCAS04BF2 = ['ZU', 'CAS04', '', 'LFN', '2020-06-02T19:00:00', '2020-07-13T19:00:00']
ZUCAS04BF3 = ['ZU', 'CAS04', '', 'LFZ', '2020-06-02T19:00:00', '2020-07-13T19:00:00']
request_list = [ZUCAS04LQ1, ZUCAS04LQ2, ZUCAS04BF1, ZUCAS04BF2, ZUCAS04BF3]
# Turn list into dataframe
request_df = pd.DataFrame(request_list, columns=m.column_names)\
[5]:
# Inspect the dataframe
print(request_df)
network station location channel start end
0 ZU CAS04 LQE 2020-06-02T19:00:00 2020-07-13T19:00:00
1 ZU CAS04 LQN 2020-06-02T19:00:00 2020-07-13T19:00:00
2 ZU CAS04 LFE 2020-06-02T19:00:00 2020-07-13T19:00:00
3 ZU CAS04 LFN 2020-06-02T19:00:00 2020-07-13T19:00:00
4 ZU CAS04 LFZ 2020-06-02T19:00:00 2020-07-13T19:00:00
[6]:
# Request the inventory information from IRIS
inventory = m.get_inventory_from_df(request_df, data=False)
[7]:
# Inspect the inventory
inventory
[7]:
(Inventory created at 2021-11-05T22:26:14.221364Z
Created by: ObsPy 1.2.2
https://www.obspy.org
Sending institution: MTH5
Contains:
Networks (1):
ZU
Stations (1):
ZU.CAS04 (Corral Hollow, CA, USA)
Channels (5):
ZU.CAS04..LFZ, ZU.CAS04..LFN, ZU.CAS04..LFE, ZU.CAS04..LQN,
ZU.CAS04..LQE,
0 Trace(s) in Stream:
)
Generate te MTH5 from the Input DataFrame and leave it open.¶
[8]:
mth5_object = m.make_mth5_from_fdsnclient(request_df, interact=True)
2021-11-05 15:26:20,273 [line 594] mth5.mth5.MTH5.open_mth5 - WARNING: ZU_CAS04.h5 will be overwritten in 'w' mode
2021-11-05 15:26:21,459 [line 653] mth5.mth5.MTH5._initialize_file - INFO: Initialized MTH5 file /Users/tronan/Desktop/Projects/aurora/tests/ZU_CAS04.h5 in mode w
2021-11-05 15:26:38,467 [line 127] mt_metadata.base.metadata.station.add_run - WARNING: Run a is being overwritten with current information
2021-11-05 15:26:41,799 [line 709] mth5.groups.base.Station.add_run - INFO: run a already exists, returning existing group.
2021-11-05 15:26:42,338 [line 224] mth5.timeseries.run_ts.RunTS.validate_metadata - WARNING: start time of dataset 2020-06-02T19:00:00+00:00 does not match metadata start 2020-06-02T18:41:43+00:00 updating metatdata value to 2020-06-02T19:00:00+00:00
2021-11-05 15:27:01,653 [line 709] mth5.groups.base.Station.add_run - INFO: run b already exists, returning existing group.
2021-11-05 15:27:25,841 [line 709] mth5.groups.base.Station.add_run - INFO: run c already exists, returning existing group.
2021-11-05 15:27:51,950 [line 709] mth5.groups.base.Station.add_run - INFO: run d already exists, returning existing group.
2021-11-05 15:27:54,225 [line 235] mth5.timeseries.run_ts.RunTS.validate_metadata - WARNING: end time of dataset 2020-07-13T19:00:00+00:00 does not match metadata end 2020-07-13T21:46:12+00:00 updating metatdata value to 2020-07-13T19:00:00+00:00
[9]:
# mth5_object.open_mth5(h5_path, 'w')
# h5_path = str(default_path)+'/ZU_CAS04.h5'
#mth5_object.close_mth5()
Extract information from the open MTH5 Object
[10]:
mth5_object
[10]:
/:
====================
|- Group: Survey
----------------
|- Group: Filters
-----------------
|- Group: coefficient
---------------------
|- Group: v to counts (electric)
--------------------------------
|- Group: v to counts (magnetic)
--------------------------------
|- Group: fap
-------------
|- Group: fir
-------------
|- Group: time_delay
--------------------
|- Group: electric time offset
------------------------------
|- Group: hx time offset
------------------------
|- Group: hy time offset
------------------------
|- Group: hz time offset
------------------------
|- Group: zpk
-------------
|- Group: electric field 1 pole butterworth high-pass
-----------------------------------------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: electric field 5 pole butterworth low-pass
----------------------------------------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: magnetic field 3 pole butterworth low-pass
----------------------------------------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: mv per km to v per m
------------------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: v per m to v
----------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: Reports
-----------------
|- Group: Standards
-------------------
--> Dataset: summary
......................
|- Group: Stations
------------------
|- Group: CAS04
---------------
|- Group: a
-----------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: b
-----------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: c
-----------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: d
-----------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
[18]:
# Collect information from the MTh5 Object and use it in the config files.
mth5_filename = mth5_object.filename
version = mth5_object.file_version
[34]:
# Edit and update the MTH5 metadata
s = mth5_object.get_station("CAS04")
s.metadata.location.declination.model = 'IGRF'
s.write_metadata()
[11]:
# Get the available stations and runs from the MTH5 object
ch_summary = mth5_object.channel_summary
[12]:
ch_summary
[12]:
| station | run | latitude | longitude | elevation | component | start | end | n_samples | sample_rate | measurement_type | azimuth | tilt | units | hdf5_reference | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | CAS04 | a | 37.633351 | -121.468382 | 329.3875 | ex | 2020-06-02 19:00:00 | 2020-06-02 22:07:46 | 11267 | 1.0 | electric | 13.2 | 0.0 | millivolts per kilometer | <HDF5 object reference> |
| 1 | CAS04 | a | 37.633351 | -121.468382 | 329.3875 | ey | 2020-06-02 19:00:00 | 2020-06-02 22:07:46 | 11267 | 1.0 | electric | 103.2 | 0.0 | millivolts per kilometer | <HDF5 object reference> |
| 2 | CAS04 | a | 37.633351 | -121.468382 | 329.3875 | hx | 2020-06-02 19:00:00 | 2020-06-02 22:07:46 | 11267 | 1.0 | magnetic | 13.2 | 0.0 | nanotesla | <HDF5 object reference> |
| 3 | CAS04 | a | 37.633351 | -121.468382 | 329.3875 | hy | 2020-06-02 19:00:00 | 2020-06-02 22:07:46 | 11267 | 1.0 | magnetic | 103.2 | 0.0 | nanotesla | <HDF5 object reference> |
| 4 | CAS04 | a | 37.633351 | -121.468382 | 329.3875 | hz | 2020-06-02 19:00:00 | 2020-06-02 22:07:46 | 11267 | 1.0 | magnetic | 0.0 | 0.0 | nanotesla | <HDF5 object reference> |
| 5 | CAS04 | b | 37.633351 | -121.468382 | 329.3875 | ex | 2020-06-02 22:24:55 | 2020-06-12 17:52:23 | 847649 | 1.0 | electric | 13.2 | 0.0 | millivolts per kilometer | <HDF5 object reference> |
| 6 | CAS04 | b | 37.633351 | -121.468382 | 329.3875 | ey | 2020-06-02 22:24:55 | 2020-06-12 17:52:23 | 847649 | 1.0 | electric | 103.2 | 0.0 | millivolts per kilometer | <HDF5 object reference> |
| 7 | CAS04 | b | 37.633351 | -121.468382 | 329.3875 | hx | 2020-06-02 22:24:55 | 2020-06-12 17:52:23 | 847649 | 1.0 | magnetic | 13.2 | 0.0 | nanotesla | <HDF5 object reference> |
| 8 | CAS04 | b | 37.633351 | -121.468382 | 329.3875 | hy | 2020-06-02 22:24:55 | 2020-06-12 17:52:23 | 847649 | 1.0 | magnetic | 103.2 | 0.0 | nanotesla | <HDF5 object reference> |
| 9 | CAS04 | b | 37.633351 | -121.468382 | 329.3875 | hz | 2020-06-02 22:24:55 | 2020-06-12 17:52:23 | 847649 | 1.0 | magnetic | 0.0 | 0.0 | nanotesla | <HDF5 object reference> |
| 10 | CAS04 | c | 37.633351 | -121.468382 | 329.3875 | ex | 2020-06-12 18:32:17 | 2020-07-01 17:32:59 | 1638043 | 1.0 | electric | 13.2 | 0.0 | millivolts per kilometer | <HDF5 object reference> |
| 11 | CAS04 | c | 37.633351 | -121.468382 | 329.3875 | ey | 2020-06-12 18:32:17 | 2020-07-01 17:32:59 | 1638043 | 1.0 | electric | 103.2 | 0.0 | millivolts per kilometer | <HDF5 object reference> |
| 12 | CAS04 | c | 37.633351 | -121.468382 | 329.3875 | hx | 2020-06-12 18:32:17 | 2020-07-01 17:32:59 | 1638043 | 1.0 | magnetic | 13.2 | 0.0 | nanotesla | <HDF5 object reference> |
| 13 | CAS04 | c | 37.633351 | -121.468382 | 329.3875 | hy | 2020-06-12 18:32:17 | 2020-07-01 17:32:59 | 1638043 | 1.0 | magnetic | 103.2 | 0.0 | nanotesla | <HDF5 object reference> |
| 14 | CAS04 | c | 37.633351 | -121.468382 | 329.3875 | hz | 2020-06-12 18:32:17 | 2020-07-01 17:32:59 | 1638043 | 1.0 | magnetic | 0.0 | 0.0 | nanotesla | <HDF5 object reference> |
| 15 | CAS04 | d | 37.633351 | -121.468382 | 329.3875 | ex | 2020-07-01 19:36:55 | 2020-07-13 19:00:00 | 1034586 | 1.0 | electric | 13.2 | 0.0 | millivolts per kilometer | <HDF5 object reference> |
| 16 | CAS04 | d | 37.633351 | -121.468382 | 329.3875 | ey | 2020-07-01 19:36:55 | 2020-07-13 19:00:00 | 1034586 | 1.0 | electric | 103.2 | 0.0 | millivolts per kilometer | <HDF5 object reference> |
| 17 | CAS04 | d | 37.633351 | -121.468382 | 329.3875 | hx | 2020-07-01 19:36:55 | 2020-07-13 19:00:00 | 1034586 | 1.0 | magnetic | 13.2 | 0.0 | nanotesla | <HDF5 object reference> |
| 18 | CAS04 | d | 37.633351 | -121.468382 | 329.3875 | hy | 2020-07-01 19:36:55 | 2020-07-13 19:00:00 | 1034586 | 1.0 | magnetic | 103.2 | 0.0 | nanotesla | <HDF5 object reference> |
| 19 | CAS04 | d | 37.633351 | -121.468382 | 329.3875 | hz | 2020-07-01 19:36:55 | 2020-07-13 19:00:00 | 1034586 | 1.0 | magnetic | 0.0 | 0.0 | nanotesla | <HDF5 object reference> |
[13]:
available_runs = ch_summary.run.unique()
sr = ch_summary.sample_rate.unique()
if len(sr) != 1:
print('Only one sample rate per run is available')
available_stations = ch_summary.station.unique()
[14]:
sr[0]
[14]:
1.0
[15]:
available_stations[0]
[15]:
'CAS04'
[16]:
mth5_object
[16]:
/:
====================
|- Group: Survey
----------------
|- Group: Filters
-----------------
|- Group: coefficient
---------------------
|- Group: v to counts (electric)
--------------------------------
|- Group: v to counts (magnetic)
--------------------------------
|- Group: fap
-------------
|- Group: fir
-------------
|- Group: time_delay
--------------------
|- Group: electric time offset
------------------------------
|- Group: hx time offset
------------------------
|- Group: hy time offset
------------------------
|- Group: hz time offset
------------------------
|- Group: zpk
-------------
|- Group: electric field 1 pole butterworth high-pass
-----------------------------------------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: electric field 5 pole butterworth low-pass
----------------------------------------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: magnetic field 3 pole butterworth low-pass
----------------------------------------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: mv per km to v per m
------------------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: v per m to v
----------------------
--> Dataset: poles
....................
--> Dataset: zeros
....................
|- Group: Reports
-----------------
|- Group: Standards
-------------------
--> Dataset: summary
......................
|- Group: Stations
------------------
|- Group: CAS04
---------------
|- Group: a
-----------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: b
-----------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: c
-----------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
|- Group: d
-----------
--> Dataset: ex
.................
--> Dataset: ey
.................
--> Dataset: hx
.................
--> Dataset: hy
.................
--> Dataset: hz
.................
Make a configuration file using the MTH5¶
[19]:
station_id = available_stations[0]
run_id = available_runs[0]
sample_rate = sr[0]
config_maker = ConfigCreator()
config_path = config_maker.create_run_config(station_id, run_id, mth5_filename, sample_rate)
[0, 1, 2, 3]
[21]:
config_path
[21]:
PosixPath('config/CAS04-a_run_config.json')
Run the Aurora Pipeline¶
[31]:
show_plot='True'
[ ]:
tf_cls = process_mth5_run(
config_path,
run_id,
mth5_path=mth5_filename,
units="MT",
show_plot=False,
z_file_path=None,
return_collection=False,
)
[41]:
tf_cls
[41]:
TF( station='CAS04', latitude=37.63, longitude=-121.47, elevation=329.39 )
Write the transfer functions to emtf emft and other output file types.¶
[42]:
tf_cls.write_tf_file(fn="emtfxml_test.xml", file_type="emtfxml")
[42]:
{
"e_m_t_f_x_m_l": {
"description": "USMTArray South Magnetotelluric Time Series (USMTArray CONUS South-USGS)",
"product_id": "CAS04",
"sub_type": "MT_TF",
"tags": "impedance, tipper"
}
}
[ ]:
tf_cls.write_tf_file(fn="emtfxml_test.xml", file_type="edi")
[ ]:
tf_cls.write_tf_file(fn="emtfxml_test.xml", file_type="zmm")