QA Metrics

Defines a structure for recording quality assurance (QA) metrics derived from data at various stages of processing to an Influx database. It is composed of individual metric objects that are associated to a single Influx field. These operate on a metadata AxisManager for a single observation ID, from which they calculate and return a quantity to be recorded. A metric for a given source should be based on the QAMetric class, which provides an interface to the Influx database. Two abstract methods should be implemented by subclasses: the _process method is passed the metadata and should calculate the metric; _get_available_obs should return a list of observation ID’s that are available to be processed (i.e. have an existing metadata entry in the Manifest file referenced in the Context).

Base Class

class sotodlib.qa.metrics.QAMetric(context, monitor, log='qa_metrics_log')

Base class for quality assurance metrics to be recorded in Influx, derived from processed metadata files.

The base class includes methods for recording a metric value to Influx, checking if a given obs_id has been recorded for this metric, and fetching a list of obs_id’s that can be recorded.

Metrics are labelled by the observation ID they describe, an Influx ‘measurement’ and ‘field’, and any number of additional tags. A measurement can have multiple fields associated with it, and here we’ve tried to organize the metrics so that a source of information (e.g. HWP angle solutions) is a measurement and relevant quantities describing it (e.g. HWP success, mean rate, etc) are fields.

Subclasses should implement the _process and _get_available_obs methods to access the source of data they are going to tap, as well as define the _influx_meas and _influx_field attributes.

exists(obs_id, tags={})

Check if a metric exists for this obs_id in Influx.

Parameters:

• obs_id (str) – The observation ID to check.

• tags (dict (optional)) – Further restrict to given tags.

Returns:

exists – Whether it exists or not.

Return type:

bool

get_existing_obs()

Get a list of observations already recorded to Influx.

process_and_record(obs_id, meta=None)

Generate a metric for this obs_id and record it to InfluxDB.

Parameters:

• obs_id (str) – The observation ID to process.

• meta (AxisManager (optional)) – The metadata for this observation ID. If not provided will read from file.

get_new_obs()

Get a list of available observations not yet recorded to InfluxDB.

Metric-recording Script

sotodlib.site_pipeline.record_qa.main(config)

Update all metrics specified in the config.

For a list of metrics provided in a config file, this script determines which observation ID’s are available and have yet to be recorded in the Influx database, loads the metadata defined in the given context file one observation at a time, calculates each metric, and records it to Influx. The configuration file should contain three blocks:

monitor

Specifies an InfluxDB monitor as defined in sotodlib.site_pipeline.monitor.Monitor()

context_file

The context file that includes all the relevant metadata.

metrics

A list of metrics, with each entry including at least a name field giving the name of the class to instantiate, with all other parameters passed to its __init__ method.

For example, a config could look like:

monitor:
    host: "daq-dev"
    port: "8086"
    database: "pipeline-dev"
    username: "monitor"
    password: "crepe-handmade-trio"
    path: ""
    ssl: False
context_file: "/path/to/context.yaml"
metrics:
  - name: "PreprocessQA"
    process_name: "det_bias_flags"
  - name: "HWPSolSuccess"
    encoder: "1"
  - name: "HWPSolNumSamples"
    encoder: "1"
  - name: "HWPSolNumFlagged"
    encoder: "1"
  - name: "HWPSolMeanRate"
    encoder: "1"
  - name: "HWPSolMeanTemplate"
    encoder: "1"
  - name: "HWPSolSuccess"
    encoder: "2"
  - name: "HWPSolNumSamples"
    encoder: "2"
  - name: "HWPSolNumFlagged"
    encoder: "2"
  - name: "HWPSolMeanRate"
    encoder: "2"
  - name: "HWPSolMeanTemplate"
    encoder: "2"
  - name: "HWPSolVersion"
  - name: "HWPSolPrimaryEncoder"
  - name: "HWPSolOffcenter"
  - name: "HWPSolOffcenterErr"

In the above config, some of the metrics have parameters like process_name or encoder that are specific to their subclass.

Metric Subclasses

Preprocess

Metrics to be derived from the output of the preprocess module share a generic class. A _PreProcess subclass that supports QA metrics should implement its gen_metric abstract method to calculate a summary quantity for its output. In the record_qa configuration, the name of the process to instantiate for a given metric should be given as the process_name argument.

class sotodlib.qa.metrics.PreprocessQA(context, monitor, process_name, process_args={}, **kwargs)

A metric derived from a preprocesstod process.

HWP Angle Solution

Some HWP solution metrics are derived for both encoder solutions separately, so the encoder argument should be specified in the config.

class sotodlib.qa.metrics.HWPSolQA(context, monitor, encoder=None, **kwargs)

Base class for metrics derived from HWP angle solutions. Subclasses should implement the _process method. Some quantities are derived twice, once for each encoder, and these will require and encoder parameter to be provided to select which one to produce a metric for. This is indicated by setting the _needs_encoder attribute to True.

class sotodlib.qa.metrics.HWPSolSuccess(context, monitor, encoder=None, **kwargs)

Records success of the HWP angle solution calculation, for each encode.

class sotodlib.qa.metrics.HWPSolNumSamples(context, monitor, encoder=None, **kwargs)

The total number of encoder samples.

class sotodlib.qa.metrics.HWPSolNumFlagged(context, monitor, encoder=None, **kwargs)

The number of encoder samples that were flagged.

class sotodlib.qa.metrics.HWPSolMeanRate(context, monitor, encoder=None, **kwargs)

The mean calculated HWP angle rate, for each encoder.

class sotodlib.qa.metrics.HWPSolMeanTemplate(context, monitor, encoder=None, **kwargs)

The mean of the calculated template magnitude.

class sotodlib.qa.metrics.HWPSolVersion(context, monitor, encoder=None, **kwargs)

The version of the solution used for the HWP angle calculation.

class sotodlib.qa.metrics.HWPSolPrimaryEncoder(context, monitor, encoder=None, **kwargs)

The primary encoder used for the HWP angle calculation.

class sotodlib.qa.metrics.HWPSolOffcenter(context, monitor, encoder=None, **kwargs)

Calculated offcentering of HWP angle solution.

class sotodlib.qa.metrics.HWPSolOffcenterErr(context, monitor, encoder=None, **kwargs)

Standard error on the offcentering of HWP angle solution.