Preprocess

The preprocess module defines a standardized interface for TOD processing operations so that they can be easily implemented in automatic data analysis scripts. The core of the system is in two parts, the _Preprocess modules and the Pipeline object. The _Preprocess modules each define how a TOD operation is run on an AxisManager TOD and the Pipeline object is used to define the order of the operations and then run them. The site-pipeline.preprocess_tod script is used to run and save Pipelines on lists of observations, grouped by detset. The site-pipeline.preprocess_obs script is used for observation-level preprocessing. This module is similar to site-pipeline.preprocess_tod but removes grouping by detset so that the entire observation is loaded, without signal. For example, pipeline steps such as DetBiasFlags requires tod-level data including signal, whereas SSOFootprint does not and uses observation-level data.

Single-layer vs. two-layer (“multilayer”) pipelines

A single process_pipe config file, run through preprocess_tod, is a single-layer pipeline. For some platforms (in particular the SATs) the full processing recipe is instead split into two layers, run through multilayer_preprocess_tod with two config files: an init (first) layer and a proc (second) layer that depends on it. The proc layer records which init layer it was built from, so if you try to run a proc config against a different init archive than the one it was created with, it will fail rather than silently mixing incompatible layers. Layers are chainable, so a proc layer’s output can itself become the init input for a further layer, though doing so currently means running that further layer interactively because there is no three_layer_preprocess_tod script.

This split was introduced to separate the pre-demodulation and post-demodulation steps for the SATs: the first (init) layer runs pointing, flagging, HWP-synchronous-signal removal, calibration, PCA relcal, and demodulation, producing calibrated demodulated Q/U timestreams; the second (proc) layer runs the filters that operate on that demodulated data (e.g. azss, estimate_t2p/subtract_t2p, sub_polyf). Splitting the pipeline this way means the second layer’s filtering can be iterated on and re-run without repeating the expensive first-layer processing. LAT configs in production are currently single-layer only (no init/proc split), reflecting that LAT does not run the HWP-demodulation/ground-pickup/T-to-P-leakage processing chain the SATs do in this pipeline.

For real, in-production examples of both designs (as opposed to the synthetic examples on this page), see the platform config directories in the site-pipeline-configs repository, e.g. satp1/preprocess_config_init.yaml + satp1/preprocess_config_proc.yaml (two-layer) and lat/preprocess_config_cmb_mf.yaml / lat/preprocess_config_cmb_uhf.yaml (single-layer).

[TOD Processing] Infrastructure Introduction has a fuller walkthrough (jobdb usage, extensive interactive/Python examples for loading, running, and saving preprocessing archives) that complements this reference page – including a deeper look at first-run-vs-rerun behavior than the summary below.

Running for the first time vs. re-running against an existing archive

The archive config key names two things together: archive.index, a ManifestDb sqlite file keyed on obs_id plus the subobs.use grouping fields (e.g. detset, or ["wafer_slot", "wafer.bandpass"]), and archive.policy.filename, the HDF5 file the actual proc_aman results are written into. Together these are “the archive” for a given config.

First run: archive.index doesn’t exist yet, so sotodlib.preprocess.preprocess_util.get_preprocess_db() creates a new, empty ManifestDb there. Every obs_id/group the query returns is processed from scratch – each pipeline step’s calc_and_save runs, and the results are written into the archive as they complete.
Re-running: if archive.index already exists, preprocess_tod inspects it first and drops any obs_id/group already present in the archive from the run list, so a second invocation over the same (or an overlapping) observation list only processes what’s new – already-archived observations are left alone rather than recomputed. Pass --overwrite to force everything in the query to be reprocessed and the existing entries replaced instead.
This is also what makes the archive reusable for simulations: when sotodlib.preprocess.pcore.Pipeline.run() is called with an existing proc_aman (as loaded from the archive), every step’s calc_and_save is skipped and only process() re-runs on top of the already-computed products – e.g. reapplying a filter fit on real data to a signal-only sim without redoing the fit.

Preprocessing job tracking (`jobdb`)

Preprocessing configs can optionally include a top-level jobdb key giving the path to a SQLite database that tracks the state (open/done/ failed) of each preprocessing job (an obs_id + group combination). Passing --run-from-jobdb to preprocess_tod/multilayer_preprocess_tod resumes a batch run from an existing jobdb’s run list instead of rebuilding one — useful for continuing a large run that was interrupted (e.g. a Slurm job hitting its walltime) without redoing already-completed or already-known-to-fail work. It also starts immediately, since the wafer/band groups don’t need to be recomputed. Jobs can be inspected programmatically via sotodlib.site_pipeline.jobdb.JobManager.

Preprocessing Pipelines

A preprocessing pipeline is series of modules, each inheriting from _Preprocess, that are defined through a configuration file and intended to be run successively on an AxisManager containing time ordered data.

class sotodlib.preprocess.pcore._Preprocess(step_cfgs)[source]

The base class for Preprocessing modules which defines the required functions and keys required in the configurations.

Each preprocess module has four overwritable functions that are called by the processing scripts in site_pipeline. These four functions are each controlled by a specific key in a configuration dictionary passed to the module on creation.

The configuration dictionary has 6 special keys: name, process, calc, save, select, and plot. name is the name used to register the module with the PIPELINE registry. The other four keys are matched to functions in the module, if the key is not present then that function will be skipped when the preprocessing pipeline is run.

There are two special AxisManagers expected to be part of the preprocessing pipeline. aman is the “standard” time ordered data AxisManager that is loaded via our default styles. proc_aman is the preprocess AxisManager, this is carry the data products that will be saved to whatever Metadata Archive is connected to the preprocessing pipeline.

process(aman, proc_aman, sim=False, data_aman=None)[source]

This function makes changes to the time ordered data AxisManager. Ex: calibrating or detrending the timestreams. This function will use any configuration information under the process key of the configuration dictionary and is not expected to change or alter proc_aman.

Parameters:

aman (AxisManager) – The time ordered data
proc_aman (AxisManager) – Any information generated by previous elements in the preprocessing pipeline.
sim (Bool) – False by default when analyzing data. Should be True when doing Transfer Function simulations and determining which steps should be run.
data_aman (AxisManager (Optional)) – An AxisManager containing the preprocessed data to be used by this process.

calc_and_save(aman, proc_aman)[source]

This function calculates data products of some sort off of the time ordered data AxisManager.

Ex: Calcuating the white noise of the timestream. This function will use any configuration information under the calc key of the configuration dictionary and can call the save function to make changes to proc_aman.

Parameters:

aman (AxisManager) – The time ordered data
proc_aman (AxisManager) – Any information generated by previous elements in the preprocessing pipeline.

save(proc_aman, *args)[source]

This function wraps new information into the proc_aman and will use any configuration information under the save key of the configuration dictionary.

Parameters:

proc_aman (AxisManager) – Any information generated by previous elements in the preprocessing pipeline.
args (any) – Any additional information calc_and_save needs to send to the save function.

select(meta, proc_aman=None, in_place=True)[source]

This function runs any desired data selection of the preprocessing pipeline results. Assumes the pipeline has already been run and that the resulting proc_aman is now saved under the preprocess key in the meta AxisManager loaded via context.

Ex: removing detectors with white noise above some limit. This function will use any configuration information under the select key.

Parameters:

meta (AxisManager) – Metadata related to the specific observation
proc_aman (AxisManager) – Optional. Any information generated by previous elements in the preprocessing pipeline.
in_place (bool) – Optional. Apply selection and return restricted axis manager if True, else return the flag array.

Returns:

meta – Metadata where non-selected detectors have been removed

Return type:

AxisManager

plot(aman, proc_aman, filename)[source]

This function creates plots using results from calc_and_save.

Ex: Plotting det bias flags. This function will use any configuration information under the plot key of the configuration dictionary.

Parameters:

aman (AxisManager) – The time ordered data
proc_aman (AxisManager) – Any information generated by previous elements in the preprocessing pipeline.
filename (str) – Filename should be a concatenation of the global plot_dir config with a name with process step number and placeholder {name} as shown in Pipeline.run().

classmethod gen_metric(meta, proc_aman)[source]

Generate a QA metric from the output of this process.

Parameters:

meta (AxisManager) – Metadata related to the specific observation
proc_aman (AxisManager) – The output of the preprocessing pipeline.

Returns:

line – InfluxDB line entry elements to be fed to site_pipeline.monitor.Monitor.record

Return type:

dict

static register(process_class)[source]: Registers a new modules with the PIPELINE

The preprocessing pipeline is defined in the Pipeline class. This class inherits from list so that you can easily find and interact with the various pipeline elements. Note that splicing a pipeline will return a list of process modules that can be used to make a new pipeline.

class sotodlib.preprocess.pcore.Pipeline(modules, plot_dir='./', logger=None, wrap_valid=True)[source]

This class is designed to create and run pipelines out of a series of different preprocessing modules (classes that inherent from _Preprocess). It inherits list object. It also contains the registration of all possible preprocess modules in Pipeline.PIPELINE

append(item)[source]: Append object to the end of the list.

insert(index, item)[source]: Insert object before index.

extend(index, other)[source]: Extend list by appending elements from the iterable.

run(aman, proc_aman=None, full_aman=None, select=True, sim=False, update_plot=False, data_amans=None)[source]

The main workhorse function for the pipeline class. This function takes an AxisManager TOD and successively runs the pipeline of preprocessing modules on the AxisManager. The order of operations called by run are:

for process in pipeline:
    process.process()
    process.calc_and_save()
        process.save() ## called by process.calc_and_save()
    process.select()

Parameters:

aman (AxisManager) – A TOD object. Generally expected to be raw, unprocessed data. This axismanager will be edited in place by the process and select functions of each preprocess module
proc_aman (AxisManager (Optional)) – A preprocess axismanager. If this is provided it is assumed that the pipeline has previously been run on this specific TOD and has returned this preprocess axismanager. In this case, calls to process.calc_and_save() are skipped as the information is expected to be present in this AxisManager.
full_aman (AxisManager (Optional)) – A preprocess axismanager. This axis manager stores the outputs of preprocessing functions (proc_aman) but without any of the detector or samps restrictions applied, thus maintaining its original shape. This is returned at the end of the pipeline. If not passed it is instantiated with the same number of dets and samps as aman.
select (boolean (Optional)) – if True, the aman detector axis is restricted as described in each preprocess module. Most pipelines are developed with select=True. Running select=False may produce unstable behavior
sim (boolean (Optional)) – if running on sim (sim=True), proccesses with the flag skip_on_sim will be skipped.
update_plot (boolean (Optional)) – if True, re-runs plotting (along with processes and selects) given proc_aman is aman.preprocess. This assumes process.calc_and_save() has been run on this aman before and has injested flags and other information into proc_aman.
data_amans (dict (Optional)) – A dictionary of AxisManagers with keys (step, process.name) filled with AxisManager processed up to step-1. This is used to pre-load all data AxisManager which could be required when processing simulations (e.g. to provide a T2P template)

Returns:

full_aman (AxisManager) – A preprocess axismanager that contains all data products calculated throughout the running of the pipeline.
success (str) – A string that stores the name of the last process step that the pipeline completed. If the pipeline successfully finishes all steps, success = ‘end’.

Processing Scripts

These scripts are designed to be the ones that interact with specific configuration files and specific manifest databases.

sotodlib.site_pipeline.preprocess_tod.preprocess_tod(configs: str | dict, obs_id: str, group: dict, verbosity: int = 0, compress: bool = False, overwrite: bool = False)[source]

Meant to be run as part of a batched script, this function calls the preprocessing pipeline a specific Observation ID and group combination and saves the results in the ManifestDb specified in the configs.

Parameters:

configs (str or dict) – Config file or loaded config dictionary.
obs_id (str or ResultSet entry) – obs_id or obs entry that is passed to context.get_obs.
group (list) – The group to be run. For example, this might be [‘ws0’, ‘f090’] if group_by (specified by the subobs->use key in the preprocess config) is [‘wafer_slot’, ‘wafer.bandpass’].
verbosity (str) – Log level. 0 = error, 1 = warn, 2 = info, 3 = debug.
compress (bool) – Whether or not to compress the preprocessing h5 files.
overwrite (bool) – If True, overwrite contents of temporary h5 files.

Returns:

out_dict (dict or None) – Dictionary output for init config from get_preproc_group_out_dict if preprocessing ran successfully for init layer or None if preprocessing was loaded or preproc_or_load_group failed.
errors (tuple) – A tuple containing the error from PreprocessError, an error message, and the traceback. Each will be None if preproc_or_load_group finished successfully.

sotodlib.site_pipeline.preprocess_tod.load_preprocess_tod_sim(obs_id, sim_map, configs='preprocess_configs.yaml', context=None, dets=None, meta=None, modulated=True, logger=<Logger preprocess (DEBUG)>)[source]

Loads the saved information from the preprocessing pipeline and runs the processing section of the pipeline on simulated data

Assumes preprocess_tod has already been run on the requested observation.

Parameters:

obs_id (multiple) – passed to context.get_obs to load AxisManager, see Notes for context.get_obs
sim_map (pixell.enmap.ndmap) – signal map containing (T, Q, U) fields
configs (string or dictionary) – config file or loaded config directory
dets (dict) – dets to restrict on from info in det_info. See context.get_meta.
meta (AxisManager) – Contains supporting metadata to use for loading. Can be pre-restricted in any way. See context.get_meta.
modulated (bool) – If True, apply the HWP angle model and scan the simulation into a modulated signal. If False, scan the simulation into demodulated timestreams.

Returns:

aman – Axis manager after running through the preprocessing steps. Returns None if all detectors are cut.

Return type:

core.AxisManager

sotodlib.site_pipeline.multilayer_preprocess_tod.multilayer_preprocess_tod(obs_id: str, configs_init: str | dict, configs_proc: str | dict, group: list, verbosity: int = 0, compress: bool = False, overwrite: bool = False)[source]

Meant to be run as part of a batched script, this function calls the preprocessing pipeline a specific Observation ID and group combination and saves the results in the ManifestDb specified in the configs.

Parameters:

obs_id (str or ResultSet entry) – obs_id or obs entry that is passed to context.get_obs
configs_init (str or dict) – Config file or loaded config dictionary for first layer database.
configs_proc (str or dict) – Config file or loaded config dictionary for second layer database.
group (list) – The group to be run. For example, this might be [‘ws0’, ‘f090’] if group_by (specified by the subobs->use key in the preprocess config) is [‘wafer_slot’, ‘wafer.bandpass’].
verbosity (str) – The log level to use. 0 = error, 1 = warn, 2 = info, 3 = debug.
compress (bool) – Whether or not to compress the preprocessing h5 files.
overwrite (bool) – If True, overwrite contents of temporary h5 files.

Returns:

out_dict_init (dict or None) – Dictionary output for init config from get_preproc_group_out_dict if preprocessing ran successfully for init layer or None if preprocessing was loaded or preproc_or_load_group failed.
out_dict_proc (dict or None) – Dictionary output for proc config from get_preproc_group_out_dict if preprocessing ran successfully for proc layer or None if preprocessing was loaded, that layer was not run or loaded, or preproc_or_load_group failed.
errors (tuple) – A tuple containing the error from PreprocessError, an error message, and the traceback. Each will be None if preproc_or_load_group finished successfully.

sotodlib.site_pipeline.preprocess_obs.preprocess_obs(obs_id, configs, overwrite=False, logger=None, obs_group=None)[source]

Meant to be run as part of a batched script, this function calls the preprocessing pipeline a specific Observation ID and saves the results in the ManifestDb specified in the configs.

Parameters:

obs_id (string or ResultSet entry) – obs_id or obs entry that is passed to context.get_obs
configs (string or dictionary) – config file or loaded config directory
overwrite (bool) – if True, overwrite existing entries in ManifestDb
logger (logging instance) – the logger to print to
obs_group (list of strings) – List of obs_ids within group

sotodlib.preprocess.preprocess_util.load_and_preprocess(obs_id, configs, context=None, dets=None, meta=None, no_signal=None, logger=None, return_full_aman=False)[source]

Loads the saved information from the preprocessing pipeline and runs the processing section of the pipeline.

Assumes preprocess_tod has already been run on the requested observation.

Parameters:

obs_id (multiple) – Passed to context.get_obs to load AxisManager, see Notes for context.get_obs
configs (string or dictionary) – Config file or loaded config directory
context (core.Context) – Optional. The Context file to use.
dets (dict) – Dets to restrict on from info in det_info. See context.get_meta.
meta (AxisManager) – Contains supporting metadata to use for loading. Can be pre-restricted in any way. See context.get_meta.
no_signal (bool) – If True, signal will be set to None. This is a way to get the axes and pointing info without the (large) TOD blob. Not all loaders may support this.
logger (PythonLogger) – Optional. Logger object. If None, a new logger is created.
return_full_aman (bool) – Optional. Return unrestricted axis manager alongside restricted aman if True, otherwise return None.

Returns:

aman (core.AxisManager or None) – Loaded and restricted axis manager with preprocessing metadata. Returns None if all detectors cut.
full_aman (core.AxisManager or None) – Unrestricted preprocessing axis manager. Used when running multilayer pipeline to ensure saved detector axis has the full size when saving metadata.

sotodlib.preprocess.preprocess_util.multilayer_load_and_preprocess(obs_id, configs_init, configs_proc, dets=None, meta=None, no_signal=None, logger=None, init_only=False, ignore_cfg_check=False, stop_for_sims=False)[source]

Loads the saved information from the preprocessing pipeline from a reference and a dependent database and runs the processing section of the pipeline for each.

Assumes preprocess_tod and multilayer_preprocess_tod have already been run on the requested observation.

Parameters:

obs_id (multiple) – Passed to context.get_obs to load AxisManager, see Notes for context.get_obs
configs_init (string or dictionary) – Config file or loaded config directory
configs_proc (string or dictionary) – Second config file or loaded config dictionary to load dependent databases generated using multilayer_preprocess_tod.py.
dets (dict) – Dets to restrict on from info in det_info. See context.get_meta.
meta (AxisManager) – Contains supporting metadata to use for loading. Can be pre-restricted in any way. See context.get_meta.
no_signal (bool) – If True, signal will be set to None. This is a way to get the axes and pointing info without the (large) TOD blob. Not all loaders may support this.
logger (PythonLogger) – Optional. Logger object or None will generate a new one.
init_only (bool) – Optional. If True, do not run the dependent pipeline.
ignore_cfg_check (bool) – If True, do not attempt to validate that configs_init is the same as the config used to create the existing init db.
stop_for_sims (bool) – Optinal. If True, will stop before each step of the pipeline with the flag use_data_aman set to True. The intended use is to prepare all necessary data products that cannot be stored in the preprocessing database, to process simulations.

Returns:

aman – Loaded and restricted axis manager with preprocessing metadata. Returns None if all detectors cut.

Return type:

core.AxisManager or None

sotodlib.preprocess.preprocess_util.multilayer_load_and_preprocess_sim(obs_id, configs_init, configs_proc, sim_map, meta=None, logger=None, init_only=False, ignore_cfg_check=False, data_amans=None, interpol=None, apply_wobble=False)[source]

Loads the saved information from the preprocessing pipeline from a reference and a dependent database, loads the signal from a (simulated) map into the AxisManager and runs the processing section of the pipeline for both databases.

Assumes preprocess_tod and multilayer_preprocess_tod have already been run on the requested observation.

Parameters:

obs_id (multiple) – Passed to context.get_obs to load AxisManager, see Notes for context.get_obs
configs_init (string or dictionary) – Config file or loaded config directory
configs_proc (string or dictionary) – Second config file or loaded config dictionary to load dependent databases generated using multilayer_preprocess_tod.py.
sim_map (numpy.ndmap or enmap.ndmap) – Input simulated map to be observed
meta (AxisManager) – Contains supporting metadata to use for loading. Can be pre-restricted in any way. See context.get_meta.
no_signal (bool) – If True, signal will be set to None. This is a way to get the axes and pointing info without the (large) TOD blob. Not all loaders may support this.
logger (PythonLogger) – Optional. Logger object or None will generate a new one.
init_only (bool) – Optional. Whether or not to run the dependent pipeline.
ignore_cfg_check (bool) – If True, do not attempt to validate that configs_init is the same as the config used to create the existing init db.
data_amans (dict (Optional)) – A dictionary of AxisManagers with keys (step, process.name) filled with AxisManager processed up to step-1. This is used to pre-load all data AxisManager which could be required when processing simulations (e.g. to provide a T2P template)
interpol (str) – Optional. The sub-pixel interpolation to use in from_map
apply_wobble (bool) – If true, apply pointing wobble to boreight pointing. This only works when all detectors belong to a single wafer_slot and bandpass. See coords.helpers.get_deflected_sightline. Defaults to False.

Returns:

aman – Loaded and restricted axis manager with preprocessing metadata. Returns None if all detectors cut.

Return type:

core.AxisManager or None

sotodlib.preprocess.preprocess_util.preproc_or_load_group(obs_id, configs_init, dets, configs_proc=None, logger=None, overwrite=False, save_archive=False, save_proc_aman=True, compress=False, skip_missing=False, ignore_cfg_check=False)[source]

This function is expected to receive a single obs_id, and dets dictionary. The dets dictionary must match the grouping specified in the preprocess config files. It accepts either one or two config strings or dicts representing an initial and a dependent pipeline stage. If the preprocess database entry for this obsid-dets group already exists then this function will just load back the processed tod calling either the load_and_preprocess or multilayer_load_and_preprocess functions. If the db entry does not exist or the overwrite flag is set to True then the full preprocessing steps defined in the configs are run and if save_proc_aman is True, the outputs are written to a unique h5 file. Any errors, the info to populate the database, the file path of the h5 file, and the process tod are returned from this function. Processed axis managers can be written to an archive and database by using cleanup_mandb (or setting save_archive to True) which consumes all of the outputs (except the processed tod), writes to the database, and moves the multiple h5 files into fewer h5 files (each <= 10 GB).

Parameters:

obs_id (str) – Obs id to process or load
configs_init (str or dict) – Filepath or dictionary containing the preprocess configuration file.
dets (dict) – Dictionary specifying which detectors/wafers to load see Context.obsdb.get_obs.
configs_proc (str or dict) – Filepath or dictionary containing a dependent preprocess configuration file.
logger (PythonLogger) – Optional. Logger object or None will generate a new one.
overwrite (bool) –

Optional. Whether or not to overwrite existing entries in the
preprocess manifest db.

save_archivebool
Call cleanup_mandb if True to save to the archive and database files in configs_init and configs_proc. Should be False if preproc_or_load_group is being called from within a parallelized script (i.e. python multiprocessing or MPI).
save_proc_aman (bool) – Whether or not to save the preprocessing axis manager. Required if saving into a preprocessing archive.
compress (bool) – Whether or not to compress the preprocessing data. Uses flacarray compression.
skip_missing (bool) – Do not attempt to run preprocessing pipeline if either of the preproc dbs don’t exist or the obs_id and group combination is not found.
ignore_cfg_check (bool) – If True, do not attempt to validate that configs_init is the same as the config used to create the existing init db when running multilayer_load_and_preprocess.

Returns:

aman (AxisManager or None) – Preprocessed axis manager if preproc_or_load_group finished successfully or None if it failed.
out_dict_init (dict or None) – Dictionary output for init config from get_preproc_group_out_dict if preprocessing ran successfully for init layer or None if preprocessing was loaded or preproc_or_load_group failed.
out_dict_proc (dict or None) – Dictionary output for proc config from get_preproc_group_out_dict if preprocessing ran successfully for proc layer or None if preprocessing was loaded, that layer was not run or loaded, or preproc_or_load_group failed.
errors (tuple) – A tuple containing the error from PreprocessError, an error message, and the traceback. Each will be None if preproc_or_load_group finished successfully.

Processing Util Functions

These functions support and are used within the driver processing scripts above and are useful for saving, loading, and verifying preprocessing archives and databases.

class sotodlib.preprocess.preprocess_util.PreprocessErrors[source]

Bases: object

Stores the various errors that can occur from the preprocessing functions.

LoadSuccess = 'load_success'

GetGroupsError = 'get_groups_error'

MetaDataError = 'get_meta_data_error'

NoDetsRemainError = 'no_dets_remain_error'

NoGroupOverlapError = 'no_group_overlap_error'

MultilayerPipelineLoadError = 'multilayer_load_and_preprocess_error'

SingleLayerPipelineLoadError = 'single_layer_load_and_preprocess_error'

PipeLineRunError = 'pipeline_run_error'

InitPipeLineRunError = 'init_pipeline_run_error'

ProcPipeLineRunError = 'proc_pipeline_run_error'

PipeLineStepError = 'pipeline_step_error'

NoInitDbError = 'no_init_db_error'

GroupOutputError = 'group_output_error'

ExecutorFutureError = 'executor_future_error'

SkipMissingError = 'skip_missing_error'

classmethod get_errors(e)[source]

sotodlib.preprocess.preprocess_util.filter_preproc_runlist_by_jobdb(jdb, jclass, db, run_list, group_by, overwrite=False, logger=None)[source]

Given a preprocess_tod or multilayer_preprocess_tod run list, checks whether that entry exists in the preprocess jobdb. If it failed or is done and overwrite is False, add it to the list of skipped obs_ids. If it doesn’t exist, is open, or is done but overwite is True, add an open job to the jobdb.

Parameters:

jdb (JobManager) – The preprocessing jobdb.
jclass (str) – The jobdb class name.
db (ManifestDb or None) – Preprocessing database.
run_list (list) – List of (obs_id, group) tuples.
group_by (list) – How grouping is being done for preprocessing. Specified in the preprocessing config through the subobs.use entry.
overwrite (bool) – Whether or not to overwrite entries in the preprocessing db.
logger (PythonLogger) – A python logger.

Returns:

run_list – Run list with the subset of skipped entries removed.

Return type:

list

sotodlib.preprocess.preprocess_util.init_logger(name, announce='', verbosity=2)[source]

Configure and return a logger for site_pipeline elements. It is disconnected from general sotodlib (propagate=False) and displays relative instead of absolute timestamps.

Parameters:

name (str) – The name of the logger
announce (str) – Initial message to be displayed after logger is instantiated.
verbosity (int) – Level of logger output 0: Error 1: Warning 2: Info 3: Debug

Returns:

logger – The initialized logger object

Return type:

PythonLogger

sotodlib.preprocess.preprocess_util.get_preprocess_context(configs, context=None)[source]

Load the provided config file and context file. To be used in preprocess_*.py site pipeline scripts. If the provided context file does not have a metadata entry for preprocess then one will be added based on the definition in the config file.

Parameters:

configs (str or dict) – The configuration file or dictionary.
context (str or core.Context, optional) – The context to use. If None, it is created from the configuration file.

Returns:

configs (dict) – The configuration dictionary.
context (core.Context) – The context file.

sotodlib.preprocess.preprocess_util.get_groups(obs_id, configs, context=None)[source]

Get subobs group method and groups. To be used in preprocess_*.py site pipeline scripts.

Parameters:

obs_id (str) – The obsid.
configs (str or dict) – The configuration dictionary.
context (core.Context) – The Context file to use.

Returns:

group_by (list of str) – The list of keys used to group the detectors.
groups (list of list of int) – The list of groups of detectors.
errors (tuple) – Tuple of errors or Nones.

sotodlib.preprocess.preprocess_util.get_preprocess_db(configs, group_by, logger=None)[source]

Get or create a ManifestDb found for a given config.

Parameters:

configs (dict) – The configuration dictionary.
group_by (list of str) – The list of keys used to group the detectors.
logger (PythonLogger) – Optional. Logger object. If None, a new logger is created.

Returns:

db – ManifestDb object

Return type:

ManifestDb

sotodlib.preprocess.preprocess_util.swap_archive(config, fpath)[source]

Update the configuration archive policy filename, create an output archive directory if it doesn’t exist, and return a copy of the config.

Parameters:

configs (dict) – The configuration dictionary.
fpath (str) – The archive policy filename to write to.

Returns:

tc – Copy of the configuration file with an updated archive policy filename

Return type:

dict

sotodlib.preprocess.preprocess_util.load_preprocess_det_select(obs_id, configs, context=None, dets=None, meta=None, logger=None)[source]

Loads the metadata information for the Observation and runs through any data selection specified by the Preprocessing Pipeline.

Parameters:

obs_id (multiple) – Passed to context.get_obs to load AxisManager, see Notes for context.get_obs
configs (string or dictionary) – Config file or loaded config directory
context (core.Context) – The Context file to use.
dets (dict) – Dets to restrict on from info in det_info. See context.get_meta.
meta (AxisManager) – Contains supporting metadata to use for loading. Can be pre-restricted in any way. See context.get_meta.
logger (PythonLogger) – Optional. Logger object. If None, a new logger is created.

Returns:

Restricted list of detector vals.

Return type:

list

sotodlib.preprocess.preprocess_util.find_db(obs_id, configs, dets, context=None, logger=None)[source]

This function checks if the manifest db from a config file exists and searches if it contains an entry for the provided Obs id and set of detectors.

Parameters:

obs_id (str) – Obs id to process or load
configs (str or dict) – Filepath or dictionary containing the preprocess configuration file.
dets (dict) – Dictionary specifying which detectors/wafers to load see Context.obsdb.get_obs.
context (core.Context) – Optional. Context object used for data loading/querying.
logger (PythonLogger) – Optional. Logger object or None will generate a new one.

Returns:

dbexist – True if db exists and entry for input detectors is found.

Return type:

bool

sotodlib.preprocess.preprocess_util.get_preproc_group_out_dict(obs_id, configs, dets, context=None, subdir='temp')[source]

This function returns a dictionary containing the data destination filename and the values to populate the manifest db.

Parameters:

obs_id (str) – Obs id to process or load
configs (str or dict) – Filepath or dictionary containing the preprocess configuration file.
dets (dict) – Dictionary specifying which detectors/wafers to load see Context.obsdb.get_obs.
context (core.Context) – Optional. Context object used for data loading/querying.
subdir (str) – Optional. Subdirectory to save the output files into. If it does not exist, it is created.

Returns:

outputs – Dictionary including output filename of data file and information for corresponding database entry.

Return type:

dict

sotodlib.preprocess.preprocess_util.save_group_and_cleanup(obs_id, configs, context=None, subdir='temp', logger=None, remove=False)[source]

This function checks if any temporary files exist from a preprocessing: run and will either add them to the config policy file and create an entry in the manifest db by calling cleanup_mandb. If the file exists but cannot be opened or if remove is True, the file will be deleted. Remove is intended to be to allow for overwrite=True in preprocess_tod.py and multilayer_preprocess_tod.py.

Parameters:

obs_id (str) – Obs id to process or load
configs (str or dict) – Filepath or dictionary containing the preprocess configuration file.
context (core.Context) – Optional. Context object used for data loading/querying.
subdir (str) – Optional. Subdirectory to save the output files into. If it does not exist, it is created.
logger (PythonLogger) – Optional. Logger object or None will generate a new one.
remove (bool) – Optional. Default is False. Whether to remove a file if found. Used when overwrite is True in driving functions.

Returns:

errors – Error from get_groups.

Return type:

tuple

sotodlib.preprocess.preprocess_util.cleanup_obs(obs_id, policy_dir, errlog, configs, context=None, subdir='temp', remove=False)[source]

For a given obs id, this function will search the policy_dir directory if it exists for any files with that obsnum in their filename. If any are found, it will run save_group_and_cleanup for that obs id.

Parameters:

obs_id (str) – Obs id to check and clean up
policy_dir (str) – Directory to temp per-group output files
errlog (str) – Filepath to error logging file.
configs (str or dict) – Filepath or dictionary containing the preprocess configuration file.
context (core.Context) – Optional. Context object used for data loading/querying.
subdir (str) – Optional. Subdirectory to save the output files into.
remove (bool) – Optional. Default is False. Whether to remove a file if found. Used when overwrite is True in driving functions.

sotodlib.preprocess.preprocess_util.cleanup_mandb(out_dict, out_meta, errors, configs, logger=None, overwrite=False, db_manager=None)[source]

Function to update the manifest db when data is collected from the preproc_or_load_group function. If used in an mpi framework this function is expected to be run from rank 0 after a comm.gather. See the preproc_or_load_group docstring for the varying expected values of errors and the associated out_dict. This function will either:

1) Update the ManifestDb sqlite file and move the h5 archive from its temporary location to its permanent path if errors[0] is None, out_dict is not``None``. Deletes the temporary h5 file.

2) Return nothing if errors[0] is PreprocessErrors.LoadSuccess or both it and out_dict are None.

Otherwise, update the error log.

Parameters:

errors (tuple) – A tuple containing the error from PreprocessError, an error message, and the traceback. Each will be None if preproc_or_load_group finished successfully.
out_meta (tuple) – The tuple (obs_id, group).
outputs (dict) – Dictionary including entries for the temporary h5 filename (‘temp_file’) and the obs_id group metadata and db entry (db_data). See save_group for more info.
configs (dict) – Preprocessing configuration dictionary.
logger (PythonLogger) – Optional. Python logger.
overwrite (bool) – Optional. Delete the entry in the archive file if it exists and replace it with the new entry.
db_manager (DbBatchManager, optional) – External database batch manager for optimized operations. If provided, uses the manager instead of creating individual connections.

sotodlib.preprocess.preprocess_util.get_pcfg_check_aman(pipe)[source]

Given a preprocess pipeline class return an axis manager containing the ordered steps of the pipeline with all arguments for each step.

Parameters:: pipe (_Preprocess class) – Preprocess pipeline class from which to build the step argument axis manager.

sotodlib.preprocess.preprocess_util.check_cfg_match(ref, loaded, logger=None)[source]

Checks that the ref and loaded axis managers containing the ordered preprocess pipelines match one another.

Parameters:

ref (AxisManager) – Reference axis manager for cross checking
loaded (AxisManager) – Loaded axis manager for cross checking.
logger (PythonLogger) – Optional. Python logger object.

Example TOD Pipeline Configuration File

Suppose we want to run a simple pipeline that runs the glitch calculator and estimates the white noise levels of the data. A configuration file for the processing pipeline would look like:

# Context for the data
context_file: 'context.yaml'

# Plot directory prefix
plot_dir: './plots'

# How to subdivide observations
subobs:
    use: ["wafer_slot", "wafer.bandpass"]
    label: "wafer_slot"

# Metadata index & archive filenaming
archive:
    index: 'preprocess_archive.sqlite'
    policy:
        type: 'simple'
        filename: 'preprocess_archive.h5'
    batch_size: 50

process_pipe:
    - name : "fft_trim"
      process:
        axis: 'samps'
        prefer: 'right'

    - name: "trends"
      calc:
        max_trend: 30
        n_pieces: 5
      save: True
      select:
        kind: "any"

    - name: "glitches"
      calc:
        t_glitch: 0.002
        hp_fc: 0.5
        n_sig: 10
        buffer: 20
      save: True
      select:
        max_n_glitch: 20
        sig_glitch: 30

    - name: "detrend"
      process:
        method: "linear"
        count: 10

    - name: "calibrate"
      process:
        kind: "array"
        cal_array: "det_cal.phase_to_pW"

    - name: "psd"
      process:
        detrend: False
        window: "hann"

    - name: "noise"
      calc:
        low_f: 5
        high_f: 10
      save: True
      select:
        max_noise: 2000

This pipeline can be run through the functions saved in site_pipeline. Each entry in “process_pipe” key will be used to generate a Preprocess module based on the name it is registered to. These entries will then be run in order through the processing pipe. The process function is always run before the calc_and_save function for each module. The plot function can be run after calc_and_save when plot: True for a module that supports it.

Example Planet TOD Pipeline Configuration File

Similar to a regular TOD pipeline, if we want to run one for planet observations, we must first flag sources in the signal and gapfill them. An example configuration file should be equivalent to non-planet data processing after a few extra first steps:

# Context for the data
context_file: 'context.yaml'

# Plot directory prefix
plot_dir: './plots'

# How to subdivide observations
subobs:
    use: wafer_slot
    label: wafer_slot

# Metadata index & archive filenaming
archive:
    index: 'preprocess_archive.sqlite'
    policy:
        type: 'simple'
        filename: 'preprocess_archive.h5'

process_pipe:
    - name : "dark_dets"
      calc: True
      save: True
      select: True

    - name: "source_flags"
      calc:
        mask: {'shape': 'circle',
              'xyr': [0, 0, 1.]}
        center_on: 'jupiter' # set to 'planet' for variable according to planet tag of each obs (must use --planet-obs argument of site-pipeline script)
        res: 20 # np.radians(20/60)
        max_pix: 4.0e+6
      save: True

    - name: "glitchfill"
      flag_aman: "sources"
      flag: "source_flags"
      process:
        nbuf: 10
        use_pca: True
        modes: 3

Example Obs Pipeline Configuration File

Suppose we want to run an observation-level pipeline that creates a SSO footprint. A configuration file for the processing pipeline would look like:

# Context for the data
context_file: 'context.yaml'

# Plot directory prefix
plot_dir: './plots'

# Metadata index & archive filenaming
archive:
    index: 'preprocess_archive.sqlite'
    policy:
        type: 'simple'
        filename: 'preprocess_archive.h5'

process_pipe:
    - name: "sso_footprint"
      calc:
        # If you want to search for nearby sources, exclude source_list
        source_list: ['jupiter']
        distance: 20
        nstep: 100
      save: True
      plot:
        wafer_offsets: {'ws0': [-2.5, -0.5],
                        'ws1': [-2.5, -13],
                        'ws2': [-13, -7],
                        'ws3': [-13, 5],
                        'ws4': [-2.5, 11.5],
                        'ws5': [8.5, 5],
                        'ws6': [8.5, -7]}
        focal_plane: 'focal_plane_positions.npz'

Process Step Glossary

Quick reference for every registered process_pipe step name (the name: string you put in a config file), grouped by what it’s for. The “What it does” column is the first line of each class’s docstring – see that class’s full entry below (including calc/save/ select/plot config options and an example config block) in the same subsection. This table is generated from sotodlib.preprocess.processes; if you add a new registered process, add a row to the relevant table below and a matching .. autoclass:: entry in the same subsection – tests/test_preprocess_docs.py will fail CI if a class is registered but has no .. autoclass:: entry anywhere on this page.

General / Utility

`name:`	Class	What it does
`fft_trim`	`FFTTrim`	Trim the AxisManager to optimize for faster FFTs later in the pipeline.
`detrend`	`Detrend`	Remove mean, median or linear trend from the data.
`move`	`Move`	Rename or remove a data field (used to replace gamma angles with those from wiregrid for example).
`trim_flag_edge`	`TrimFlagEdge`	Trim edge until given flags of all detectors are False.

class sotodlib.preprocess.processes.FFTTrim(step_cfgs)[source]

Trim the AxisManager to optimize for faster FFTs later in the pipeline. All processing configs go to fft_trim

Example config block:

- name: "fft_trim"
  process:
    axis: "samps"
    prefer: "right"

fft_trim(tod, axis='samps', prefer='right')

Restrict AxisManager sample range so that FFTs are efficient. This uses the find_inferior_integer function.

Parameters:

tod (AxisManager) – Target, which is modified in place.
axis (str) – Axis to target.
prefer (str) – One of [‘left’, ‘right’, ‘center’], indicating whether to trim away samples from the end, the beginning, or !equally at the beginning and end (respectively).

Returns:

The (start, stop) indices to use to slice an array and get these samples.

class sotodlib.preprocess.processes.Detrend(step_cfgs)[source]

Detrend the signal. All processing configs go to detrend_tod

Example config block:

- name: "detrend"
  signal: "signal" # optional
  process:
    method: "linear"

detrend_tod(tod, method='linear', axis_name='samps', signal_name='signal', in_place=True, wrap_name=None, count=10)

Returns detrended data. Detrends data in place by default but pass in_place=False if you would like a copied array (such as if you’re just looking to use this in an FFT).

Using this with method =’mean’ and axis_name=’dets’ will remove a common mode from the detectors Using this with method =’median’ and axis_name=’dets’ will remove a common mode from the detectors with the median rather than the mean

Parameters:

tod (axis manager)
method (str) – method of detrending can be ‘linear’, ‘mean’, or median
axis_name (str) – the axis along which to detrend. default is ‘samps’
signal_name (str) – the name of the signal to detrend. defaults to ‘signal’. Can have any shape as long as axis_name can be resolved.
in_place (bool.) – If False it makes a copy of signal before detrending and returns the copy.
wrap_name (str or None.) – If not None, wrap the detrended data into tod with this name.
count (int) – Number of samples to use, on each end, when measuring mean level for ‘linear’ detrend. Values larger than 1 suppress the influence of white noise.

Returns:

signal – Detrended signal. Done in place or on a copy depend on in_place argument.

Return type:

array of type tod[signal_name]

class sotodlib.preprocess.processes.Move(step_cfgs)[source]

Rename or remove a data field. To delete the field, pass new_name=None. If proc_aman is True, move a data field of proc_aman.

Example config block:

- name: "move"
  proc_aman: False
  process:
    name: "name"
    new_name: "new_name"

move(self, name, new_name)

Rename or remove a data field. To delete the field, pass new_name=None.

Example usage:

aman.move('hwp_angle', None)
Deletes the field hwp_angle from aman.

aman.move('hwp_angle', 'angle')
Renames the field hwp_angle to angle.

aman.move('preprocess.t2p.t2p_stats', None)
Deletes the field t2p_stats from the sub-AxisManager aman.preprocess.t2p.

class sotodlib.preprocess.processes.TrimFlagEdge(step_cfgs)[source]

Trim edge until given flags of all detectors are False To find first and last sample id that has False (i.e., no flags applied) for all detectors. This is for avoiding glitchfill problem for data whose edge has flags of True.

Example config block:

- name: "trim_flag_edge"
  process:
    flags: "pca_exclude"

find_common_edge_idx(flags)

Find the common valid range across multiple RangesMatrix objects.

Parameters:: flags (RangesMatrix) – An instance of so3g.proj.RangesMatrix indicating flagged time ranges.
Returns:: minmum and maximum indices that has False flag across all detectros.
Return type:: min_idx, max_idx

Detector Cuts and Flags

`name:`	Class	What it does
`det_bias_flags`	`DetBiasFlags`	Derive poorly biased detectors from IV and Bias Step data.
`trends`	`Trends`	Check for large linear ramping in the data to look for unlocked detectors.
`ptp_flags`	`PTPFlags`	Find (and cut) detectors with anomalous peak-to-peak signal.
`inv_var_flags`	`InvVarFlags`	Find (and cut) detectors with too high inverse variance.
`cut_bad_dist`	`CutBadDistribution`	Detector cuts to keep a statistic (i.e white noise, peak-peak, fknee, etc.) within some bounds of a gaussian distribution.
`detcal_nan_cuts`	`DetcalNanCuts`	Remove detectors with NaN values in the specified det_cal metadata fields.
`fp_flags`	`FocalplaneNanFlags`	Cut detectors which have nans in their pointing information.
`dark_dets`	`DarkDets`	Cut dark detectors in the data.
`acu_drop_flags`	`AcuDropFlags`	Expands ACU drop (bad ACU/platform pointing data) flag fields in aman to all detectors.
`smurfgaps_flags`	`SmurfGapsFlags`	Expand smurfgaps (bad smurf data) flag of each stream_id to all detectors.
`load_premade_flags`	`LoadPremadeFlags`	Load premade flags from aman.
`tod_stats`	`GetStats`	Get basic statistics from a TOD or its power spectrum to use for flags and cuts.

class sotodlib.preprocess.processes.DetBiasFlags(step_cfgs)[source]

Derive poorly biased detectors from IV and Bias Step data. Save results in proc_aman under the “det_bias_flags” field.

Data selection cuts detectors flagged by any of the bias-range checks (see get_det_bias_flags below for what each checks).

Example config block:

- name: "det_bias_flags"
  calc:
    rfrac_range: [0.2, 0.8]
    # psat_range: [0.1, 10]  # optional, required if plot: True
  save: True
  select: True
  plot: True

get_det_bias_flags(aman, detcal=None, rfrac_range=(0.1, 0.7), psat_range=None, rn_range=None, si_range=None, phase_to_pW=None, merge=True, overwrite=True, name='det_bias_flags', full_output=False)

Function for selecting detectors in appropriate bias range.

Parameters:

aman (AxisManager) – Input axis manager.
detcal (AxisManager) – AxisManager containing detector calibration information from bias steps and IVs. If None defaults to aman.det_cal.
rfrac_range (Tuple) – Tuple (lower_bound, upper_bound) for rfrac det selection.
psat_range (Tuple) – Tuple (lower_bound, upper_bound) for P_SAT from IV analysis. P_SAT in the IV analysis is the bias power at 90% Rn in pW. If None, no flags are not applied from P_SAT.
rn_range (Tuple) – Tuple (lower_bound, upper_bound) for r_n det selection.
si_range (Tuple) – Tuple (lower_bound, upper_bound) for s_i det selection.
phase_to_pW (Tuple) – Tuple (lower_bound, upper_bound) for phase_to_pW det selection.
merge (bool) – If true, merges the generated flag into aman.
overwrite (bool) – If true, write over flag. If false, don’t.
name (str) – Name of flag to add to aman.flags if merge is True.
full_output (bool) – If true, returns the full output with separated RangesMatrices

Returns:

msk_aman – AxisManager containing RangesMatrix shaped N_dets x N_samps that is True if the detector is flagged to be cut and false if it should be kept based on the rfrac, and psat ranges. To create a boolean mask from the RangesMatrix that can be used for aman.restrict() use keep = ~has_all_cut(mask) and then restrict with aman.restrict('dets', aman.dets.vals[keep]). If full_output is True, this will contain multiple RangesMatrices.

Return type:

AxisManager

class sotodlib.preprocess.processes.Trends(step_cfgs)[source]

Calculate the trends in the data to look for unlocked detectors. All calculation configs go to get_trending_flags.

Saves results in proc_aman under the “trend” field.

Data selection can have key “kind” equal to “any” or “all.”

Example config block:

- name : "trends"
  signal: "signal" # optional
  calc:
    max_trend: 2.5
    t_piece: 100
  save: True
  plot: True
  select:
    kind: "any"

get_trending_flags(aman, max_trend=1.2, t_piece=500, max_samples=500, signal=None, timestamps=None, merge=True, overwrite=True, name='trends', full_output=False)

Flag Detectors with trends larger than max_trend. This function can be used to find unlocked detectors. Note that this is a rough cut and unflagged detectors can still have poor tracking.

Parameters:

aman (AxisManager) – The tod
max_trend (float) – Slope at which detectors are unlocked. The default is for use with phase units.
t_piece (float) – Duration in seconds of each pieces to cut the timestream in to to look for trends
max_samples (int) – Maximum samples to compute the slope with.
signal (array) – (Optional). Signal to use to generate flags, if None default is aman.signal.
timestamps (array) – (Optional). Timestamps to use to generate flags, default is aman.timestamps.
merge (bool) – If true, merges the generated flag into aman.
overwrite (bool) – If true, write over flag. If false, don’t.
name (str) – Name of flag to add to aman.flags if merge is True.
full_output (bool) – If true, returns calculated slope sizes

Returns:

cut (RangesMatrix) – RangesMatrix of trending regions
trends (AxisManager) – If full_output is true, calculated slopes and the sample edges where they were calculated.

class sotodlib.preprocess.processes.PTPFlags(step_cfgs)[source]

Find detectors with anomalous peak-to-peak signal.

Saves results in proc_aman under the “ptp_flags” field.

Example config block:

- name : "ptp_flags"
  calc:
    signal_name: "dsT"
    kurtosis_threshold: 6
  save: True
  select: True

get_ptp_flags(aman, signal_name='signal', kurtosis_threshold=5, merge=False, overwrite=False, ptp_flag_name='ptp_flag', outlier_range=(0.5, 2.0))

Returns a ranges matrix that indicates if the peak-to-peak (ptp) of the tod is valid based on the kurtosis of the distribution of ptps. The threshold is set by kurtosis_threshold.

Parameters:

aman (AxisManager) – The tod
signal_name (str) – Signal to estimate flags off of. Default is signal.
kurtosis_threshold (float) – Maximum allowable kurtosis of the distribution of peak-to-peaks. Default is 5.
merge (bool) – Merge RangesMatrix into aman.flags. Default is False.
overwrite (bool) – Whether to write over any existing data in aman.flags[ptp_flag_name] if merge is True. Default is False.
ptp_flag_name (str) – Field name used when merge is True. Default is ptp_flag.
outlier_range (tuple) – (lower, upper) bound of the initial cut before estimating the kurtosis.

Returns:

mskptps – RangesMatrix of detectors with acceptable peak-to-peaks. All ones if the detector should be cut.

Return type:

RangesMatrix

class sotodlib.preprocess.processes.InvVarFlags(step_cfgs)[source]

Find detectors with too high inverse variance.

Saves results in proc_aman under the “inv_var_flags” field.

Example config block:

- name : "inv_var_flags"
  calc:
    signal_name: "demodQ"
    nsigma: 6
  save: True
  select: True

get_inv_var_flags(aman, signal_name='signal', nsigma=5, merge=False, overwrite=False, inv_var_flag_name='inv_var_flag')

Returns a ranges matrix that indicates if the inverse variance (inv_var) of the tod is greater than nsigma away from the median.

Parameters:

aman (AxisManager) – The tod
signal_name (str) – Signal to estimate flags off of. Default is signal.
nsigma (float) – Maximum allowable deviation from the median inverse variance. Default is 5.
merge (bool) – Merge RangesMatrix into aman.flags. Default is False.
overwrite (bool) – Whether to write over any existing data in aman.flags[inv_var_flag_name] if merge is True. Default is False.
inv_var_flag_name (str) – Field name used when merge is True. Default is inv_var_flag.

Returns:

mskptps – RangesMatrix of detectors with acceptable inverse variance. All ones if the detector should be cut.

Return type:

RangesMatrix

class sotodlib.preprocess.processes.CutBadDistribution(step_cfgs)[source]

Detector cuts to keep a statistic within some bounds of a gaussian distribution.

Example config:

- name: "cut_bad_dist"
  select:
    param_name: wn_signal
    outlier_range: [0.5, 2.0]
    kurtosis_threshold: 2.0
    blame_max: False
    blame_min: False

For parameter options see: sotodlib.tod_ops.flags.get_good_distribution_flags()

class sotodlib.preprocess.processes.DetcalNanCuts(step_cfgs)[source]

Remove detectors with NaN values in the specified det_cal metadata fields.

Example config file entry:

- name: "detcal_nan_cuts"
  calc:
      fields: [tau_eff, phase_to_pW]
  save: True
  select: True

class sotodlib.preprocess.processes.FocalplaneNanFlags(step_cfgs)[source]

Find additional detectors which have nans: in their focal plane coordinates.

Saves results in proc_aman under the “fp_flags” field.

Example config block:

- name : "fp_flags"
  signal: "signal" # optional
  calc:
      merge: False
  save: True
  select: True

get_focalplane_flags(aman, merge=True, overwrite=True, invalid_flags_name='fp_flags')

Generate flags for invalid detectors in the focal plane.

Parameters:

aman (AxisManager) – Axismanager containing the focal plane AxisManager.
merge (bool) – If True, merges the generated flag into aman.
overwrite (bool) – If True, write over flag. If False, don’t.
invalid_flags_name (str) – Name of flag to add to aman.flags if merge is True.

Returns:

msk_invalid_fp – RangesMatrix of invalid detectors in the focal plane.

Return type:

RangesMatrix

class sotodlib.preprocess.processes.DarkDets(step_cfgs)[source]

Find dark detectors in the data.

Saves results in proc_aman under the “dark_dets” field.

Example config block:

- name : "dark_dets"
  signal: "signal" # optional
  calc: True
  save: True
  select: True

get_dark_dets(aman, merge=True, overwrite=True, dark_flags_name='darks')

Identify and flag dark detectors in the given aman object.

Parameters:

aman (AxisManager) – The tod.
merge (bool, optional) – If True, merge the dark detector flags into the aman.flags. Default is True.
overwrite (bool, optional) – If True, overwrite existing flags with the same name. Default is True.
dark_flags_name (str, optional) – The name to use for the dark detector flags in aman.flags. Default is ‘darks’.

Returns:

mskdarks – A matrix of ranges indicating the dark detectors.

Return type:

RangesMatrix

Raises:

ValueError – If merge is True and dark_flags_name already exists in aman.flags and overwrite is False.

class sotodlib.preprocess.processes.AcuDropFlags(step_cfgs)[source]

Expands ACU drop flag fields in aman to all detectors. ACU drop flags indicate where samples are missing from the ACU data due to aggregator failures.

Example config block:

- name: "acu_drop_flags"
  calc:
    buffer: 200 # disable buffering by setting to False or None.
    name: "acu_drop_flags"
    merge: True
  save: True

class sotodlib.preprocess.processes.SmurfGapsFlags(step_cfgs)[source]

Expand smurfgaps flag of each stream_id to all detectors smurfgaps flags indicates the samples of each stream_id where the lost frames are filled in the bookbinding process.

Example config block:

- name: "smurfgaps_flags"
  calc:
    buffer: 200
    name: "smurfgaps"
    merge: True
  save: True

expand_smurfgaps_flags(aman, buffer=200, name='smurfgaps', merge=True)

smurfgaps flags indicates the samples of each stream_id where the lost frames are filled in the bookbinding process. See sotodlib.io.bookbinder.bind.

This function expands smurfgaps flags of each stream_id to all detectors.

Parameters:

aman (AxisManager) – Input AxisManager
buffer (int) – Amount of buffer to apply on smurfgaps
name (str) – Name of flag to add to aman.flags if merge is True.
merge (bool) – If true, merges the generated flag into aman.

Returns:

smurfgaps – smurfgaps flag with ‘dets’ and ‘samps’ axis

Return type:

RangesMatrix

class sotodlib.preprocess.processes.LoadPremadeFlags(step_cfgs)[source]

Load premade flags from aman.

Saves results in proc_aman under the “premade_flags_name” field of aman. In addtion, you can select detectors based on the loaded flags. This is mainly used for simulation for planet mapmaking. When simulated planet is made, we need corresponding flags to carry out the same preprocessing as the real planet mapmaking. This class is useful whenever you want to premake custum flags and use it.

E.g., if aman contains ‘sim_jupiter_flag’, you can load it to proc_aman as follows:

Example config block:

- name : "load_premade_flags"
    load_premade_flag_name: 'sim_jupiter_flag'
    calc:
        inv_flag: True
    save: True
    select: # optional
        kind: "any"
        invert: True

class sotodlib.preprocess.processes.GetStats(step_cfgs)[source]

Get basic statistics from a TOD or its power spectrum.

Example config block:

- name : "tod_stats"
  signal: "signal"  # optional
  wrap: "tod_stats" # optional
  calc:
    stat_names: ["median", "std"]
    split_subscans: False  # optional
    psd_mask:  # optional, for cutting a power spectrum in frequency
      freqs: "psd.freqs"
      low_f: 1
      high_f: 10
  save: True

Glitches & Jumps

`name:`	Class	What it does
`glitches`	`GlitchDetection`	Run glitch detection algorithm to find glitches.
`glitchfill`	`GlitchFill`	Fill glitches.
`jumps`	`Jumps`	Run generic jump finding and fixing algorithm.
`fix_jumps`	`FixJumps`	Repairs the jump heights given a set of jump flags and heights.

class sotodlib.preprocess.processes.GlitchDetection(step_cfgs)[source]

Run glitch detection algorithm to find glitches. All calculation configs go to get_glitch_flags

Saves results in proc_aman under the “glitches” field.

Data selection should define a glitch significance “sig_glitch”, a maximum number of glitches “max_n_glitch”, and a maximum fraction of TOD samples “max_t_frac” that is allowed to be flagged by glitches.

Example configuration block:

- name: "glitches"
  glitch_name: "my_glitches"
  calc:
    signal_name: "hwpss_remove"
    t_glitch: 0.00001
    buffer: 10
    hp_fc: 1
    n_sig: 10
    subscan: False
  save: True
  plot:
      plot_ds_factor: 50
  select:
    max_n_glitch: 10
    sig_glitch: 10
    max_t_frac: 0.1

get_glitch_flags(aman, t_glitch=0.002, hp_fc=0.5, n_sig=10, buffer=200, detrend=None, signal_name=None, merge=True, overwrite=False, name='glitches', full_output=False, edge_guard=2000, subscan=False)

Find glitches with fourier filtering. Translation from moby2 as starting point

Parameters:

aman (AxisManager) – The tod.
t_glitch (float) – Gaussian filter width.
hp_fc (float) – High pass filter cutoff.
n_sig (int or float) – Significance of detection.
buffer (int) – Amount to buffer flags around found location
detrend (str) – Detrend method to pass to fourier_filter
signal_name (str) – Field name in aman to detect glitches on if None, defaults to signal
merge (bool)) – If true, add to aman.flags
name (string) – Name of flag to add to aman.flags
overwrite (bool) – If true, write over flag. If false, raise ValueError if name already exists in AxisManager
full_output (bool) – If true, return sparse matrix with the significance of the detected glitches
edge_guard (int) – Number of samples at the beginning and end of the tod to exclude from the returned glitch RangesMatrix. Defaults to 2000 samples (10 sec).
subscan (bool) – If True, compute the glitch threshold on a per-subscan basis. Includes turnarounds.

Returns:

flag – RangesMatrix object containing glitch mask.

Return type:

RangesMatrix

class sotodlib.preprocess.processes.GlitchFill(step_cfgs)[source]

Fill glitches. All process configs go to fill_glitches. Notes on flags. If flags are provided as step_cfgs, proc_aman.get(flags) is used. If provided as process_cfgs, aman.get(glitch_flags) is used instead.

Example configuration block:

- name: "glitchfill"
  signal: "hwpss_remove"
  flags: "glitches.glitch_flags" # optional
  process:
    nbuf: 10
    use_pca: False
    modes: 1
    in_place: True
    glitch_flags: "glitch_flags"
    wrap: None

fill_glitches(aman, nbuf=10, use_pca=False, modes=3, signal=None, glitch_flags=None, in_place=True, wrap=None)

This function fills pre-computed glitches provided by the caller in time-ordered data using either a polynomial (default) or PCA-based approach. Wraps the other functions in the tod_ops.gapfill module.

Parameters:

aman (AxisManager) – AxisManager to fill glitches in
nbuf (int) – Number of buffer samples to use in polynomial gap filling.
use_pca (bool) – Whether or not to fill glitches using pca model. Default is False
modes (int) – Number of modes in the pca to use if pca=True. Default is 3.
signal (ndarray or None) – Array of data to fill glitches in. If None then uses aman.signal. Default is None.
glitch_flags (str or RangesMatrix or None) – RangesMatrix containing flags to use for gap filling. If provided by a string, aman.flags.get(flags) is used for the flags. If None then uses aman.flags.glitches.
in_place (bool) – If False it makes a copy of signal before gap filling and returns the copy.
wrap (str or None) – If not None, wrap the gap filled data into tod with this name.

Returns:

signal – Returns ndarray with gaps filled from input signal.

Return type:

ndarray

class sotodlib.preprocess.processes.Jumps(step_cfgs)[source]

Run generic jump finding and fixing algorithm.

calc_cfgs should have ‘function’ defined as one of ‘find_jumps’, ‘twopi_jumps’ or ‘slow_jumps’. Any additional configs to the jump function goes in ‘jump_configs’.

Saves results in proc_aman under the “jumps” field.

Data section should define a maximum number of jumps “max_n_jumps”.

Example config block:

- name: "jumps"
  calc:
    function: "twopi_jumps"
  save:
    jumps_name: "jumps_2pi"
  plot:
      plot_ds_factor: 50
  select:
      max_n_jumps: 5

find_jumps(aman, signal, min_sigma, min_size, win_size, exact, fix: Literal[False] = False, inplace=False, merge=True, overwrite=False, name='jumps', ds=10, clean=80, **filter_pars) → Tuple[RangesMatrix, csr_array]

find_jumps(aman, signal, min_sigma, min_size, win_size, exact, fix: Literal[True], inplace, merge, overwrite, name, ds, clean, **filter_pars) → Tuple[RangesMatrix, csr_array, ndarray[Any, dtype[floating]]]

Find jumps in aman.signal_name with a matched filter for edge detection. Expects aman.signal_name to be 1D of 2D.

Parameters:

aman – axis manager.
signal – Signal to jumpfind on. If None than aman.signal is used.
min_sigma – Number of standard deviations to count as a jump, note that the standard deviation here is computed by std_est and is the white noise standard deviation, so it doesn’t include contributions from jumps or 1/f. If min_size is provided it will be used instead of this.
min_size – The smallest jump size counted as a jump. By default this is set to None and min_sigma is used instead, if set this will override min_sigma. If both min_sigma and min_size are None then the IQR is used as min_size.
win_size – Size of window used when peak finding. Also used for height estimation, should be of order jump width.
exact – If True search for the exact jump location. If False flag allow some undertainty within the window (cheaper).
fix – Set to True to fix.
inplace – Whether of not signal should be fixed inplace.
merge – If True will wrap ranges matrix into aman.flags.<name>
overwrite – If True will overwrite existing content of aman.flags.<name>
name – String used to populate field in flagmanager if merge is True.
ds – Downsample factor used when computing noise level, the actual factor used is ds*win_size.
clean – Cleaning value to pass to estimate_heights. See that function for details.
**filter_pars – Parameters to pass to _filter

Returns:

RangesMatrix containing jumps in signal,: if signal is 1D Ranges in returned instead. There is some uncertainty on order of a few samples. Jumps within a few samples of each other may not be distinguished.

heights: csr_array of jump heights.

fixed: signal with jump fixed. Only returned if fix is set.

Return type:

jumps

class sotodlib.preprocess.processes.FixJumps(step_cfgs)[source]

Repairs the jump heights given a set of jump flags and heights.

Example config block:

- name: "fix_jumps"
  signal: "signal" # optional
  process:
  jumps_aman: "jumps_2pi"

jumpfix_subtract_heights(x: ndarray[Any, dtype[floating]], jumps: RangesInt32 | RangesMatrix | ndarray[Any, dtype[bool_]], inplace: bool = False, heights: ndarray[Any, dtype[floating]] | csr_array | None = None, **kwargs) → ndarray[Any, dtype[floating]]

Naive jump fixing routine where we subtract known heights between jumps. Note that you should exepect a glitch at the jump locations. Works best if you buffer the jumps mask by a bit.

Parameters:

x – Data to jumpfix on, expects 1D or 2D.
jumps – Boolean mask or Ranges(Matrix) of jump locations. Should be the same shape at x.
inplace – Whether of not x should be fixed inplace.
heights – Array of jump heights, can be sparse. If None will be computed.
**kwargs – Additional arguments to pass to estimate_heights if heights is None.

Returns:

x with jumps removed.: If inplace is True this is just a reference to x.

Return type:

x_fixed

Noise & PSD

`name:`	Class	What it does
`psd`	`PSDCalc`	Calculate the PSD of the data and add it to the AxisManager.
`noise_ratio`	`NoiseRatio`	Compute ratios of “signal band” to white noise in PSDs (simple fit-independent way to check for excessive 1/f channels).
`noise`	`Noise`	Estimate the white noise levels in the data.

class sotodlib.preprocess.processes.PSDCalc(step_cfgs)[source]

Calculate the PSD of the data and add it to the AxisManager under the “psd” field.

Note: noverlap = 0 amd full_output = True are recommended to get unbiased: median white noise estimation by Noise.

Example config block:

- "name : "psd"
  "signal: "signal" # optional
  "wrap": "psd" # optional
  "process":
    "nperseg": 1024 # optional
    "noverlap": 0 # optional
    "wrap_name": "psd" # optional
    "subscan": False # optional
    "full_output": True # optional

calc_psd(aman, signal=None, timestamps=None, max_samples=262144, prefer='center', freq_spacing=None, merge=False, merge_suffix=None, overwrite=True, subscan=False, full_output=False, label_axis='dets', **kwargs)

Calculates the power spectrum density of an input signal using signal.welch(). Data defaults to aman.signal and times defaults to aman.timestamps. By default the nperseg will be set to power of 2 closest to the 1/50th of the samples used, this can be overridden by providing nperseg or freq_spacing.

Parameters:

aman (AxisManager) – with (dets, samps) OR (channels, samps)axes.
signal (float ndarray) – data signal to pass to scipy.signal.welch().
timestamps (float ndarray) – timestamps associated with the data signal.
max_samples (int) – maximum samples along sample axis to send to welch.
prefer (str) – One of [‘left’, ‘right’, ‘center’], indicating what part of the array we would like to send to welch if cuts are required.
freq_spacing (float) – The approximate desired frequency spacing of the PSD. If None the default nperseg of ~1/50th the signal length is used. If an nperseg is explicitly passed then that will be used.
merge (bool) – if True merge results into axismanager.
merge_suffix (str, optional) – Suffix to append to the Pxx field name in aman. Defaults to None (merged as Pxx).
overwrite (bool) – if true will overwrite f, Pxx axes.
subscan (bool) – if True, compute psd on subscans.
full_output – if True this also outputs nseg, the number of segments used for welch, for correcting bias of median white noise estimation by calc_wn.
label_axis (str) – The name of LabelAxis in the input aman. Default is dets.
**kwargs – keyword args to be passed to signal.welch().

Returns:

array of frequencies corresponding to PSD calculated from welch. Pxx: array of PSD values. nseg: number of segments used for welch. this is returned if full_output is True.

Return type:

freqs

class sotodlib.preprocess.processes.NoiseRatio(step_cfgs)[source]

Compute ratios of “signal band” to white noise in PSDs.

Example config block:

- name: "noise_ratio"
  psd: "psdQ"
  wrap: "noise_ratio_Q"
  subscan: False
  calc:
    f_sel: [0.04, 0.14]
    f_wn: [0.6, 1.0]
  save: True
  select:
    r_max: 1.19
    select_per_detector: True

noise_ratio(aman, pxx, freqs, f_sig=(0.04, 0.14), f_wn=(0.6, 1.0), subscan=False)

Compute the ratio of the mean PSD in two frequency regions to evaluate the noise.

Parameters:

aman (AxisManager) – Only used for matching the dets and subscans dimensions in the output aman.
pxx (np.ndarray[float]) – Input PSD. Can be [dets, nufreq] or [dets, nufreq, subscans] (NOT just [nufreq]).
freqs (np.ndarray[float]) – frequency information related to the psd.
f_sig (tuple) – 2-tuple of frequencies giving the range of the numerator (“signal band”)
f_wn (tuple) – 2-tuple of frequencies giving the range of the denominator (“white noise”)
subscan (bool) – True if the PSD is split by subscans

Returns:

calc_aman – Axis manager with fields “rdets” giving the per-detector ratio and “rmean” giving the ratio for the mean (over detectors) PSD.

Return type:

AxisManager

class sotodlib.preprocess.processes.Noise(step_cfgs)[source]

Estimate the white noise levels in the data. Assumes the PSD has been wrapped into the preprocessing AxisManager. All calculation configs go to calc_wn.

Saves the results into the “noise” field of proc_aman.

Can select detectors on the minimum and maximum white noise (min_noise and max_noise respectively) and with a maximum allowed fknee value (max_fknee; only if fitting). These may be passed as scalars or as a dictionary where the keys are the bandpass names.

When fit: True, the parameter wn_est can be a float or the name of an axis manager containing an array named white_noise. If not specified, the white noise is calculated with calc_wn() and used for wn_est. The calculated white noise will be stored in the noise fit axis manager.

Example config block for fitting PSD:

- name: "noise"
  fit: True
  subscan: False
  calc:
    fwhite: (5, 10)
    lowf: 1
    f_max: 25
    mask: True
    wn_est: noise
    fixed_param: 'wn'
    binning: True
    fit_method: log_curve_fit # or likelihood
    curve_fit_kwargs:
        maxfev: 20000
  save: True
  select:
    min_noise:
       f090: 18e-6
       f150: 18e-6
    max_noise: 80e-6
    max_fknee: 7
    require_finite_fit: True

Set select.require_finite_fit to True to drop detectors whose fit parameters contain NaNs (indicating a failed noise fit).

Example config block for calculating white noise only:

- name: "noise"
  fit: False
  subscan: False
  calc:
    low_f: 5
    high_f: 20
  save: True
  select:
    min_noise: 18e-6
    max_noise: 80e-6

If fit: True this operation will run sotodlib.tod_ops.fft_ops.fit_noise_model(), else it will run sotodlib.tod_ops.fft_ops.calc_wn().

Calibration

`name:`	Class	What it does
`calibrate`	`Calibrate`	Calibrate the timestreams based on some provided information (Abscal, relcal, bias step)–just a multiplication.
`pca_relcal`	`PCARelCal`	Estimate the relcal factor from the atmosphere using PCA.
`correct_iir_params`	`CorrectIIRParams`	Correct missing iir_params (readout downsampling filter) by default values.

class sotodlib.preprocess.processes.Calibrate(step_cfgs)[source]

Calibrate the timestreams based on some provided information.

Type of calibration is decided by process[“kind”]

1. “single_value” : multiplies entire signal by the single value process[“val”]

2. “array” : takes the dot product of the array with the entire signal. The array is specified by process["cal_array"], which must exist in aman. The array can be nested within additional AxisManager objects, for instance det_cal.phase_to_pW.

Example config block(s):

- name: "calibrate"
  process:
    kind: "single_value"
    divide: True # If true will divide instead of multiply.
    # phase_to_pA: 9e6/(2*np.pi)
    val: 1432394.4878270582
- name: "calibrate"
  process:
    kind: "array"
    cal_array: "cal.array"
  select:
    cut_array: "cal.missing_cal" # should be 0 where cal is good 1 where missing.

class sotodlib.preprocess.processes.PCARelCal(step_cfgs)[source]

Estimate the relcal factor from the atmosphere using PCA.

Example configuration file entry:

- name: 'pca_relcal'
  signal: 'lpf_sig'
  pca_run: 'run1'
  calc:
      pca:
          xfac: 2
          yfac: 1.5
          calc_good_medianw: True
      lpf:
          type: "sine2"
          cutoff: 1
          trans_width: 0.1
      trim_samps: 2000
  save: True
  plot:
      plot_ds_factor: 20

See tod_ops.pca for more details on the method.

class sotodlib.preprocess.processes.CorrectIIRParams(step_cfgs)[source]

Correct missing iir_params by default values. This corrects iir_params only when the observation is within the time_range that is known to have problem.

Example config block:

- name: "correct_iir_params"
  process: True

correct_iir_params(aman, ignore_time=False, check_srate=-1)

Correct missing iir_params by default values. This corrects iir_params only when the observation is within the time_range that is known to have problem.

See sotodlib.tod_ops.filters.iir_filter for more details of iir_params

Parameters:

aman (AxisManager of observation)
ignore_time (Boolean. True if we don't want to check if the observation is within) – a known bad time range.
check_srate (If greater than 0 will check that the observations sample rate is within) – check_srate Hz of 200 Hz. If less than 0 the check is skipped.

Return type:

List of field names that have no iir_params

HWP & Demodulation

`name:`	Class	What it does
`hwp_angle_model`	`HWPAngleModel`	Apply hwp angle model (from metadata) to the TOD.
`estimate_hwpss`	`EstimateHWPSS`	Builds a HWPSS (HWP-synchronous-signal) template.
`subtract_hwpss`	`SubtractHWPSS`	Subtracts a HWPSS template from signal.
`demodulate`	`Demodulate`	Demodulate the TOD.
`a2_stats`	`A2Stats`	Calculate statistical metrics for A2, the 2f-demodulated Q and U signals.
`get_tau_hwp`	`GetTauHWP`	Analyze observation with hwp spinning up or spinning down to estimate time constant.

class sotodlib.preprocess.processes.HWPAngleModel(step_cfgs)[source]

Apply hwp angle model to the TOD.

Saves results in proc_aman under the “hwp_angle” field.

Example config block:

- name : "hwp_angle_model"
  process: True
  calc:
    on_sign_ambiguous: 'fail'
  save: True

apply_hwp_angle_model(tod, on_sign_ambiguous='fail')

Applies hwp_angle_model to the hwp_solution and construct hwp angle calibrated in the telescope frame. This will populate the calibrated hwp angle as hwp_angle attribute. hwp_solution is the hwp angle measured in the encoder frame. hwp_angle_model corrects the sign and offset of hwp_solution.

Parameters:

tod – AxisManager
on_sign_ambiguous – Tolerance options for sign ambiguous fail: raise an error if there is any sign ambiguous pid: raise an error if pid sign does not exist offcenter: raise an error if offcenter sign does not exist

Returns:

AxisManager with hwp_angle attribute

Return type:

tod

class sotodlib.preprocess.processes.EstimateHWPSS(step_cfgs)[source]

Builds a HWPSS Template. Calc configs go to hwpss_model. Results of fitting saved if field specified by calc[“name”].

Example config block:

- "name : "estimate_hwpss"
  "calc":
    "signal_name": "signal" # optional
    "hwpss_stats_name": "hwpss_stats"
  "save": True

get_hwpss(aman, signal=None, hwp_angle=None, bin_signal=True, bins=360, lin_reg=True, modes=[1, 2, 3, 4, 5, 6, 7, 8], apply_prefilt=True, prefilt_cfg=None, prefilt_detrend='linear', flags=None, apodize_edges=True, apodize_edges_samps=1600, apodize_flags=True, apodize_flags_samps=200, apo_type='C1', merge_stats=True, hwpss_stats_name='hwpss_stats', merge_model=True, hwpss_model_name='hwpss_model')

Extracts HWP synchronous signal (HWPSS) from a time-ordered data (TOD) using linear regression or curve-fitting. The curve-fitting or linear regression are either run on the full time ordered data vs hwp angle or the time ordered data binned in hwp_angle. If the curve-fitting option is used it must be performed on the binned data.

Parameters:

aman (AxisManager object) – The TOD to extract HWPSS from.
signal (str or None) – The field name in the axis manager to use for the TOD signal. If not provided, signal will be used.
hwp_angle (array-like, optional) – The HWP angle for each sample in aman. If not provided, aman.hwp_angle will be used.
bin_signal (bool, optional) – Whether to bin the TOD signal into HWP angle bins before extracting HWPSS. Default is True.
bins (int, optional) – The number of HWP angle bins to use if bin_signal is True. Default is 360.
lin_reg (bool, optional) – Whether to use linear regression to extract HWPSS from the binned signal. If False, curve-fitting will be used instead. Default is True.
modes (list of int, optional) – The HWPSS harmonic modes to extract. Default is [1, 2, 3, 4, 5, 6, 7, 8].
apply_prefilt (bool, optional) – Whether to apply a high-pass filter to signal before extracting HWPSS. Default is True. If run through preprocess and signal is not aman.signal then default to False.
prefilt_cfg (dict, optional) – The configuration of the high-pass filter, in Hz. Only used if apply_prefilt is True. Default is sine2 filter of with cutoff frequency of 1.0 Hz and trans_width of 1.0 Hz.
prefilt_detrend (str or None) – Method of detrending when you apply prefilter. Default is linear. If data is already detrended or you do not want to detrend, set it to None.
flags (str or RangesMatrix or Ranges, optional) – Flags to be masked out before extracting HWPSS. If Default is None, and no mask will be applied. If provided by a string, aman.flags.get(flags) is used for the flags.
apodize_edges (bool, optional) – If True, applies an apodization window to the edges of the signal. Defaults to True.
apodize_edges_samps (int, optional) – The number of samples over which to apply the edge apodization window. Defaults to 1600.
apodize_flags (bool, optional) – If True, applies an apodization window based on the flags. Defaults to True.
apodize_flags_samps (int, optional) – The number of samples over which to apply the flags apodization window. Defaults to 200.
apo_type (str, optional) – Type of apodization, default is C1. See tod_ops.apodize for all options.
merge_stats (bool, optional) – Whether to add the extracted HWPSS statistics to aman as new axes. Default is True.
hwpss_stats_name (str, optional) – The name to use for the new field containing the HWPSS statistics if merge_stats is True. Default is ‘hwpss_stats’.
merge_extract (bool, optional) – Whether to add the extracted HWPSS to aman as a new signal field. Default is True.
hwpss_extract_name (str, optional) – The name to use for the new signal field containing the extracted HWPSS if merge_extract is True. Default is ‘hwpss_extract’.

Returns:

hwpss_stats –

The extracted HWPSS and its statistics. The statistics include:

coeffs (n_dets x n_modes) : coefficients of the model

\[\sum_n \mathrm{coeffs}[2n]\sin{(\mathrm{modes}[n] \chi_{\mathrm{hwp}})} + \mathrm{coeffs}[2n+1]\cos{(\mathrm{modes}[n] \chi_{\mathrm{hwp}})}\]

where the sum on n range(len(modes)). Note: n_modes is 2*len(modes)

covars (n_dets x n_modes x n_modes) : variance covariance matrix of the fitted coefficients for each detector.

redchi2 (n_dets) : reduced chi^2 of the fit for each detector.

In the binned case the following are returned:

binned_angle (n_bins) : binned version of hwp_angle in range (0, 2pi] with number of bins set by bins argument.

bin_counts (n_dets x n_bins): sample counts of each bin for each detector.

binned_signal (n_dets x n_bins) : binned signal for each detector.

sigma_bin (n_dets) : average over all bins of the standard deviation of the signal within each bin.

In the non-binned case the following are returned:

sigma_tod (n_dets) : estimate of the standard deviation of the signal using function estimate_sigma_tod

Return type:

AxisManager object

class sotodlib.preprocess.processes.SubtractHWPSS(step_cfgs)[source]

Subtracts a HWPSS template from signal.

Example config block:

- name: "subtract_hwpss"
  hwpss_stats: "hwpss_stats"
  process:
    subtract_name: "hwpss_remove"

subtract_hwpss(aman, signal='signal', hwpss_template_name='hwpss_model', subtract_name='hwpss_remove', in_place=False, remove_template=True)

Subtract the half-wave plate synchronous signal (HWPSS) template from the signal in the given axis manager.

Parameters:

aman (AxisManager) – The axis manager containing the signal and the HWPSS template.
signal (str, optional) – The name of the field in the axis manager containing the signal to be processed. Defaults to ‘signal’.
hwpss_template_name (str, optional) – The name of the field in the axis manager containing the HWPSS template. Defaults to ‘hwpss_model’.
subtract_name (str, optional) – The name of the field in the axis manager that will store the HWPSS-subtracted signal. Only used if in_place is False. Defaults to ‘hwpss_remove’.
in_place (bool, optional) – If True, the subtraction is done in place, modifying the original signal in the axis manager. If False, the result is stored in a new field specified by subtract_name. Defaults to False.
remove_template (bool, optional) – If True, the HWPSS template field is removed from the axis manager after subtraction. Defaults to True.

Return type:

None

class sotodlib.preprocess.processes.Demodulate(step_cfgs)[source]

Demodulate the TOD. All process configs go to demod_tod.

Example config block:

- name: "demodulate"
  process:
    trim_samps: 6000
    demod_cfgs:
      bpf_cfg: {'type': 'sine2', 'center': 8, 'width': 3.8, 'trans_width': 0.1}
      lpf_cfg: {'type': 'sine2', 'cutoff': 1.9, 'trans_width': 0.1}

If you want to set filters with respect to actual HWP rotation frequency, you can pass strings like below. * is needed after the number you want to multiply HWP freq by:

- name: "demodulate"
  process:
    trim_samps: 6000
    demod_cfgs:
      # You can set float number or str (i.e., ``'4*f_HWP'``) as configs
      bpf_cfg: {'type': 'sine2', 'center': '4*f_HWP', 'width': '3.8*f_HWP', 'trans_width': 0.1}
      lpf_cfg: {'type': 'sine2', 'cutoff': '1.9*f_HWP', 'trans_width': 0.1}

demod_tod(aman, signal=None, demod_mode=4, bpf_cfg=None, lpf_cfg=None, wrap=True)

Demodulate TOD based on HWP angle

Parameters:

aman (AxisManager) – The AxisManager object
signal (str, optional) – Axis name of the signal to demodulate in aman. Default is ‘signal’.
demod_mode (int, optional) – Demodulation mode. Default is 4 (i.e. 4th harmonic of HWP).
bpf_cfg (dict) – Configuration for Band-pass filter applied to the TOD data before demodulation. If not specified, a sine-squared bandwidth filter of (demod_mode * HWP speed) +/- 0.95*(HWP speed) is used with transition width 0.1. Example) bpf_cfg = {‘type’: ‘sine2’, ‘center’: 8.0, ‘width’: 3.8, ‘trans_width’: 0.1} or bpf_cfg = {‘type’: ‘sine2’, ‘center’: ‘4*f_HWP’, ‘width’: ‘1.9*f_HWP’, ‘trans_width’: 0.1} See filters.get_bpf for details.
lpf_cfg (dict) – Configuration for Low-pass filter applied to the demodulated TOD data. If not specified, a sine-squared filter with a cutoff frequency of 0.95*(HWP speed) and transition width 0.1 is used. Example) lpf_cfg = {‘type’: ‘sine2’, ‘cutoff’: 1.9, ‘trans_width’: 0.1} or lpf_cfg = {‘type’: ‘sine2’, ‘cutoff’: ‘0.95*f_HWP’, ‘trans_width’: 0.1} See filters.get_lpf for details.
wrap (bool, optional) – If True, the demodulated signal is wrapped and stored in the input aman container. If False, the demodulated signal is returned.

Returns:

The demodulated TOD data is added to the input aman container as new signals: ‘dsT’ for the original signal filtered with lpf, ‘demodQ’ for the demodulated signal real component filtered with lpf and multiplied by 2, and ‘demodU’ for the demodulated signal imaginary component filtered with lpf and multiplied by 2.

Return type:

None

class sotodlib.preprocess.processes.A2Stats(step_cfgs)[source]

Calculate statistical metrics for A2, the 2f-demodulated Q and U signals.

Takes the following calc config options:

Stat_names:: (list) List of strings identifying which statistics to calculate. Refer to sotodlib.tod_ops.flags.get_stats (below) for available stats. Default is ["mean", "median", "var", "ptp"].
Subscan:: (bool) Whether to calculate stats for each subscan separately. Default is False.

Example config block:

- name: "a2_stats"
  calc:
    stat_names: ["mean", "median", "var", "ptp"]
    subscan: True
  save: True

demod_tod(aman, signal=None, demod_mode=4, bpf_cfg=None, lpf_cfg=None, wrap=True)

Demodulate TOD based on HWP angle

Parameters:

aman (AxisManager) – The AxisManager object
signal (str, optional) – Axis name of the signal to demodulate in aman. Default is ‘signal’.
demod_mode (int, optional) – Demodulation mode. Default is 4 (i.e. 4th harmonic of HWP).
bpf_cfg (dict) – Configuration for Band-pass filter applied to the TOD data before demodulation. If not specified, a sine-squared bandwidth filter of (demod_mode * HWP speed) +/- 0.95*(HWP speed) is used with transition width 0.1. Example) bpf_cfg = {‘type’: ‘sine2’, ‘center’: 8.0, ‘width’: 3.8, ‘trans_width’: 0.1} or bpf_cfg = {‘type’: ‘sine2’, ‘center’: ‘4*f_HWP’, ‘width’: ‘1.9*f_HWP’, ‘trans_width’: 0.1} See filters.get_bpf for details.
lpf_cfg (dict) – Configuration for Low-pass filter applied to the demodulated TOD data. If not specified, a sine-squared filter with a cutoff frequency of 0.95*(HWP speed) and transition width 0.1 is used. Example) lpf_cfg = {‘type’: ‘sine2’, ‘cutoff’: 1.9, ‘trans_width’: 0.1} or lpf_cfg = {‘type’: ‘sine2’, ‘cutoff’: ‘0.95*f_HWP’, ‘trans_width’: 0.1} See filters.get_lpf for details.
wrap (bool, optional) – If True, the demodulated signal is wrapped and stored in the input aman container. If False, the demodulated signal is returned.

Returns:

The demodulated TOD data is added to the input aman container as new signals: ‘dsT’ for the original signal filtered with lpf, ‘demodQ’ for the demodulated signal real component filtered with lpf and multiplied by 2, and ‘demodU’ for the demodulated signal imaginary component filtered with lpf and multiplied by 2.

Return type:

None

get_stats(aman, signal, stat_names, split_subscans=False, mask=None, name='stats', merge=False)

Calculate basic statistics on a TOD or power spectrum.

The statistics currently implemented are: - 'mean' - 'median' - 'ptp' (peak to peak) - 'std' (standard deviation) - 'var' (variance) - 'kurtosis' - 'skew'

Parameters:

aman (AxisManager) – Input AxisManager.
signal (Array) – Input signal. Statistics will be computed over axis 1.
stat_names (list) – List of strings identifying which statistics to run.
split_subscans (bool) – If True statistics will be computed on subscans. Assumes aman.subscan_info exists already.
mask (Array) – Mask to apply before computation. 1d array for advanced indexing (keep True), or a slice object.
name (str) – Name of axis manager to add to aman if merge is True.

class sotodlib.preprocess.processes.GetTauHWP(step_cfgs)[source]

Analyze observation with hwp spinning up or spinning down and compute the timeconstant of detectors from hwp speed dependence of the angle of half-wave plate synchronous signal.

Example config block:

- name: "get_tau_hwp"
  calc:
    width: 1000
    apodize_samps: 2000
    trim_samps: 2000
    min_fhwp: 1
    max_fhwp: 2
    demod_mode: 4
    name: "tau_hwp"
    merge: False
  save: True

get_tau_hwp(aman, signal='signal', lpf_cfg=None, bpf_cfg=None, width=1000, apodize_samps=2000, apo_type='C1', trim_samps=2000, min_fhwp=1.0, max_fhwp=2.0, demod_mode=4, wn=None, flags=None, full_output=False, merge=False, name='tau_hwp')

Analyze observation with hwp spinning up or spinning down and compute the timeconstant of detectors. This demoulate tod and estimate timeconstant from hwp rotation speed depdndence of half-wave plate synchronous signal.

Parameters:

aman (AxisManager) – AxisManager object containing the TOD data.
signal (std optional) – Name of signal to process. Default is signal.
lpf_cfg (dict optional) – Configuration for Low-pass filter applied before demodulation.
bpf_cfg (dict optional) – Configuration for Band-pass filter applied before demodulation.
width (int optional) – width of single section of TOD.
apodize_samps (int optional) – Number of samples on tod ends to apodize.
apo_type (str, optional) – Type of apodization, default is C1. See tod_ops.apodize for all options.
trim_samps (int optional) – Number of samples on tod ends to trim.
min_fhwp (float optional) – Mininum rotation frequency of hwp to be used for calculation.
max_fhwp (float optional) – Maximum rotation frequency of hwp to be used for calculation.
demod_mode (int, optional) – Demodulation mode. Default is 4.
flags (str optional) – Name of flags in aman.flags to use for fitting.
wn (str or None) – Precomputed white noise level of signal to be used for weights of fitting. If none, the fitting weights will be 1.
full_output (bool optional) – Whether to output all the statistics used for fitting.
merge (bool optional) – Whether to merge calculated statistics to aman. Default is False.
name (str optional) – Name under which to wrap the calculated statistics. Default is ‘tau_hwp’.

Returns:

result – An AxisManager containing time constants, their errors, and reduced chi-squared statistics.

Return type:

AxisManager

Ground Pickup (AzSS)

`name:`	Class	What it does
`azss`	`AzSS`	Estimates Azimuth Synchronous Signal (AzSS) by binning signal by azimuth of boresight and subtract.
`subtract_azss_template`	`SubtractAzSSTemplate`	Subtract Azimuth Synchronous Signal (AzSS) common template.

class sotodlib.preprocess.processes.AzSS(step_cfgs)[source]

Estimates Azimuth Synchronous Signal (AzSS) by binning signal by azimuth of boresight and subtract. All process configs go to get_azss. If method is ‘interpolate’, no fitting applied and binned signal is directly used as AzSS model. If method is ‘fit’, Legendre polynominal fitting will be applied and used as AzSS model. If subtract is True in process, subtract AzSS model from signal in place.

Example configuration block:

- name: "azss"
  calc:
    signal: 'demodQ'
    azss_stats_name: 'azss_statsQ'
    azrange: [-1.57079, 7.85398]
    bins: 1080
    flags: 'glitch_flags'
    merge_stats: True
    merge_model: False
  save: True
  process:
    subtract: True

If we estimate and subtract azss in left going scans only, make union of glitch_flags and scan_flags first:

- name : "union_flags"
  process:
    flag_labels: ['glitches.glitch_flags', 'turnaround_flags.right_scan']
    total_flags_label: 'glitch_flags_left'

- name: "azss"
  calc:
    signal: 'demodQ'
    azss_stats_name: 'azss_statsQ_left'
    azrange: [-1.57079, 7.85398]
    bins: 1080
    flags: 'glitch_flags_left'
    scan_flags: 'left_scan'
    merge_stats: True
    merge_model: False
  save: True
  process:
    subtract: True

get_azss(aman, signal='signal', az=None, azrange=None, bins=100, flags=None, scan_flags=None, apodize_edges=True, apodize_edges_samps=1600, apodize_flags=True, apodize_flags_samps=200, apo_type='C1', apply_prefilt=True, prefilt_cfg=None, prefilt_detrend='linear', method='interpolate', max_mode=None, modes_axis_name='azss_modes', subtract_in_place=False, merge_stats=True, azss_stats_name='azss_stats', merge_model=True, azss_model_name='azss_model', coverage_threshold=0.95, exclude_turnarounds=True, return_det_mask=False)

Derive azss (Azimuth Synchronous Signal) statistics and model from the given axismanager data. NOTE: This function does not modify the signal unless subtract_in_place = True.

Parameters:

aman (TOD) – core.AxisManager
signal (array-like, optional) – A numpy array representing the signal to be used for azss extraction. If not provided, the signal is taken from aman.signal.
az (array-like, optional) – A 1D numpy array representing the azimuth angles. If not provided, the azimuth angles are taken from aman.boresight.az.
azrange (list, optional) – A list specifying the range of azimuth angles to consider for binning. Defaults to [-np.pi, np.pi]. If None, [min(az), max(az)] will be used for binning.
bins (int or sequence of scalars) – If bins is an int, it defines the number of equal-width bins in the given azrange (100, by default). If bins is a sequence, it defines the bin edges, including the rightmost edge, allowing for non-uniform bin widths. If bins is a sequence, bins overwrite azrange.
flags (str or Rannges or RangesMatrix, optional) – Flag indicating whether to exclude flagged samples when binning the signal. Default is no mask applied.
scan_flags (str or Ranges, optional) – Subtract in the scan/time region specified by flags. Typically flags.left_scan or flags.right_scan will be used. If we estimate and subtract azss in left going scan only, then flags should be a “union of glitch_flags and flags.right_scan (or ~flags.left_scan)”, and scan_flags should be flags.left_scan.
apodize_edges (bool, optional) – If True, applies an apodization window to the edges of the signal. Defaults to True.
apodize_edges_samps (int, optional) – The number of samples over which to apply the edge apodization window. Defaults to 1600.
apodize_flags (bool, optional) – If True, applies an apodization window based on the flags. Defaults to True.
apodize_flags_samps (int, optional) – The number of samples over which to apply the flags apodization window. Defaults to 200.
apo_type (str, optional) – Type of apodization, default is C1. See tod_ops.apodize for all options.
apply_prefilt (bool, optional) – If True, applies a pre-filter to the signal before azss extraction. Defaults to True.
prefilt_cfg (dict, optional) – Configuration for the pre-filter. Defaults to {‘type’: ‘sine2’, ‘cutoff’: 0.005, ‘trans_width’: 0.005}.
prefilt_detrend (str, optional) – Method for detrending before filtering. Defaults to ‘linear’.
method (str) – The method to use for azss modeling. Options are ‘interpolate’ and ‘fit’. In ‘interpolate’, binned signal is used directly. In ‘fit’, fitting is applied to the binned signal. Defaults to ‘interpolate’.
max_mode (integer, optional) – The number of Legendre modes to use for azss when method is ‘fit’. Required when method is ‘fit’.
modes_axis_name (string, optional) – The name assigned to the LabelAxis of azss legendre modes when method is ‘fit’. Set a unique name when fitting azss with different max_mode values to avoid axis name conflicts. Defaults to ‘azss_modes’.
subtract_in_place (bool) – If True, it subtract the modeled tod from original signal. The aman.signal will be modified.
merge_stats (boolean, optional) – Boolean flag indicating whether to merge the azss statistics with aman. Defaults to True.
azss_stats_name (string, optional) – The name to assign to the merged azss statistics. Defaults to ‘azss_stats’.
merge_model (boolean, optional) – Boolean flag indicating whether to merge the azss model with the aman. Defaults to True.
azss_model_name (string, optional) – The name to assign to the merged azss model. Defaults to ‘azss_model’.
coverage_threshold (float, optional) – Minimum azimuth coverage fraction required. Default 0.8
exclude_turnarounds (bool, optional) – Exclude turnarounds when checking coverage. Default True
return_det_mask (bool, optional) – If True, return detector mask along with model. Default False

Returns:

azss_stats: core.AxisManager
- azss statistics including: azumith bin centers, bin counts, binned signal, std of each detector-az bin, std of each detector.
- If method=fit then also includes: binned legendre model, legendre bin centers, fit coefficients, reduced chi2.
- If return_det_mask=True then also includes: bad_dets, coverages.
model_sig_tod: numpy.array
- azss model as a function of time either from fits or interpolation depending on method argument.

Return type:

Tuple

class sotodlib.preprocess.processes.SubtractAzSSTemplate(step_cfgs)[source]

Subtract Azimuth Synchronous Signal (AzSS) common template. Make common template by weighted mean or pca. This requires to calculate AzSS beforehand.

Example configuration block:

- name: "subtract_azss_template"
  process:
    signal: 'signal'
    azss: 'azss_stats_left'
    method: 'interpolate'
    scan_flags: 'left_scan'
    pca_modes: 1
    subtract: True

subtract_azss_template(aman, signal='signal', azss='azss_stats', method='interpolate', scan_flags=None, pca_modes=None, subtract=True)

Make azss template model that is “common” to all detectors and subtract If pca_modes are specified, make template by pca, otherwise make template by weighted mean.

Parameters:

aman (AxisManager) – The axis manager containing the signal and the azss template.
signal (str or array-like) – numpy array of signal to be binned. If None, the signal is taken from aman.signal.
azss (str or azss_stats AxisManager) – azss_stats AxisManager generated by azss.get_azss
method (str) – The method to use for azss modeling. Option is ‘interpolate’ only now.
scan_flags (str or flags, optional) – Subtract template model in the time region specified by flags. Typically flags.left_scan or flags.right_scan will be used.
pca_modes (integer, optinal) – Number of pca modes for making azss template
subtract (boolean, optional) – If true subtract azss template from signal

Return type:

azss_template_model

Filtering

`name:`	Class	What it does
`fourier_filter`	`FourierFilter`	Applies a chain of Fourier filters (defined in fft_ops) to the data.
`sub_polyf`	`SubPolyf`	Fit TOD in each subscan with polynomial of given order and subtract it.
`apodize`	`Apodize`	Apodize the edges of a signal.
`scan_freq_cut`	`ScanFreqCut`	Apply high-pass cut at the scan frequency.
`pca_filter`	`PCAFilter`	Applies a pca filter to the data.
`get_common_mode`	`GetCommonMode`	Calculate common mode (average over detectors not PCA filtered).
`joint_qu_nmat_model`	`JointQUNmatModel`	Fit a joint demodulated Q/U Fourier Nmat operator; store it in proc_aman.
`joint_qu_nmat_filter`	`JointQUNmatFilter`	Apply a stored joint demodulated Q/U Fourier Nmat operator.

class sotodlib.preprocess.processes.FourierFilter(step_cfgs)[source]

Applies a chain of Fourier filters (defined in fft_ops) to the data.

Example config file entry for one filter:

- name: "fourier_filter"
  process:
    filt_function: "timeconst_filter"
    filter_params:
      timeconst: "det_cal.tau_eff"
      invert: True

Example for passing in a different signal name and wrapping into a new field:

- name: "fourier_filter"
      wrap_name: "lpf_demodQ"
      signal_name: "demodQ"
      process:
        filt_function: "sine2"
        filter_params:
          cutoff: 1
          trans_width: 0.1

Example config file entry for two filters:

- name: "fourier_filter"
  process:
    filters:
      - name: "iir_filter"
        filter_params:
          invert: True
      - name: "timeconst_filter"
        filter_params:
          timeconst: "det_cal.tau_eff"
          invert: True

Or with params from a noise fit:

- name: "fourier_filter"
  process:
    noise_fit_array: "noiseQ_fit"
    filters:
      - name: "iir_filter"
        filter_params:
          invert: True
      - name: "timeconst_filter"
        filter_params:
          timeconst: "det_cal.tau_eff"
          invert: True

See Fourier space filters documentation for more details.

class sotodlib.preprocess.processes.SubPolyf(step_cfgs)[source]

Fit TOD in each subscan with polynominal of given order and subtract it.: All process configs go to sotodlib.tod_ops.sub_polyf.

Example config block:

- name: "sub_polyf"
  process:
    degree: 0
    method: "polyfit"
    in_place: True

subscan_polyfilter(aman, degree, signal_name='signal', exclude_turnarounds=False, mask=None, exclusive=True, method='legendre', in_place=True)

Apply polynomial filtering to subscan segments in a data array. This function applies polynomial filtering to subscan segments within signal for each detector. Subscan segments are defined based on the presence of flags such as ‘left_scan’ and ‘right_scan’. Polynomial filtering is used to remove low-degree polynomial trends within each subscan segment.

Parameters:

aman (AxisManager)
degree (int) – The degree of the polynomial to be removed.
signal_name (string, optional) – The name of TOD signal to use. If not provided, aman.signal will be used.
exclude_turnarounds (bool) – Optional. If True, turnarounds are excluded from subscan identification. Default is False.
mask (str or RangesMatrix) – Optional. A mask used to select specific data points for filtering. If None, no mask is applied. If the mask is given in str, aman.flags['mask'] is used as mask. Arbitrary mask can be specified in the style of RangesMatrix.
exclusive – Optional. If True, the mask is used to exclude data from fitting. If False, the mask is used to include data for fitting. Default is True.
method (str) – Optioal. Method to model the baseline of TOD. In legendre method, baseline model is constructed using orthonormality of Legendre function. In polyfit method, numpy.polyfit is used. legendre is faster. Default is legendre.
in_place (bool) – Optional. If True, aman.signal is overwritten with the processed signal.

Returns:

signal – The processed signal.

Return type:

array-like

class sotodlib.preprocess.processes.Apodize(step_cfgs)[source]

Apodize the edges of a signal. All process configs go to apodize_cosine. If flags is provided, apodize based on it; otherwise, apodize the edge of the timestream.

Example config block:

- name: "apodize"
  process:
    signal_name: signal
    apodize_samps: 2000
    flags: glitch_flags
    apo_type: C1

apodize_cosine(aman, signal_name='signal', apodize_samps=1600, in_place=True, apo_axis='apodized', window=None, flags=None, apo_type='C1')

Function to smoothly filter the timestream to 0’s on the ends with a cosine function. If window is provided, multiply the window function to aman[signal_name]. If flags is provided, generate an apodization window based on flag values instead of ends of timestream.

Parameters:

signal_name (str) – Axis to apodize
apodize_samps (int) – Number of samples on tod ends to apodize.
in_place (bool) – writes over signal with apodized version
apo_axis (str) – Axis to store the apodized signal if not in place.
window (numpy.ndarray) – Precomputed apodization window.
flags (str or RangesMatrix or Ranges) – flag value to compute apodization window.
apo_type (str) –
Type of apodization window applied to the edges. Options are:
- 'C1': Standard cosine (Hann) taper, i.e. a half-cosine that goes smoothly from 1 to 0 as 0.5 * (1 + cos(x)) over the apodization region. This is the default.
- 'old_default': Legacy quarter-cosine taper, cos(x) over [0, pi/2]. Retained for backward compatibility.

class sotodlib.preprocess.processes.ScanFreqCut(step_cfgs)[source]

Apply high-pass cut at the scan frequency.

Example config block:

- name : 'scan_freq_cut'
  process:
    signal_name_T: 'dsT'
    signal_name_Q: 'demodQ'
    signal_name_U: 'demodU'

class sotodlib.preprocess.processes.PCAFilter(step_cfgs)[source]

Applies a pca filter to the data. model_signal is used to calculate the PCA modes, which are then subtracted from signal. If model_signal is not provided, signal is used for both. An example use case is to use a low-pass filtered version of the signal to calculate the PCA modes.

example config file entry:

- name: "pca_filter"
  model_signal: "lpf_signal" # optional, if not provided, use signal
  signal: "signal"
  process:
    n_modes: 10

See tod_ops.pca for more details on the method.

class sotodlib.preprocess.processes.GetCommonMode(step_cfgs)[source]

Calculate common mode.

example config file entry:

- name: "get_common_mode"
  wrap_name: "common_demodQ"
  calc:
      noise_fit: True
      f_max: 2.0
      signal: "signal"
      method: "median"
  save: True

If noise_fit is True, the 1/f noise fit parameters of the common mode is wrapped together. .. autofunction:: sotodlib.tod_ops.pca.get_common_mode

class sotodlib.preprocess.processes.JointQUNmatModel(step_cfgs)[source]

Fit a joint demodulated Q/U Fourier Nmat operator.

The Q and U detector streams are whitened and analyzed together. Modes inconsistent with independent noise according to a Marchenko–Pastur plus Tracy–Widom threshold are identified, and the whitened noise model N(f) = D(f) + V E(f) V.T is stored in proc_aman.

This is calculated from real data only (skip_on_sim: True) so that joint_qu_nmat_filter can reload it when running on simulations, following the same pattern as noise and fourier_filter.

Example configuration:

- name: "joint_qu_nmat_model"
  skip_on_sim: True
  signal_Q: "demodQ"
  signal_U: "demodU"
  calc:
    fmin: 0.0015
    fmax: 0.2
    noise_band: [0.5, 1.75]
    bin_width_hz: 0.2
    mp_significance: 0.999
    n_modes_max: 18
    singleness_max: 0.55
    profile_n_bins: 40
    profile_min_nfreq: 20
    profile_diagonal_floor: 0.05
  save:
    wrap_name: "nmat_qu"

See sotodlib.tod_ops.nmat_filter.fit_joint_qu_nmat_operator.

class sotodlib.preprocess.processes.JointQUNmatFilter(step_cfgs)[source]

Apply a stored joint demodulated Q/U Fourier Nmat operator.

Reads the operator fit by joint_qu_nmat_model from proc_aman and applies it, so simulations reuse the operator derived from real data rather than refitting it on signal-only timestreams.

Example configuration:

- name: "joint_qu_nmat_filter"
  skip_on_sim: False
  signal_Q: "demodQ"
  signal_U: "demodU"
  process:
    nmat_model: "nmat_qu"
    psd_scale: 1.0

Setting use_data_aman: True instead fits the operator directly from the supplied real-data AxisManager rather than reloading a stored one, with the fit parameters given under process.fit.

See sotodlib.tod_ops.nmat_filter.apply_joint_qu_nmat_operator.

Pointing & Focal-Plane Geometry

`name:`	Class	What it does
`pointing_model`	`PointingModel`	Apply pointing model to the TOD.
`rotate_focal_plane`	`RotateFocalPlane`	Interpret the boresight rotation effect as a focal plane rotation.
`rotate_qu`	`RotateQU`	Rotate Q and U components to/from telescope coordinates.
`subtract_qu_common_mode`	`SubtractQUCommonMode`	Subtract Q and U common mode.

class sotodlib.preprocess.processes.PointingModel(step_cfgs)[source]

Apply pointing model to the TOD.

Saves results in proc_aman under the “pointing” field.

Example config block:

- name : "pointing_model"
  process: True

apply_pointing_model(tod, pointing_model=None, ancil=None, wrap=None)

Applies a static pointing model to compute corrected boresight position and orientation in horizon coordinates. The encoder values in tod.ancil are consumed as raw data, and the computed values are stored in tod.boresight.

Parameters:

tod (AxisManager) – the observation data.
pointing_model (AxisManager) – if None, the pointing_model parameters are read from tod.pointing_model.
ancil (AxisManager) – if None, the encoders are read from tod.ancil.
wrap (str) – If specified, the name in tod where corrected boresight should be stored. If None, the default of ‘boresight’ is used. Pass wrap=False to not store the result in tod.

Returns:

the corrected boresight.

Return type:

AxisManager

class sotodlib.preprocess.processes.RotateFocalPlane(step_cfgs)[source]

Interpret the boresight rotation effect as a focal plane rotation and update them accordingly. This applies constant rotation to focal plane. If hwp=True, rotations of gamma are reflected. This updates boresight.roll and focal_plane, in place.

Example config block:

- name : "rotate_focal_plane"
  process:
    hwp: True

rotate_focal_plane(tod, hwp=True): Interpret the boresight rotation effect as a focal plane rotation and update them accordingly. This applies constant rotation to focal plane. If hwp=True, rotations of gamma are reflected. This updates tod.boresight.roll and tod.focal_plane, in place.

class sotodlib.preprocess.processes.RotateQU(step_cfgs)[source]

Rotate Q and U components to/from telescope coordinates.

sign: 1 (the default) rotates each detector’s demodQ/demodU out of its own polarization-angle frame and into the shared telescope frame, zeroing focal_plane.gamma when update_focal_plane: True. sign: -1 undoes that, rotating back from the shared telescope frame into each detector’s own polarization-angle frame.

Example config block:

- name : "rotate_qu"
  process:
    sign: 1
    offset: 0
    update_focal_plane: True

rotate_demodQU(tod, sign=1, offset=0, radial=False, update_focal_plane=True)

Apply detectors’ polarization angle calibration to the HWP demodulated Q and U timestreams to place all detectors’ Q and U timestreams in a common telescope frame. This updates tod.demodQ and tod.demodU, in place. To get Qr Ur timestreams, run rotate_demodQU(tod) and then rotate_demodQU(tod, radial=True). To restore the Q U timestreams, run rotate_demodQU(tod, sign=-1, radial=True).

Parameters:

tod – an axisManager object
update_focal_plane (bool, optional) – Whether to update focal_plane.gamma angles consistent with new coordinate reference. Make this True for polarization mapmaking using make_map.
offset – float, optional The rotation angle in degrees to apply (default is 0).
sign – int, optional A sign factor to control the direction of the rotation (default is +1).
radial – bool, optional If True and the Q U timestreams in tod are in a common telescope flame, this function turns Q U into Qr Ur.

class sotodlib.preprocess.processes.SubtractQUCommonMode(step_cfgs)[source]

Subtract Q and U common mode.

If calc is set, computes the median Q/U template and each detector’s coupling coefficient to it (via sotodlib.tod_ops.deproject.get_qu_common_mode_coeffs()) and saves them under the qu_common_mode_coeffs field of proc_aman. process then subtracts that scaled template from each detector’s Q/U signal. If calc was not run (or its result wasn’t saved), process falls back to computing the template/coefficients on the fly from the current aman instead of using a saved proc_aman archive.

Example config block:

- name : 'subtract_qu_common_mode'
  signal_name_Q: 'demodQ'
  signal_name_U: 'demodU'
  process: True
  calc: True
  save: True

get_qu_common_mode_coeffs(aman, Q_signal=None, U_signal=None, merge=False)

Gets the median signal (template) and coefficients for the coupling to that signal for each detector for both the Q and U signals. Returns an AxisManager with the template and coefficients wrapped.

Arguments:

aman: AxisManager: Contains the signal to operate on.
Q_signal: ndarray or str: array or string with field in aman containing the demodulated Q signal.
U_signal: ndarray or str: array or string with field in aman containing the demodulated U signal.
merge: bool: If True wrap the returned AxisManager into aman.

Returns:

output_aman: AxisManager: Contains the template signals for Q/U and coefficients coupling each detector to the templates.

subtract_qu_common_mode(aman, Q_signal=None, U_signal=None, coeff_aman=None, merge=False)

Subtracts the median signal (template) from each detector scaled by the a coupling coefficient per detector.

Arguments:

aman: AxisManager: Contains the signal to operate on.
Q_signal: ndarray or str: array or string with field in aman containing the demodulated Q signal.
U_signal: ndarray or str: array or string with field in aman containing the demodulated U signal.
coeff_aman: AxisManager: contains the coefficients and templates to use for subtraction. See get_qu_common_mode_coeffs.
merge: bool: If True wrap the returned AxisManager into aman.

Sources & Planets

`name:`	Class	What it does
`source_flags`	`SourceFlags`	Calculate the source flags in the data.
`filter_for_sources`	`FilterForSources`	Mask and gap-fill the signal at samples flagged by source_flags.
`sso_footprint`	`SSOFootprint`	Find nearby sources within a given distance and get SSO footprint and plot.

class sotodlib.preprocess.processes.SourceFlags(step_cfgs)[source]

Calculate the source flags in the data. All calculation configs go to get_source_flags.

Saves results in proc_aman under the “source_flags” field.

Example config block:

- name : "source_flags"
  source_flags_name: "my_source_flags"
  calc:
    mask: {'shape': 'circle',
           'xyr': [0, 0, 1.]}
    center_on: ['jupiter', 'moon'] # list of str
    res: 20 # arcmin
    max_pix: 4000000 # max number of allowed pixels in map
    distance: 0 # max distance of footprint from source in degrees
  save: True
  select: True # optional
    select_source: 'jupiter' # list of str or str. If not provided, all sources from center_on are selected.
    kind: 'any' # 'any', 'all', or float (0.0 < kind < 1.0)
    invert: False # optional, if True logic is filipped.
    Examples:
        1. invert=False, kind='any' → Select detectors with **no** True flags (e.g., for Moon cut).
        2. invert=True, kind='any' → Select detectors with **any** True flags (e.g., for planet selection).
        3. invert=False, kind=0.4 → Select detectors with <40% of True flags.

get_source_flags(aman, merge=True, overwrite=True, source_flags_name=None, mask=None, center_on=None, res=None, max_pix=None)

class sotodlib.preprocess.processes.FilterForSources(step_cfgs)[source]

Mask and gap-fill the signal at samples flagged by source_flags. Then PCA the resulting time ordered data.

example config file entry:

- name: "filter_for_sources"
  signal: "signal"
  process:
    n_modes: 10
    source_flags: "source_flags"
    edge_guard: 10 # Number of samples to make the first and last flags False
    trim_samps: 100
    pca_wrap: "pca_model" # optional, if provided, the PCA model is wrapped into aman under this key

filter_for_sources(tod=None, signal=None, source_flags=None, n_modes=10, low_pass=None, wrap=None, pca_wrap=None, edge_guard=None)

Mask and gap-fill the signal at samples flagged by source_flags. Then PCA the resulting time ordered data. Restore the flagged signal, remove the strongest modes from PCA.

If signal is not passed in tod.signal will be modified directly. To leave tod.signal intact, pass in signal=tod.signal.copy().

Parameters:

tod – AxisManager from which defaults will be drawn.
signal – Time-ordered data to operate on. Defaults to tod.signal.
source_flags – RangesMatrix to use for source flagging. Defaults to tod.source_flags.
low_pass – Frequency, in Hz, at which to apply low pass filter to signal. If None, no filtering is done. You can pass in a filter from tod_ops.filters if you want.
n_modes (int) – Number of eigenmodes to remove… interface subject to change.
wrap (str) – If specified, the result will be stored at tod[wrap].
pca_wrap (str) – If specified, the PCA model modes and weights calculated for subtraction will be wrapped into tod[pca_wrap].
edge_guard (int) – Number of samples at the beginning and end of the flags to change them False. Default is None. (Nothing happens.)

Returns:

The filtered signal.

class sotodlib.preprocess.processes.SSOFootprint(step_cfgs)[source]

Find nearby sources within a given distance and get SSO footprint and plot each source on the focal plane.

Example config block:

- name: "sso_footprint"
  calc:
      # Note: all distances in degrees
      source_list: ['jupiter', 'moon', 'saturn'] # remove to find nearby sources
      distance: 20 # distance from boresight center
      nstep: 100
      telescope_flavor: 'sat' # options: ['sat', 'lat']
      wafer_hit_threshold: 10 # number of planet-wafer distances to consider being a source hit
      # for SATs:
      wafer_radius: 6
      wafer_centers: {'ws0': [-0.19791037, 0.08939717],
                      'ws1': [-0.014455856, -12.528095],
                      'ws2': [-10.867158, -6.2621593],
                      'ws3': [-10.835234, 6.2727923],
                      'ws4': [0.11142064, 12.461107],
                      'ws5': [10.878714, 6.273904],
                      'ws6': [10.870621, -6.2822847]}
      # for LAT:
      wafer_radius: 0.5
      wafer_centers: {'c1_ws0': [-0.36504516, 1.9619369e-05],
                      'c1_ws1': [0.18297304, 0.3164044],
                      'c1_ws2': [0.18297556, -0.31638196],
                      'i1_ws0': [-1.9073119, -0.8932063],
                      'i1_ws1': [-1.357702, -0.57522374],
                      'i1_ws2': [-1.3556796, -1.20838],
                      'i3_ws0': [1.1854928, -0.8960549],
                      'i3_ws1': [1.7332374, -0.57540596],
                      'i3_ws2': [1.7351116, -1.2087585],
                      'i4_ws0': [1.1789553, 0.89766216],
                      'i4_ws1': [1.7351091, 1.208781],
                      'i4_ws2': [1.7332398, 0.5754285],
                      'i5_ws0': [-0.35970667, 1.7832578],
                      'i5_ws1': [0.19053483, 2.0997307],
                      'i5_ws2': [0.1866497, 1.4668859],
                      'i6_ws0': [-1.9017702, 0.89197767],
                      'i6_ws1': [-1.3556821, 1.2084025],
                      'i6_ws2': [-1.3564061, 0.5815026]}
  save: True
  plot:
      # for SATs:
      wafer_offsets: {'ws0': [-2.5, -0.5],
                      'ws1': [-2.5, -13],
                      'ws2': [-13, -7],
                      'ws3': [-13, 5],
                      'ws4': [-2.5, 11.5],
                      'ws5': [8.5, 5],
                      'ws6': [8.5, -7]}
      focal_plane: '/so/home/msilvafe/shared_files/sat_hw_positions.npz'
      # for LAT:
      wafer_offsets: {'c1_ws0': [-0.6, 0.0],
                      'c1_ws1': [-0.0, 0.3],
                      'c1_ws2': [-0.0, -0.3],
                      'i1_ws0': [-2.1, -0.9],
                      'i1_ws1': [-1.6, -0.6],
                      'i1_ws2': [-1.6, -1.2],
                      'i3_ws0': [1.0, -0.9],
                      'i3_ws1': [1.5, -0.6],
                      'i3_ws2': [1.5, -1.2],
                      'i4_ws0': [1.0, 0.9],
                      'i4_ws1': [1.5, 1.2],
                      'i4_ws2': [1.5, 0.6],
                      'i5_ws0': [-0.6, 1.8],
                      'i5_ws1': [-0.0, 2.1],
                      'i5_ws2': [-0.0, 1.5],
                      'i6_ws0': [-2.1, 0.9],
                      'i6_ws1': [-1.6, 1.2],
                      'i6_ws2': [-1.6, 0.6]}
      focal_plane: '/so/home/dnguyen/repos/scripts/lat_hw_positions.npz'

get_sso(aman, sso, nstep=100)

Function for getting xi, eta position of given sso.

Parameters:

aman (AxisManager) – Input axis manager.
sso (str) – Name of input sso.
nstep (int) – Number of steps to downsample the TOD.

Returns:

xi (array) – Array of xi positions.
eta (array) – Array of eta positions.

T-to-P Leakage

`name:`	Class	What it does
`estimate_t2p`	`EstimateT2P`	Estimate T to P leakage coefficients.
`subtract_t2p`	`SubtractT2P`	Subtract T to P leakage.

class sotodlib.preprocess.processes.EstimateT2P(step_cfgs)[source]

Estimate T to P leakage coefficients.

Saves results in proc_aman under the “t2p” field.

Example config block:

- name : "estimate_t2p"
  fit_in_freq : False
  calc:
    T_sig_name: 'dsT'
    Q_sig_name: 'demodQ'
    U_sig_name: 'demodU'
    joint_fit: True
    trim_samps: 2000
    lpf_cfgs:
      type: 'sine2'
      cutoff: 0.5
      trans_width: 0.1
    flag_name: 'exclude' # a field in aman.flags can combine with union_flags.
  save: True

get_t2p_coeffs(aman, T_sig_name='dsT', Q_sig_name='demodQ', U_sig_name='demodU', joint_fit=True, wn_demod=None, f_lpf_cutoff=2.0, flag_name=None, ds_factor=100, subtract_sig=False, merge_stats=True, t2p_stats_name='t2p_stats')

Compute the leakage coefficients from temperature (T) to polarization (Q and U) by either a joint fit of both or individually. Optionally subtract this leakage. Return an axismanager of the coefficients with their statistical uncertainties and reduced chi-squared values for the fit.

Parameters:

aman (AxisManager) – AxisManager object containing the TOD data.
T_sig_name (str) – Name of the temperature signal in aman. Default is ‘dsT’.
Q_sig_name (str) – Name of the Q polarization signal in aman. Default is ‘demodQ’.
U_sig_name (str) – Name of the U polarization signal in aman. Default is ‘demodU’.
joint_fit (bool) – Whether to fit Q and U leakage coefficients as parameters in a single model or fit independently. Default is True.
wn_demod (float or str or None) – Precomputed white noise level for demodulated signals. If None, it will be calculated. If provided by a string, aman.get(wn_demod) is used.
f_lpf_cutoff (float) – Cutoff frequency of low pass filter in demodulation. Used for error bar estimation by combination with wn_demod. Default is 2.0.
flag_name (str) – Name of the flag field in aman to use for masking data. If None, no masking is applied.
ds_factor (float or None) – Factor by which to downsample the TODs prior to fitting. If None, the low pass filter frequency is used to estimate the factor.
subtract_sig (bool) – Whether to subtract the calculated leakage from the polarization signals. Default is False.
merge_stats (bool) – Whether to merge the calculated statistics back into aman. Default is True.
t2p_stats_name (str) – Name under which to wrap the output AxisManager containing statistics. Default is ‘t2p_stats’.

Returns:

out_aman – An AxisManager containing leakage coefficients, their errors, and reduced chi-squared statistics.

Return type:

AxisManager

class sotodlib.preprocess.processes.SubtractT2P(step_cfgs)[source]

Subtract T to P leakage.

Example config block:

- name : "subtract_t2p"
  process: {}

subtract_t2p(aman, t2p_aman, T_signal=None)

Subtract T to P leakage.

Parameters:

aman (AxisManager) – The tod.
t2p_aman (AxisManager) – Axis manager with Q and U leakage coeffients. If joint fitting was used in get_t2p_coeffs, Q coeffs are in fields lamQ and AQ and U coeffs are in lamU and AU. Otherwise Q coeff is in field coeffsQ and U coeff in coeffsU.
T_signal (array) – Temperature signal to scale and subtract from Q/U. Default is aman['dsT'].

Scan / Turnaround Flags

`name:`	Class	What it does
`flag_turnarounds`	`FlagTurnarounds`	From the Azimuth encoder data, flag turnarounds, left-going, and right-going.
`noisy_subscan_flags`	`BadSubscanFlags`	Identifies and flags bad subscans (statistics of the subscan non-gaussian, e.g., high kurtosis).

class sotodlib.preprocess.processes.FlagTurnarounds(step_cfgs)[source]

From the Azimuth encoder data, flag turnarounds, left-going, and right-going.: All process configs go to get_turnaround_flags. If the method key is not included in the preprocess config file calc configs then it will default to ‘scanspeed’.

Saves results in proc_aman under the “turnaround_flags” field, with sub-fields turnarounds, left_scan, and right_scan.

The example block below includes optional arguments such as t_buffer, az_throw_threshold, and a min_ta. The az_throw_threshold and min_ta (minimum number of turnarounds) values as shown would cut stare observations.

Example config block:

- name: "flag_turnarounds"
  skip_on_sim: False
  process:
    method: "scanspeed"
    t_buffer: 4.
    az_throw_threshold: 1.
  calc:
    method: "scanspeed"
    t_buffer: 4.
    az_throw_threshold: 1.
  save: True
  select:
    min_ta: 1.

get_turnaround_flags(aman, az=None, method='scanspeed', name='turnarounds', merge=True, merge_lr=True, overwrite=True, t_buffer=2.0, az_buffer=None, smooth_seconds=0.5, kernel_size=400, peak_threshold=0.1, rel_distance_peaks=0.3, truncate=False, qlim=1, merge_subscans=True, turnarounds_in_subscan=False, az_throw_threshold=0.0)

Compute turnaround flags for a dataset.

Parameters:

aman (AxisManager) – Input axis manager.
az (Array) – (Optional). Azimuth data for turnaround flag computation. If not provided, it uses aman.boresight.az.
method (str) – (Optional). The method for computing turnaround flags. Options are az or scanspeed.
name (str) – (Optional). The name of the turnaround flag in aman.flags. Default is turnarounds
merge (bool) – (Optional). Merge the computed turnaround flags into aman.flags if True.
merge_lr (bool) – (Optional). Merge left and right scan flags as aman.flags.left_scan and aman.flags.right_scan if True.
overwrite (bool) – (Optional). Overwrite an existing flag in aman.flags with the same name.
t_buffer (None or float or tuple (float, float)) – (Optional). Buffer time (in seconds) for flagging turnarounds in the scanspeed method. If a single float is provided, half of the value is applied to each side (before and after) of the turnarounds. If a tuple (before, after) is provided, each value is applied to the corresponding side.
az_buffer (None or float or tuple (float, float)) – (Optional). Buffer angle (in degree) for flagging turnarounds. If a single float is provided, half of the value is applied to each side (before and after) of the turnarounds. If a tuple (before, after) is provided, each value is applied to the corresponding side.
smooth_seconds (float) – Time window in seconds to smooth the azimuth data before differentiating, in the az method.
kernel_size (int) – (Optional). Size of the step-wise matched filter kernel used in scanspeed method.
peak_threshold (float) – (Optional). Peak threshold for identifying peaks in the matched filter response. It is a value used to determine the minimum peak height in the signal.
rel_distance_peaks (float) – (Optional). Relative distance between peaks. It specifies the minimum distance between peaks as a fraction of the approximate number of samples in one scan period.
truncate (bool) – (Optional). Truncate unstable scan segments if True in scanspeed method.
qlim (float) – (Optional). Azimuth threshold percentile for az method turnaround detection.
merge_subscans (bool) – (Optional). Also merge an AxisManager with subscan information.
turnarounds_in_subscan (bool) – (Optional). Turnarounds are included as part of a subscan.
az_throw_threshold (float) – (Optional). Minimum azimuth throw (deg) required to attempt turnaround flagging. Returns no turnarounds if below threshold.

Returns:

Ranges – The turnaround flags as a Ranges object.

Return type:

RangesMatrix

class sotodlib.preprocess.processes.BadSubscanFlags(step_cfgs)[source]

Identifies and flags bad subscans.

Saves results in proc_aman under the “noisy_subscan_flags” (valid subscans, dets x samps) and “noisy_dets_flags” (valid detectors, dets) fields. calc.subscan_stats must list signal names whose last character (T/Q/U) selects a prior tod_stats result: for each entry sig this looks up proc_aman[stats_name + "_" + sig[-1]], so e.g. "demodQ" selects proc_aman.tod_stats_Q. This requires tod_stats (GetStats, see above) to have already been run once per signal with a matching wrap name (tod_stats_T/tod_stats_Q/tod_stats_U for the default stats_name: tod_stats).

Example config block:

- name : "tod_stats"
  signal: "dsT"
  wrap: "tod_stats_T"
  calc:
    stat_names: ["std", "ptp"]
    split_subscans: True
  save: True

- name : "tod_stats"
  signal: "demodQ"
  wrap: "tod_stats_Q"
  calc:
    stat_names: ["median", "std", "skew", "kurtosis", "ptp"]
    split_subscans: True
  save: True

- name : "tod_stats"
  signal: "demodU"
  wrap: "tod_stats_U"
  calc:
    stat_names: ["median", "std", "skew", "kurtosis", "ptp"]
    split_subscans: True
  save: True

- name : "noisy_subscan_flags"
  stats_name: tod_stats # optional
  calc:
    subscan_stats: ["demodQ", "demodU", "dsT"]
    nstd_lim: 3.0
    ptp_lim: 0.8
    kurt_lim: 0.5
    skew_lim: 0.5
    noisy_detector_lim: 0.5
    merge: False
  save: True
  select: True

get_noisy_subscan_flags(aman, subscan_stats, nstd_lim=None, ptp_lim=None, kurt_lim=None, skew_lim=None, noisy_detector_lim=0.9, merge=False, overwrite=False, name='noisy_subscan')

Identify and flag bad subscans based on various statistical thresholds.

Parameters:

aman (AxisManager) – The tod.
subscan_stats (dict) – Dictionary containing statistical metrics for subscans. Keys should include ‘std’, ‘ptp’, ‘kurtosis’, and ‘skew’.
nstd_lim (float [optional]) – Threshold for standard deviation.
ptp_lim (float [optional]) – Threshold for peak-to-peak values in pW.
kurt_lim (float [optional]) – Threshold for kurtosis.
skew_lim (float [optional]) – Threshold for skewness.
noisy_detector_lim (float [optional]) – If > noisy_detector_lim fraction of samps are in flagged subscans then the detector is added to noisy_detector_flags.
merge (bool) – If true, merges the generated flag into aman.
overwrite (bool) – If true, write over flag. If false, don’t.
name (str) – Name of flag to add to aman.flags if merge is True.

Returns:

noisy_subscan_flags (RangesMatrix) – RangesMatrix of bad subscans. (dets, samps), True for flagged (bad) samples.
noisy_detector_flags (ndarray) – Array indicating detectors that are too noisy for more than noisy_detector_lim fraction of the obs duration. (dets,). We return ~noisy_detector_flags which is True for good detectors.

Flag Combination (Mapmaking / Splits)

`name:`	Class	What it does
`union_flags`	`UnionFlags`	Deprecated – use `combine_flags` instead. Kept only to load old process archives.
`combine_flags`	`CombineFlags`	Do the combination of relevant flags for mapping (generalizes `union_flags`).
`split_flags`	`SplitFlags`	Get flags used for map splitting/bundling.

class sotodlib.preprocess.processes.UnionFlags(step_cfgs)[source]

Do the union of relevant flags for mapping Typically you would include turnarounds, glitches, etc.

Deprecated since version Use: the more general CombineFlags instead. UnionFlags is kept only so archives built with older process configs can still be loaded; process() raises a deprecation warning when run.

Saves results for aman under the “flags.[total_flags_label]” field.

Example config block:

- name : "union_flags"
  process:
    flag_labels: ['jumps_2pi.jump_flag', 'glitches.glitch_flags', 'turnaround_flags.turnarounds']
    total_flags_label: 'glitch_flags'

class sotodlib.preprocess.processes.CombineFlags(step_cfgs)[source]

Do the combination of relevant flags for mapping

Saves results for aman under the “flags.[total_flags_label]” field.

Example config block:

- name : "combine_flags"
  process:
    flag_labels: ['glitches.glitch_flags', 'source_flags.jupiter_inv']
    total_flags_label: 'glitch_flags'
    method: 'union' # You can select a method from ['union', '+', 'intersect', '*', 'except', '-'].
    #method: ['+', '*'] # Or you can pass individual method for each flags as a list.
       # Length of list must match the length of flag_labels.
       # If a list, the first method must be '+', as if adding the first flag set to an empty flag set.
       # Operations are performed strictly from Left to Right, '*' are not performed first.

class sotodlib.preprocess.processes.SplitFlags(step_cfgs)[source]

Get flags used for map splitting/bundling.

Saves results in proc_aman under the “split_flags” field.

Example config block:

- name : "split_flags"
  calc:
    high_gain: 0.115
    high_noise: 3.5e-5
    high_tau: 1.5e-3
    det_A: A
    pol_angle: 35
    crossover: BL
    high_leakage: 1.0e-3
    high_2f: 1.5e-3
    right_focal_plane: 0
    top_focal_plane: 0
    central_pixels: 0.071
  save: True

get_split_flags(aman, proc_aman=None, split_cfg=None)

Function returning flags used for null splits consumed by the mapmaking and bundling codes. Fields labeled field_name_flag contain boolean masks and _avg are the mean of the numerical based split flags to be used for observation level splits.

Parameters:

aman (AxisManager) – Main axis manager containing signal.
proc_aman (AxisManager) – Preprocess axis manager, usually loaded in aman.preprocess.
split_cfg (dict) – Dictionary containing the thresholds used for cutting

Returns:

split_aman – Axis manager containing splitting flags. cuts field is a FlagManager containing the detector and subscan based splits used in the mapmaker. <split_name>_threshold fields contain the threshold used for the split. Other fields conatain info for obs-level splits.

Return type:

AxisManager

Preprocess

Single-layer vs. two-layer (“multilayer”) pipelines

Running for the first time vs. re-running against an existing archive

Preprocessing job tracking (jobdb)

Preprocessing Pipelines

Processing Scripts

Processing Util Functions

Example TOD Pipeline Configuration File

Example Planet TOD Pipeline Configuration File

Example Obs Pipeline Configuration File

Process Step Glossary

General / Utility

Detector Cuts and Flags

Glitches & Jumps

Noise & PSD

Calibration

HWP & Demodulation

Ground Pickup (AzSS)

Filtering

Pointing & Focal-Plane Geometry

Arguments:

Returns:

Arguments:

Sources & Planets

T-to-P Leakage

Scan / Turnaround Flags

Flag Combination (Mapmaking / Splits)

Preprocessing job tracking (`jobdb`)