DetMatch
The sotodlib.coords.det_match module allows us to map resonators from one
source to another, using information such as resonator frequency, bias-line
assignments, and pointing information. This is particularly useful to create
a map from resonators in a SMuRF tune-file to real detectors from either a
design-file, or a handmade solutions file based on a separate tune-file.
This works by translating the detector matching problem into an instance
of the well-studied assignment problem, in which, given a bipartite graph and
edge weights, one can efficiently find the minimum-cost matching.
Here, the two sets of the bipartite graph are the two resonator sets, with
additional nodes added to represent the possibility of a resonator being
unmatched. The edge weights computed using a cost-function that uses
resonator properties to determine resonator-to-resonator costs, and
resonator-to-unmatched costs. The function
scipy.optimize.linear_sum_assignment is then used to find the minimum-cost
match.
Usage
Below is an example of how to match a resonator-set based on a smurf tune-file and sodetlib bgmap to a resonator-set based on a handmade solution file, and save the output.
from sotodlib.coords import det_match as dm
tunefile = <path_to_tune_file>
bgmap_file = <path_to_bgmap_file>
sol_file = <path_to_solution_file>
src = dm.ResSet.from_tunefile(
tune_file, north_is_highband=False, bgmap_file=bgmap_file,
name=f'SAT Tuning'
)
dst = dm.ResSet.from_solutions(
sol_file, north_is_highband=True, name='solution'
)
match = dm.Match(src, dst)
match.save('match.h5')
To validate, you can check match.stats to see information such as how many
resonators in each set were matched or left unassigned, and to see how many
matches had bias-line mismaps, etc. If the majority of resonators have bias-line
mismaps, then it is likely the north_is_highband flag was set incorrectly,
and you are attempting to match opposite sides of the UFM.
To validate, you can also use the dm.plot_match_freqs to view how well the
resonators match in frequency space. With a proper solution file, this should
be rather well, as is seen below
Tuning Matching Params
Depending on what data you have available, you may want to tune the matching algorithm and the cost function. For instance, once we have accurate pointing data we’ll want to rely less solely on the frequency pairings, and more on pointing information. We can do this by passing in a MatchParams object to set various parameters. For instance, below is an example of how to tell the match function to be stricter with the pointing cost penalties, and more lenient with the frequency penalty:
from sotodlib.io.coords import det_match as dm
tunefile = <path_to_tune_file>
bgmap_file = <path_to_bgmap_file>
sol_file = <path_to_solution_file>
src = dm.ResSet.from_tunefile(
tune_file, north_is_highband=False, bgmap_file=bgmap_file,
name=f'SAT Tuning'
)
dst = dm.ResSet.from_solutions(
sol_file, north_is_highband=True, name='solution'
)
mpars = dm.MatchParams(
freq_width=5 #MHz
dist_width=np.deg2rad(0.3), # radians
)
match = dm.Match(src, dst, match_params=mpars)
Det Match Solutions
The det_match_solutions script can be used to generate “handmade” detector match
solutions sets for all wafers. A solution set is defined as a mapping from a tune file
with the addition of pointing information from fits to a point source in that observation to
the design wafer information (i.e. matching tune readout IDs to design detector IDs with
xi and eta constraints). Solutions are useful due to the potentially alrge frequency shifts
between the tunesets and design frequencies.It is performs multiple matches
sequentially while correcting for frequency and pointing offsets between them.
The major steps in this script are:
Load pointing xi and eta information from fits to observations of point sources. These are derived by fitting TODs or maps of observations targeting point sources (planets or the Moon) and are stored as a structured array in an
hdf5file under a group namedfocal_planeand should include entries for all det_ids from the matching tune (NaNs are allowed). It should also include an estimate of the coefficient of determination, R2 for excluding bad fits. Multiple pointing files may be input in which case they will a match will be performed and the median xi and eta values will be used from all matched resonators.Do the first match for the wafer using pointing, frequency, and bias line information.
Subtract the median xi and eta offset from matched detectors. Also remove frequency offsets through box median interpolation.
Run a second match after offset correction.
Perform a grid based pointing offset given a selection radius in the config file.
Run the third match after second pointing offset correction.
API
- sotodlib.coords.det_match.map_band_chans(b1, c1, b2, c2, chans_per_band=512)[source]
Returns an index mapping of length nchans1 from one set of bands and channels to another. Note that unmapped indices are returned as -1, so this must be handled before (or after) indexing or else you’ll get weird results. :param b1: Array of length nchans1 containing the smurf band of each channel :type b1: np.ndarray :param c1: Array of length nchans1 containing the smurf channel of each channel :type c1: np.ndarray :param b2: Array of length nchans2 containing the smurf band of each channel :type b2: np.ndarray :param c2: Array of length nchans2 containing the smurf channel of each channel :type c2: np.ndarray :param chans_per_band: Lets just hope this never changes. :type chans_per_band: int
- sotodlib.coords.det_match.get_north_is_highband(bands, bgs)[source]
Checks if north is highband based on bgmapping. This will tell you if the majority of dets on the north side of the ufm (bgs 0-5) belong to highband (bands 4-7).
- class sotodlib.coords.det_match.PointingConfig(fp_file: str, wafer_slot: str, tel_type: str, zemax_path: str | None = None, roll: float | None = 0, tube_slot: str | int | None = None)[source]
Bases:
objectHelper class for getting pointing info from an optics model.
- Parameters:
fp_file (str) – Path to focal-plane file that is used by the optics module.
wafer_slot (str) – Wafer slot of the UFM. For example: “ws0”
tel_type (str) – Tel type for the optics model. Either “SAT” or “LAT”
zemax_path (str) – If running for a “LAT” tel_type, the path to the zemax file must be specified.
roll (float) – Rotation about the line of sight. For the LAT this is elev - 60 - corotator. For the SAT this is -1*boresight.
tube_slot (str/int) – If running for a “LAT” tel_type, the tube slot must be specified. Either the tube name as a string or the tube number as an int.
- class sotodlib.coords.det_match.Resonator(idx: int, is_north: int, res_freq: float, res_qi: float = nan, smurf_res_idx: int = -1, smurf_band: int = -1, smurf_channel: int = -1, smurf_subband: int = -1, readout_id: str = '', xi: float = nan, eta: float = nan, gamma: float = nan, bg: int = -1, det_x: float = nan, det_y: float = nan, det_row: int | None = None, det_col: int | None = None, pixel_num: int = 0, det_rhomb: str | None = None, det_pol: str = '', det_freq: int = 0, det_bandpass: str = '', det_angle_raw_deg: float = nan, det_angle_actual_deg: float = nan, det_type: str = '', det_id: str = 'NO_MATCH', is_optical: int = 1, mux_bondpad: int = 0, mux_subband: str = '', mux_band: int = -1, mux_channel: int = -1, mux_layout_pos: int = -1, matched: int = 0, match_idx: int = -1)[source]
Bases:
objectData structure to hold any resonator information.
- sotodlib.coords.det_match.apply_design_properties(smurf_res, design_res, in_place=False, apply_pointing=True)[source]
Combines two resonators into one, taking smurf-properties such as res-idx, smurf-band, and smurf-channel one, and design properties such as det position and polarization from the other.
- class sotodlib.coords.det_match.ResSet(resonances: List[Resonator], name=None)[source]
Bases:
objectClass to hold a group of resonances. This provides easy interfaces for accessing Resonance fields as np arrays for fast computations and provides initialization functions from different data sources.
- classmethod from_array(arr, name=None, ignore_extra_fields=True)[source]
Creates a ResSet from a numpy structured array (resulting from
as_arraymethod).- Parameters:
- classmethod from_aman(aman, stream_id, det_cal=None, name=None, pointing: AxisManager | None = None)[source]
Load a resonator set from a Context object based on an obs_id
- Parameters:
aman (AxisManager) – Axis manager containing metadata
stream_id (str) – Stream id for ResSet to load
det_cal (AxisManager) – Detector calibration metadata. If not specified, will default to
aman.det_calpointing (Optional[AxisManager]) – AxisManager containing pointing metadata. If set, this should be an AxisManager containing the fields
xiandeta, and resonator pointing information will be added from here.
- classmethod from_tunefile(tunefile, name=None, north_is_highband=True, resfit_file=None, bgmap_file=None)[source]
Creates an instance based on a smurf-tune file. If a resfit or bgmap file is included, that data will be added to the Resonance objects as well.
- classmethod from_wafer_info_file(wafer_info_file, array_name, name=None, pt_cfg: PointingConfig | None = None)[source]
Initialize a ResSet from a wafer info file. This is a file that contains detector design information.
- Parameters:
wafer_info_file (str) – Path to wafer info file
array_name (str) – Array name, which is the key in the wafer-info-file. For example: “mv7”.
name (str) – Name to assign to the ResSet
pt_cfg (PointingConfig) – If set, this will be used to get pointing info based on the optics model. If not set, pointing info will not be included in the ResSet.
- classmethod from_solutions(sol_file, north_is_highband=True, name=None, fp_pars=None, platform='SAT', zemax_path=None)[source]
Creates an instance from an input-solution file. This will include both design data, along with smurf-band and smurf-channel info. Resonance frequencies used here are the VNA freqs measured by Kaiwen.
- Parameters:
sol_file (str) – Path to solutions file
name (str) – Name to label this ResSet
north_is_highband (bool) – True if the north-side of the array corresponds to bands 4-8
fp_pars (dict) – Result of the function
sotododlib.coords.optics.get_ufm_to_fp_pars. If this is None, detector positions will not be mapped to pointing angles.platform (str) – ‘SAT’ or ‘LAT’. Used to determine which focal plane function to use for pointing
zemax_path (str) – zemax path, required to get pointing for LAT optics
- class sotodlib.coords.det_match.MatchParams(unassigned_slots: int = 1000, freq_offset_mhz: float = 0.0, freq_width: float = 2.0, dist_width: float = 0.01, unmatched_good_res_pen: float = 10.0, good_res_qi_thresh: float = 100000.0, enforce_pointing_reqs: bool = False, allow_unassigned_to_assigned: bool = True, assigned_bg_unmatched_pen: float = 100000, unassigned_bg_unmatched_pen: float = 10000, assigned_bg_mismatch_pen: float = 100000, unassigned_bg_mismatch_pen: float = 1)[source]
Bases:
objectAny constants / hardcoded values go here to be fiddled with
- Parameters:
unassigned_slots (int) – Number of additional “unassigned” node to use per-side
freq_offset_mhz (float) – constant offset between resonator-set frequencies to use in matching.
freq_width (float) – width of exponential to use in the frequency cost function (MHz).
dist_width (float) – width of exponential to use in the pointing cost function (rad)
unmatched_good_res_pen (float) – penalty to apply to leaving a resonator with a good qi unassigned
good_res_qi_thresh (float) – qi threshold that is considered “good”
enforce_pointing_reqs (bool) –
If this is enabled, it will enforce the following requirements when matching:
Resonators with OPTC det_type must have pointing data.
Resonators with UNRT, SQID, or BARE det_types must _not_ have pointing data.
Resonators with DARK or SLOT det_types may or may not have pointing data.
allow_unassigned_to_assigned (bool) – Allow resonators from dst that do not have an assigned bias group to be matched to one in src that has an assigned bias group.
assigned_bg_unmatched_pen (float) – Penalty to apply to leaving a resonator with an assigned bg unmatched
unassigned_bg_unmatched_pen (float) – Penalty to apply to leaving a resonator with an unassigned bg unmatched
assigned_bg_mismatch_pen (float) – Penalty to apply for matching a resonator with an assigned bias line to another one with a mis-matched bias line.
unassigned_bg_mismatch_pen (float) – Penalty to apply for matching a resonator with no bias line to another one with a mis-matched bias line.
- class sotodlib.coords.det_match.MatchingStats(unmatched_src: int = 0, unmatched_dst: int = 0, unmatched_src_with_pointing: int = 0, matched_chans: int = 0, mismatched_bg: int = 0, freq_diff_avg: float = 0.0, freq_err_avg: float = 0.0, pointing_err_avg: float = 0.0)[source]
Bases:
object
- class sotodlib.coords.det_match.Match(src: ResSet, dst: ResSet, match_pars: MatchParams | None = None, apply_dst_pointing=True)[source]
Bases:
objectClass for performing a Resonance Matching between two sets of resonators, labeled src and dst. In the matching algorithm there is basically no difference between src and dst res-sets, except:
When merged, smurf-data such as band, channel, and res-idx will be taken from the
srcres-set, while detector information will be taken from thedstset.
- Parameters:
src (ResSet) – The source resonator set
dst (ResSet) – The dest resonator set
match_pars (MatchParams) – MatchParams object used in the matching algorithm. This can be used to tune the cost-function and matching.
apply_dst_pointing (bool) – If True, the
mergedres-set will take its pointing information fromdstinstead ofsrc.
- match_pars
MatchParams object used in the matching algorithm. This can be used to tune the cost-function and matching.
- Type:
- matching
A 2xN array of indices, where the first row corresponds to the indices of the src resonators, and the second row corresponds to the indices of the dst resonators. If the index is larger than the size of the corresponding res-set, that means the paired resonator was not matched.
- Type:
np.ndarray
- merged
A ResSet containing the merged resonators. This is created by applying the “design” properties of the dst resonators to the source resonators. If a source resonator is not matched to any dest resonator, it will be copied as-is.
- Type:
- stats
A MatchingStats object containing some statistics about the matching.
- Type:
- get_match_iter(include_unmatched=True) Iterator[Tuple[Resonator | None, Resonator | None]][source]
Returns an iterator over matched resonators (r1, r2).
- Parameters:
include_unmatched (bool) – If True, will include unmatched resonators, with the pair set to None.
- get_stats() MatchingStats[source]
Gets stats associated with current matching.
- sotodlib.coords.det_match.plot_match_freqs(m: Match, is_north=True, show_offset=False, xlim=None)[source]
Plots src and dst ResSet freqs in a match, along with their matching assignments.
- class sotodlib.coords.det_match_solutions.SolutionsCfg(ctx_path: str, pointing_results_dir: str, results_dir: str, wafer_info_path: str, tel_type: str, base_obs_id: str | None = None, zemax_path: str | None = None, apply_roll: bool = True, pointing_field: str = 'tod_pointing', site_pipeline_cfg_dir: str = '$SITE_PIPELINE_CONFIG_DIR', finite_xi_thresh: int = 500, min_r2: float = 0.9, sel_rad: float = 2.0, unassigned_slots: int = 1200, wafer_map_path: str | None = None, match_pars: ~typing.Dict[str, dict] = <factory>, initial_pointing_offset: ~typing.Tuple[float, float] = (0, 0), ufm_to_fp_path: str | None = None, freq_correct_by_muxband: bool = True)[source]
Bases:
object- Parameters:
ctx_path (str) – Path to context file to use to pull tod metadata.
pointing_results_dir (str) – Results to directory that contains pointing results. Files in directory should look like:
` focal_plane_<obs_id>_<wafer_slot>.hdf `results_dir (str) – Directory where results should be stored.
wafer_info_path (str) – Path to the wafer_info h5 file.
tel_type (str) – Tel type for the optics model. Either “SAT” or “LAT”
base_obs_id (str) – Obs_id to use as a base for matching when merging multiple pointing obs_ids for a wafer. Will default to the pointing obs_id with the greatest number of detectors above the min_R2 threshold.
zemax_path (str) – If running for a “LAT” tel_type, the path to the zemax file must be specified.
apply_roll (bool) – Whether or not to apply the obs_id roll angle. Some pointing sets may already be corrected for roll angle.
pointing_field (str) – Name of sub axis manager in pointing tune axis maanger containng the pointing information.
site_pipeline_cfg_dir (str) – Path to site-pipeline-config dir. Defaults to the env var
$SITE_PIPELINE_CONFIG_DIR.finite_xi_thresh (int) – Minimum number of dets a pointing result must have to add it to the analysis.
min_r2 (float) – Minimum R-squared for det pointing to be considered.
sel_rad (float) – Selection radius for grid-based interpolation pointing offset subtraction.
unassigned_slots (int:) – Number of additional “unassigned” node to use per-side
wafer_map_path (str) – Path to the wafer map file. Defaults to
<site-pipeline-config>/shared/detmatpping/wafer_map.yaml.match_pars (dict) –
Dictionary of match parameters to use for pointing obs_id merging and each match iteration. Should have the form:
- match_pars:
- pointing:
freq_width: 0.4 dist_width: 2.0
- match0:
freq_width: 200 dist_width: 0.4
- match1:
freq_width: 50 dist_width: 0.8
- match2:
freq_width: 5 dist_width: 0.1
offset (Initial pointing) – Estimated pointing offset for the boresight. This should be (xi_offset, eta_offset) where both are in radians.
ufm_to_fp_path (str) – Path to file that maps wafer_slot to position on focal plane.
freq_correct_by_muxband (bool) – If true, apply the same freq offset correction to all resonators in a mux-band.
- classmethod from_yaml(path: str) SolutionsCfg[source]
- class sotodlib.coords.det_match_solutions.PointingInfo(pointing: numpy.ndarray, obs_id: str, obs: dict, meta: sotodlib.core.axisman.AxisManager, preprocessed: bool = False)[source]
Bases:
object- pointing: ndarray
- meta: AxisManager
- sotodlib.coords.det_match_solutions.get_meta(cfg: SolutionsCfg, obs_id: str, wafer_slot: str | None = None)[source]
- sotodlib.coords.det_match_solutions.load_good_pointing_info(cfg: SolutionsCfg, wafer_slot: str) List[PointingInfo][source]
Load pointing data for each pointing measurement on disk
- sotodlib.coords.det_match_solutions.pointing_preprocess(cfg: SolutionsCfg, pinfo: PointingInfo)[source]
Add tod_pointing to PointingInfo metadata, adjusting for boresight angle and pointing offset.
- sotodlib.coords.det_match_solutions.merge_pointing_info(cfg: SolutionsCfg, pinfos: List[PointingInfo], base_idx=0)[source]
Combine all pointing measurements into a single resonator set, with the median pointing info from all. This requires a base_idx to be specified, which will be the index of the PointingInfo to use to create the ResSet template. For all other PointingInfo objects, resonators will be matched to the base resset based on resonance frequency, to compile all pointing measurements for a given detector. The median of all measurements will be used as the real value.
- sotodlib.coords.det_match_solutions.get_best_tod_pointing(cfg: SolutionsCfg, pinfos: List[PointingInfo]) AxisManager[source]
- class sotodlib.coords.det_match_solutions.MatchSolution(match: sotodlib.coords.det_match.Match, am: sotodlib.core.axisman.AxisManager, match_iterations: List[sotodlib.coords.det_match.Match] = <factory>)[source]
Bases:
object- am: AxisManager
- sotodlib.coords.det_match_solutions.get_pt_offset_interp(match, sel_rad=0.03490658503988659) Tuple[Any, Any][source]
- sotodlib.coords.det_match_solutions.get_foffset_interp(match, is_north, box_size=50, box_step=25) Callable[[float], float][source]
- class sotodlib.coords.det_match_solutions.MatchSolutionResult(results: Dict[str, sotodlib.coords.det_match_solutions.MatchSolution | None], am: sotodlib.core.axisman.AxisManager | None = None, traceback: str | None = None)[source]
Bases:
object- results: Dict[str, MatchSolution | None]
- am: AxisManager | None = None
- sotodlib.coords.det_match_solutions.match_wafer(cfg: SolutionsCfg, am: AxisManager, stream_id: str, meas_rset: ResSet | None) MatchSolution[source]
Create a match solution for a given wafer slot.
- Parameters:
cfg (SolutionsCfg) – Configuration object
am (AxisManager) – Axis manager containing detector info about relevant wafer slot, along with measured pointing data.
stream_id (str) – Stream Id of the wafer
- class sotodlib.coords.det_match_solutions.FullWaferSolution(match_solution: sotodlib.coords.det_match_solutions.MatchSolution, pointing_results: List[sotodlib.coords.det_match_solutions.PointingInfo], meta: sotodlib.core.axisman.AxisManager, stream_id: str)[source]
Bases:
object- match_solution: MatchSolution
- pointing_results: List[PointingInfo]
- meta: AxisManager
- sotodlib.coords.det_match_solutions.save_wafer_solution(cfg: SolutionsCfg, solution: FullWaferSolution)[source]
- sotodlib.coords.det_match_solutions.get_wafer_solution(cfg: SolutionsCfg, wafer_slot: str, save=False) FullWaferSolution | None[source]