Utilities
- class sotodlib.utils.procs_pool.get_exec_env(nprocs: int = None, priority: List[str] = ['mpi', 'process_pool'])[source]
This function sets up the execution environment for parallel processing based on the specified priority list.
Since this function returns a rank value the main function needs to go under an >>> if rank == 0
block to ensure that the code is only run by the master process.
In addition the main function needs to take the executor and as_completed_callable as arguments.
Let’s assume that the main function is called main_func and it current state is as follows: >>> def main_func(): >>> with ProcessPoolExecutor() as executor: >>> futures = [] >>> for i in range(10): >>> futures.append(executor.submit(some_func, i)) >>> for future in as_completed(futures): >>> result = future.result() >>> print(result) >>> if __name__ == “__main__”: >>> main_func()
Utilizing different executors requires to change the main function as follows: >>> def main_func(executor, as_completed_callable): >>> futures = [] >>> for i in range(10): >>> futures.append(executor.submit(some_func, i)) >>> for future in as_completed_callable(futures): >>> result = future.result() >>> print(result) >>> if __name__ == “__main__”: >>> rank, executor, as_completed_callable = get_exec_env(nprocs=4) >>> if rank == 0: >>> main_func(executor, as_completed_callable)
- Parameters:
- Returns:
int – The rank of the process that is the master process for this type of execution
Union[“MPICommExecutor”, “ProcessPoolExecutor”] – The executor that is used to run the code in parallel. This can be either an MPICommExecutor or a ProcessPoolExecutor depending on the execution environment.
Callable – The as_completed function that is used to get the results of the parallel execution based on the type of the executor
- Raises:
ValueError – When a valid executor is not available based on the execution priority list.