Python Interface for MVM Processing Code

The hapmultisequencer module is part of the drizzlepac package and calls utilities which are intended to be used with Multi-visit Mosaic (MVM) data and to be imported into other Python programs or a Python session for interactive data analysis.

hapmultisequencer

This module provides the primary user interface for performing MVM processing.

This script defines the HST Advanced Products (HAP) Multi-visit (MVM) generation portion of the calibration pipeline. This portion of the pipeline produces mosaic products. This script provides similar functionality as compared to the Hubble Legacy Archive (HLA) pipeline in that it provides the overall sequence of the processing.

Note regarding logging… During instantiation of the log, the logging level is set to NOTSET which essentially means all message levels (debug, info, etc.) will be passed from the logger to the underlying handlers, and the handlers will dispatch all messages to the associated streams. When the command line option of setting the logging level is invoked, the logger basically filters which messages are passed on to the handlers according the level chosen. The logger is acting as a gate on the messages which are allowed to be passed to the handlers.

Some environment variables can be used to specify whether or not to include some types of data when creating MVM products. By default, the code will generate SkyCell layers from all data specified by the user. These variables allow the user to ignore types of data from the input during MVM processing.

MVM_INCLUDE_SMALL : Generate MVM SkyCell layers from ACS/HRC and ACS/SBC data. These exposures typically only cover a miniscule fraction of a SkyCell and the plate scale of the SkyCell would result in a degraded representation of the original ACS/HRC and ACS/SBC data. Therefore, this variable can be set to ‘off’ or ‘false’ to turn off generation of layers from ACS/HRC and ACS/SBC data.
MVM_ONLY_CTE : Generate MVM SkyCell layers using only CTE-corrected input files. If ‘off’ or ‘false’, then both CTE-corrected and non-CTE-corrected data may end up in the same SkyCell layer (image).

The output products can be evaluated to determine the quality of the alignment and output data through the use of the environment variable:

MVM_QUALITY_TESTING : Turn on quality assessment processing. This environment variable, if found with an affirmative value, will turn on processing to generate a JSON file which contains the results of evaluating the quality of the generated products.

drizzlepac.hapmultisequencer.run_mvm_processing(input_filename, skip_gaia_alignment=True, diagnostic_mode=False, use_defaults_configs=True, input_custom_pars_file=None, output_custom_pars_file=None, phot_mode='both', custom_limits=None, output_file_prefix=None, log_level=20)[source]

Run the HST Advanced Products (HAP) generation code. This routine is the sequencer or controller which invokes the high-level functionality to process the multi-visit data.

Parameters:

input_filename: string: The ‘poller file’ where each line contains information regarding an exposures considered part of the multi-visit.
skip_gaia_alignmentbool, optional: Skip alignment of all input images to known Gaia/HSC sources in the input image footprint? If set to ‘True’, the existing input image alignment solution will be used instead. The default is False.
diagnostic_modebool, optional: Allows printing of additional diagnostic information to the log. Also, can turn on creation and use of pickled information.
use_defaults_configs: bool, optional: If True, use the configuration parameters in the ‘default’ portion of the configuration JSON files. If False, use the configuration parameters in the “parameters” portion of the file. The default is True.
input_custom_pars_file: string, optional: Represents a fully specified input filename of a configuration JSON file which has been customized for specialized processing. This file should contain ALL the input parameters necessary for processing. If there is a filename present for this parameter, the ‘use_defaults_configs’ parameter is ignored. The default is None.
output_custom_pars_file: string, optional: Fully specified output filename which contains all the configuration parameters available during the processing session. The default is None.
phot_modestr, optional: Which algorithm should be used to generate the sourcelists? ‘aperture’ for aperture (point) photometry; ‘segment’ for isophotal photometry; ‘both’ for both ‘segment’ and ‘aperture’. Default value is ‘both’.
custom_limitslist, optional: 4-element list containing the mosaic bounding rectangle X min and max and Y min and max values for custom mosaics
output_file_prefixstr, optional: ‘Text string that will be used as the filename prefix all files created by hapmultisequencer.py during the MVM custom mosaic generation process. If not explicitly specified, all output files will start with the following formatted text string: “hst-skycell-p<pppp>-ra<##>d<####>-dec<n|s><##>d<####>”, where p<pppp> is the projection cell ID, ra<##>d<####> are the whole-number and decimal portions of the right ascention, respectively, and dec<n|s><##>d<####> are the whole-number and decimal portions of the declination, respectively. Note that the “<n|s>” denotes if the declination is north (positive) or south (negative). Example: For skycell = 1974, ra = 201.9512, and dec = +26.0012, The filename prefix would be “skycell-p1974-ra201d9512-decn26d0012”.
log_levelint, optional: The desired level of verboseness in the log statements displayed on the screen and written to the .log file. Default value is 20, or ‘info’.

Returns:

return_value: integer: A return exit code used by the calling Condor/OWL workflow code: 0 (zero) for success, 1 for error

haputils.cell_utils

Module for defining ProjectionCells and SkyCells for a set of exposures.

drizzlepac.haputils.cell_utils.get_sky_cells(visit_input, input_path=None, scale=None, cell_size=None, diagnostic_mode=False)[source]

Return all sky cells that overlap the exposures in the input.

Parameters:

visit_inputstr or list: Input specifying the exposures from a single visit; either a poller output file or a simple list of exposure filenames. Exposures in an input list are assumed to be in the current working directory when running the code, unless input_path has been provided which points to the location of the exposures to be processed.
input_pathstr, optional: Location of input exposures, if provided. If not provided, location will be assumed to be the current working directory.
scalefloat, optional: User-defined value for the pixel scale, in arcseconds/pixel, of the sky cells and projection cells. If None, default value from grid definition file will be used.
cell_sizefloat, optional: User-specified size, in degrees, for each projection cell. If None, default value from grid definition file will be used.
diagnostic_modebool, optional: Parameter which specifies whether or not to generate additional output useful for debugging internal computations.

Returns:

sky_cellslist of objects: List of SkyCell objects for all sky cells which overlap the exposures provided in visit_input.

drizzlepac.haputils.cell_utils.interpret_scells(sky_cells)[source]

Return dict of filenames each with the skycell name they overlap

Parameters:

sky_cellsdict: Dictionary of sky-cell objects from get_sky_cells

Returns:

sky_cell_filesdict: Dictionary of ALL sky-cell IDs as a ‘;’-delimited string for each exposure(sky cell member), with exposure filenames as keys.

class drizzlepac.haputils.cell_utils.ProjectionCell(index=None, band=None, scale=None, nxy=None, overlap=None)[source]

Class which defines a projection cell product based on the position on the sky using the PanSTARRs-derived definitions.

This class has the following members for defining the projection cell:

wcs : astropy.wcs.WCS object defining the WCS for the cell

footprint : footprint of the projection cell on the sky from wcs.calc_footprint()

corners : (RA, DEC) of the corners of the projection cell on the sky.

Build projection cell for cell with name skycell_NNNNN

If band is not specified, it will open the grid definitions file to obtain the band definition for the cell with the specified index.

Parameters:

indexint: ProjectionCell index on the sky. This index is 0-based. It either represents the index in a single declination band of ProjectionCells or the ID number of the ProjectionCell across the entire sky (from 0 to 2643, given the default GridDefs table).
bandlist, optional: Definition of spacing of projection cells along a line of constant declination. These defintions, if not provided, are extracted from the projection cell FITS table.
scalefloat, optional: Plate scale to use for defining the projection cell WCS. If not specified here, will use values from the projection cell FITS table ‘PC_SCALE’ header keyword.
nxyint, optional: Number of sky cells in X and Y to use in subdividing the projection cell. If not specified here, will use values from the projection cell FITS table ‘SC_NXY’ header keyword.
overlapint, optional: Number of pixels of overlap between sky cells. If not specified here, will use values from the projection cell FITS table ‘SC_OLAP’ header keyword.

class drizzlepac.haputils.cell_utils.SkyCell(projection_cell=None, x=None, y=None, scale='fine')[source]

Definition of the sky cell within a ProjectionCell object.

This class defines the following attributes of a sky cell:

sky_cell_id : string representation of the sky cell ID

projection_cell : name of projection cell this sky cell belongs to

wcsastropy.wcs.WCS definition of the WCS of this sky cell as a
subarray within the tangent plane defined by the projection cell’s WCS

corners : (RA, Dec) array of the edges of the exposures within the sky cell

mask : polygon representation exposures within the sky cell

Define sky cell at position x,y within projection cell.

Parameters:

projection_cellobject, optional: ProjectionCell instance which this SkyCell will be based upon.
xint, optional: X position (1-based) of this SkyCell within the ProjectionCell
yint, optional: Y position (1-based) of this SkyCell within the ProjectionCell
scalestr or float, optional: Plate scale to be used to define the SkyCell WCS. The strings ‘fine’ or ‘coarse’ can be used to refer to default plate scales. Default values are specified in the dict cell_utils.SUPPORTED_SCALES. Alternatively, floating-point values can be provided to specify the exact pixel size in arcseconds/pixel should be used.

Examples

The SkyCell object can be initialized in one of 2 ways.

Using an already existing SkyCell name with:

>>> skycell = SkyCell.from_name('skycell-p0197x18y16')

Using separate projection cell ID, along with X and Y indices with:

>>> skycell = SkyCell(projection_cell=ProjectionCell(index=197), x=18, y=16)

haputils.make_poller_files

Module for creating CSV-formatted input ‘poller’ files for MVM processing.

Generates a poller file that will be used as input to runsinglehap.py, hapsequencer.py, runmultihap.py or hapmultisequencer.py based on the files or rootnames listed user-specified list file.

USAGE

>>> python drizzlepac/haputils/make_poller_files.py <input filename> -[ost]
- input filename: Name of a file containing a list of calibrated fits files (ending with "_flt.fits" or
  "_flc.fits") or rootnames (9 characters, usually ending with a "q" to process. The corresponding
  flc.fits or flt.fits files must exist in the user-specified path, the current working directory or the
  online cache

The ‘-o’ optional input allows users to input the name of an output poller file that will be created. If not explicitly specified, the poller file will be named “poller_file.out”.
The ‘-s’ optional input allows users to input the Name of the skycell. The correct syntax for skycell names is “skycell-pNNNNxXXyXX”, where NNNN is the 4-digit projection cell number, and XX and YY are the two-digit X and Y skycell indices, respectively. NOTE: this input argument is not needed for SVM poller file creation, but REQUIRED for MVM poller file creation. Users can determine the skycell(s) that their observations occupy using the haputils.which_skycell script.
The ‘-t’ optional input allows users to specify the type of poller file that will be created. The valid input options are “svm” to create a poller file for use with the single-visit mosaics pipeline or “mvm” to create a poller file for use with the multiple-visit mosaics pipeline. If not explicitly specified, the default value is “svm”. NOTE: if creating a MVM poller file, one must specify the skycell name using the “-s” input argument.

Python USAGE:

>>> python
>>> from drizzlepac.haputils import make_poller_files
>>> make_poller_files.generate_poller_file(input_list, poller_file_type='svm', output_poller_filename="poller_file.out", skycell_name=None):

drizzlepac.haputils.make_poller_files.generate_poller_file(input_list, poller_file_type='svm', output_poller_filename='poller_file.out', skycell_name=None)[source]

Creates a properly formatted SVM or MVM poller file.

Parameters:

input_liststr: Name of the text file containing the list of filenames or rootnames to process
poller_file_typestr, optional: Type of poller file to create. ‘svm’ for single visit mosaic, ‘mvm’ for multi-visit mosaic. Default value is ‘svm’.
output_poller_filenamestr, optional: Name of the output poller file that will be created. Default value is ‘poller_file.out’.
skycell_namestr, optional: Name of the skycell to use when creating a MVM poller file. skycell_name is REQUIRED for the creation of a MVM poller file, but completely unnecessary for the creation of a SVM poller file. The correct syntax for skycell names is ‘skycell-pNNNNxXXyXX’, where NNNN is the 4-digit projection cell number, and XX and YY are the two-digit X and Y skycell indices, respectively. Default value is logical ‘None’. NOTE: this input argument is not needed for SVM poller file creation, but REQUIRED for MVM poller file creation. Users can determine the skycell(s) that their observations occupy using the haputils.which_skycell script.

Returns:

Nothing.

haputils.poller_utils

Module for interpreting input ‘poller’ files for MVM processing.

drizzlepac.haputils.poller_utils.interpret_mvm_input(results, log_level, layer_method='all', exp_limit=2.0, user_table=None, include_small=True, only_cte=False)[source]

Parameters:

resultsstr or list: Input poller file name or Python list of dataset names to be processed for a multi-visit. Dataset names have to be either the filename of a singleton (non-associated exposure) or the name of an ASN file (e.g., jabc01010_asn.fits).
log_levelint: The desired level of verboseness in the log statements displayed on the screen and written to the .log file.
exp_limitfloat, optional: This specifies the ratio between long and short exposures that should be used to define each exposure time bin (short, med, long). If None, all exposure times will be combined into a single bin (all). Splitting of the bins gets computed using Kmeans clustering for 3 output bins.
layer_methodstr: Specify what type of interpretation should be done for the definition of each layer. Options include: ‘all’, ‘hard’, ‘kmeans’. The value ‘all’ simply puts all exposures regardless of exposure time. The value of ‘hard’ relies on pre-defined limits for ‘short’, ‘med’, and ‘long’ exposure times. The value of ‘kmeans’ relies on using ‘kmeans’ analysis for defining sub-layers based on exposure time.
user_tabledict, optional: User-supplied table generated using build_poller_table. This table may have been edited to meet the unique needs of the user for this custom processing run.
include_smallbool, optional: Specify whether or not to create products from ACS/HRC and ACS/SBC data.
only_ctebool, optional: Specify whether or not to create products only using CTE-corrected data.

Notes

Interpret the contents of the MVM poller file for the values to use in generating the names of all the expected output products.

Input will have format of:

ib4606c5q_flc.fits,11665,B46,06,1.0,F555W,UVIS,skycell-p2381x05y09,NEW,/ifs/archive/ops/hst/public/ib46/ib4606c5q/ib4606c5q_flc.fits

which are:

filename, proposal_id, program_id, obset_id, exptime, filters, detector, skycell-p<PPPP>x<XX>y<YY>, [OLD|NEW], pathname

The SkyCell ID will be included in this input information to allow for grouping of exposures into the same SkyCell layer based on filter, exptime, and year.

Output dict will have only have definitions for each defined sky cell layer to be either created or updated based on the input exposures being processed. There will be a layer defined for each unique combination of filter/exp class/year. An example would be:

obs_info_dict["layer 00": {"info": '11665 06 wfc3 uvis f555w short 2011 drc',
                           "files": ['ib4606c5q_flc.fits', 'ib4606cgq_flc.fits']}

obs_info_dict["layer 01": {"info": '11665 06 wfc3 uvis f555w med 2011 drc',
                           "files": ['ib4606c6q_flc.fits', 'ib4606cdq_flc.fits']}

obs_info_dict["layer 02": {"info": '11665 06 wfc3 uvis f555w long 2011 drc',
                           "files": ['ib4606c9q_flc.fits', 'ib4606ceq_flc.fits']}

Caution

The input (“results”) parameter was originally designed to be able to accept a Python list of pipeline product dataset names (e.g., ib4606cdq_flc.fits) for MVM proessing. However, this was later modified. MVM processing needs the SVM FLT/FLC files as input.

haputils.product

Module for defining classes for MVM processing.

drizzlepac.haputils.product.SkyCellExposure(prop_id, obset_id, instrument, detector, aperture_from_poller, filename, layer, filetype, log_level)[source]

An SkyCell Exposure Product is an individual exposure/image (flt/flc).

The “sce” is short hand for ExposureProduct.

drizzlepac.haputils.product.SkyCellProduct(prop_id, obset_id, instrument, detector, aperture_from_poller, skycell_name, layer, filetype, log_level)[source]

A SkyCell Product is a mosaic comprised of images acquired during a multiple visits with one instrument, one detector, a single filter, and all exposure times. The exposure time characteristic may change - TBD.

The “scp” is short hand for SkyCellProduct.

haputils.generate_custom_svm_mvm_param_file

Module for defining custom sets of parameters for SVM and MVM processing.

generate_custom_mvm_svm_param_file - a module to create a template SVM/MVM processing pipeline parameter file based on the observations present in the current working directory for the user to customize

Command-line USAGE:

>>> python generate_custom_svm_mvm_param_file.py [-clop] poller_filename

poller_filename: (required) the name of the input SVM/MVM poller file
-c: (optional) If turned on, existing files with the same name as the output custom SVM parameter file created by this script will be overwritten.
-l: (optional) The desired level of verboseness in the log statements displayed on the screen and written to the .log file. Valid input options are ‘critical’, ‘error’, ‘warning’, ‘info’, or ‘debug’.
-o: (optional) Name of the output configuration JSON file which will be created for specialized processing. This file will contain ALL the input parameters necessary for processing. If not explicitly specified, the default name for the output file is “custom_parameters.json”.
-p: (optional) Name of the pipeline that the configurations will be prepared for. Valid options are ‘mvm’ (for the HAP multi-visit mosaics pipeline) or ‘svm’ (for the HAP single-visit mosaic pipeline). If not explicitly stated, the default value is ‘svm’

Python USAGE:

>>> python
>>> from drizzlepac.haputils import generate_custom_svm_mvm_param_file
>>> generate_custom_svm_mvm_param_file.make_svm_input_file(input_filename,
                                                       clobber=False,
                                                       log_level=logutil.logging.INFO
                                                       output_custom_pars_file='custom_parameters.json',
                                                       hap_pipeline_name='svm')

Note

This script only generates a template input parameter file populated with default values based on the observations present in the current working directory. It is entirely incumbent on the user to modify this file with non-default values.

drizzlepac.haputils.generate_custom_svm_mvm_param_file.make_svm_input_file(input_filename, hap_pipeline_name='svm', output_custom_pars_file='custom_parameters.json', clobber=False, log_level=20)[source]

create a custom SVM processing pipeline parameter file based on the observations present in the current working directory using config_utils.HapConfig() and optionally update_ci_values() to adjust CI upper and lower limits for filter products

Parameters:

input_filename: str: The ‘poller file’ where each line contains information regarding an exposures taken during a single visit.
hap_pipeline_namestr, optional: Name of the pipeline that the configurations will be prepared for. Valid options are ‘mvm’ (for the HAP multi-visit mosaics pipeline) or ‘svm’ (for the HAP single-visit mosaic pipeline). If not explicitly stated, the default value is ‘svm’
output_custom_pars_file: str, optional: Fully specified output filename which contains all the configuration parameters available during the processing session. Default is ‘custom_parameters.json’.
clobberBool, optional: If set to Boolean ‘True’, existing files with the same name as output_custom_pars_file, the output custom SVM parameter file created by this script will be overwritten. Default value is Boolean ‘False’.
log_levelint, optional: The desired level of verboseness in the log statements displayed on the screen and written to the .log file. Default value is 20, or ‘info’.

Returns:

Nothing.