Python Interface for MVM Processing Code

The hapmultisequencer.py module is part of the drizzlepac package and calls utilities which are intended to be used with Multi-visit Mosaic (MVM) data and to be imported into other Python programs or a Python session for interactive data analysis.

hapmultisequencer

This module provides the primary user interface for performing MVM processing.

This script defines the HST Advanced Products (HAP) Multi-visit (MVM) generation portion of the calibration pipeline. This portion of the pipeline produces mosaic products. This script provides similar functionality as compared to the Hubble Legacy Archive (HLA) pipeline in that it provides the overall sequence of the processing.

Note regarding logging… During instantiation of the log, the logging level is set to NOTSET which essentially means all message levels (debug, info, etc.) will be passed from the logger to the underlying handlers, and the handlers will dispatch all messages to the associated streams. When the command line option of setting the logging level is invoked, the logger basically filters which messages are passed on to the handlers according the level chosen. The logger is acting as a gate on the messages which are allowed to be passed to the handlers.

Some environment variables can be used to specify whether or not to include some types of data when creating MVM products. By default, the code will generate SkyCell layers from all data specified by the user. These variables allow the user to ignore types of data from the input during MVM processing.

  • MVM_INCLUDE_SMALL : Generate MVM SkyCell layers from ACS/HRC and ACS/SBC data. These exposures typically only cover a miniscule fraction of a SkyCell and the plate scale of the SkyCell would result in a degraded representation of the original ACS/HRC and ACS/SBC data. Therefore, this variable can be set to ‘off’ or ‘false’ to turn off generation of layers from ACS/HRC and ACS/SBC data.

  • MVM_ONLY_CTE : Generate MVM SkyCell layers using only CTE-corrected input files. If ‘off’ or ‘false’, then both CTE-corrected and non-CTE-corrected data may end up in the same SkyCell layer (image).

The output products can be evaluated to determine the quality of the alignment and output data through the use of the environment variable:

  • MVM_QUALITY_TESTING : Turn on quality assessment processing. This environment variable, if found with an affirmative value, will turn on processing to generate a JSON file which contains the results of evaluating the quality of the generated products.

drizzlepac.hapmultisequencer.run_mvm_processing(input_filename, skip_gaia_alignment=True, diagnostic_mode=False, use_defaults_configs=True, input_custom_pars_file=None, output_custom_pars_file=None, phot_mode='both', custom_limits=None, output_file_prefix=None, log_level=20)[source]

Run the HST Advanced Products (HAP) generation code. This routine is the sequencer or controller which invokes the high-level functionality to process the multi-visit data.

Parameters
input_filename: string

The ‘poller file’ where each line contains information regarding an exposures considered part of the multi-visit.

skip_gaia_alignmentbool, optional

Skip alignment of all input images to known Gaia/HSC sources in the input image footprint? If set to ‘True’, the existing input image alignment solution will be used instead. The default is False.

diagnostic_modebool, optional

Allows printing of additional diagnostic information to the log. Also, can turn on creation and use of pickled information.

use_defaults_configs: bool, optional

If True, use the configuration parameters in the ‘default’ portion of the configuration JSON files. If False, use the configuration parameters in the “parameters” portion of the file. The default is True.

input_custom_pars_file: string, optional

Represents a fully specified input filename of a configuration JSON file which has been customized for specialized processing. This file should contain ALL the input parameters necessary for processing. If there is a filename present for this parameter, the ‘use_defaults_configs’ parameter is ignored. The default is None.

output_custom_pars_file: string, optional

Fully specified output filename which contains all the configuration parameters available during the processing session. The default is None.

phot_modestr, optional

Which algorithm should be used to generate the sourcelists? ‘aperture’ for aperture (point) photometry; ‘segment’ for isophotal photometry; ‘both’ for both ‘segment’ and ‘aperture’. Default value is ‘both’.

custom_limitslist, optional

4-element list containing the mosaic bounding rectangle X min and max and Y min and max values for custom mosaics

output_file_prefixstr, optional

‘Text string that will be used as the filename prefix all files created by hapmultisequencer.py during the MVM custom mosaic generation process. If not explicitly specified, all output files will start with the following formatted text string: “hst-skycell-p<pppp>-ra<##>d<####>-dec<n|s><##>d<####>”, where p<pppp> is the projection cell ID, ra<##>d<####> are the whole-number and decimal portions of the right ascention, respectively, and dec<n|s><##>d<####> are the whole-number and decimal portions of the declination, respectively. Note that the “<n|s>” denotes if the declination is north (positive) or south (negative). Example: For skycell = 1974, ra = 201.9512, and dec = +26.0012, The filename prefix would be “skycell-p1974-ra201d9512-decn26d0012”.

log_levelint, optional

The desired level of verboseness in the log statements displayed on the screen and written to the .log file. Default value is 20, or ‘info’.

Returns
return_value: integer

A return exit code used by the calling Condor/OWL workflow code: 0 (zero) for success, 1 for error

haputils.cell_utils

Module for defining ProjectionCells and SkyCells for a set of exposures.

drizzlepac.haputils.cell_utils.get_sky_cells(visit_input, input_path=None, scale=None, cell_size=None)[source]

Return all sky cells that overlap the exposures in the input.

Parameters
visit_inputstr or list

Input specifying the exposures from a single visit; either a poller output file or a simple list of exposure filenames. Exposures in an input list are assumed to be in the current working directory when running the code, unless input_path has been provided which points to the location of the exposures to be processed.

input_pathstr, optional

Location of input exposures, if provided. If not provided, location will be assumed to be the current working directory.

scalefloat, optional

User-defined value for the pixel scale, in arcseconds/pixel, of the sky cells and projection cells. If None, default value from grid definition file will be used.

cell_sizefloat, optional

User-specified size, in degrees, for each projection cell. If None, default value from grid definition file will be used.

Returns
sky_cellslist of objects

List of SkyCell objects for all sky cells which overlap the exposures provided in visit_input.

drizzlepac.haputils.cell_utils.interpret_scells(sky_cells)[source]

Return dict of filenames each with the skycell name they overlap

Parameters
sky_cellsdict

Dictionary of sky-cell objects from get_sky_cells

Returns
sky_cell_filesdict

Dictionary of ALL sky-cell IDs as a ‘;’-delimited string for each exposure(sky cell member), with exposure filenames as keys.

class drizzlepac.haputils.cell_utils.ProjectionCell(index=None, band=None, scale=None, nxy=None, overlap=None)[source]

Class which defines a projection cell product based on the position on the sky using the PanSTARRs-derived definitions.

This class has the following members for defining the projection cell:

  • wcs : astropy.wcs.WCS object defining the WCS for the cell

  • footprint : footprint of the projection cell on the sky from wcs.calc_footprint()

  • corners : (RA, DEC) of the corners of the projection cell on the sky.

Build projection cell for cell with name skycell_NNNNN

If band is not specified, it will open the grid definitions file to obtain the band definition for the cell with the specified index.

Parameters
indexint

ProjectionCell index on the sky

bandlist, optional

Definition of spacing of projection cells along a line of constant declination. These defintions, if not provided, are extracted from the projection cell FITS table.

scalefloat, optional

Plate scale to use for defining the projection cell WCS.

If not specified here,

will use values from the projection cell FITS table ‘PC_SCALE’ header keyword.

nxyint, optional

Number of sky cells in X and Y to use in subdividing the projection cell. If not specified here, will use values from the projection cell FITS table ‘SC_NXY’ header keyword.

overlapint, optional

Number of pixels of overlap between sky cells. If not specified here, will use values from the projection cell FITS table ‘SC_OLAP’ header keyword.

class drizzlepac.haputils.cell_utils.SkyCell(projection_cell=None, x=None, y=None, scale='fine')[source]

Definition of the sky cell within a ProjectionCell object.

This class defines the following attributes of a sky cell:

  • sky_cell_id : string representation of the sky cell ID

  • projection_cell : name of projection cell this sky cell belongs to

  • wcsastropy.wcs.WCS definition of the WCS of this sky cell as a

    subarray within the tangent plane defined by the projection cell’s WCS

  • corners : (RA, Dec) array of the edges of the exposures within the sky cell

  • mask : polygon representation exposures within the sky cell

Define sky cell at position x,y within projection cell.

Parameters
namestr, optional

Name of the sky cell in the format ‘skycell-p1234x01y01’

projection_cellobject, optional

ProjectionCell instance which this SkyCell will be based upon.

xint, optional

X position (1-based) of this SkyCell within the ProjectionCell

yint, optional

Y position (1-based) of this SkyCell within the ProjectionCell

scalestr or float, optional

Plate scale to be used to define the SkyCell WCS. The strings ‘fine’ or ‘coarse’ can be used to refer to default plate scales. Default values are specified in the dict cell_utils.SUPPORTED_SCALES. Alternatively, floating-point values can be provided to specify the exact pixel size in arcseconds/pixel should be used.

haputils.make_poller_files

Module for creating CSV-formatted input ‘poller’ files for MVM processing.

Generates a poller file that will be used as input to runsinglehap.py, hapsequencer.py, runmultihap.py or hapmultisequencer.py based on the files or rootnames listed user-specified list file.

USAGE
>>> python drizzlepac/haputils/make_poller_files.py <input filename> -[ost]
- input filename: Name of a file containing a list of calibrated fits files (ending with "_flt.fits" or
  "_flc.fits") or rootnames (9 characters, usually ending with a "q" to process. The corresponding
  flc.fits or flt.fits files must exist in the user-specified path, the current working directory or the
  online cache
  • The ‘-o’ optional input allows users to input the name of an output poller file that will be created. If not explicitly specified, the poller file will be named “poller_file.out”.

  • The ‘-s’ optional input allows users to input the Name of the skycell. The correct syntax for skycell names is “skycell-pNNNNxXXyXX”, where NNNN is the 4-digit projection cell number, and XX and YY are the two-digit X and Y skycell indices, respectively. NOTE: this input argument is not needed for SVM poller file creation, but REQUIRED for MVM poller file creation. Users can determine the skycell(s) that their observations occupy using the haputils.which_skycell script.

  • The ‘-t’ optional input allows users to specify the type of poller file that will be created. The valid input options are “svm” to create a poller file for use with the single-visit mosaics pipeline or “mvm” to create a poller file for use with the multiple-visit mosaics pipeline. If not explicitly specified, the default value is “svm”. NOTE: if creating a MVM poller file, one must specify the skycell name using the “-s” input argument.

Python USAGE:
>>> python
>>> from drizzlepac.haputils import make_poller_files
>>> make_poller_files.generate_poller_file(input_list, poller_file_type='svm', output_poller_filename="poller_file.out", skycell_name=None):
drizzlepac.haputils.make_poller_files.generate_poller_file(input_list, poller_file_type='svm', output_poller_filename='poller_file.out', skycell_name=None)[source]

Creates a properly formatted SVM or MVM poller file.

Parameters
input_liststr

Name of the text file containing the list of filenames or rootnames to process

poller_file_typestr, optional

Type of poller file to create. ‘svm’ for single visit mosaic, ‘mvm’ for multi-visit mosaic. Default value is ‘svm’.

output_poller_filenamestr, optional

Name of the output poller file that will be created. Default value is ‘poller_file.out’.

skycell_namestr, optional

Name of the skycell to use when creating a MVM poller file. skycell_name is REQUIRED for the creation of a MVM poller file, but completely unnecessary for the creation of a SVM poller file. The correct syntax for skycell names is ‘skycell-pNNNNxXXyXX’, where NNNN is the 4-digit projection cell number, and XX and YY are the two-digit X and Y skycell indices, respectively. Default value is logical ‘None’. NOTE: this input argument is not needed for SVM poller file creation, but REQUIRED for MVM poller file creation. Users can determine the skycell(s) that their observations occupy using the haputils.which_skycell script.

Returns
Nothing.

haputils.poller_utils

Module for interpreting input ‘poller’ files for MVM processing.

Utilities to interpret the pipeline poller obset information and generate product filenames

The function, interpret_obset_input, parses the file generated by the pipeline poller, and produces a tree listing of the output products. The function, parse_obset_tree, converts the tree into product catagories.

drizzlepac.haputils.poller_utils.interpret_mvm_input(results, log_level, layer_method='all', exp_limit=2.0, user_table=None, include_small=True, only_cte=False)[source]
Parameters
resultsstr or list

Input poller file name or Python list of dataset names to be processed for a multi-visit. Dataset names have to be either the filename of a singleton (non-associated exposure) or the name of an ASN file (e.g., jabc01010_asn.fits).

log_levelint

The desired level of verboseness in the log statements displayed on the screen and written to the .log file.

exp_limitfloat, optional

This specifies the ratio between long and short exposures that should be used to define each exposure time bin (short, med, long). If None, all exposure times will be combined into a single bin (all). Splitting of the bins gets computed using Kmeans clustering for 3 output bins.

layer_methodstr

Specify what type of interpretation should be done for the definition of each layer. Options include: ‘all’, ‘hard’, ‘kmeans’. The value ‘all’ simply puts all exposures regardless of exposure time. The value of ‘hard’ relies on pre-defined limits for ‘short’, ‘med’, and ‘long’ exposure times. The value of ‘kmeans’ relies on using ‘kmeans’ analysis for defining sub-layers based on exposure time.

include_smallbool, optional

Specify whether or not to create products from ACS/HRC and ACS/SBC data.

only_ctebool, optional

Specify whether or not to create products only using CTE-corrected data.

Notes

Interpret the contents of the MVM poller file for the values to use in generating the names of all the expected output products.

Input will have format of:

ib4606c5q_flc.fits,11665,B46,06,1.0,F555W,UVIS,skycell-p2381x05y09,NEW,/ifs/archive/ops/hst/public/ib46/ib4606c5q/ib4606c5q_flc.fits

which are filename, proposal_id, program_id, obset_id, exptime, filters, detector, skycell-p<PPPP>x<XX>y<YY>, [OLD|NEW], pathname

The SkyCell ID will be included in this input information to allow for grouping of exposures into the same SkyCell layer based on filter, exptime, and year.

Output dict will have only have definitions for each defined sky cell layer to be either created or updated based on the input exposures being processed. There will be a layer defined for each unique combination of filter/exp class/year. An example would be:

obs_info_dict[“layer 00”: {“info”: ‘11665 06 wfc3 uvis f555w short 2011 drc’,

“files”: [‘ib4606c5q_flc.fits’, ‘ib4606cgq_flc.fits’]}

.

obs_info_dict[“layer 01”: {“info”: ‘11665 06 wfc3 uvis f555w med 2011 drc’,

“files”: [‘ib4606c6q_flc.fits’, ‘ib4606cdq_flc.fits’]}

. obs_info_dict[“layer 02”: {“info”: ‘11665 06 wfc3 uvis f555w long 2011 drc’,

“files”: [‘ib4606c9q_flc.fits’, ‘ib4606ceq_flc.fits’]}

.

Caution: The input (“results”) parameter was originally designed to be able to accept a Python list of pipeline product dataset names (e.g., ib4606cdq_flc.fits) for MVM proessing. However, this was later modified. MVM processing needs the SVM FLT/FLC files as input.

drizzlepac.haputils.poller_utils.build_poller_table(input, log_level, all_mvm_exposures=[], poller_type='svm', include_small=True, only_cte=False)[source]

Create a poller file from dataset names.

Parameters
inputstr, list

Filename with list of dataset names, or just a Python list of dataset names, provided by the user.

log_levelint

The desired level of verboseness in the log statements displayed on the screen and written to the . log file.

all_mvm_exposuresstr, list

List of all the MVM FLT/FLC exposure filenames in the list. This will include the filenames which are dropped because they have been processed previously.

poller_typestr, optional

The type of poller file being processed. Either ‘svm’ for single visit mosaic, or ‘mvm’ for multi-visit mosaic. Unless explicitly specified, the default value is ‘svm’.

Returns
poller_tableTable

Astropy table object with the same columns as a poller file.

haputils.product

Module for defining classes for MVM processing.

Definition of Super and Subclasses for the mosaic output image_list

Classes which define the total (“white light” image), filter, and exposure drizzle products.

These products represent different levels of processing with the levels noted in the ‘HAPLEVEL’ keyword. The ‘HAPLEVEL’ values are:

  • 1 : calibrated (FLT/FLC) input images and exposure level drizzle products with improved astrometry

  • 2 : filter and total products combined using the improved astrometry, consistent pixel scale, and oriented to North

  • 3 : unused

  • 4 : multi-visit mosaics aligned to a common tangent plane

drizzlepac.haputils.product.HAPProduct(prop_id, obset_id, instrument, detector, filename, filetype, log_level)[source]

HAPProduct is the base class for the various products generated during the astrometry update and mosaicing of image data.

drizzlepac.haputils.product.FilterProduct(prop_id, obset_id, instrument, detector, filename, filters, filetype, log_level)[source]

A Filter Detection Product is a mosaic comprised of images acquired during a single visit with one instrument, one detector, a single filter, and all exposure times. The exposure time characteristic may change - TBD.

The “fdp” is short hand for FilterProduct.

drizzlepac.haputils.product.SkyCellExposure(prop_id, obset_id, instrument, detector, filename, layer, filetype, log_level)[source]

An SkyCell Exposure Product is an individual exposure/image (flt/flc).

The “sce” is short hand for ExposureProduct.

drizzlepac.haputils.product.SkyCellProduct(prop_id, obset_id, instrument, detector, skycell_name, layer, filetype, log_level)[source]

A SkyCell Product is a mosaic comprised of images acquired during a multiple visits with one instrument, one detector, a single filter, and all exposure times. The exposure time characteristic may change - TBD.

The “scp” is short hand for SkyCellProduct.