API for runsinglehap

The task runsinglehap serves as the primary interface for processing data from a single-visit into a uniform set of images.

runsinglehap.py - Module to control processing of single-visit mosaics

License:LICENSE

USAGE: runsinglehap [-d] inputFilename

The ‘-d’ option will run this task in DEBUG mode producing additional outputs.

Python USAGE:
python from drizzlepac import runsinglehap runsinglehap.perform(inputFilename,debug=True)

This script defines the HST Advanced Products (HAP) generation portion of the calibration pipeline. This portion of the pipeline produces mosaic and catalog products. This script provides similar functionality as compared to the Hubble Legacy Archive (HLA) pipeline in that it acts as the controller for the overall sequence of processing.

Note regarding logging… During instantiation of the log, the logging level is set to NOTSET which essentially means all message levels (debug, info, etc.) will be passed from the logger to the underlying handlers, and the handlers will dispatch all messages to the associated streams. When the command line option of setting the logging level is invoked, the logger basically filters which messages are passed on to the handlers according the level chosen. The logger is acting as a gate on the messages which are allowed to be passed to the handlers.

Creation of source catalogs can be controlled through the use of environment variables:

  • SVM_CATALOG_HRC
  • SVM_CATALOG_SBC
  • SVM_CATALOG_WFC
  • SVM_CATALOG_UVIS
  • SVM_CATALOG_IR

These variables can be defined using values of:

  • ‘on’, ‘true’, ‘yes’ : Create catalogs
  • ‘off’, ‘false’, ‘no’ : Turn off generation of catalogs

The output products can be evaluated to determine the quality of the alignment and output data through the use of the environment variable:

  • SVM_QUALITY_TESTING : Turn on quality assessment processing. This environment variable, if found with an affirmative value, will turn on processing to generate a JSON file which contains the results of evaluating the quality of the generated products.

NOTE: Step 9 compares the output HAP products to the Hubble Legacy Archive (HLA) products. In order for step 9 (run_sourcelist_comparison()) to run, the following environment variables need to be set: - HLA_CLASSIC_BASEPATH - HLA_BUILD_VER

Alternatively, if the HLA classic path is unavailable, The comparison can be run using locally stored HLA classic files. The relevant HLA classic imagery and sourcelist files must be placed in a subdirectory of the current working directory called ‘hla_classic’.

drizzlepac.hapsequencer.run_hap_processing(input_filename, diagnostic_mode=False, use_defaults_configs=True, input_custom_pars_file=None, output_custom_pars_file=None, phot_mode='both', log_level=20)[source]

Run the HST Advanced Products (HAP) generation code. This routine is the sequencer or controller which invokes the high-level functionality to process the single visit data.

Parameters:
input_filename: string

The ‘poller file’ where each line contains information regarding an exposures taken during a single visit.

diagnostic_mode : bool, optional

Allows printing of additional diagnostic information to the log. Also, can turn on creation and use of pickled information.

use_defaults_configs: bool, optional

If True, use the configuration parameters in the ‘default’ portion of the configuration JSON files. If False, use the configuration parameters in the “parameters” portion of the file. The default is True.

input_custom_pars_file: string, optional

Represents a fully specified input filename of a configuration JSON file which has been customized for specialized processing. This file should contain ALL the input parameters necessary for processing. If there is a filename present for this parameter, the ‘use_defaults_configs’ parameter is ignored. The default is None.

output_custom_pars_file: string, optional

Fully specified output filename which contains all the configuration parameters available during the processing session. The default is None.

phot_mode : str, optional

Which algorithm should be used to generate the sourcelists? ‘aperture’ for aperture photometry; ‘segment’ for segment map photometry; ‘both’ for both ‘segment’ and ‘aperture’. Default value is ‘both’.

log_level : int, optional

The desired level of verboseness in the log statements displayed on the screen and written to the .log file. Default value is 20, or ‘info’.

Returns:
return_value: integer

A return exit code used by the calling Condor/OWL workflow code: 0 (zero) for success, 1 for error

Supporting code

These modules and functions provide the core functionality for the single-visit processing.

drizzlepac.haputils.product

Definition of Super and Subclasses for the mosaic output image_list

Classes which define the total (“white light” image), filter, and exposure drizzle products.

These products represent different levels of processing with the levels noted in the ‘HAPLEVEL’ keyword. The ‘HAPLEVEL’ values are:

  • 1 : calibrated (FLT/FLC) input images and exposure level drizzle products with improved astrometry
  • 2 : filter and total products combined using the improved astrometry,
    consistent pixel scale, and oriented to North.
  • 3 : (future) multi-visit mosaics aligned to common tangent plane
class drizzlepac.haputils.product.HAPProduct(prop_id, obset_id, instrument, detector, filename, filetype, log_level)[source]

HAPProduct is the base class for the various products generated during the astrometry update and mosaicing of image data.

class drizzlepac.haputils.product.TotalProduct(prop_id, obset_id, instrument, detector, filename, filetype, log_level)[source]

A Total Detection Product is a ‘white’ light mosaic comprised of images acquired with one instrument, one detector, all filters, and all exposure times. The exposure time characteristic may change - TBD.

The “tdp” is short hand for TotalProduct.

class drizzlepac.haputils.product.FilterProduct(prop_id, obset_id, instrument, detector, filename, filters, filetype, log_level)[source]

A Filter Detection Product is a mosaic comprised of images acquired during a single visit with one instrument, one detector, a single filter, and all exposure times. The exposure time characteristic may change - TBD.

The “fdp” is short hand for FilterProduct.

class drizzlepac.haputils.product.ExposureProduct(prop_id, obset_id, instrument, detector, filename, filters, filetype, log_level)[source]

An Exposure Product is an individual exposure/image (flt/flc).

The “edp” is short hand for ExposureProduct.

drizzlepac.haputils.poller_utils

Utilities to interpret the pipeline poller obset information and generate product filenames

The function, interpret_obset_input, parses the file generated by the pipeline poller, and produces a tree listing of the output products. The function, parse_obset_tree, converts the tree into product catagories.

drizzlepac.haputils.poller_utils.interpret_obset_input(results, log_level)[source]
Parameters:
results : str or list

Input poller file name or Python list of dataset names to be processed as a single visit. Dataset names have to be either the filename of a singleton (non-associated exposure) or the name of an ASN file (e.g., jabc01010_asn.fits).

log_level : int

The desired level of verboseness in the log statements displayed on the screen and written to the .log file.

Notes

Interpret the database query for a given obset to prepare the returned values for use in generating the names of all the expected output products.

Input will have format of:
ib4606c5q_flc.fits,11665,B46,06,1.0,F555W,UVIS,/ifs/archive/ops/hst/public/ib46/ib4606c5q/ib4606c5q_flc.fits which are filename, proposal_id, program_id, obset_id, exptime, filters, detector, pathname
Output dict will have format (as needed by further code for creating the

product filenames) of:

obs_info_dict[“single exposure product 00”: {“info”: ‘11665 06 wfc3 uvis ib4606c5q f555w drc’,
“files”: [‘ib4606c5q_flc.fits’]}

. obs_info_dict[“single exposure product 08”: {“info”: ‘11665 06 wfc3 ir ib4606clq f110w drz’,

“files”: [‘ib4606clq_flt.fits’]}
obs_info_dict[“filter product 00”: {“info”: ‘11665 06 wfc3 uvis ib4606c5q f555w drc’,
“files”: [‘ib4606c5q_flc.fits’, ‘ib4606c6q_flc.fits’]},

. obs_info_dict[“filter product 01”: {“info”: ‘11665 06 wfc3 ir ib4606cmq f160w drz’,

“files”: [‘ib4606cmq_flt.fits’, ‘ib4606crq_flt.fits’]},
obs_info_dict[“total detection product 00”: {“info”: ‘11665 06 wfc3 uvis ib4606c5q f555w drc’,
“files”: [‘ib4606c5q_flc.fits’, ‘ib4606c6q_flc.fits’]}

. obs_info_dict[“total detection product 01”: {“info”: ‘11665 06 wfc3 ir ib4606cmq f160w drz’,

“files”: [‘ib4606cmq_flt.fits’, ‘ib4606crq_flt.fits’]}
drizzlepac.haputils.poller_utils.parse_obset_tree(det_tree, log_level)[source]

Convert tree into products

Tree generated by create_row_info() will always have the following levels:

  • detector
    • filters
      • exposure

Each exposure will have an entry dict with keys ‘info’ and ‘filename’.

Products created will be:
  • total detection product per detector
  • filter products per detector
  • single exposure product
drizzlepac.haputils.poller_utils.build_obset_tree(obset_table)[source]

Convert obset table into a tree listing all products to be created.

drizzlepac.haputils.poller_utils.build_poller_table(input, log_level, poller_type='svm')[source]

Create a poller file from dataset names.

Parameters:
input : str, list

Filename with list of dataset names, or just a Python list of dataset names, provided by the user.

log_level : int

The desired level of verboseness in the log statements displayed on the screen and written to the . log file.

poller_type : str, optional

The type of poller file being processed. Either ‘svm’ for single visit mosaic, or ‘mvm’ for multi-visit mosaic. Unless explicitly specified, the default value is ‘svm’.

Returns:
poller_table : Table

Astropy table object with the same columns as a poller file.

drizzlepac.haputils.catalog_utils

This script contains code to support creation of photometric sourcelists using two techniques: aperture photometry and segmentation-map based photometry.

class drizzlepac.haputils.catalog_utils.CatalogImage(filename, num_images_mask, log_level)[source]
class drizzlepac.haputils.catalog_utils.HAPCatalogs(fitsfile, param_dict, param_dict_qc, num_images_mask, log_level, diagnostic_mode=False, types=None, tp_sources=None)[source]

Generate photometric sourcelist for specified TOTAL or FILTER product image.

class drizzlepac.haputils.catalog_utils.HAPCatalogBase(image, param_dict, param_dict_qc, diagnostic_mode, tp_sources)[source]

Virtual class used to define API for all catalogs

class drizzlepac.haputils.catalog_utils.HAPPointCatalog(image, param_dict, param_dict_qc, diagnostic_mode, tp_sources)[source]

Generate photometric sourcelist(s) for specified image(s) using aperture photometry of point sources.

class drizzlepac.haputils.catalog_utils.HAPSegmentCatalog(image, param_dict, param_dict_qc, diagnostic_mode, tp_sources)[source]

Generate a sourcelist for a specified image by detecting both point and extended sources using the image segmentation process.

Parameters:
image : CatalogImage object

The white light (aka total detection) or filter drizzled image

param_dict : dictionary

Configuration values for catalog generation based upon input JSON files

diagnostic_mode : bool

Specifies whether or not to generate the regions file used for ds9 overlay

tp_sources: dictionary

Dictionary containing computed information for each catalog type

drizzlepac.haputils.photometry_tools

Tools for aperture photometry with non native bg/error methods

This function serves to ease the computation of photometric magnitudes and errors using PhotUtils by replicating DAOPHOT’s photometry and error methods. The formula for DAOPHOT’s error is:

err = sqrt (Poisson_noise / epadu + area * stdev**2 + area**2 * stdev**2 / nsky)

Which gives a magnitude error:

mag_err = 1.0857 * err / flux

Where epadu is electrons per ADU (gain), area is the photometric aperture area, stdev is the uncertainty in the sky measurement and nsky is the sky annulus area. To get the uncertainty in the sky we must use a custom background tool, which also enables computation of the mean and median of the sky as well (more robust statistics). All the stats are sigma clipped. These are calculated by the functions in aperture_stats_tbl.

Note

Currently, the background computations will fully include a pixel that has ANY overlap with the background aperture (the annulus). This is to simplify the computation of the median, as a weighted median is nontrivial, and slower. Copied from https://grit.stsci.edu/HLA/software/blob/master/HLApipeline/HLApipe/scripts/photometry_tools.py

Authors

  • Varun Bajaj, January 2018

Use

from photometry_tools import iraf_style_photometry
phot_aps = CircularAperture((sources['xcentroid'], sources['ycentroid']),r=10.)
bg_aps = CircularAnnulus((sources['xcentroid'], sources['ycentroid']), r_in=13., r_out=15.)

Simplest call:

photometry_tbl = iraf_style_photometry(phot_aps, bg_aps, data)

Pass in pixelwise error and set background to mean

photometry_tbl = iraf_style_photometry(phot_aps, bg_aps, data, error_array=data_err, bg_method='mean')

Can also set the gain (if image units are DN)

photometry_tbl = iraf_style_photometry(phot_aps, bg_aps, data, epadu=2.5)

Classes and Functions

drizzlepac.haputils.photometry_tools.iraf_style_photometry(phot_apertures, bg_apertures, data, photflam, photplam, error_array=None, bg_method='mode', epadu=1.0)[source]

Computes photometry with PhotUtils apertures, with IRAF formulae

Parameters:
phot_apertures : photutils PixelAperture object (or subclass)

The PhotUtils apertures object to compute the photometry. i.e. the object returned via CirularAperture.

bg_apertures : photutils PixelAperture object (or subclass)

The phoutils aperture object to measure the background in. i.e. the object returned via CircularAnnulus.

data : array

The data for the image to be measured.

photflam : float

inverse sensitivity, in ergs/cm2/angstrom/electron

photplam : float

Pivot wavelength, in angstroms

error_array : array

(Optional) The array of pixelwise error of the data. If none, the Poisson noise term in the error computation will just be the square root of the flux/epadu. If not none, the aperture_sum_err column output by aperture_photometry (divided by epadu) will be used as the Poisson noise term.

bg_method: string

{‘mean’, ‘median’, ‘mode’}, optional. The statistic used to calculate the background. All measurements are sigma clipped. Default value is ‘mode’. NOTE: From DAOPHOT, mode = 3 * median - 2 * mean.

epadu : float

(optional) Gain in electrons per adu (only use if image units aren’t e-). Default value is 1.0

Returns:
An astropy Table with columns as follows:

X-Center Y-Center RA DEC ID MagAp1 MagErrAp1 MagAp2 MagErrAp2 MSkyAp2 StdevAp2 FluxAp2 CI Flags

drizzlepac.haputils.photometry_tools.compute_phot_error(flux_variance, bg_phot, bg_method, ap_area, epadu=1.0)[source]

Computes the flux errors using the DAOPHOT style computation. Originally taken from https://github.com/spacetelescope/wfc3_photometry/blob/527815bf580d0361754281b608008e539ed84368/photometry_tools/photometry_with_errors.py#L137

Parameters:
flux_variance : array

flux values

bg_phot : array

background brightness values.

bg_method : string

background method

ap_area : array

the area of the aperture in square pixels

epadu : float

(optional) Gain in electrons per adu (only use if image units aren’t e-). Default value is 1.0

Returns:
flux_error : array

an array of flux errors

drizzlepac.haputils.photometry_tools.convert_flux_to_abmag(in_flux, photflam, photplam)[source]

converts flux (in units of electrons/sec) to ABMAG

Parameters:
in_flux : astropy.table.column.Column object

flux values to convert to ABMAG, in electrons/second

photflam : float

inverse sensitivity, in ergs/cm2/angstrom/electron

photplam : float

pivot wavelength, in angstroms

Returns:
abmag : astropy.table.column.Column object

input flux values converted to ABMAG

drizzlepac.haputils.processing_utils

Utilities to support creation of Hubble Advanced Pipeline(HAP) products.

drizzlepac.haputils.processing_utils.refine_product_headers(product, total_obj_list)[source]

Refines output product headers to include values not available to AstroDrizzle.

A few header keywords need to have values computed to reflect the type of product being generated, which in some cases can only be done using information passed in from the calling routine. This function insures that all header keywords have been populated with values appropriate for the type of product being processed.

Parameters:
product : str or object

Filename or HDUList object for product to be updated

total_obj_list: list

List of TotalProduct objects which are composed of Filter and Exposure Product objects

drizzlepac.haputils.processing_utils.compute_sregion(image, extname='SCI')[source]

Compute the S_REGION keyword for a given WCS.

Parameters:
image : Astropy io.fits HDUList object

Image to update with the S_REGION keyword in each of the SCI extensions.

extname : str, optional

EXTNAME value for extension containing the WCS(s) to be updated