API for runsinglehap

The task runsinglehap serves as the primary interface for processing data from a single-visit into a uniform set of images.

runsinglehap.py - Module to control processing of single-visit mosaics

License:: License

USAGE:

>>> runsinglehap [-cdl] inputFilename

The ‘-c’ option allows the user to specify a customized configuration JSON file which has been tuned for specialized processing. This file should contain ALL the input parameters necessary for processing. If not specified, default configuration values will be used.
The ‘-d’ option will run this task in a more verbose diagnostic mode producing additional log messages will be displayed and additional files will be created.
The ‘-l’ option allows the user to set the desired level of verboseness in the log statements displayed on the screen and written to the .log file. Specifying “critical” will only record/display “critical” log statements, and specifying “error” will record/display both “error” and “critical” log statements, and so on. Valid inputs: ‘critical’, ‘error’, ‘warning’, ‘info’, or ‘debug’.

Python USAGE:

>>> python
>>> from drizzlepac import runsinglehap
>>> runsinglehap.perform(inputFilename, debug=False, input_custom_pars_file=None, log_level='info')

drizzlepac.runsinglehap.perform(input_filename, **kwargs)[source]

Main calling subroutine for the runsinglehap task.

Parameters:

input_filenamestr: Name of the input csv file containing information about the files to be processed
debugbool, optional: display all tracebacks, and debug information? If not specified, the default value is Boolean ‘False’.
input_custom_pars_filestr, optional: Represents a fully specified input filename of a configuration JSON file which has been customized for specialized processing. This file should contain ALL the input parameters necessary for processing. If not specified, default configuration parameter values will be used.
log_levelstr, optional: The desired level of verboseness in the log statements displayed on the screen and written to the .log file. Valid inputs: ‘critical’, ‘error’, ‘warning’, ‘info’, or ‘debug’. If not specified, the default value is ‘info’.

This script defines the HST Advanced Products (HAP) generation portion of the calibration pipeline. This portion of the pipeline produces mosaic and catalog products. This script provides similar functionality as compared to the Hubble Legacy Archive (HLA) pipeline in that it acts as the controller for the overall sequence of processing.

Note regarding logging… During instantiation of the log, the logging level is set to NOTSET which essentially means all message levels (debug, info, etc.) will be passed from the logger to the underlying handlers, and the handlers will dispatch all messages to the associated streams. When the command line option of setting the logging level is invoked, the logger basically filters which messages are passed on to the handlers according the level chosen. The logger is acting as a gate on the messages which are allowed to be passed to the handlers.

Creation of source catalogs can be controlled through the use of environment variables:

SVM_CATALOG_HRC

SVM_CATALOG_SBC

SVM_CATALOG_WFC

SVM_CATALOG_UVIS

SVM_CATALOG_IR

These variables can be defined using values of:

‘on’, ‘true’, ‘yes’ : Create catalogs

‘off’, ‘false’, ‘no’ : Turn off generation of catalogs

The output products can be evaluated to determine the quality of the alignment and output data through the use of the environment variable:

SVM_QUALITY_TESTING : Turn on quality assessment processing. This environment variable, if found with an affirmative value, will turn on processing to generate a JSON file which contains the results of evaluating the quality of the generated products.

NOTE: Step 9 compares the output HAP products to the Hubble Legacy Archive (HLA) products. In order for step 9 (run_sourcelist_comparison()) to run, the following environment variables need to be set: - HLA_CLASSIC_BASEPATH - HLA_BUILD_VER

Alternatively, if the HLA classic path is unavailable, The comparison can be run using locally stored HLA classic files. The relevant HLA classic imagery and sourcelist files must be placed in a subdirectory of the current working directory called ‘hla_classic’.

drizzlepac.hapsequencer.run_hap_processing(input_filename, diagnostic_mode=False, input_custom_pars_file=None, output_custom_pars_file=None, phot_mode='both', log_level=20)[source]

Run the HST Advanced Products (HAP) generation code. This routine is the sequencer or controller which invokes the high-level functionality to process the single visit data.

Parameters:

input_filenamestr: The ‘poller file’ where each line contains information regarding an exposures taken during a single visit.
diagnostic_modebool, optional: Allows printing of additional diagnostic information to the log. Also, can turn on creation and use of pickled information.
input_custom_pars_filestr, optional: Represents a fully specified input filename of a configuration JSON file which has been customized for specialized processing. This file should contain ALL the input parameters necessary for processing. If not specified, default configuration parameter values will be used. The default is None.
output_custom_pars_filestr, optional: Fully specified output filename which contains all the configuration parameters available during the processing session. The default is None.
phot_modestr, optional: Which algorithm should be used to generate the sourcelists? ‘aperture’ for aperture photometry; ‘segment’ for segment map photometry; ‘both’ for both ‘segment’ and ‘aperture’. Default value is ‘both’.
log_levelint, optional: The desired level of verboseness in the log statements displayed on the screen and written to the .log file. Default value is 20, or ‘info’.

Returns:

return_value: int: A return exit code used by the calling Condor/OWL workflow code: 0 (zero) for success, 1 for error

drizzlepac.hapsequencer.create_catalog_products(total_obj_list, log_level, diagnostic_mode=False, phot_mode='both', catalog_switches=None)[source]

This subroutine utilizes haputils/catalog_utils module to produce photometric sourcelists for the specified total drizzle product and it’s associated child filter products.

Parameters:

total_obj_listTotalProduct

total drizzle product that will be processed by catalog_utils. catalog_utils will also create photometric sourcelists for the child filter products of this total product.

log_levelint, optional

The desired level of verboseness in the log statements displayed on the screen and written to the .log file.

diagnostic_modebool, optional

generate ds9 region file counterparts to the photometric sourcelists? Default value is False.

phot_modestr, optional

Which algorithm should be used to generate the sourcelists? ‘aperture’ for aperture photometry; ‘segment’ for segment map photometry; ‘both’ for both ‘segment’ and ‘aperture’. Default value is ‘both’.

catalog_switchesdict, optional

Specify which, if any, catalogs should be generated at all, based on detector. This dictionary needs to contain values for all instruments; namely:

SVM_CATALOG_HRC, SVM_CATALOG_SBC, SVM_CATALOG_WFC, SVM_CATALOG_UVIS, SVM_CATALOG_IR, SVM_CATALOG_PC

These variables can be defined with values of ‘on’/’off’/’yes’/’no’/’true’/’false’.

Returns:

product_listlist: list of all catalogs generated.

drizzlepac.hapsequencer.create_drizzle_products(total_obj_list)[source]

Run astrodrizzle to produce products specified in the total_obj_list.

Parameters:

total_obj_listlist: List of TotalProduct objects, one object per instrument/detector combination is a visit. The TotalProduct objects are comprised of FilterProduct and ExposureProduct objects.

Returns:

product_listlist: A list of output products

Supporting code

These modules and functions provide the core functionality for the single-visit processing.

drizzlepac.haputils.product

Definition of Super and Subclasses for the mosaic output image_list

Classes which define the total (“white light” image), filter, and exposure drizzle products.

These products represent different levels of processing with the levels noted in the ‘HAPLEVEL’ keyword. The ‘HAPLEVEL’ values are:

1 : calibrated (FLT/FLC) input images and exposure level drizzle products with improved astrometry

2 : filter and total products combined using the improved astrometry, consistent pixel scale, and oriented to North

3 : unused

4 : multi-visit mosaics aligned to a common tangent plane

class drizzlepac.haputils.product.HAPProduct(prop_id, obset_id, instrument, detector, aperture_from_poller, filename, filetype, log_level)[source]: HAPProduct is the base class for the various products generated during the astrometry update and mosaicing of image data.

class drizzlepac.haputils.product.TotalProduct(prop_id, obset_id, instrument, detector, aperture_from_poller, filename, filetype, log_level)[source]

A Total Detection Product is a ‘white’ light mosaic comprised of images acquired with one instrument, one detector, all filters, and all exposure times. The exposure time characteristic may change - TBD.

The “tdp” is short hand for TotalProduct.

class drizzlepac.haputils.product.FilterProduct(prop_id, obset_id, instrument, detector, aperture_from_poller, filename, filters, filetype, log_level)[source]

A Filter Detection Product is a mosaic comprised of images acquired during a single visit with one instrument, one detector, a single filter, and all exposure times. The exposure time characteristic may change - TBD.

The “fdp” is short hand for FilterProduct.

class drizzlepac.haputils.product.ExposureProduct(prop_id, obset_id, instrument, detector, aperture_from_poller, filename, filters, filetype, log_level)[source]

An Exposure Product is an individual exposure/image (flt/flc).

The “edp” is short hand for ExposureProduct.

drizzlepac.haputils.poller_utils

Utilities to interpret the pipeline poller obset information and generate product filenames

The function, interpret_obset_input, parses the file generated by the pipeline poller, and produces a tree listing of the output products. The function, parse_obset_tree, converts the tree into product catagories.

drizzlepac.haputils.poller_utils.interpret_obset_input(results: str, log_level)[source]

parses the file generated by the pipeline poller, and produces a tree listing of the output products.

Parameters:

resultsstr or list: Input poller file name or Python list of dataset names to be processed as a single visit. Dataset names have to be either the filename of a singleton (non-associated exposure) or the name of an ASN file (e.g., jabc01010_asn.fits).
log_levelint: The desired level of verboseness in the log statements displayed on the screen and written to the .log file.

Returns:

obset_dictdict: Dictionary of obset objects
tdp_listlist: list of total detection products

Notes

Interpret the database query for a given obset to prepare the returned values for use in generating the names of all the expected output products.

Input will have formated rows of one of the following three options

Full poller file

ib4606c5q_flc.fits,11665,B46,06,1.0,F555W,UVIS,/foo/bar/ib4606c5q_flc.fits

Simpler poller file (list); other info taken from file header keywords

ib4606c5q_flc.fits

For updating the WFPC2 SVM aperture keyword using the poller file; it
is important that there are no spaces within the poller aperture keyword(s)

ib4606c5q_flc.fits, PC1-FIX;F160BN15

Full poller files contain filename, proposal_id, program_id, obset_id, exptime, filters, detector, pathname.

The output dict will have format (as needed by further code for creating the product filenames) of

obs_info_dict["single exposure product 00": {"info": '11665 06 wfc3 uvis empty_aperture ib4606c5q f555w drc',
                                             "files": ['ib4606c5q_flc.fits']}
.
.
.
obs_info_dict["single exposure product 08": {"info": '11665 06 wfc3 ir empty_aperture ib4606clq f110w drz',
                                             "files": ['ib4606clq_flt.fits']}

obs_info_dict["filter product 00": {"info": '11665 06 wfc3 uvis empty_aperture ib4606c5q f555w drc',
                                    "files": ['ib4606c5q_flc.fits', 'ib4606c6q_flc.fits']},
.
.
.
obs_info_dict["filter product 01": {"info": '11665 06 wfc3 ir empty_aperture ib4606cmq f160w drz',
                                    "files": ['ib4606cmq_flt.fits', 'ib4606crq_flt.fits']},

obs_info_dict["total detection product 00": {"info": '11665 06 wfc3 uvis empty_aperture ib4606c5q f555w drc',
                                             "files": ['ib4606c5q_flc.fits', 'ib4606c6q_flc.fits']}
.
.
.
obs_info_dict["total detection product 01": {"info": '11665 06 wfc3 ir empty_aperture ib4606cmq f160w drz',
                                             "files": ['ib4606cmq_flt.fits', 'ib4606crq_flt.fits']}

The aperture keyword, which has a default value of ‘empty_aperture’ is filled in the case of WFPC2 observations, and the use of the two-column format.

drizzlepac.haputils.poller_utils.parse_obset_tree(det_tree, log_level)[source]

Convert tree into products

Tree generated by create_row_info() will always have the following levels:

detector

filters

exposure

Each exposure will have an entry dict with keys ‘info’ and ‘filename’.

Products created will be:

total detection product per detector

filter products per detector

single exposure product

grism/prism single exposure product [only if Grism/Prism exposures]

Grism/Prism images were added to the “doProcess” list after most of this code was developed as they need to have the same Primary WCS as the direct images from the same detector in the visit. These images are deliberately not added to the output obset_products as they will not be drizzled. They are added as individual GrismExposureProduct objects to the TotalProduct only. The GrismExposureProduct list in the TotalProduct is separate and distinct from the ExposureProduct list of direct images.

drizzlepac.haputils.poller_utils.build_obset_tree(obset_table)[source]: Convert obset table into a tree listing all products to be created.

drizzlepac.haputils.poller_utils.build_poller_table(input: str, log_level, all_mvm_exposures=[], poller_type='svm', include_small=True, only_cte=False)[source]

Create a poller file from dataset names for either SMV or MVM processing. Information is either gathered from the poller file or by using the filename to open the file and pulling information from the header keywords. The code treats WFPC2 differently, by uses both approaches. For WFPC2, We use simple poller files with a second column that includes the aperture. The code gathers the rest of the relevant informaiton from the header keywords.

Parameters:

inputstr, list: Filename with list of dataset names, or just a Python list of dataset names, provided by the user.
log_levelint: The desired level of verboseness in the log statements displayed on the screen and written to the . log file.
all_mvm_exposuresstr, list: List of all the MVM FLT/FLC exposure filenames in the list. This will include the filenames which are dropped because they have been processed previously.
poller_typestr, optional: The type of poller file being processed. Either ‘svm’ for single visit mosaic, or ‘mvm’ for multi-visit mosaic. Unless explicitly specified, the default value is ‘svm’.

Returns:

poller_tableTable: Astropy table object with the same columns as a poller file.

drizzlepac.haputils.catalog_utils

This script contains code to support creation of photometric sourcelists using two techniques: aperture photometry and segmentation-map based photometry.

class drizzlepac.haputils.catalog_utils.CatalogImage(filename, num_images_mask, log_level)[source]

class drizzlepac.haputils.catalog_utils.HAPCatalogs(fitsfile, param_dict, param_dict_qc, num_images_mask, log_level, diagnostic_mode=False, types=None, tp_sources=None)[source]: Generate photometric sourcelist for specified TOTAL or FILTER product image.

class drizzlepac.haputils.catalog_utils.HAPCatalogBase(image, param_dict, param_dict_qc, diagnostic_mode, tp_sources)[source]: Virtual class used to define API for all catalogs

class drizzlepac.haputils.catalog_utils.HAPPointCatalog(image, param_dict, param_dict_qc, diagnostic_mode, tp_sources)[source]: Generate photometric sourcelist(s) for specified image(s) using aperture photometry of point sources.

class drizzlepac.haputils.catalog_utils.HAPSegmentCatalog(image, param_dict, param_dict_qc, diagnostic_mode, tp_sources)[source]

Generate a sourcelist for a specified image by detecting both point and extended sources using the image segmentation process.

Parameters:

imageCatalogImage object: The white light (aka total detection) or filter drizzled image
param_dictdictionary: Configuration values for catalog generation based upon input JSON files
diagnostic_modebool: Specifies whether or not to generate the regions file used for ds9 overlay
tp_sources: dictionary: Dictionary containing computed information for each catalog type

drizzlepac.haputils.photometry_tools

Tools for aperture photometry with non native bg/error methods

This function serves to ease the computation of photometric magnitudes and errors using PhotUtils by replicating DAOPHOT’s photometry and error methods. The formula for DAOPHOT’s error is:

err = sqrt (Poisson_noise / epadu + area * stdev**2 + area**2 * stdev**2 / nsky)

Which gives a magnitude error:

mag_err = 1.0857 * err / flux

Where epadu is electrons per ADU (gain), area is the photometric aperture area, stdev is the uncertainty in the sky measurement and nsky is the sky annulus area. To get the uncertainty in the sky we must use a custom background tool, which also enables computation of the mean and median of the sky as well (more robust statistics). All the stats are sigma clipped. These are calculated by the functions in aperture_stats_tbl.

Note

Currently, the background computations will fully include a pixel that has ANY overlap with the background aperture (the annulus). This is to simplify the computation of the median, as a weighted median is nontrivial, and slower. Copied from https://grit.stsci.edu/HLA/software/blob/master/HLApipeline/HLApipe/scripts/photometry_tools.py

Authors

Varun Bajaj, January 2018

Use

from photometry_tools import iraf_style_photometry
phot_aps = CircularAperture((sources['xcentroid'], sources['ycentroid']),r=10.)
bg_aps = CircularAnnulus((sources['xcentroid'], sources['ycentroid']), r_in=13., r_out=15.)

Simplest call:

photometry_tbl = iraf_style_photometry(phot_aps, bg_aps, data)

Pass in pixelwise error and set background to mean

photometry_tbl = iraf_style_photometry(phot_aps, bg_aps, data, error_array=data_err, bg_method='mean')

Can also set the gain (if image units are DN)

photometry_tbl = iraf_style_photometry(phot_aps, bg_aps, data, epadu=2.5)

Classes and Functions

drizzlepac.haputils.photometry_tools.iraf_style_photometry(phot_apertures, bg_apertures, data, photflam, photplam, error_array=None, bg_method='mode', epadu=1.0)[source]

Computes photometry with PhotUtils apertures, with IRAF formulae

Parameters:

phot_aperturesphotutils PixelAperture object (or subclass): The PhotUtils apertures object to compute the photometry. i.e. the object returned via CirularAperture.
bg_aperturesphotutils PixelAperture object (or subclass): The phoutils aperture object to measure the background in. i.e. the object returned via CircularAnnulus.
dataarray: The data for the image to be measured.
photflamfloat: inverse sensitivity, in ergs/cm2/angstrom/electron
photplamfloat: Pivot wavelength, in angstroms
error_arrayarray: (Optional) The array of pixelwise error of the data. If none, the Poisson noise term in the error computation will just be the square root of the flux/epadu. If not none, the aperture_sum_err column output by aperture_photometry (divided by epadu) will be used as the Poisson noise term.
bg_method: string: {‘mean’, ‘median’, ‘mode’}, optional. The statistic used to calculate the background. All measurements are sigma clipped. Default value is ‘mode’. NOTE: From DAOPHOT, mode = 3 * median - 2 * mean.
epadufloat: (optional) Gain in electrons per adu (only use if image units aren’t e-). Default value is 1.0

Returns:

An astropy Table with columns as follows:
X-Center Y-Center RA DEC ID MagAp1 MagErrAp1 MagAp2 MagErrAp2 MSkyAp2 StdevAp2 FluxAp2 CI Flags

drizzlepac.haputils.photometry_tools.compute_phot_error(flux_variance, bg_phot, bg_method, ap_area, epadu=1.0)[source]

Computes the flux errors using the DAOPHOT style computation. Originally taken from https://github.com/spacetelescope/wfc3_photometry/blob/527815bf580d0361754281b608008e539ed84368/photometry_tools/photometry_with_errors.py#L137

Parameters:

flux_variancearray: flux values
bg_photarray: background brightness values.
bg_methodstring: background method
ap_areaarray: the area of the aperture in square pixels
epadufloat: (optional) Gain in electrons per adu (only use if image units aren’t e-). Default value is 1.0

Returns:

flux_errorarray: an array of flux errors

drizzlepac.haputils.photometry_tools.convert_flux_to_abmag(in_flux, photflam, photplam)[source]

converts flux (in units of electrons/sec) to ABMAG

Parameters:

in_fluxastropy.table.column.Column object: flux values to convert to ABMAG, in electrons/second
photflamfloat: inverse sensitivity, in ergs/cm2/angstrom/electron
photplamfloat: pivot wavelength, in angstroms

Returns:

abmagastropy.table.column.Column object: input flux values converted to ABMAG

Associated Helper Code

These modules and functions assist with some of the logistics associated with single-visit processing.

drizzlepac.haputils.processing_utils

Utilities to support creation of Hubble Advanced Pipeline(HAP) products.

drizzlepac.haputils.processing_utils.refine_product_headers(product, total_obj_list)[source]

Refines output product headers to include values not available to AstroDrizzle.

A few header keywords need to have values computed to reflect the type of product being generated, which in some cases can only be done using information passed in from the calling routine. This function insures that all header keywords have been populated with values appropriate for the type of product being processed.

Parameters:

productstr or object: Filename or HDUList object for product to be updated
total_obj_list: list: List of TotalProduct objects which are composed of Filter and Exposure Product objects

drizzlepac.haputils.processing_utils.compute_sregion(image, extname='SCI')[source]

Compute the S_REGION keyword for a given WCS.

Parameters:

imageAstropy io.fits HDUList object: Image to update with the S_REGION keyword in each of the SCI extensions.
extnamestr, optional: EXTNAME value for extension containing the WCS(s) to be updated