Utilities for Manipulating MVM Products

The hapcut_utils.py module contains utilities which are intended to be used with Multi-visit Mosaic (MVM) data and to be imported into other Python programs or a Python session for interactive data analysis. The utilities are wrappers around lower-level functionality which actually perform the core tasks. The core tasks referenced here are astroquery, used for MAST inquiries and retrievals, and astrocut, used to create the cutouts and cutout combinations. The hapcut_utils.py role is to customize the use of these core tasks specifically for working with MVM products which can be large and unwieldy. At this time there are four utilities which are designed to be invoked in sequence as is evident from their function signatures, return values, and output products: mvm_id_filenames, mvm_retrieve_files, make_the_cut, and mvm_combine.

A brief overview of the utilities is described in this list.

  1. mvm_id_filenames: Query MAST to identify all MVM files which overlap the specified cutout region on the sky

  2. mvm_retrieve_files: Download all requested MVM files from MAST to the working directory

  3. make_the_cut: Extract cutouts from all specified MVM files

  4. mvm_combine: Create a mosaic from the specified cutouts that span multiple SkyCells and fit the qualifications

MVM Cutout Utilities

mvm_id_filenames

This utility is invoked by providing sky coordinates and a cutout box size. The sky coordinates represent the central location about which the search for data of interest is to be done. The cutout box size represents the search box extent encompassing the sky coordinates. The utility retrieves from MAST a table containing MVM filenames for ACS and WFC3 detectors, as well as other relevant image specific information, one row per filename. While the utility returns an astropy table object, an ascii ECSV file is also written to disk with the output filename being constructed from the query coordinates and box size parameters. The output filename is of the form:

mvm_query-ra<###>d<####>-dec<n|s><##>d<####>_<radius>_cutout.ecsv

The ra and dec are the right ascension and declination, respectively, in decimal degrees with <n|s> representing north or south of the celestial equator. The <radius> is the computed diagonal of the cutout box which is basically the square root of the sum of the box sides squared divided by two.

The in-memory table or the ascii file can be refined by the user to suit specific needs. The MVM filenames within the query file are self-descriptive. The filter-level drizzled filename is of the form:

hst_skycell-p<PPPP>x<XX>y<YY>_<instr>_<detector>_<filter>_<label>_dr[cz].fits

where -p is followed by the four digit projection cell value, and x and y are each followed by a two digit sky cell value. The possible label values at this time are all or coarse-all where all indicates all exposures while coarse-all indicates all the exposures at a coarse platescale (i.e., the WFC3/IR platescale of 0.12”/pixel) in the visit were used in the generation of the product.

While the standard pipeline processing does not generate MVM exposure-level, drizzled products, the user can activate this capability when performing custom processing. As such, the MAST query will not contain any exposure-level drizzled product filenames. However, an example of an exposure-level drizzled filename created by custom processing would be of the form:

hst_skycell-p<PPPP>x<XX>y<YY>_<instr>_<detector>_<filter>_<label>-ipppssoo_dr[cz].fits

An example of a real MVM filter-level, drizzled filename is:

hst_skycell-p0081x14y14_wfc3_uvis_f225w_all_drc.fits

mvm_retrieve_files

This utility is designed to use the in-memory table generated by mvm_id_filenames and retrieve any file listed in the table from MAST. Since the images can be quite large, the clobber parameter is set to False by default, indicating no download is done for a filename that already exists on disk. The utility returns a Python list of filenames on disk available for further analysis.

make_the_cut

The make_the_cut utility will perform the FITS cutouts on the input MVM filter- and exposure-level images. This MVM customized utility is layered on fits_cut, a function which is part of Astrocut, a Python package for making astronomical cutouts (https://astrocut.readthedocs.io/en/latest/). The make_the_cut function is invoked by providing the list of files on disk for which one wants to make cutouts, as well as the sky coordinates, and the cutout box size. Ideally, the sky coordinates and cutout box size are the same values used for the mvm_id_filenames utility. For each input file designated for a cutout, there will be a corresponding output file. Since both the SCI and WHT extensions of the input files are actually cut, individual output cutout FITS files will contain two image extensions, a SCI followed by a WHT. The cutouts are explicitly generated by looping over the input file list in case one of the files experiences an exception so the remaining files in the list still have the opportunity to create a cutout. Each filter-level output filename will be of the form:

hst_cutout_skycell-p<pppp>-ra<##>d<####>-dec<n|s><##>d<####>_instrument_detector_filter[_platescale].fits

Each exposure-level filename generated by custom user processing will be of the form:

hst_cutout_skycell-p<pppp>-ra<##>d<####>-dec<n|s><##>d<####>_instrument_detector_filter[_platescale]-ipppssoo.fits

The components of the output filename follow the pattern discussed for mvm_id_filenames. The additional platescale component has the possible value of coarse. When there is no platescale component present in the filename, the platescale has the implicit default value of fine. The fine platescale is 0.04”/pixel and is the standardized platescale for all MVM products. The platescale of coarse is only applicable to the WFC3/IR data.

The primary header data unit (PHDU) from the input file is retained as the PHDU for the output cutout file with an updated FILENAME keyword and a few additional keywords. The new keywords RA_OBJ/DEC_OBJ were added by the Astrocut utility representing the right ascension and declination of the object in decimal degrees. Another new keyword is ORIG_FLE which represents the original input filename from which the cutout was extracted. In each output file, the image extensions have been explicitly named for for clarity.

>>> from astropy.io import fits
>>> hdu= fits.open("hst_cutout_skycell-p0081x14y15-ra84d8208-decs69d8516_wfc3_uvis_f275w.fits")
>>> hdu.info()
Filename: hst_cutout_skycell-p0081x14y15-ra84d8208-decs69d8516_wfc3_uvis_f275w.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     872   ()
  1  SCI           1 ImageHDU       106   (12500, 12500)   float32
  2  WHT           1 ImageHDU        56   (12500, 12500)   float32

mvm_combine

The final MVM utility at this time is mvm_combine and is designed to combine filter-level cutout images originating from multiple skycells where all the images were acquired with the same detector and using the same filter. The utility will also combine exposure-level cutout images from multiple skycells where the images were acquired with the same detector, using the same filter, and sharing the same ipppssoo and created by user custom processing. Put more straightforwardly, the cutout images must have originated from the same MVM drizzled image which spanned the border between multiple skycells. Cutout images in the input list which do not adhere to these noted requirements will trigger a message of explanation and be ignored for the combination process.

The mvm_combine utility is invoked by providing a list of cutout files, and the utility will handle segregating the files into filter- and exposure-level sets and determining whether or not the cutout file is eligible for the combination process. Once all the input images have been prepared, the mvm_combine will call the Astrocut function CutoutsCombiner to handle the actual combination task. At this time, only the default mean combiner is available for use where every output pixel is the mean of all input pixels that have data at that point.

Since both the SCI and WHT extensions of the input files are combined, individual output combined FITS files will contain two image extensions, a SCI followed by a WHT. The utility writes to the specified output directory one combined file per set of compatible images. The output filenames are constructed from the RA, Dec, instrument, detector, filter, and exposure as appropriate. Each filter-level output filename will be of the form:

hst_combined_skycell-ra<##>d<####>-dec<n|s><##>d<####>_instrument_detector_filter.fits

Each exposure-level filename where the cutout images were generated by custom user processing will be of the form:

hst_combined_skycell-ra<##>d<####>-dec<n|s><##>d<####>_instrument_detector_filter_ipppssoo.fits

The keyword, FILENAME, has been added to the PHDU to self-document the name of the file. The Extension Header Data Units (EHDUs) contain custom keywords which document the filenames and extension names of the originating files. These custom keywords are created by the CutoutsCombiner and are of the form Fnn_[K|V|C]mm representing the keywords EXTNAMEs and ORIG_FLEs from the input cutout files. The nn represents the header number, and mm represents a count of the custom keywords. For example,

>>> hdu = fits.open("hst_combined_skycells-ra84d9402-decs69d8514_acs_wfc_f658n.fits")
>>> hdu.info()
Filename: hst_combined_skycells-ra84d9402-decs69d8514_acs_wfc_f658n.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU      10   ()
  1  SCI           1 ImageHDU       167   (250, 250)   float64
  2  WHT           1 ImageHDU        70   (250, 250)   float64

where the first extention or combined SCI extension header contains
O_EXT_NM= 'SCI     '           / Original extension name.
ORIG_EXT=                    1 / Extension in original file.
F01_K01 = 'EXTNAME '           / Keyword
F01_V01 = 'SCI'                / Value
F01_C01 = 'extension name'     / Comment
F02_K01 = 'EXTNAME '           / Keyword
F02_V01 = 'SCI'                / Value
F02_C01 = 'extension name'     / Comment
F01_V02 = 'hst_skycell-p0081x14y15_acs_wfc_f658n_all_drz.fits' / Value
F01_C02 = 'Original image'     / Comment
F02_K02 = 'ORIG_FLE'           / Keyword
F02_V02 = 'hst_skycell-p0081x14y14_acs_wfc_f658n_all_drc.fits' / Value
F02_C02 = 'Original image'     / Comment
EXTNAME = 'SCI'

and similarly for the second or combined WHT extension.

A Full Worked Example

In the example below, the ellipses (”…”) are used to indicate the omission of output similar to what has already been documented for the reader. Note that as of this writing, there are actually no MVM files in MAST upon which to perform a query or request a retrieval so details on output messages may vary from the true output.

>>> from drizzlepac.haputils import hapcut_utils as hu
>>> from astropy.coordinates import SkyCoord
>>> query_table = hu.mvm_id_filename(SkyCoord(84.9208, -69.1467, unit="deg"), [256, 256])
2021295152553 INFO src=hapcut- Radius for query: 182.0 arcsec.
2021295152553 INFO src=hapcut- Performing query for ACS images.
2021295152621 INFO src=hapcut- Performing query for WFC3 images.
2021295152646 INFO src=hapcut- Get the product list for all entries in the query table.
2021295152646 INFO src=hapcut- Filter the product list table for only ['DRZ', 'DRC'] filenames.
2021295152646 INFO src=hapcut- Writing out the MVM product list table to mvm_query-ra84d9207-decs69d1467_182_cutout.ecsv.
2021295152646 INFO src=hapcut- Number of entries in table: 26.

The mvm_retrieve_files simply gets the requested files, in this case every file in the query_table, from MAST unless the files are already found to be resident on disk. It should be noted any individual MVM file could be rather large and unwieldy.

>>> requested_files = hu.mvm_retrieve_files(query_table)

The make_the_cut function loops over the input list of files and tries to make the requested cutout. If a cutout cannot be made, an error message will be generated, and the function will continue processing. All cutout files have the prefix hst_cutout.

>>> cutout_files = hu.make_the_cut(requested_files, SkyCoord(84.9208, -69.1467, unit="deg"), [256, 256])
Number of input files: 1
Cutting out [1, 2] extension(s)
Center coordinate: 84.9208 -69.1467 deg
Cutout size: [256. 256.] arcsec

Cutting out hst_skycell-p0081x14y14_acs_wfc_f658n_all_drc.fits
Original image shape: (21953, 11503)
xmin,xmax: [ 8093 14493]
ymin,ymax: [18554 24954]
Image cutout shape: (6400, 6400)
Original image shape: (21953, 11503)
xmin,xmax: [ 8093 14493]
ymin,ymax: [18554 24954]
Image cutout shape: (6400, 6400)
Returning cutouts as individual FITS
Total time: 4.3 sec

...

2021295103421 INFO src=hapcut- Cutout FITS filename: ./hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_wfc3_ir_f110w_coarse-iby03fg6.fits
2021295103422 INFO src=hapcut- Cutout FITS filename: ./hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f658n.fits

The mvm_combine function loops over the input list of cutout files and organizes the data according to instrument/detector and filter for filter-level combinations. The data is organized by instrument/detector, filter, and ipppssoo for exposure-level combinations. Images in the input list which do not share an instrument/detector and filter with any other image will be ignored. Individual exposures from a single skycell will also be ignored.

>>> hu.mvm_combine(cutout_files)
2021295104231 INFO src=hapcut- Input cutout files:
2021295104231 INFO src=hapcut- File: hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f555w-jby001r0.fits
2021295104231 INFO src=hapcut- File: hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f555w.fits
2021295104231 INFO src=hapcut- File: hst_cutout_skycell-p0081x14y15-ra84d9207-decs69d8533_acs_wfc_f814w-jbp505jg.fits
2021295104231 INFO src=hapcut- File: hst_cutout_skycell-p0081x14y15-ra84d9207-decs69d8533_acs_wfc_f814w.fits

...

2021295104231 INFO src=hapcut- === Combining filter-level files ===
2021295104231 WARNING src=hapcut- There is only one file for this detector/filter[/ipppssoo] combination, so there is nothing to combine.
2021295104231 WARNING src=hapcut- File ['hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f555w.fits'] will be ignored for combination purposes.

2021295104231 WARNING src=hapcut- There is only one file for this detector/filter[/ipppssoo] combination, so there is nothing to combine.
2021295104231 WARNING src=hapcut- File ['hst_cutout_skycell-p0081x14y15-ra84d9207-decs69d8533_acs_wfc_f814w.fits'] will be ignored for combination purposes.

2021295104231 INFO src=hapcut- Combining the SCI and then the WHT extensions of the input cutout files.
2021295104237 INFO src=hapcut- The combined output filename is ./hst_combined_skycells-ra84d9207-decs69d8533_acs_wfc_f658n.fits.

...

2021295104242 INFO src=hapcut-
2021295104242 INFO src=hapcut- === Combining exposure-level files ===
2021295104242 WARNING src=hapcut- There is only one file for this detector/filter[/ipppssoo] combination, so there is nothing to combine.
2021295104242 WARNING src=hapcut- File ['hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f555w-jby001r0.fits'] will be ignored for combination purposes.

...

2021295104242 INFO src=hapcut- Combining the SCI and then the WHT extensions of the input cutout files.
2021295104246 INFO src=hapcut- The combined output filename is ./hst_combined_skycells-ra84d9207-decs69d8533_wfc3_ir_f110w_iby03fg4.fits.

...

2021295104255 INFO src=hapcut- Cutout combination is done.
_images/SkyCell14_14.png

This cutout is hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f658n.fits scaled in size by 75%. The region in common with skycell-p0081x14y15 is in the upper-right corner.

_images/SkyCell14_15.png

This cutout is hst_cutout_skycell-p0081x14y15-ra84d9207-decs69d8533_acs_wfc_f658n.fits scaled in size by 75%. The region in common with skycell-p0081x14y14 is in the lower-right corner.

_images/SkyCellCombo.png

This figure is the combination of the two cutouts scaled in size by 75%.

_images/ZoomSkyCellCombo.png

Finally, this figure is the combination of the two cutouts scaled in size by 75% and zoomed in to the overlap region.