Utilities for Manipulating MVM Products
hapcut_utils.py module contains utilities which are intended to be used with
Multi-visit Mosaic (MVM) data and to be imported into other Python programs or
a Python session for interactive data analysis. The utilities are wrappers
around lower-level functionality which actually perform the core tasks.
The core tasks referenced here are astroquery,
used for MAST inquiries and retrievals,
used to create the cutouts and cutout combinations.
hapcut_utils.py role is to customize the use of these core
tasks specifically for working with MVM products which can be large and unwieldy.
At this time there are four utilities which are designed to be invoked in
sequence as is evident from their function signatures, return values, and output
A brief overview of the utilities is described in this list.
mvm_id_filenames: Query MAST to identify all MVM files which overlap the specified cutout region on the sky
mvm_retrieve_files: Download all requested MVM files from MAST to the working directory
make_the_cut: Extract cutouts from all specified MVM files
mvm_combine: Create a mosaic from the specified cutouts that span multiple SkyCells and fit the qualifications
MVM Cutout Utilities
This utility is invoked by providing sky coordinates and a cutout box size. The sky coordinates represent the central location about which the search for data of interest is to be done. The cutout box size represents the search box extent encompassing the sky coordinates. The utility retrieves from MAST a table containing MVM filenames for ACS and WFC3 detectors, as well as other relevant image specific information, one row per filename. While the utility returns an astropy table object, an ascii ECSV file is also written to disk with the output filename being constructed from the query coordinates and box size parameters. The output filename is of the form:
dec are the right ascension and
declination, respectively, in decimal degrees with
<n|s> representing north or
south of the celestial equator. The
<radius> is the computed diagonal
of the cutout box which is basically the square root of the sum of the box sides
squared divided by two.
The in-memory table or the ascii file can be refined by the user to suit specific needs. The MVM filenames within the query file are self-descriptive. The filter-level drizzled filename is of the form:
-p is followed by the four digit projection cell value, and
are each followed by a two digit sky cell value. The possible
at this time are
all indicates all exposures while
coarse-all indicates all the exposures at a coarse platescale
(i.e., the WFC3/IR platescale of 0.12”/pixel) in the visit were used in the
generation of the product.
While the standard pipeline processing does not generate MVM exposure-level, drizzled products, the user can activate this capability when performing custom processing. As such, the MAST query will not contain any exposure-level drizzled product filenames. However, an example of an exposure-level drizzled filename created by custom processing would be of the form:
An example of a real MVM filter-level, drizzled filename is:
This utility is designed to use the in-memory table generated by
and retrieve any file listed in the table from MAST. Since the images can be quite
clobber parameter is set to False by default, indicating
no download is done for a filename that already exists on disk. The utility returns
a Python list of filenames on disk available for further analysis.
make_the_cut utility will perform the FITS cutouts on the input MVM
filter- and exposure-level images. This MVM customized utility is layered
fits_cut, a function which is part of Astrocut, a Python package for
making astronomical cutouts (https://astrocut.readthedocs.io/en/latest/). The
make_the_cut function is invoked
by providing the list of files on disk for which one wants to make cutouts, as
well as the sky coordinates, and the cutout box size. Ideally, the
sky coordinates and cutout box size are the same values used for the
mvm_id_filenames utility. For each input file designated for a cutout,
there will be a corresponding output file. Since both the SCI and WHT
extensions of the input files are actually cut, individual output cutout FITS
files will contain two image extensions, a SCI followed by a WHT. The
cutouts are explicitly generated by looping over the input file list in case one
of the files experiences an exception so the remaining files in the list still
have the opportunity to create a cutout.
Each filter-level output filename will be of the form:
Each exposure-level filename generated by custom user processing will be of the form:
The components of the output filename follow the pattern discussed for
mvm_id_filenames. The additional
platescale component has
the possible value of
coarse. When there is no
platescale component present
in the filename, the
platescale has the implicit default value of
fine platescale is 0.04”/pixel and is the standardized platescale for all
MVM products. The platescale of
coarse is only applicable to the WFC3/IR data.
The primary header data unit (PHDU) from the input file is retained as the PHDU for the output cutout file with an updated FILENAME keyword and a few additional keywords. The new keywords RA_OBJ/DEC_OBJ were added by the Astrocut utility representing the right ascension and declination of the object in decimal degrees. Another new keyword is ORIG_FLE which represents the original input filename from which the cutout was extracted. In each output file, the image extensions have been explicitly named for for clarity.
>>> from astropy.io import fits >>> hdu= fits.open("hst_cutout_skycell-p0081x14y15-ra84d8208-decs69d8516_wfc3_uvis_f275w.fits") >>> hdu.info() Filename: hst_cutout_skycell-p0081x14y15-ra84d8208-decs69d8516_wfc3_uvis_f275w.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 872 () 1 SCI 1 ImageHDU 106 (12500, 12500) float32 2 WHT 1 ImageHDU 56 (12500, 12500) float32
The final MVM utility at this time is
mvm_combine and is designed to combine filter-level
cutout images originating from multiple skycells where all the images were acquired with the same
detector and using the same filter. The utility will also combine exposure-level cutout
images from multiple skycells where the images were acquired with the same
detector, using the same filter, and sharing the same ipppssoo and created by user custom
processing. Put more straightforwardly,
the cutout images must have originated from the same MVM drizzled image which spanned
the border between multiple skycells. Cutout images in the
input list which do not adhere to these noted requirements will trigger a message of
explanation and be ignored for the combination process.
mvm_combine utility is invoked by providing a list of cutout files, and the utility
will handle segregating the files into filter- and exposure-level sets and determining
whether or not the cutout file is eligible for the combination process. Once all the
input images have been prepared, the mvm_combine will call the Astrocut function
to handle the actual combination task. At this time, only the default
mean combiner is available for use where every output pixel is the mean of all input
pixels that have data at that point.
Since both the SCI and WHT extensions of the input files are combined, individual output combined FITS files will contain two image extensions, a SCI followed by a WHT. The utility writes to the specified output directory one combined file per set of compatible images. The output filenames are constructed from the RA, Dec, instrument, detector, filter, and exposure as appropriate. Each filter-level output filename will be of the form:
Each exposure-level filename where the cutout images were generated by custom user processing will be of the form:
The keyword, FILENAME, has been added to the PHDU to self-document the
name of the file.
The Extension Header Data Units (EHDUs) contain custom keywords which document the
filenames and extension names of the originating files. These custom keywords are
created by the
CutoutsCombiner and are of the form
Fnn_[K|V|C]mm representing the
keywords EXTNAMEs and ORIG_FLEs from the input cutout files. The
nn represents the
header number, and
mm represents a count of the custom keywords.
>>> hdu = fits.open("hst_combined_skycells-ra84d9402-decs69d8514_acs_wfc_f658n.fits") >>> hdu.info() Filename: hst_combined_skycells-ra84d9402-decs69d8514_acs_wfc_f658n.fits No. Name Ver Type Cards Dimensions Format 0 PRIMARY 1 PrimaryHDU 10 () 1 SCI 1 ImageHDU 167 (250, 250) float64 2 WHT 1 ImageHDU 70 (250, 250) float64 where the first extention or combined SCI extension header contains O_EXT_NM= 'SCI ' / Original extension name. ORIG_EXT= 1 / Extension in original file. F01_K01 = 'EXTNAME ' / Keyword F01_V01 = 'SCI' / Value F01_C01 = 'extension name' / Comment F02_K01 = 'EXTNAME ' / Keyword F02_V01 = 'SCI' / Value F02_C01 = 'extension name' / Comment F01_V02 = 'hst_skycell-p0081x14y15_acs_wfc_f658n_all_drz.fits' / Value F01_C02 = 'Original image' / Comment F02_K02 = 'ORIG_FLE' / Keyword F02_V02 = 'hst_skycell-p0081x14y14_acs_wfc_f658n_all_drc.fits' / Value F02_C02 = 'Original image' / Comment EXTNAME = 'SCI'
and similarly for the second or combined WHT extension.
A Full Worked Example
In the example below, the ellipses (”…”) are used to indicate the omission of output similar to what has already been documented for the reader. Note that as of this writing, there are actually no MVM files in MAST upon which to perform a query or request a retrieval so details on output messages may vary from the true output.
>>> from drizzlepac.haputils import hapcut_utils as hu >>> from astropy.coordinates import SkyCoord >>> query_table = hu.mvm_id_filename(SkyCoord(84.9208, -69.1467, unit="deg"), [256, 256]) 2021295152553 INFO src=hapcut- Radius for query: 182.0 arcsec. 2021295152553 INFO src=hapcut- Performing query for ACS images. 2021295152621 INFO src=hapcut- Performing query for WFC3 images. 2021295152646 INFO src=hapcut- Get the product list for all entries in the query table. 2021295152646 INFO src=hapcut- Filter the product list table for only ['DRZ', 'DRC'] filenames. 2021295152646 INFO src=hapcut- Writing out the MVM product list table to mvm_query-ra84d9207-decs69d1467_182_cutout.ecsv. 2021295152646 INFO src=hapcut- Number of entries in table: 26.
mvm_retrieve_files simply gets the requested files, in this case every file in the
query_table, from MAST unless the files are already found to be resident on disk. It should be
noted any individual MVM file could be rather large and unwieldy.
>>> requested_files = hu.mvm_retrieve_files(query_table)
make_the_cut function loops over the input list of files and tries to make the
requested cutout. If a cutout cannot be made, an error message will be generated, and
the function will continue processing. All cutout files have the prefix
>>> cutout_files = hu.make_the_cut(requested_files, SkyCoord(84.9208, -69.1467, unit="deg"), [256, 256]) Number of input files: 1 Cutting out [1, 2] extension(s) Center coordinate: 84.9208 -69.1467 deg Cutout size: [256. 256.] arcsec Cutting out hst_skycell-p0081x14y14_acs_wfc_f658n_all_drc.fits Original image shape: (21953, 11503) xmin,xmax: [ 8093 14493] ymin,ymax: [18554 24954] Image cutout shape: (6400, 6400) Original image shape: (21953, 11503) xmin,xmax: [ 8093 14493] ymin,ymax: [18554 24954] Image cutout shape: (6400, 6400) Returning cutouts as individual FITS Total time: 4.3 sec ... 2021295103421 INFO src=hapcut- Cutout FITS filename: ./hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_wfc3_ir_f110w_coarse-iby03fg6.fits 2021295103422 INFO src=hapcut- Cutout FITS filename: ./hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f658n.fits
mvm_combine function loops over the input list of cutout files and organizes the
data according to instrument/detector and filter for filter-level combinations. The data is
organized by instrument/detector, filter, and ipppssoo for exposure-level combinations.
Images in the input list which do not share an instrument/detector and filter with any other
image will be ignored. Individual exposures from a single skycell will also be ignored.
>>> hu.mvm_combine(cutout_files) 2021295104231 INFO src=hapcut- Input cutout files: 2021295104231 INFO src=hapcut- File: hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f555w-jby001r0.fits 2021295104231 INFO src=hapcut- File: hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f555w.fits 2021295104231 INFO src=hapcut- File: hst_cutout_skycell-p0081x14y15-ra84d9207-decs69d8533_acs_wfc_f814w-jbp505jg.fits 2021295104231 INFO src=hapcut- File: hst_cutout_skycell-p0081x14y15-ra84d9207-decs69d8533_acs_wfc_f814w.fits ... 2021295104231 INFO src=hapcut- === Combining filter-level files === 2021295104231 WARNING src=hapcut- There is only one file for this detector/filter[/ipppssoo] combination, so there is nothing to combine. 2021295104231 WARNING src=hapcut- File ['hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f555w.fits'] will be ignored for combination purposes. 2021295104231 WARNING src=hapcut- There is only one file for this detector/filter[/ipppssoo] combination, so there is nothing to combine. 2021295104231 WARNING src=hapcut- File ['hst_cutout_skycell-p0081x14y15-ra84d9207-decs69d8533_acs_wfc_f814w.fits'] will be ignored for combination purposes. 2021295104231 INFO src=hapcut- Combining the SCI and then the WHT extensions of the input cutout files. 2021295104237 INFO src=hapcut- The combined output filename is ./hst_combined_skycells-ra84d9207-decs69d8533_acs_wfc_f658n.fits. ... 2021295104242 INFO src=hapcut- 2021295104242 INFO src=hapcut- === Combining exposure-level files === 2021295104242 WARNING src=hapcut- There is only one file for this detector/filter[/ipppssoo] combination, so there is nothing to combine. 2021295104242 WARNING src=hapcut- File ['hst_cutout_skycell-p0081x14y14-ra84d9207-decs69d8533_acs_wfc_f555w-jby001r0.fits'] will be ignored for combination purposes. ... 2021295104242 INFO src=hapcut- Combining the SCI and then the WHT extensions of the input cutout files. 2021295104246 INFO src=hapcut- The combined output filename is ./hst_combined_skycells-ra84d9207-decs69d8533_wfc3_ir_f110w_iby03fg4.fits. ... 2021295104255 INFO src=hapcut- Cutout combination is done.