MIRI MRS extract1d reference file format is not JSON

[stscijirahub]

Issue JP-4195 was created on JIRA by Taylor Bell:

If a user tries to run calwebb_tso3 with MIRI/MRS TSO data, they get an error message saying "RuntimeError: Invalid extract1d reference file: must be JSON." (the full traceback is copied below). After a bit of digging, I found that it is because the MIRI/MRS extract1d reference file is jwst_miri_extract1d_0004.asdf which is indeed not in a JSON file format.

While calwebb_tso3 processing of MRS data isn't supported at present, this still raises a couple of questions/concerns in my mind. Ideally, trying to run calwebb_tso3 with unsupported modes would raise an informative warning/error saying that the mode can't be processed or something. But also, it is strange to me that the extract_1d step would exclusively allow JSON reference files while a small number of modes (including MIRI MRS) do not have JSON extract1d reference files; it seems to me that either the extract_1d step should allow ASDF reference files or those modes with non-JSON reference files should switch to JSON file format.

 

Full error traceback:

 

RuntimeError                              Traceback (most recent call last)
Cell In[1], line 5
      3 pipe = Tso3Pipeline()
      4 pipe.output_dir = "./tso3"
----> 5 pipe.run("./jw01633-o001_20251201_tso3_00001_asn.json")
 
File ~/miniconda3/envs/jwst_v1.20.2/lib/python3.12/site-packages/jwst/stpipe/core.py:299, in JwstStep.run(self, *args, **kwargs)
    282 @wraps(_Step.run)
    283 def run(self, *args, **kwargs):
    284     """
    285     Run the step.
    286
   (...)    297         The step output
    298     """
--> 299     result = super().run(*args, **kwargs)
    300     if not self.parent:
    301         log.info(f"Results used jwst version: {__version__}")
 
File ~/miniconda3/envs/jwst_v1.20.2/lib/python3.12/site-packages/stpipe/step.py:594, in Step.run(self, *args)
    592     self.prefetch(*args)
    593 try:
--> 594     step_result = self.process(*args)
    595 except TypeError as e:
    596     if "process() takes exactly" in str(e):
 
File ~/miniconda3/envs/jwst_v1.20.2/lib/python3.12/site-packages/jwst/pipeline/calwebb_tso3.py:144, in Tso3Pipeline.process(self, input_data)
    141 # Process spectroscopic TSO data
    142 # extract 1D
    143 log.info("Extracting 1-D spectra ...")
--> 144 result = self.extract_1d.run(cube)
    145 for row in cube.int_times:
    146     # Subtract one to assign 1-indexed int_nums to int_times array locations
    147     x1d_result.int_times[row[0] - 1] = row
 
File ~/miniconda3/envs/jwst_v1.20.2/lib/python3.12/site-packages/jwst/stpipe/core.py:299, in JwstStep.run(self, *args, **kwargs)
    282 @wraps(_Step.run)
    283 def run(self, *args, **kwargs):
    284     """
    285     Run the step.
    286
   (...)    297         The step output
    298     """
--> 299     result = super().run(*args, **kwargs)
    300     if not self.parent:
    301         log.info(f"Results used jwst version: {__version__}")
 
File ~/miniconda3/envs/jwst_v1.20.2/lib/python3.12/site-packages/stpipe/step.py:594, in Step.run(self, *args)
    592     self.prefetch(*args)
    593 try:
--> 594     step_result = self.process(*args)
    595 except TypeError as e:
    596     if "process() takes exactly" in str(e):
 
File ~/miniconda3/envs/jwst_v1.20.2/lib/python3.12/site-packages/jwst/extract_1d/extract_1d_step.py:433, in Extract1dStep.process(self, input_data)
    430     extracted = self._extract_ifu(model, exp_type, extract_ref, apcorr_ref)
    431 else:
    432     # Call the general extraction routine
--> 433     extracted, profile, scene_model, residual = extract.run_extract1d(
    434         model,
    435         extract_ref,
    436         apcorr_ref,
    437         psf_ref,
    438         self.extraction_type,
    439         self.smoothing_length,
    440         self.bkg_fit,
    441         self.bkg_order,
    442         self.log_increment,
    443         self.subtract_background,
    444         self.use_source_posn,
    445         self.position_offset,
    446         self.model_nod_pair,
    447         self.optimize_psf_location,
    448         self.save_profile,
    449         self.save_scene_model,
    450         self.save_residual_image,
    451     )
    453 # Set the step flag to complete in each model
    454 extracted.meta.cal_step.extract_1d = "COMPLETE"
 
File ~/miniconda3/envs/jwst_v1.20.2/lib/python3.12/site-packages/jwst/extract_1d/extract.py:2115, in run_extract1d(input_model, extract_ref_name, apcorr_ref_name, psf_ref_name, extraction_type, smoothing_length, bkg_fit, bkg_order, log_increment, subtract_background, use_source_posn, position_offset, model_nod_pair, optimize_psf_location, save_profile, save_scene_model, save_residual_image)
   2112 exp_type = meta_source.meta.exposure.type
   2114 # Read in the extract1d reference file.
-> 2115 extract_ref_dict = read_extract1d_ref(extract_ref_name)
   2117 # Check for non-null PSF reference file
   2118 if psf_ref_name == "N/A":
 
File ~/miniconda3/envs/jwst_v1.20.2/lib/python3.12/site-packages/jwst/extract_1d/extract.py:122, in read_extract1d_ref(refname)
    120     else:
    121         log.error("Invalid Extract1d reference file: must be JSON.")
--> 122         raise RuntimeError("Invalid extract1d reference file: must be JSON.")
    124 return ref_dict
 
RuntimeError: Invalid extract1d reference file: must be JSON.

 

Sorry if you're not the right person to assign Melanie Clarke - it wouldn't allow me to use the Automatic assignee and I also couldn't assign David Law who was my first guess as to who to assign

[stscijirahub]

Comment by Melanie Clarke on JIRA:

The error message could definitely be clearer, but failures are expected either way, since MRS data is not supported in tso3.

The ASDF reference file for MRS is used for extraction from resampled IFU cubes, which is the only extraction type currently supported for IFU data. Since the tso3 pipeline does not run cube_build, it can't support IFU spectral extraction.

The error message you're seeing is because the extract_1d step is checking the input model data type, seeing that it's not an IFU cube, and attempting standard spectral extraction from a 2D image instead, which requires a JSON-type reference file. I'm curious if you might be able to convince the step to do some form of extraction from an IFU calints file with a manually created override reference file in the JSON format, but that's definitely an off-nominal use, not yet supported.

See the docs here for more information on the two extract1d reference file styles:
https://jwst-pipeline.readthedocs.io/en/latest/jwst/extract_1d/reference_files.html#extract1d-reference-file

[stscijirahub]

Comment by Melanie Clarke on JIRA:

Following up out of curiosity, it looks like I can get tso3 to run on MIRI MRS data with a little effort. Using calints from MAST for jw1633-o001, I had to manually set the PIXAR_SR keyword (meta.photometry.pixelarea_steradians in the datamodel) to 1.0. Since it’s set to 0.0 currently, it will zero out all extracted flux otherwise. Then, run tso3 as:

strun calwebb_tso3 jw01633-o001_20251201_tso3_00001_asn.json
--steps.extract_1d.override_extract1d=N/A --steps.extract_1d.apply_apcorr=False

This will skip aperture correction and override the ASDF extract1d default file for MIRI MRS with no reference file at all, which causes it to extract all data with valid wavelengths and values, collapsed over columns. The resulting spectra are a mess, but the whitelight curve at the end does at least show the transit. This is obviously not recommended for science, but maybe would be useful for quicklook.

I did also try making a JSON reference file to extract a set aperture from each slice. This works for making more sensible spectra, but I think it's only possible to define one slice aperture at a time, so you'd have to either pick your favorite slice or process each file multiple times to extract all the relevant data.