JP-4047: Update regtests for niriss and nircam wfss

[emolter]

Resolves JP-4047

Closes #9596

This PR:

• fixes a bug where casting the DQ array to float was causing bad behavior where that array was not defined: zero-padding on Mac and padding with the max value of a uint32 on Linux, when it should be 1-padded to represent DO NOT USE.
• adds regression test coverage of level 3 products for NIRCam WFSS.
• updates the association requested by the regression test for level3 NIRISS WFSS to be reflective of the current asn rules for that mode, including Use actual target id instead of placeholder for WFSS level 3 associations #9614

Once merged, the following files should be removed from Artifactory:

• truth/test_niriss_wfss/*gr150c-gr150r*
• niriss/wfss/jw01324-o001_20220629t171902_spec3_003_asn.json
• niriss/wfss/jw01324-o001_20220629t171902_spec2_002_asn.json
• jw01324001001_03101_00002_nis_rate.fits

Tasks

• If you have a specific reviewer in mind, tag them.
• add a build milestone, i.e. Build 12.0 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.47%. Comparing base (d4724c0) to head (83eabc8).
⚠️ Report is 418 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9602      +/-   ##
==========================================
- Coverage   80.66%   79.47%   -1.20%     
==========================================
  Files         368      368              
  Lines       37294    37224      -70     
==========================================
- Hits        30085    29585     -500     
- Misses       7209     7639     +430     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[emolter]

https://github.com/spacetelescope/RegressionTests/actions/runs/15887901780

Small regression test differences are due to Mac vs Linux - they pass on my Mac machine. These can be ok-ified after merge.

Regression test differences were due to Mac vs Linux, but as @tapastro spotted, they were hiding a bug in casting between data types. This should be fixed in 74e3e05

[emolter]

retrying regtests here https://github.com/spacetelescope/RegressionTests/actions/runs/16055159745

I found a simpler and more robust fix that takes the defaults from the schema defaults, e.g. here. This way of doing it makes it so no special handling of different column datatypes is needed, and no hard-coding of default values has to be done for those dtypes. Regression tests restarted here:
https://github.com/spacetelescope/RegressionTests/actions/runs/16056215256

There's a good chance this may also affect the TSO multispec products, since the fix was applied to a shared utility. The new behavior should be appropriate for those products too, but with a caveat: the default DQ column for TSO multispec data was set to 0 instead of 1. It makes more sense for the default to be 1, i.e. DO_NOT_USE, since all the other column defaults are NaN. PR has been submitted here to make that one-liner change, and the regtests above are pointing to that stdatamodels branch.

Obviously we should be careful evaluating the differences with TSO products, and I'll take a look at them locally, but in principle they can be ok-ified.

Suggested workflow would be to merge the stdatamodels change first, as I don't expect any regtest failures from that one, and then to re-run the jwst regtests to get a cleanly okify-able run (although those failures should be the same as from the run above).

[emolter]

This did NOT affect the TSO regtests. I'm still working on exactly why: it may be that the regtest data has only spectra of the same length so this is not touched. But in any case this simplifies things; we don't need to make any stdatamodels default value changes before this can be merged.

The WFSS spec2 x1d regtest failures from https://github.com/spacetelescope/RegressionTests/actions/runs/16056215256 look as expected. That regtest was pinned to spacetelescope/stdatamodels#522 branch, but the change therein should not have made any difference.

Nevertheless, starting a (hopefully final) regtest, this one from stdatamodels main

https://github.com/spacetelescope/RegressionTests/actions/runs/16057873804

[melanieclarke]

the regtest data has only spectra of the same length

This is expected for TSO data - all integrations in an extension have the same wavelength solution. We currently keep spectra from different detectors, which would have different spectral lengths, in separate extensions. I think the spectral padding for TSO data should still be available as a fallback, in case this convention changes in the future, but it should never be used under current assumptions.

[tapastro]

Tests look good, RTs show expected difference from update to default DQ handling.

[tapastro]

@meeseeksdev backport to release/1.19.x