JP-3982: Ignore fitsdiffs for atoca spectra

[melanieclarke]

Resolves JP-3982

The auxiliary output from ATOCA appears to be radically different in different build environments - the fit diverges wildly in unpredictable places for Mac vs Linux, different numpy versions, etc. I think it's not reasonable to do a fitsdiff at all on this product. This PR changes the regtest to just check for existence of the output file instead of checking the contents.

Merging this should fix the last of the regtest failures with devdeps.

Tasks

• If you have a specific reviewer in mind, tag them.
• add a build milestone, i.e. Build 12.0 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 75.39%. Comparing base (5bc305b) to head (1c93ca5).
Report is 660 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #9388   +/-   ##
=======================================
  Coverage   75.39%   75.39%           
=======================================
  Files         368      368           
  Lines       36842    36842           
=======================================
  Hits        27777    27777           
  Misses       9065     9065           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[melanieclarke]

Regression test run for just the ATOCA extras tests is here (attempt 6 for current version of tests):
https://github.com/spacetelescope/RegressionTests/actions/runs/14365371034

Full regtests here:
https://github.com/spacetelescope/RegressionTests/actions/runs/14383695768

[pllim]

For just Ubuntu, this was passing on stable deps but failing on devdeps, right? Or is it something else now? If former, I guess we have no choice.

[melanieclarke]

It is still passing on stable Linux, but failing on devdeps and failing on my local Mac. The diffs are reasonable for some of the well-formed spectra contained in the file, but entirely unreasonable for some bad pixels and some empty spectra -- like many orders of magnitude relative difference. I'm not super familiar with this algorithm, but it looks to me like there is a diverging fit somewhere that is responding drastically to minor numerical changes.

I tried to see if there was a way I could diff just the reasonable extensions in the file, but there is no way to do that in the FITSDiff interface, because all the extensions have the same name.

In the end, this is just an auxiliary product that probably doesn't see much use, which is why I proposed just not diffing the product for now. If interest picks up in making this product stable and useful, we should revisit the test.

[pllim]

I don't know what Atoca is but it is sure not Ibiza. Yes better to get devdeps green again for other work for now. I'll let you all decide if a follow up issue is necessary lest we forgot about this. Thanks!