Skip to content

Conversation

@braingram
Copy link
Collaborator

@braingram braingram commented Jul 20, 2023

The test_flatfield_step_interface was unskipped in #7615 right before the frequent but unpredictable failures for FlatField.

This test modifies FlatFieldStep.reference_file_types (and does not undo the modification). Subsequent tests (in the same process) that use FlatField may fail (and not consider f/s/dflats) which would explain the errors seen in regression tests following the merge of the above changes.

This minimal regression test run (2 tests in one process) will hopefully help to explain the error:
https://plwishmaster.stsci.edu:8081/blue/organizations/jenkins/RT%2FJWST-Developers-Pull-Requests/detail/JWST-Developers-Pull-Requests/799/tests
The order of tests was controlled so that test_flatfield_step_interface was called prior to test_nis_wfss_spec2 (which fails with the expected error).

This PR modifies the test to save, modify then restore FlatFieldStep.reference_file_types so that it doesn't interfere with other tests.

Checklist for maintainers

  • added entry in CHANGES.rst within the relevant release section
  • updated or added relevant tests
  • updated relevant documentation
  • added relevant milestone
  • added relevant label(s)
  • ran regression tests, post a link to the Jenkins job below.
    How to run regression tests on a PR
  • Make sure the JIRA ticket is resolved properly

@braingram
Copy link
Collaborator Author

braingram commented Jul 20, 2023

@braingram braingram marked this pull request as ready for review July 20, 2023 23:06
@braingram braingram requested a review from a team as a code owner July 20, 2023 23:06
@codecov
Copy link

codecov bot commented Jul 20, 2023

Codecov Report

Patch and project coverage have no change.

Comparison is base (d1a60ee) 76.60% compared to head (c021c8e) 76.60%.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #7752   +/-   ##
=======================================
  Coverage   76.60%   76.60%           
=======================================
  Files         456      456           
  Lines       36950    36950           
=======================================
+ Hits        28304    28305    +1     
+ Misses       8646     8645    -1     
Flag Coverage Δ *Carryforward flag
nightly 77.43% <ø> (ø) Carriedforward from d1a60ee

*This pull request uses carry forward flags. Click here to find out more.

see 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@braingram braingram marked this pull request as draft July 21, 2023 17:37
@braingram braingram changed the title re-skip test that modifies FlatField step Modify test_flatfield_step_interface step to not interfere with other tests Jul 21, 2023
@braingram braingram marked this pull request as ready for review July 21, 2023 17:42
Copy link
Collaborator

@nden nden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😱
Good catch!

@braingram
Copy link
Collaborator Author

Thanks for the review!

It looks like I'm not authorized to merge this. The regression tests are almost done (stuck on the last 98%) but should finish in about 30 minutes (and so far no errors have been hit).

I did run these tests with --durations=10 to see the names of the slowest tests. Based on previous runs I expect it's one of these (based off their previous runtimes) which have not so far shown the d/s/fflat errors.

1hr9min [stable-deps] test_spec3_multi[jw01249005001_03101_00001_nrs1_o005_crf.fits] – jwst.regtest.test_nirspec_ifu_spec3
1hr [stable-deps] test_residual_fringe_cal – jwst.regtest.test_miri_residual_fringe 

Copy link
Collaborator

@hbushouse hbushouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works! YAY!!!

@hbushouse hbushouse merged commit e840dfc into spacetelescope:master Jul 24, 2023
@braingram braingram deleted the skip_flat_test branch July 24, 2023 12:55
mairanteodoro pushed a commit to mairanteodoro/jwst that referenced this pull request Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants