JP-3775: Expand usage of resource_tracker for benchmarking

[tapastro]

Resolves JP-3775

This PR expands coverage of resource tracking to more regression tests.

Tasks

• If you have a specific reviewer in mind, tag them.
• add a build milestone, i.e. Build 12.0 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.93%. Comparing base (5973747) to head (b2d3e3d).
⚠️ Report is 678 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9433      +/-   ##
==========================================
+ Coverage   77.92%   77.93%   +0.01%     
==========================================
  Files         362      362              
  Lines       36283    36252      -31     
==========================================
- Hits        28272    28253      -19     
+ Misses       8011     7999      -12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[emolter]

Maybe you already plan to do this, but could you add one that covers calwebb_ami3 also? It would help track e.g. JP-3776

[tapastro]

Maybe you already plan to do this, but could you add one that covers calwebb_ami3 also? It would help track e.g. JP-3776

Done - coverage is effectively complete now. I'll pull this out of draft as review-ready.

Latest RT run here: https://github.com/spacetelescope/RegressionTests/actions/runs/14888971507

[emolter]

I'm a bit puzzled by the summary of the regression test. I only see the output of jwst.regtest.test_nirspec_mos_spec3::test_log_tracked_resources. Do these only appear in the summary if they have changed by some threshold?

[tapastro]

I'm a bit puzzled by the summary of the regression test. I only see the output of jwst.regtest.test_nirspec_mos_spec3::test_log_tracked_resources. Do these only appear in the summary if they have changed by some threshold?

My understanding is that entries will only exist in the RT summary for tests that have an existing value on main - that excludes all tests added in this PR. Of the existing tests on main (~five, added in #9357 ), it appears that the MOS spec3 test went above the threshold set in the workflow: https://github.com/spacetelescope/RegressionTests/blob/01de5ada29e4e8914902576f8ea9558b37b4b494/.github/workflows/test.yml#L465

Interestingly, in the set of RTs I ran before the last set of commits, that test showed marginally higher memory usage - https://github.com/spacetelescope/RegressionTests/actions/runs/14886917550

That test appears to have some variability in memory usage.

[emolter]

Ok that makes sense - it isn't displayed because there's nothing to compare with on main.

That test appears to have some variability in memory usage.

It would be nice if we didn't see the report in this case, if this variability is benign. Is there a way to either make the test not change memory usage (I myself have no clue why it would be happening in the first place) or change the tolerance such that we don't see this? Would it help to make the tolerance relative, rather than an absolute number of megabytes?

[tapastro]

Ok that makes sense - it isn't displayed because there's nothing to compare with on main.

That test appears to have some variability in memory usage.

It would be nice if we didn't see the report in this case, if this variability is benign. Is there a way to either make the test not change memory usage (I myself have no clue why it would be happening in the first place) or change the tolerance such that we don't see this? Would it help to make the tolerance relative, rather than an absolute number of megabytes?

Yes, I think a tolerance increase will be a necessary next step. We can probably let the new set of tests inform the update as well. I'm not opposed to a relative tolerance, though I a flat level of 10-100 MB may also be feasible. Curious to see how these tests behave
.

[emolter]

Overall LGTM. I had one comment about perhaps splitting a fixture into spec2 and spec3 components. It's not possible to see what the outputs will look like until merging, but I assume they'll be mostly fine since you've already made a proof-of-concept

[braingram]

FWIW: the resource measurements can be seen from the runs by looking at the results.xml directly.

For https://github.com/spacetelescope/RegressionTests/actions/runs/14886917550 near the bottom of the summary is a link in the "Artifacts" section for results-jwst-1.16.0-1576-gf6ba9c8ab-Linux-X64-py3.12-1947.xml.

Do these look reasonable?

Test	mem	time
jwst.regtest.test_miri_coron3.test_log_tracked_resources_coron3	532	226.02
jwst.regtest.test_miri_coron_image2.test_log_tracked_resources_image2	107	10.72
jwst.regtest.test_miri_image.test_log_tracked_resources_det1	12265	220.57
jwst.regtest.test_miri_image.test_log_tracked_resources_image2	313	25.05
jwst.regtest.test_miri_image.test_log_tracked_resources_image3	306	64.62
jwst.regtest.test_miri_lrs_slit_spec2.test_log_tracked_resources_spec2	395	33.86
jwst.regtest.test_miri_lrs_slit_spec3.test_log_tracked_resources_spec3[default_wcs]	297	33.89
jwst.regtest.test_miri_lrs_slit_spec3.test_log_tracked_resources_spec3[user_wcs+shape1]	297	32.38
jwst.regtest.test_miri_lrs_slit_spec3.test_log_tracked_resources_spec3[user_wcs+shape]	297	31.47
jwst.regtest.test_miri_lrs_slit_spec3.test_log_tracked_resources_spec3[user_wcs]	297	32.00
jwst.regtest.test_miri_lrs_slitless.test_log_tracked_resources_spec2	66	55.07
jwst.regtest.test_miri_lrs_slitless.test_log_tracked_resources_spec3	35	48.80
jwst.regtest.test_miri_mrs_spec2.test_log_tracked_resources_spec2	1224	495.73
jwst.regtest.test_miri_mrs_spec3.test_log_tracked_resources_spec3long	1565	712.40
jwst.regtest.test_miri_mrs_spec3.test_log_tracked_resources_spec3short	1841	901.23
jwst.regtest.test_miri_mrs_tso.test_log_tracked_resources_spec2	3251	102.06
jwst.regtest.test_nircam_coron3.test_log_tracked_resources	403	66.26
jwst.regtest.test_nircam_dhs.test_log_tracked_resources_det1	939	28.72
jwst.regtest.test_nircam_image.test_log_tracked_resources_detector1	7773	81.94
jwst.regtest.test_nircam_image.test_log_tracked_resources_image2	1176	23.99
jwst.regtest.test_nircam_image.test_log_tracked_resources_image3	1608	211.36
jwst.regtest.test_nircam_tsgrism.test_log_tracked_resources_pipelines	13434	294.83
jwst.regtest.test_nircam_tsimg.test_log_tracked_resources_tsimg	225	6.28
jwst.regtest.test_nircam_wfss_spec2.test_log_tracked_resources_spec2	619	356.18
jwst.regtest.test_niriss_ami3.test_log_tracked_resources_ami3	1199	748.23
jwst.regtest.test_niriss_image.test_log_tracked_resources_det1_cfn	10792	332.95
jwst.regtest.test_niriss_image.test_log_tracked_resources_det1_mp	11638	270.77
jwst.regtest.test_niriss_soss.test_log_tracked_resources_spec2	1627	447.87
jwst.regtest.test_niriss_soss.test_log_tracked_resources_spec3	1314	865.03
jwst.regtest.test_niriss_wfss.test_log_tracked_resources_spec2	546	164.22
jwst.regtest.test_niriss_wfss.test_log_tracked_resources_spec3	125	160.66
jwst.regtest.test_nirspec_brightobj.test_log_tracked_resources_spec2	893	118.84
jwst.regtest.test_nirspec_fs_spec2.test_log_tracked_resources_spec2[jw01245-o002_20230107t223023_spec2_00001_asn.json]	842	52.56
jwst.regtest.test_nirspec_fs_spec2.test_log_tracked_resources_spec2[jw01309-o022_20230113t025924_spec2_00001_asn.json]	971	178.52
jwst.regtest.test_nirspec_fs_spec2.test_log_tracked_resources_spec2[jw02072-o002_20221206t143745_spec2_00001_asn.json]	1494	196.17
jwst.regtest.test_nirspec_fs_spec3.test_log_tracked_resources_spec3	178	258.68
jwst.regtest.test_nirspec_ifu_spec2.test_log_tracked_resources_spec2	2483	893.28
jwst.regtest.test_nirspec_ifu_spec3.test_log_tracked_resources_spec3	4753	2277.70
jwst.regtest.test_nirspec_irs2_detector1.test_log_tracked_resources_det1	11389	499.29
jwst.regtest.test_nirspec_mos_fs_spec2.test_log_tracked_resources_spec2	1467	236.24
jwst.regtest.test_nirspec_mos_spec2.test_log_tracked_resources	1729	329.47
jwst.regtest.test_nirspec_mos_spec3.test_log_tracked_resources	200	407.66

[emolter]

Thanks Brett that is very helpful. If the runtime changes substantially, will that also cause the line to appear in the summary? Or only the memory usage?

[braingram]

The runtime is highly variable (in part because the tests run in parallel) so it's not shown but is recorded in the results.xml (in case it's useful).

[emolter]

I figured that was probably the case, thanks

[tapastro]

Fresh RTs just in case: https://github.com/spacetelescope/RegressionTests/actions/runs/14911058249

[emolter]

LGTM pending clean regtest run

[tapastro]

Regtests were clean; merging. I'll run another set immediately after for a sonarscan smoke test.