Backport PR #9501 on branch release/1.18.x (JP-3116: Use duplication checking to prevent undesirable associations in DMS operations)

[zacharyburnett]

backport #9501

regtest: https://github.com/spacetelescope/RegressionTests/actions/runs/15540530462/job/43750158547

[codecov]

Codecov Report

❌ Patch coverage is 41.17647% with 10 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release/1.18.x@1b78507). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...t/associations/generator/generate_per_candidate.py	41.17%	10 Missing ⚠️

Additional details and impacted files

@@                Coverage Diff                @@
##             release/1.18.x    #9529   +/-   ##
=================================================
  Coverage                  ?   74.61%           
=================================================
  Files                     ?      369           
  Lines                     ?    37154           
  Branches                  ?        0           
=================================================
  Hits                      ?    27722           
  Misses                    ?     9432           
  Partials                  ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tapastro]

This backport accumulated some extraneous changes - the regtest file diff is mostly a refactor introduced by https://github.com/spacetelescope/jwst/pull/9470/files

I think it'd be best to revert changes made in that file, though I'm sure the STASIS tests will be difficult. I can make a PR/commit to your backport PR branch with suggested changes.

[tapastro]

My recollection of the expected diffs with 1.18.0 artifacts led me to add a single asn to the truth output for the jw04225{...}DMS_pool test - hopefully when run against the 1.18.0 artifacts this will come back clean 🤞

[zacharyburnett]

new regtest against 1.18.0 and jwst_1364.pmap: https://github.com/spacetelescope/RegressionTests/actions/runs/15542486090/job/43757128635

[zacharyburnett]

the only test NEWLY failing (not also failing on main) is jwst.regtest.test_nircam_image::test_nircam_image_detector1_with_clean_flicker_noise[flicker_noise]:
https://github.com/spacetelescope/RegressionTests/actions/runs/15542486090/attempts/2#summary-43757128635