fix: Async OpenAI caching order and more common args

[kcz358]

Before you open a pull-request, please check if a similar issue already exists or has been closed before.

When you open a pull-request, please be sure to include the following

• A descriptive title: [xxx] XXXX
• A detailed description

If you meet the lint warnings, you can use following scripts to reformat code.

pip install pre-commit
pre-commit install
pre-commit run --all-files

Thank you for your contributions!

Summary by CodeRabbit

• Bug Fixes
  • Fixed an issue where partially cached chat generations were overwritten; outputs now correctly combine cached and newly generated content in order.
  • Improved model cache identification by considering additional attributes (e.g., performance and size limits), reducing cache misses and misidentification across model variants.
  • Enhances reliability and consistency of results when resuming or mixing cached and fresh computations.

[coderabbitai]

Caution

Review failed

The pull request is closed.

Walkthrough

Updates adjust model cache key resolution to consider additional attributes and modify async OpenAI chat generation to merge previously cached outputs with newly computed results, preserving order and returning a combined list.

Changes

Cohort / File(s)	Summary of Changes
Model cache name resolution lmms_eval/api/model.py	Extended _resolve_model_name_for_cache to check additional attributes: fps, max_pixels, min_pixels. Returns first non-empty among model_name, model_version, model_id, pretrained, fps, max_pixels, min_pixels; else str(model) then class name. No public API changes.
Async OpenAI result merging lmms_eval/models/chat/async_openai.py	generate_until now preserves cached results and appends newly computed async run() outputs. Sorts new results by index and returns concatenation: cached results + new contents, instead of overwriting with only new outputs.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant C as Caller
  participant G as generate_until
  participant Cache as Cache Layer
  participant A as Async run()

  C->>G: generate_until(prompts, cache_state)
  G->>Cache: Fetch cached results (by prompt indices)
  Note right of G: Identify missing indices<br/>needing computation
  G->>A: Run async for missing prompts
  A-->>G: eval_results (indexed)
  Note over G: Sort eval_results by index<br/>Extract contents
  G->>G: Merge cached_results + new_contents
  G-->>C: Combined results (cached first, then new)

Loading

sequenceDiagram
  autonumber
  participant M as Model instance
  participant R as _resolve_model_name_for_cache

  M->>R: Resolve cache key
  alt model_name/model_version/model_id/pretrained/fps/max_pixels/min_pixels present
    R-->>M: First non-empty attribute value
  else str(model) available
    R-->>M: str(model)
  else
    R-->>M: class name
  end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• [Feat] Adding cache api for model #814 — Also modifies model caching logic around _resolve_model_name_for_cache, overlapping the same functional area.

Suggested reviewers

• Luodian

Poem

I twitch my whiskers, cache in tow,
Old bits saved, new bits flow.
Keys resolved with careful care,
Pixels, frames—now all there.
Hop, merge, hop—results align,
Async carrots in a line.
Thump! The pipeline’s looking fine. 🥕🐇

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 9c569d0 and 0f975e5.

📒 Files selected for processing (2)

• lmms_eval/api/model.py (1 hunks)
• lmms_eval/models/chat/async_openai.py (1 hunks)

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch dev/cache

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}