[Fix] Aria and LLama Vision and OpenAI compatible models #641
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…dling - Added `system_prompt`, `interleave_visuals`, and `max_length` parameters to Qwen2_VL class. - Simplified device assignment logic for single process scenarios. - Improved visual processing by refining how visuals are handled and ensuring proper mapping to contexts. - Enhanced message construction to support interleaving of visuals and text based on placeholders. - Set default generation parameters and refined handling of generated outputs to ensure proper trimming and formatting.
… consistency - Updated string formatting to use double quotes for consistency. - Enhanced the `VisualPuzzles_doc_to_text` and `parse_response` functions for better clarity and structure. - Simplified conditional checks and improved whitespace handling in response parsing. - Ensured consistent handling of options and answers throughout the utility functions.
…Compatible - Replaced AutoModelForCausalLM and AutoProcessor with AriaForConditionalGeneration and AriaProcessor in the Aria model. - Updated the pretrained model string in LlamaVision to "meta-llama/Llama-3.2-11B-Vision-Instruct". - Enhanced OpenAICompatible to support AzureOpenAI and modified API key handling for better flexibility. - Adjusted timeout parameter and refined token handling in OpenAICompatible for improved functionality.
Luodian
changed the title
Contributor
Author
|
We should close the last PR then could merge this since it's developed based on that. |
kcz358
approved these changes
Apr 19, 2025
Collaborator
kcz358
left a comment
kcz358
left a comment
There was a problem hiding this comment.
Most of the codes look fine to me as this is the only way to hardcode under current logic. I will merge this PR.
A small notice for others:
We plan to rfc the doc to text these things in the future with messages format so these hardcoded models will gradually being deprecated. Please stay tuned for our update
dadwadw233
pushed a commit
to dadwadw233/lmms-eval
that referenced
this pull request
Apr 28, 2025
…s-Lab#641) * Enhance Qwen model with additional parameters and improved visual handling - Added `system_prompt`, `interleave_visuals`, and `max_length` parameters to Qwen2_VL class. - Simplified device assignment logic for single process scenarios. - Improved visual processing by refining how visuals are handled and ensuring proper mapping to contexts. - Enhanced message construction to support interleaving of visuals and text based on placeholders. - Set default generation parameters and refined handling of generated outputs to ensure proper trimming and formatting. * Refactor VisualPuzzles utility functions for improved readability and consistency - Updated string formatting to use double quotes for consistency. - Enhanced the `VisualPuzzles_doc_to_text` and `parse_response` functions for better clarity and structure. - Simplified conditional checks and improved whitespace handling in response parsing. - Ensured consistent handling of options and answers throughout the utility functions. * Update model imports and parameters for Aria, LlamaVision, and OpenAICompatible - Replaced AutoModelForCausalLM and AutoProcessor with AriaForConditionalGeneration and AriaProcessor in the Aria model. - Updated the pretrained model string in LlamaVision to "meta-llama/Llama-3.2-11B-Vision-Instruct". - Enhanced OpenAICompatible to support AzureOpenAI and modified API key handling for better flexibility. - Adjusted timeout parameter and refined token handling in OpenAICompatible for improved functionality.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before you open a pull-request, please check if a similar issue already exists or has been closed before.
When you open a pull-request, please be sure to include the following
If you meet the lint warnings, you can use following scripts to reformat code.
Thank you for your contributions!