Add MEGA-Bench #496
Merged
Add MEGA-Bench #496
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Luodian
approved these changes
Jan 13, 2025
MichalCiesiolka
pushed a commit
to MichalCiesiolka/lmms-eval-llmzszl
that referenced
this pull request
Apr 3, 2025
* init megabench query-response and submission generation * submission creation & core set evaluation script * megabench evaluator for open set w/ eval resuming * finish megabench score generation, update readme * update megabench score generation readme * update megabench readme, reorg folder structure * Update README.md --------- Co-authored-by: Li Bo <[email protected]>
dadwadw233
pushed a commit
to dadwadw233/lmms-eval
that referenced
this pull request
Apr 28, 2025
* init megabench query-response and submission generation * submission creation & core set evaluation script * megabench evaluator for open set w/ eval resuming * finish megabench score generation, update readme * update megabench score generation readme * update megabench readme, reorg folder structure * Update README.md --------- Co-authored-by: Li Bo <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi, lmms-eval team,
This PR incorporates our recent work, MEGA-Bench, a multimodal evaluation suite with over 500 real-world tasks and 45 metrics.
The evaluation process involves two steps: 1) run lmms-eval to produce the response/submission file; 2) run our evaluator with 45 diverse metrics to get the scores and multi-dimensional breakdown results.
Example response/submission file generation:
# Core set (440 tasks) python3 -m accelerate.commands.launch \ --num_processes=8 \ -m lmms_eval \ --model llava_onevision \ --tasks megabench_core \ --batch_size 1 \ --log_samples \ --log_samples_suffix llava_ov_megabench_core \ --output_path ./logs/ \ --model_args=pretrained=lmms-lab/llava-onevision-qwen2-7b-ov,conv_template=qwen_1_5,model_name=llava_qwenDetailed steps are recorded in this README file.
Please advise on the format of the README or other details, thanks!