UPSTREAM PR #15641: Modern Bert Support

[loci-dev]

Mirrored from ggml-org/llama.cpp#15641

adding support to run granite embedding small, and it primarily pulls the modern bert architecture - https://huggingface.co/ibm-granite/granite-embedding-small-english-r2, currently working on it still, havent figured out the pre-tokenizer type or if I need to impliment it, also for the ubatch size the assert fails in llama-graph.cpp, hacked it to accept ubatch size of 1 for testing, but it seems to keep failing there and not sure why,

if I comment out of the line in llama-graph.cpp

assert(!ubatch.equal_seqs());

then it works

[loci-agentic-ai]

Explore the complete analysis inside the Version Insights

Performance Analysis Summary: PR #362 - Modern BERT Support

Overview

This PR introduces Modern BERT architecture support across 15 files with 308 additions. The changes add a new model type without modifying existing inference paths. Performance analysis shows the primary binary libllama.so achieved a 0.85% reduction in estimated power consumption (1,656 nJ saved), indicating net positive energy efficiency despite localized function-level variations.

Key Findings

Performance-Critical Functions Impact

The analysis identified performance variations in several functions, but these are not caused by this PR. The changes stem from build configuration or compiler differences affecting STL implementations across all models:

Most Impacted Functions (Absolute Changes):

• llama_model_ftype_name: +901 ns response time (from 1,482 ns to 2,383 ns)
• std::vector<llama_token_data>::end(): +113 ns response time (from 82 ns to 195 ns)
• std::__make_move_if_noexcept_iterator: -101 ns response time (improvement, from 195 ns to 94 ns)

These functions are not in the inference hot path. The PR adds new code paths for Modern BERT without touching existing tokenization or inference functions.

Tokens Per Second Impact

No impact on tokens per second. The core inference functions remain unchanged:

• llama_decode: Not modified
• llama_encode: Not modified
• llama_tokenize: Not modified

The PR adds a new graph builder (llm_build_modern_bert) that only executes for Modern BERT models. Existing models use their original code paths with zero performance change. The new Modern BERT implementation adds one modulo operation per layer for RoPE frequency selection, contributing negligible overhead (1-2 CPU cycles per layer).

Power Consumption Analysis

Binary-Level Impact:

• libllama.so: -0.85% power consumption (193,964 nJ → 192,307 nJ)
• All other binaries: No measurable change

The power reduction in libllama.so reflects improved energy efficiency from the overall codebase state, not specifically from this PR's additions.

Code Changes Assessment

The PR implements:

• Hybrid attention pattern (global + sliding window)
• Dual RoPE frequency bases for position encoding
• Fused QKV attention weights
• GPT2-style BPE tokenization reuse

All changes are isolated to new code paths. The architecture follows established patterns from existing BERT variants (NOMIC_BERT, NEO_BERT), ensuring consistency with the codebase design.