ggml-cpu: parallelize tensor repacking with OpenMP #18239
+89
−55
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add OpenMP parallelization to tensor repack functions to significantly speed up model loading on many-core CPUs. Measured on AMD EPYC 9655 (96 cores): | Model Size | Before | After | Speedup | |------------|--------|-------|---------| | 6.8GB Q4_K | 5.0s | 3.3s | 1.5x | | 19GB Q4_K | 11.9s | 5.3s | 2.2x | Key changes: - Convert pointer-increment loops to explicit indexing - Add #pragma omp parallel for to outer loops - Each thread processes independent row groups - Move thread-local dst_tmp arrays inside parallel region Functions parallelized: - repack_q4_0_to_q4_0_4_bl - repack_q4_K_to_q4_K_8_bl - repack_q2_K_to_q2_K_8_bl - repack_q4_0_to_q4_0_8_bl - repack_iq4_nl_to_iq4_nl_4_bl - repack_iq4_nl_to_iq4_nl_8_bl
github-actions
bot
added
the
ggml
pestopoppa
added a commit
to pestopoppa/amd-epyc-inference
that referenced
this pull request
Dec 21, 2025
Key changes: - patches/: OpenMP parallelization of tensor repack functions - PR submitted: ggml-org/llama.cpp#18239 - Measured: 19GB model loads in 5.3s vs 11.9s (2.2x faster) - scripts/lib/executor.py: Remove OMP_NUM_THREADS=1 - Enables parallel repack in benchmark scripts - Also improves prompt processing 2.4x (49 → 119 t/s) - README.md: Add modded llama.cpp fork reference - Fork: https://github.com/pestopoppa/llama.cpp - Instructions for reproducing the setup - CLAUDE.md: Document fork and patches directory - research/RESULTS_SUMMARY.md: Add parallel repack section - orchestration/progress/PROGRESS_2025-12-21.md: Progress report 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Add OpenMP parallelization to tensor repack functions to significantly speed up model loading on many-core CPUs.
Motivation
On systems with many cores (e.g., AMD EPYC), tensor repacking for AVX-512 optimization was single-threaded and became a significant bottleneck during model loading. The repack functions convert quantized tensors from storage layout to SIMD-optimized interleaved layout.
Benchmark Results
Measured on AMD EPYC 9655 "Turin" (96 cores, 192 threads):
Speedup increases with model size as repack time dominates over I/O.
Changes
#pragma omp parallel forto outer loopsdst_tmparrays inside parallel regionFunctions Parallelized
repack_q4_0_to_q4_0_4_bl- Q4_0 x4 interleaverepack_q4_K_to_q4_K_8_bl- Q4_K models (most common)repack_q2_K_to_q2_K_8_bl- Q2_K modelsrepack_q4_0_to_q4_0_8_bl- Q4_0 x8 interleaverepack_iq4_nl_to_iq4_nl_4_bl- IQ4_NL x4repack_iq4_nl_to_iq4_nl_8_bl- IQ4_NL x8Testing