thek0tyara/llama-cpp-turboquant
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 4

kv-cache : pad the cache size to 256 for performance (#17046)

Browse source 
* kv-cache : pad the size of the small SWA cache for performance

* context : pad the total context to 256

* cont : future-proof the swa pad

* server : adjust test params to new logic
This commit is contained in:
Georgi Gerganov 2025-11-07 20:03:25 +02:00 • committed by GitHub
parent 9eb9a1331d
commit 16bcc1259d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 14 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

4
src/llama-context.cpp
View file

@ -114,10 +114,14 @@ llama_context::llama_context(
}
}
// ref: https://github.com/ggml-org/llama.cpp/pull/17046#discussion_r2503085732
cparams.n_ctx = GGML_PAD(cparams.n_ctx, 256);
if (cparams.kv_unified) {
cparams.n_ctx_seq = cparams.n_ctx;
} else {
cparams.n_ctx_seq = cparams.n_ctx / cparams.n_seq_max;
cparams.n_ctx_seq = GGML_PAD(cparams.n_ctx_seq, 256);
if (cparams.n_ctx_seq == 0) {
throw std::runtime_error("n_ctx_seq == 0");