thek0tyara/llama-cpp-turboquant
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 4

rpc : cache and reuse compute graphs (#15405)

Browse source 
Store the last computed graph and reuse it when possible.
Also do not return response from GRAPH_COMPUTE and assume it always
completes successfully. If this this is not the case, the server closes
the connection. This saves us a network round trip to the server.
This commit is contained in:
Radoslav Gerganov 2025-11-28 10:33:51 +02:00 • committed by GitHub
parent 6bca76ff5e
commit 15d2b46b4d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 92 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

2
ggml/include/ggml-rpc.h
View file

@ -8,7 +8,7 @@ extern "C" {
#endif
#define RPC_PROTO_MAJOR_VERSION 3
#define RPC_PROTO_MINOR_VERSION 0
#define RPC_PROTO_MINOR_VERSION 5
#define RPC_PROTO_PATCH_VERSION 0
#define GGML_RPC_MAX_SERVERS 16