KV Cache Calculator

Size the cache for a model, a context length, and a sequence count. The config is read from HuggingFace or ModelScope.

Provider

Model id

Reading the config

Context length

Tokens per sequence.

Sequences

Concurrent requests in the batch.

KV cache dtype

hf_token

Only needed for gated or private repos. Sent to HuggingFace and nowhere else.

Reading the model config