KV Cache Calculator

Size the cache for a model, a context length, and a sequence count. The config can come from HuggingFace, from ModelScope, from a config.json you paste, or from values you enter yourself.

Model source

Read the config from HuggingFace or ModelScope.

Provider

Model id

Reading the config

hf_token

Only needed for gated or private repos. Sent to HuggingFace and nowhere else.

Context length

Tokens per sequence.

Sequences

Concurrent requests in the batch.

KV cache dtype

Reading the model config