Alauda AI

DeepSeek-V4-Flash (W8A8)

deepseek_v4-architecture model — a 256-expert MoE with MLA + DSA sparse attention and a native MTP speculative head, served as W8A8 (int8 attention + expert, ~280 GB weights, 70 shards, 43 layers). At ~280 GB the weights do not fit a single 8 × 32 GB node, so it was validated on Ascend 910B3 (64 GB/card) across two nodes = 16 cards as a single aggregated DP2 × TP=8 + EP16 service, with the vLLM-Ascend nightly engine through Alauda AI's InferNex surface. It adds a mooncake cross-rank KV store so the two data-parallel ranks share prefix KV, and both benchmark scenarios were driven through the MaaS gateway (API-key) ingress as a concurrency sweep (8 / 16 / 32).

TOC

Model identityValidated hardware × stackModel configurationDeployment specDeployBenchmark results

Model identity

FieldValue
PublisherDeepSeek
Architecturedeepseek_v4 — 256-expert MoE (6 experts/token + 1 shared) + MLA + DSA sparse attention + native MTP
QuantizationW8A8 (W8A8_DYNAMIC, int8 attention + expert; ~280 GB weights, 70 shards, 43 layers); MIT license
Model source (W8A8)https://www.modelscope.cn/models/Eco-Tech/DeepSeek-V4-Flash-w8a8-mtp
ModelCar (OCI Image Layout tar)http://<model-package-host>/packages/aml-models-oci/v0.1.0/DeepSeek-V4-Flash-w8a8-mtp/DeepSeek-V4-Flash-w8a8-mtp.oci.tar
NOTE

The W8A8 ModelCar is distributed as an OCI Image Layout tar on the internal package store, not a public Docker Hub image. Replace <model-package-host> in the link above with the package store address supplied by Alauda for the environment. The cluster pulls an OCI image, not the tar, so import it into your own registry once and point model.uri at that — see the Deploy section below. The benchmark itself was run with the weights staged on a node-local PVC per node; the ModelCar is the same weights repackaged (the manifest header documents that PVC alternative).

Validated hardware × stack

PlatformEngineVersion / configStatus
Ascend 910B3 64 GB × 16, 2 nodes (1 instance, DP2 × TP=8 + EP16)vLLM-Ascendnightly-main-openeuler (vLLM 0.23.0)✅ closed-loop, 2-scenario concurrency sweep (8/16/32), 480 req/tier, 0 error / 0 mismatch, agg + mooncake KV store, via MaaS gateway
NOTE

The nightly-main image carries the full deepseek_v4 stack and the DP2 cross-node graph + DSA-CP prefix-cache fixes. enable_dsa_cp is intentionally on — it is a prerequisite for a non-zero prefix-cache hit, and this build is stable with it. With the full hit gate satisfied (see the config table), prefix caching works — ~90.9 % single-rank hit at a 21k prefix.

Model configuration

ParameterValue
Tensor parallelism (tensor-parallel-size)8
Data parallelism (data-parallel-size)2 (1 rank per node, cross-node)
Replicas (instances)1 aggregated service (leader + worker = 16 cards)
Expert parallelism (enable-expert-parallel)on (EP16)
max-model-len200000 (agent long-context)
max-num-batched-tokens8192
max-num-seqs32
gpu-memory-utilization0.92 (64 GB card — ~20.6 GB/card weights leave ample KV room)
block-size128
Quantizationascend (W8A8_DYNAMIC)
Speculative decoding (MTP)mtp, 1 token (enforce_eager on the draft head)
Decode graphFULL_DECODE_ONLY (enable_npugraph_ex)
DSA context parallel (enable_dsa_cp)on (prefix-cache hit prerequisite)
Prefix cachingenabled and effective (~90.9 % single-rank hit; requires VLLM_ASCEND_ENABLE_FLASHCOMM1=1 + VLLM_PREFIX_CACHE_RETENTION_INTERVAL=16384 + prefix ≥ 16384)
Tool calling / reasoning--tool-call-parser deepseek_v4 + --enable-auto-tool-choice + --reasoning-parser deepseek_v4 (thinking off by default)

Deployment spec

Served as agg-mc-kv — a single DP2 × TP=8 aggregation plus a mooncake KV store shared by the two DP ranks. DP2 random routing scatters each user's growing history across the two ranks, so per-rank local multi-turn hit is only ~41 %; the mooncake store lets one rank pull prefix KV the other already computed, lifting merged effective prefix reuse to ~84 %. The router stays random and the cache-indexer is off: a single aggregated endpoint needs no global KV index or KV-cache-aware routing — this matches the GLM-5.2 / Qwen3-8B aggregation paradigm (store shares KV; router stays random).

Componentagg-mc-kv
hermes-router (EPP)✅ started (single leader endpoint, random no-op)
Routing strategyrandom
mooncake KV store (AscendStoreConnector)✅ (shared across the two DP ranks)
cache-indexer / KV-cache-aware routing— (not needed for a single aggregated endpoint)
Cross-rank prefix KV reuseyes (~84 % merged effective)
NOTE

The store writes asynchronously (~30–60 s to land a 21k prefix). Cross-rank hits land for multi-turn agent sessions with turn gaps of roughly a minute or more; back-to-back sub-second reuse only hits the same rank's local prefix cache. A separate attempt at KV-cache-aware routing (to steer follow-ups back to the rank holding the cache) was evaluated and reverted — it gave no benefit on this stack at this scale, so the production form stays random + mooncake store.

Deploy

Self-contained InferNex manifest (engine inlined in the LLMInferenceService leader + worker templates + hermes-router preset, DP2 × TP=8 across two nodes with the mooncake store):

SpecFile
agg-mc-kv, DP2 × TP=8deepseek-v4-flash-w8a8-agg-mc-kv-llmisvc.yaml
base=https://raw.githubusercontent.com/alauda/aml-docs/master/docs/en/plan/validated_models/assets/deepseek-v4-flash-w8a8

# 1. Import the ModelCar (OCI Image Layout tar) into your own registry once — the cluster
#    pulls an OCI image, not the tar. skopeo does the multi-arch (amd64+arm64) push in one go:
curl -sO http://<model-package-host>/packages/aml-models-oci/v0.1.0/DeepSeek-V4-Flash-w8a8-mtp/DeepSeek-V4-Flash-w8a8-mtp.oci.tar
skopeo copy --all \
  oci-archive:DeepSeek-V4-Flash-w8a8-mtp.oci.tar:v0.1.0 \
  docker://<your-registry>/modelcar-deepseek-v4-flash-w8a8-mtp:v0.1.0

# 2. Edit the manifest: set the namespace, set model.uri to the registry ref you just
#    pushed, and the image tag (match your CANN / host driver). Then apply — the
#    storageInitializer pulls the ModelCar to /mnt/models in each DP rank.
#    (Alternative: stage the weights on a node-local PVC per node — see the manifest header.)
kubectl apply -f $base/deepseek-v4-flash-w8a8-agg-mc-kv-llmisvc.yaml

# Internal KServe ingress (no auth):
curl -s http://<gateway>/<namespace>/deepseek-v4-flash-w8a8-agg-mc-kv/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4-flash-w8a8","messages":[{"role":"user","content":"hello"}]}'

# Product MaaS gateway (OpenAI-compatible, API-key auth + token rate limiting):
curl -s http://<maas-gateway>/v1/chat/completions \
  -H "Authorization: Bearer $MAAS_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4-flash-w8a8","messages":[{"role":"user","content":"hello"}]}'

Benchmark results

Closed-loop aiperf 0.7.0, DP2 × TP=8 (16 × 910B3 64 GB), driven through the product MaaS gateway. Concurrency sweep 8 / 16 / 32, 480 requests per tier (all tiers 0 error / 0 mismatch). The headline numbers are the production agg-mc-kv configuration — FULL_DECODE_ONLY graph + mooncake cross-rank KV store, the exact config the manifest above deploys — on scenario ② (multi-turn nested long context), the agent-style workload the store targets. TTFT / E2E in s, ITL in ms; decode = output-only tok/s (measured), TPS = total tokens/s (input + output, computed; prefill-dominated).

Scenario ② — multi-turn dialogue, production agg-mc-kv (ISL ~17.5k / OSL 128)

ConcurrencyTTFT avg (s)ITL avg (ms)E2E avg (s)Decode (tok/s)TPS (in+out)
81.767210.993.613198
161.369012.8159.322467
321.9412317.6230.932559

The mooncake cross-rank KV store is what makes this fast: it lifts multi-turn prefix reuse from ~41 % (per-rank local) to ~84 % (merged effective). Stepping the same scenario ② at concurrency 32 through the configs the deployment evolved from shows where the gain comes from:

Config evolution — Scenario ②, concurrency 32

ConfigTTFT avg (s)E2E avg (s)Decode (tok/s)TPS (in+out)
agg (base — eager, no store)10.283.748.16781
+ FULL_DECODE_ONLY graph3.229.3138.719554
+ mooncake KV store (production)1.917.6230.932559

→ graph + store together are TTFT ~5.3× / E2E ~4.8× / TPS ~4.8× better than the base aggregation at the same concurrency.

Scenario ① — fixed-length system-prompt reuse (ISL ~8k / OSL 128), base agg

The 8k fixed-prompt scenario was measured only on the base aggregation config (graph

  • mooncake were not separately re-run for it), so treat these as a lower bound — the production config would lift them much like scenario ②:
ConcurrencyTTFT avg (s)ITL avg (ms)E2E avg (s)Decode (tok/s)TPS (in+out)
83.1930541.924.21539
163.4235047.842.32689
325.7149368.359.23759
NOTE

How to read these. Every tier completed 480/480 with zero errors and zero output mismatches. As concurrency rises, system throughput (decode / TPS) climbs (sub-linear, saturating) while tail latency (TTFT / ITL / E2E) also climbs — the queueing cost of more in-flight requests. TPS is the total-token (input + output) caliber and is prefill-dominated (ISL 8k / 17.5k, OSL 128); the decode-only output rate is the separate "Decode" column.

These numbers are not directly comparable to the other models in this guide: it is a concurrency sweep (not the fixed concurrency-4 run the Qwen models use), on 910B3 64 GB cards (not 910B4 32 GB), with a 16-card DP2 × TP=8 topology. Treat them as the operating envelope of this specific large-MoE deployment, not a cross-model ranking.