[Canonical HTML page](https://githits.com/blog/deepinfer-vs-vllm/)

[Back to blog](/blog.md)

October 22, 2026 · 8 min read

# DeepInfer vs vLLM: What the Source Shows

We compared DeepSeek's DeepInfer with vLLM v0.32.0 through GitHits, tracing scheduling, hardware backends and speculative decoding to source.

DeepSeek open-sourced [DeepInfer](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/README.md#L3) this week under Apache 2.0. The README calls it the inference framework behind its production services, and credits [vLLM, with inspiration from SGLang and TensorRT-LLM](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/README.md#L95). Twenty-three files still open with `# Derived from vLLM; modified by DeepSeek.` The interesting question is how far the fork has actually drifted.

At [`774ce244`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01), three commits in and not yet tagged, DeepInfer is an offline engine for one model. [vLLM v0.32.0](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be) is a general server, and accelerators beyond its five built-in platforms load as plugins. We read [`vllm-ascend`](https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d) at `v0.30.0rc2`, the newest tag on that plugin the day we looked. On confidence-based speculative verification, both maximize expected accepted tokens per unit of step cost. We read the source through GitHits and did not run either engine. The performance numbers below are the ones the repositories publish.

| Question                          | DeepInfer (774ce244)                                                                     | vLLM (v0.32.0)                                                                                      |
| --------------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| Scope                             | One supported model, DeepSeek-V4.1-Flash, on NVIDIA SM100 and Ascend 950                 | A model registry with many architectures, including DeepSeek V2 through V4.1                        |
| How you call it                   | `LLM.generate()` in Python; the only HTTP route is `POST /profile`                       | OpenAI-compatible HTTP routes and gRPC, plus `LLM.generate()`                                       |
| Scheduler                         | C++ header; data-parallel ranks vote on prefill or decode each step                      | Python; no prefill or decode phase, one token budget per step, chunked prefill                      |
| KV cache blocks                   | C++ reference-counted free list; admission answers OK, LATER or NEVER                    | Python block pool with prefix caching on by default                                                 |
| Parallelism                       | Expert, data and context parallel plus an “engram” group; no tensor or pipeline parallel | Tensor, pipeline, data, prefill-context and decode-context parallel, with an expert-parallel switch |
| Expert load balancing             | Greedy two-stage replica placement in Python                                             | Default policy adapted from DeepSeek’s EPLB                                                         |
| Adaptive speculative verification | C++, scheduler side, opt-in                                                              | Python, GPU worker side, opt-in, DSpark only                                                        |
| CPU-GPU sync guard                | On by default; syncs in forward and sample raise                                         | Off by default; `warn` or `error` after warmup                                                      |
| Ascend                            | In-tree `PlatformNPU`                                                                    | Separate `vllm-ascend` plugin repository                                                            |

## One way in, or a server

**DeepInfer vs vLLM request path**

DeepInfer is called from a Python script through LLM.generate and an LLMEngine step loop into a C++ scheduler where data-parallel ranks vote on whether the step is prefill or decode, with a C++ block manager. vLLM is called through an OpenAI-compatible API server or LLM.generate, both reaching AsyncLLM or LLMEngine and then EngineCore, whose Python scheduler fills a token budget with prefill chunks and decodes and uses a KV cache manager with prefix caching.

Both request paths, traced with GitHits at DeepInfer 774ce244 and vLLM v0.32.0.

DeepInfer is a script. [`generate()`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/entrypoints/llm.py#L90) adds the requests and calls [`llm_engine.step()` until nothing is left](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/entrypoints/llm.py#L176-L178). The only HTTP route is [`POST /profile`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/engine/profile_server.py#L1-L12), a profiler trigger, and the batch is DeepSeek-V4.1-Flash. vLLM keeps the offline [`LLM.generate()`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/llm.py#L418-L424) and also speaks [`/v1/chat/completions`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/chat_completion/api_router.py#L41-L42), [`/v1/completions`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/completion/api_router.py#L35-L36), [`/v1/models`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/entrypoints/openai/models/api_router.py#L20), gRPC, and others. Both front ends reach `EngineCore` through an `EngineCoreClient`, from [`llm_engine.py`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/llm_engine.py#L116) and [`async_llm.py`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/async_llm.py#L184). A batch can stop at DeepInfer. A client on the network needs vLLM.

DeepInfer’s scheduler is a [C++ header](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/scheduler.hpp#L308-L318). Under data parallelism each rank is a replica with its own batch. The ranks allreduce until they agree the next step is prefill, processing a prompt, or decode, producing the next token, and every rank runs that kind of step. The replicas stay in lockstep. A prompt and a generation do not share it.

vLLM’s scheduler has [no decoding phase and no prefill phase](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/core/sched/scheduler.py#L594-L600). Each request remembers how many tokens are already computed, and each step hands out a token budget so requests can catch up. Long prompts are split into chunks. `enable_chunked_prefill` [defaults to true on the config class](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/scheduler.py#L134-L140), though the docstring says the engine arguments normally set it. One step on a shared GPU can hold a prompt and someone else’s next token.

When a vLLM rank has nothing ready and the others are still running, `DPEngineCoreProc` runs [a dummy pass](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/engine/core.py#L2353-L2363). With `enable_balance_scheduling`, [`patch_balance_schedule.py`](https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/vllm_ascend/patch/__init__.py#L35-L50) patches vLLM’s `Scheduler`, because prefill and decode in one pass “can impact the overall system throughput and performance in some scenarios”.

DeepInfer’s [block manager](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/block_manager.hpp#L14-L18) is a C++ reference-counted free list, and the answer is `OK`, `LATER`, or `NEVER`. vLLM’s [`BlockPool`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/core/block_pool.py#L136-L146) keeps free blocks in eviction order so a cached prefix can be found by hash, and [prefix caching is on by default](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/cache.py#L141).

## NVIDIA and Ascend

**DeepInfer vs vLLM hardware backends**

DeepInfer keeps NVIDIA and Ascend in one repository: platform.py selects PlatformGPU or PlatformNPU, a 32-line dispatcher picks CUDA or Ascend TileLang kernels, and the vendor libraries are installed under the same four import names. vLLM ships built-in platform plugins for TPU, CUDA, ROCm, XPU and CPU and loads others through the vllm.platform\_plugins entry point; vllm-ascend is a separate repository that registers NPUPlatform, patches vLLM classes and carries its own DeepSeek V4.1 model code.

Where the NVIDIA/Ascend split lives in each project. Paths at DeepInfer 774ce244, vLLM v0.32.0, vllm-ascend v0.30.0rc2.

NVIDIA and Ascend both live in DeepInfer’s tree, and the model code does not branch on which one it got. [`PlatformGPU`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/platform.py#L55-L63) selects the `cuda` kernels, `nccl`, and `int32` FP8 scales. [`PlatformNPU`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/platform.py#L86-L90) selects `asc` and `hccl`, with [`int16` scales](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/platform.py#L138). A [32-line dispatcher](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/ops.py#L15-L32) loads the kernel set, and [`build_vendor.sh`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/scripts/build_vendor.sh#L7-L22) builds a different library per vendor under the same four import names. `flash_mla` or `deep_gemm` is the same import on NVIDIA SM100 and on Ascend 950.

[Five platforms are built in](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/platforms/__init__.py#L231-L237): TPU, CUDA, ROCm, XPU, and CPU. vLLM also loads [any installed package registered under `vllm.platform_plugins`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/platforms/__init__.py#L250-L286). vllm-ascend joins as [`ascend = vllm_ascend:register`](https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/setup.py#L507), which returns [`NPUPlatform`](https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/vllm_ascend/__init__.py#L72-L75). The platform sets [`device_name` to `"npu"` and uses `PrivateUse1`](https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/vllm_ascend/platform.py#L89-L99), the PyTorch dispatch key reserved for devices that ship out of tree. It carries its own [`models/deepseek_v41/`](https://github.com/vllm-project/vllm-ascend/blob/8fde6d4e335fe300e2b949aab0e31659cce0811d/vllm_ascend/models/deepseek_v41/model.py) and a `patch/` package that edits vLLM classes at startup. The plugin is versioned alongside vLLM, and the tags we found were `v0.30.0rc2` against `v0.32.0`. Choosing Ascend on this side means following both repositories.

vLLM already registers [`DeepseekV4ForCausalLM`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/model_executor/models/registry.py#L96) and [`DeepseekV41ForCausalLM`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/model_executor/models/registry.py#L373-L376), with per-vendor code under `vllm/models/deepseek_v41/` for NVIDIA and AMD. DeepInfer’s list is the one model. “Engram” shows up in DeepInfer’s [parallel state](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/parallel_utils/parallel_state.py#L70-L118) with no explanation beside it. The clearest note we found is vLLM’s, [“n-gram hash lookups gated into the hyper-connection stream”](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/models/deepseek_v41/common/engram.py#L3-L8), ported from DeepSeek’s reference code.

`ParallelConfig` is the wider menu: [tensor, pipeline, prefill-context, and data parallel sizes](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py#L139-L148), [`enable_expert_parallel`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py#L189-L190), [`enable_eplb`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py#L205-L207), and [decode-context parallelism](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/parallel.py#L385-L387). We searched DeepInfer for tensor or pipeline parallel names and found none. DeepSeek’s [deployment Q\&A](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference_deployment_qa.md#L53) says production does not use tensor parallelism. The groups here are expert, data, and context, plus engram. A tensor or pipeline setting from a vLLM config has nothing to map onto.

Expert load balancing, copying busy experts onto more GPUs, is where the credit runs the other way. vLLM’s default policy says it was [adapted from DeepSeek’s EPLB](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/distributed/eplb/policy/default.py#L3-L11). DeepInfer has its own [greedy two-stage replication and placement](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/eplb/policy.py#L12-L120).

KV formats overlap and are spelled differently. vLLM’s `cache_dtype` accepts [`fp8_e4m3`, `fp8_e5m2`, `fp8_ds_mla`, and `nvfp4_ds_mla`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/cache.py#L39-L49), and quantization includes [`deepseek_v4_fp8`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/model_executor/layers/quantization/__init__.py#L38). DeepInfer declares [five cache formats](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/layers/tilelang_ops/cache_common.py#L109-L155) for its one model, with block scales stored per 16, 32 or 128 elements.

## Speculative verification

**DeepInfer vs vLLM adaptive speculative verification**

Both engines size speculative verification from per-token draft confidence and choose the size that maximizes expected accepted tokens per unit of step cost. DeepInfer does this in a C++ MTPVarlen component on the scheduler side using rolling medians of measured step times, behind DEEPINFER\_ENABLE\_MTP\_VARLEN. vLLM does it in a Python AdaptiveVerificationManager on the GPU worker using cost tables seeded from startup profiling, behind enable\_adaptive\_verification for the dspark method.

Both projects choose a variable verification length from draft confidence. GitHits found both implementations.

Speculative decoding drafts several tokens with a cheap model, then checks them in one forward pass of the full model. DeepInfer sizes that check in C++ beside the scheduler. vLLM sizes it in Python on the GPU worker. Both take that size from the draft model’s own confidence.

A [`ConfidenceHead`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/model_executor/models/mtp/dflash.py#L46-L79) predicts acceptance per draft token. `MTPVarlen` [sorts pending drafts by that rate](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/mtp_varlen.hpp#L213-L214) and [keeps the batch size with the most expected tokens per predicted step time](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/mtp_varlen.hpp#L242-L249). Step time is a [median over a 5,000-step window](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/csrc/engine/mtp_varlen.hpp#L48-L131), so the cost is whatever this process has actually been paying. The path is [opt-in, behind `DEEPINFER_ENABLE_MTP_VARLEN`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/environ.py#L129-L131).

`AdaptiveVerificationManager` scores each draft slot by the [running product of confidences](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/worker/gpu/spec_decode/adaptive_verification.py#L309-L313), prices the step from cost tables [seeded by profiling runs at startup](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/worker/gpu/spec_decode/adaptive_verification.py#L188-L230), and takes the [argmax of expected accepted tokens over that cost](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/v1/worker/gpu/spec_decode/adaptive_verification.py#L325-L344). It is off by default, and [only for `method="dspark"`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/config/speculative.py#L551-L553).

DeepSeek’s Q\&A, which we did not reproduce, says this [cuts draft tokens verified per step by about 30%, from 5.00 to 3.2–4.4, with accepted length roughly unchanged](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference_deployment_qa.md#L247), and estimates +4–7% throughput at about 60 concurrent sequences per rank from EP64/DP64 benchmark data. The [table there](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/doc/inference_deployment_qa.md#L249-L256) shows a 1.5% loss at 16 sequences per rank. Those figures are DeepSeek’s. We found no comparable figure in vLLM’s source.

Both use torch’s sync debug mode, which is process-wide. DeepInfer wraps forward and sample in [`forbid_cuda_sync()`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/model_runner.py#L322-L325) and raises unless [`DEEPINFER_DISABLE_SYNC_GUARD` is set](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/utils.py#L119-L132). vLLM’s [`gpu_sync_debug`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/utils/gpu_sync_debug.py#L3-L16) catches those syncs and non-blocking copies through pageable memory, [arms only after warmup](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/utils/gpu_sync_debug.py#L51-L63), and stays [off unless `VLLM_GPU_SYNC_CHECK` is `warn` or `error`](https://github.com/vllm-project/vllm/blob/5df63a7203cdb9328d824a27b682e77c7b9928be/vllm/envs.py#L656-L658).

## What the credit line still matches

The headers never name a vLLM version. We compared each of the 23 files with its counterpart at 16 tags from `v0.2.0` to `v0.32.0` and counted how many of DeepInfer’s non-comment lines still appear verbatim. It is a rough measure. Sixteen of the 23 are closest to a release from `v0.2.0` to `v0.5.0`, and the median overlap is 19%. The nearest are the shared-memory broadcast queue, 73% and best against `v0.6.6` and `v0.7.3`, and `model_executor/models/__init__.py` at 70%. Most of the rest kept the filename and rewrote the body. The buffer-moving bits still look like vLLM. The engine around them is DeepInfer’s.

vLLM `v0.3.3`’s [`CacheEngine`](https://github.com/vllm-project/vllm/blob/v0.3.3/vllm/worker/cache_engine.py#L141-L144) is 178 lines and swaps blocks to CPU. DeepInfer’s is 978 lines, does not swap, and adds a [`KVCacheManager`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/cache_engine.py#L309) plus [sliding-window state](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/worker/cache_engine.py#L683). `sequence.py` replaces vLLM’s sequence-group classes with [`MTPPredictInfo`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/sequence.py#L77) and [`SampledInfo`](https://github.com/deepseek-ai/DeepInfer/blob/774ce244b87cc0eb8d4b94ab9f5de2ab22049f01/deepinfer/sequence.py#L114). `sampling_params.py` removes [beam search](https://github.com/vllm-project/vllm/blob/v0.2.7/vllm/sampling_params.py#L104). Two of the 23 have no counterpart at `v0.32.0`, because vLLM’s V0 `worker/` directory is gone. The percentages are how you tell a small patch from a file that grew up.

## Use the GitHits API

Each link above is a file and a line at the revision we read, the same jump the API gives an agent. The [API overview](https://docs.githits.com/api/overview) maps those calls. Create a key in the token settings of your [GitHits account](https://app.githits.com/) and send it as a bearer token. The [authentication guide](https://docs.githits.com/api/authentication) covers storing that key for CI, and the [quickstart](https://docs.githits.com/api/quickstart) is the first request.

If you’re a software factory interested in using our API, we’d love for you to [book a call with us](https://calendly.com/d/dvxq-7dn-msq/meeting-with-githits-select-15-or-30-minutes) to discuss what it would look like.

[Open Source](/blog/tag/open-source.md) · [Code Navigation](/blog/tag/code-navigation.md) · [Engineering](/blog/tag/engineering.md)

Nathan Burg — Co-founder, CPO

Keep exploring

* [The Solution](/the-solution.md)
* [Documentation](https://docs.githits.com/)
* [Changelog](https://docs.githits.com/changelog/overview)

Get started

## GitHits in one command

Install GitHits or refresh an existing setup with the same command.

`npx -y githits@latest init`