Posts | 木叶吟

Helix: Automating Communication-Computation Overlap with Graph Scheduling

Mon, 18 May 2026 12:00:00 +0800

Large models are rarely trained or served with one clean parallelism strategy. Tensor parallelism splits matrix operations. Pipeline parallelism splits layers. Sequence parallelism stretches context length across devices. Expert parallelism routes tokens through distributed experts. Real deployments increasingly compose several of these dimensions at once.

That composition is powerful, but it creates a familiar systems tax: communication bubbles. When an AllReduce, AllGather, ReduceScatter, or All-to-All sits on the critical path, GPU compute units wait. In a dense tensor-parallel block, the waiting may come from sharded matrix results. In a long-context sequence-parallel block, it may come from exchanging sequence chunks. In a MoE layer, it may come from routing tokens across experts. The parallel strategy changes, but the shape of the problem is the same: computation and communication are both present, yet the execution graph does not expose enough safe overlap.

Helix is built around one idea: communication-computation overlap should be a graph scheduling problem, not a hand-written kernel trick for each new parallel pattern. Helix is currently a WIP research project starts from early 2026. Please check back for updates.

TL;DR. Helix makes communication-computation overlap a compiler scheduling problem. Instead of hand-coding overlap for tensor, sequence, or expert parallelism separately, it exposes compute, communication, waits, and dependencies in one graph, then schedules that graph to reduce bubbles under a memory budget.

Why Manual Overlap Does Not Scale

The fastest overlap techniques often go deep into kernels. They split an operation into small pieces, launch communication early, and fuse enough computation around it to hide latency. This can work extremely well for one pattern. Ring-style attention can overlap sequence exchange with local attention blocks. Tensor-parallel kernels can pipeline collectives with partial matrix multiplications. MoE systems can schedule expert computation around token dispatch.

The problem is that each of these optimizations tends to encode assumptions about the model, the collective, the tiling shape, and the synchronization protocol. Once the model architecture changes, or a deployment combines tensor, sequence, and expert parallelism, the optimization becomes harder to reuse. A local trick may also miss a global opportunity: a communication operation produced by one parallel dimension might be hidden under computation from another dimension, but a pattern-specific optimizer will not necessarily see that.

Helix moves the optimization boundary up to the compiled execution graph. After torch.compile captures the model-parallel program, Helix sees compute operators, communication operators, waits, and dependency edges in one intermediate representation. That unified graph is the key abstraction. The compiler no longer needs a separate overlap recipe for every parallel strategy; it can schedule visible compute and communication nodes under the same correctness rules.

Key observation. The overlap opportunity is often cross-dimensional: communication from one parallel strategy may be hidden under computation from another. A pattern-specific optimizer can miss that; a graph scheduler can see it.

The Scheduling Objective

At a high level, Helix treats the model-parallel program as a directed graph. Nodes are compute or communication operators. Edges are precedence constraints: an operator can run only after the values it depends on are ready.

The optimization goal is straightforward but constrained:

reduce the graph makespan by hiding communication under independent computation;
preserve every data dependency in the original graph;
keep peak memory below the available device budget.

That last constraint matters. Aggressive overlap is not free. If the compiler launches too much work early, intermediate activations and communication buffers live longer. A schedule that looks faster on the timeline can become unusable because it inflates peak memory. Helix therefore optimizes both time and memory, guided by a lightweight graph simulator.

The system uses three compiler passes: tiling, reordering, and bucketing.

Pass	What it changes	What it protects
Tiling	Splits coarse operators into tile streams	Creates overlap opportunities without violating local dependencies
Reordering	Interleaves tile-stream segments around waits	Moves communication earlier while preserving synchronization
Bucketing	Merges compatible fragments back together	Recovers kernel efficiency under the memory budget

Tiling: Create Overlap Opportunities

The original execution graph is often too coarse. A large compute operator may wait for a large communication operator, even though smaller chunks of the work could have been interleaved. Helix first applies graph tiling: it partitions operators into multiple tile streams while preserving the dependency structure inside each stream.

By default, the paper tiles along the batch dimension because it is broadly applicable and easy to reason about. Other dimensions, such as sequence length, can also be used when correctness is guaranteed for that region of the graph.

Tiling has two benefits. First, it exposes overlap. Communication from one tile stream can be launched while computation from another tile stream is still running. This turns one rigid graph into several smaller streams that can be woven together. Second, it can reduce activation memory. Smaller tiles mean smaller live inputs and intermediate tensors, so the peak memory footprint can drop when lifetimes are well controlled.

But tiling also has a cost. Smaller compute kernels can lose efficiency, especially for memory-bound operators such as normalization, softmax, and pointwise functions. Smaller communication messages can also lose effective bandwidth. The paper’s profiling shows that a small tiling factor is usually the practical choice; Helix uses K = 2 by default because it exposes useful overlap without creating excessive fragmentation.

Reordering: Make Overlap Safe

After tiling, the compiler has several independent tile streams. The next question is launch order.

A naive schedule would simply execute the streams one by one. That preserves correctness, but it leaves communication bubbles exposed. An overly aggressive schedule would launch many asynchronous operations early, which may improve overlap but keep too many tensors alive and push peak memory upward.

Helix uses Segmented Round-Robin Reordering to sit between those extremes. The key observation is that explicit wait operators are natural segment boundaries. Within a tile stream, Helix groups contiguous non-blocking compute and communication operators into a segment until it reaches a wait. It then schedules segments across streams in a round-robin style. Communication from one stream can be injected into the compute-heavy region of another stream, but waits still force the graph to respect the original data dependencies.

The scheduler is aggressive only between waits. The wait operators keep the original dependency contract visible.

This segment-level granularity is important. It is coarse enough to avoid the memory explosion of operator-by-operator eager scheduling, because segments are flushed at synchronization boundaries and their intermediates can be released. It is also fine enough to move communication earlier than the original graph would allow under strict serial execution.

In practice, this is where Helix gets much of its generality. The scheduler does not need to know that a node came from tensor parallelism, sequence parallelism, or expert parallelism. If the node is visible in the graph and its dependencies are explicit, the reordering pass can reason about it.

Bucketing: Recover Kernel Efficiency

Tiling creates flexibility, but too much fragmentation hurts hardware efficiency. The bucketing pass repairs that damage selectively.

The idea is to merge compatible operators across tile streams back into larger buckets when doing so improves end-to-end performance. This sounds simple, but it creates a trade-off. Bucketing can reduce kernel-launch overhead and improve compute or communication efficiency. At the same time, it may reintroduce synchronization, reduce scheduling freedom, and extend tensor lifetimes by moving some work earlier.

Helix treats bucketing as a constrained search. For a candidate merge, the graph simulator estimates two quantities: the new makespan and the new peak memory. A merge is useful only if the saved time is worth the additional memory cost and does not destroy the overlap created by tiling and reordering. The implementation uses dynamic programming over candidate buckets, choosing the set of merges that gives the best schedule under the memory budget.

This pass is the reason Helix is not just “split everything and hope.” It deliberately creates overlap granularity, then fuses back the pieces that should not remain separate.

The Simulator Is the Control Loop

The graph simulator is small but central. It runs at compile time and estimates both runtime and peak memory for candidate schedules. For compute and communication cost, it combines graph-visible operator semantics, tensor shapes, analytical modeling, and automated benchmarking. For memory, it simulates execution order and tracks the lifetimes of tensors and communication buffers.

Simulator estimate	Why the optimizer needs it
Candidate makespan	Decide whether a schedule actually hides communication
Peak memory	Reject schedules that create too many long-lived tensors or buffers
Operator and communication costs	Compare tiling, reordering, and bucketing choices before real execution

The simulator does not have to be perfect to be useful. It needs to rank scheduling choices well enough that the compiler avoids obviously bad trade-offs. The paper reports close agreement between estimated and real traces across GPT-3, LLaMA3, and Qwen3-MoE configurations. For example, on a GPT-3 Curie setup with TP=2 and SP=4, the estimated runtime is 6.80 seconds versus 6.41 seconds measured, and the estimated peak memory is 65.9 GiB versus 66.0 GiB measured.

That fidelity matters because the optimizer is making decisions before the real run. Without a simulator, the compiler would either need expensive trial execution or rely on brittle heuristics.

What It Buys

Across GPT-3, LLaMA3, and Qwen3-MoE workloads, Helix shows the same pattern: once communication is exposed to graph scheduling, bubbles shrink and useful GPU work rises. End-to-end training throughput improves by 4% to 9% within a node, and by 12% to 30% when communication crosses nodes. At the layer level, communication bubbles are often reduced by more than 60%, which is the direct evidence that the scheduler is hiding communication rather than merely shifting overhead around.

What Helix buys.

It improves end-to-end training throughput by 4% to 9% within a node.
It improves throughput by 12% to 30% when communication crosses nodes.
It reduces layer-level communication bubbles by more than 60% in many cases.
It reduces long-context inference activation memory by up to 30%.

The memory result is also important. In long-context inference, Helix reduces activation memory by up to 30%, lowering peak memory from 23.5 GiB to 21.4 GiB in the measured trace. This comes from the same design principle as the performance gain: the graph scheduler controls when tiles become live and when their intermediates can be released, instead of letting overlap inflate memory lifetime accidentally.

Helix also compares favorably with hand-tuned tensor-parallel overlap. On large GPT and LLaMA training runs, it reaches 17% and 16% speedups over the baseline, while AsyncTP reports 12% and 13% in the same comparison. The point is not that compiler scheduling makes specialized kernels obsolete. The point is that a graph-level optimizer can find cross-dimensional overlap while keeping correctness, synchronization, kernel efficiency, and memory lifetime in one place.

That is the technical core of Helix: make communication visible, make dependencies explicit, and let the compiler schedule the overlap that manual implementations would otherwise have to rediscover for each workload.

ResiHP: Surviving LLM Training Failures with Dynamic Hybrid Parallelism

Sun, 17 May 2026 14:00:00 +0800

Reference reading: 大模型训练遇到 GPU 故障怎么办？我们的做法是动态调整 3D 并行.

Large-scale LLM training is not one distributed system problem. It is several stacked on top of each other.

At the scale of hundreds or thousands of GPUs, failures are no longer rare events. Some devices disappear completely. Others stay alive but become slower. The second case is especially unpleasant: a fail-slow GPU does not crash the job, but it drags the whole synchronous training iteration behind it. In hybrid parallel training, that delay can propagate through tensor parallelism, pipeline parallelism, and data parallelism until one weak device quietly dictates the speed of the entire job.

ResiHP is built for this setting. Its central idea is to make hybrid parallelism dynamic. Instead of treating the 3D parallel layout as fixed after launch, ResiHP detects unhealthy devices and reshapes the training plan around them.

TL;DR. ResiHP turns resilience into a dynamic parallelism problem: first distinguish real fail-slow behavior from normal sequence-length noise, then reshape TP, PP, and DP together so one bad device does not define the global training speed.

Why Failure Detection Is Hard

The naive signal is iteration time. If one iteration becomes much slower, maybe a device is failing.

That logic is too brittle for LLM training.

Modern LLM workloads often use variable-length sequences. Even when the token budget is controlled by sequence packing, the true attention cost still depends on sequence lengths inside each micro-batch. A packed batch with many long sequences can naturally take longer than a packed batch with shorter ones. Pipeline scheduling adds another layer of noise: the observed iteration time is not just one micro-batch cost, but the critical path of forward, backward, and weight-update chunks across pipeline stages.

This is the point emphasized in the Zhihu writeup: the detector cannot stare at raw iteration time and call every spike a failure. It first needs to ask what the iteration should have cost if all devices were healthy.

Key observation. In LLM training, time is noisy because the workload is noisy. A detector must normalize by expected FLOPs and pipeline schedule before treating slowdown as hardware failure.

FLOPs-Aware Detection

ResiHP’s Detector normalizes iteration time by expected computation.

At the micro-batch level, it estimates the work from the packed sequence structure. Attention is not linear in sequence length, so the model considers the quadratic attention cost rather than only counting tokens. At the pipeline level, ResiHP simulates the schedule of forward, backward, and weight-update chunks to predict the critical path for a healthy iteration.

Only after this normalization does ResiHP compare observed time with expected time. If the gap remains abnormal, the system treats it as a fail-slow signal rather than ordinary sequence-length variation. Fail-stop cases are handled separately through missing heartbeats.

This distinction matters because false positives are costly. A resilient training system that constantly misidentifies normal workload skew as hardware failure will keep reshaping the job for no reason. ResiHP tries to make detection lightweight enough for online use, but accurate enough that adaptation is reserved for real trouble.

Why Hybrid Parallelism Makes Recovery Tricky

Once a device is identified as unhealthy, the simple response is to remove it.

That is rarely enough.

In pure data parallelism, losing one worker mostly reduces replica count. In hybrid parallelism, a device participates in a structure. It may be one rank of a tensor-parallel group, one stage of a pipeline, and one member of a data-parallel replica at the same time. If a tensor-parallel rank fails, the whole TP group is affected. If one pipeline stage slows down, upstream and downstream stages wait. If one data-parallel replica lags, synchronization suffers.

The failure is local, but the performance damage is global.

ResiHP therefore adapts at multiple levels instead of applying one generic workaround. It changes parallelism group sizes, repartitions model layers across pipeline stages, adjusts workload scheduling, and reallocates work among replicas.

A fail-slow GPU is local, but in hybrid parallel training its damage travels through TP groups, PP stages, and DP synchronization.

Dynamic Hybrid Parallelism

The Scheduler is the part of ResiHP that turns detection into a new training plan.

For tensor parallelism, ResiHP can shrink or re-form TP groups around healthy devices. The goal is not simply to drop every device in the affected group, because that may waste too many healthy GPUs. Instead, the scheduler searches for a better group size and membership that preserves useful computation while avoiding the slow or failed rank.

For pipeline parallelism, ResiHP can rebalance model partitioning. A slow stage should not keep the same layer load as healthy stages. If one stage becomes slower, the scheduler can assign it fewer layers and shift work to healthier stages, reducing the pipeline bottleneck.

For data parallelism, ResiHP uses workload migration. If one replica is falling behind while another has capacity, the scheduler can move work so progress becomes more balanced. This is especially useful because data-parallel replicas are logically symmetric, but their actual speed may diverge after a device failure or performance degradation.

The important engineering point is that these adaptations are coordinated. Adjusting TP alone may create pipeline imbalance. Adjusting PP alone may leave healthy GPUs underused. Adjusting DP alone may not remove the original bottleneck. ResiHP treats the layout as a connected 3D object.

Parallel dimension	What can be adapted	Why it helps
Tensor parallelism	Group size and membership	Avoids letting a slow or failed rank poison an entire TP group
Pipeline parallelism	Layer partitioning across stages	Moves work away from slow stages to reduce the pipeline bottleneck
Data parallelism	Workload placement across replicas	Balances progress when replicas diverge in effective speed

Executor Support

A new plan is only useful if the runtime can execute it without turning recovery into a second failure.

ResiHP’s Executor handles the mechanics of dynamic reconfiguration. It reconstructs model and optimizer states under the new parallel layout, updates communication strategies, and supports efficient data movement for the adapted groups. This is where the system moves from scheduling policy to actual fault-tolerant training.

The Executor also matters for fail-stop recovery. If a GPU disappears, the system must preserve training continuity while redistributing the affected model shards and workloads. If a GPU merely slows down, the system must avoid overreacting while still reducing its influence on the global critical path.

What ResiHP Buys

ResiHP was evaluated on a 256-GPU cluster under diverse failure scenarios. The paper reports near-optimal failure detection accuracy and a training throughput improvement of 1.13x to 2.22x compared with state-of-the-art resilient training systems.

What ResiHP buys.

It separates real fail-slow signals from sequence-length and pipeline-schedule noise.
It adapts TP, PP, and DP as one connected 3D layout rather than as isolated knobs.
It improves training throughput by 1.13x to 2.22x in the evaluated failure scenarios.

The broader lesson is that resilience for LLM training cannot be bolted on as a checkpoint-and-restart loop. Hybrid parallelism is already the structure that makes training possible at scale, so resilience has to understand that structure. ResiHP does this by separating three questions:

Is this slowdown a real failure or just sequence-length variation?
Which part of the 3D parallel layout is actually damaged?
How should TP, PP, and DP change together so the job keeps making progress?

That is the shift I like in ResiHP: it treats failure handling as a dynamic parallelism problem, not merely as a device replacement problem.

Paper: ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
Preprint: arXiv:2605.06374

CONCUR: Controlling Mid-Phase Thrashing in Agentic Batch Inference

Sun, 17 May 2026 13:10:00 +0800

Batch inference for LLMs used to be shaped by requests. A request arrives, the server schedules prefill and decode, the KV cache grows for that sequence, and the request eventually leaves.

Agentic workloads are different. An agent is not one request. It is a long-running loop of planning, tool calls, observations, and follow-up generations. Many agents can stay alive at the same time, and each one gradually accumulates KV state. The server may still see individual requests, but the resource pressure is created by agent lifetimes.

CONCUR focuses on the pathology that appears in this setting: mid-phase thrashing.

TL;DR. CONCUR treats agentic batch inference as a congestion-control problem. The key move is to control how many agents are active at once, before their accumulated KV histories push the serving system into eviction and recomputation collapse.

What Is Mid-Phase Thrashing?

A long-running batch of agents does not fail immediately. At the beginning, most agents have short histories, KV cache demand is modest, and throughput looks healthy. Near the end, many agents have already completed, so pressure drops again.

The hard part is the middle.

In the mid phase, many agents are still active and their histories have grown. The aggregate KV cache footprint becomes large, but the GPU memory may not be completely exhausted yet. This is what makes the problem subtle: the system can look feasible by a capacity check, while the cache is already becoming inefficient.

The failure mode. Mid-phase thrashing is not simply out-of-memory. It is a feedback loop: eviction removes histories that live agents soon need again, recomputation consumes GPU time, and the extra churn causes more eviction.

When KV pressure crosses a threshold, request-level cache management starts fighting itself. A serving system may evict old KV blocks to make room for new ones. But agentic workloads soon return to those evicted histories, because the same agents keep generating, calling tools, and continuing. The server then has to recompute or reload context, which consumes GPU time and causes more cache churn. More churn leads to more eviction. More eviction leads to more recomputation. Throughput collapses before memory capacity is formally exhausted.

That collapse is mid-phase thrashing.

Why Request-Level Control Is Too Late

The root cause is a mismatch of control granularity. The serving runtime manages individual requests, but the pressure source is the number of active agents.

If too many agents are admitted together, each agent continues to grow its own history. A reactive cache policy can only respond after the KV cache is already congested. LRU-style eviction may be locally reasonable for a single request stream, but it is a poor global signal for agentic workloads. It does not know that an evicted block belongs to a still-living agent that will likely need it again soon.

In other words, the system is not just running out of memory. It is admitting too many long-lived state machines into a shared cache.

CONCUR changes the question from “which KV block should we evict now?” to “how many agents should be active at the same time?”

Control point	What it sees	Why it is too late or useful
Request-level cache policy	Individual KV blocks and request streams	Reacts after congestion has already started
Static batch sizing	Initial workload shape	Misses the fact that agent histories grow over time
Agent-level admission	Number of active long-lived agents	Acts on the entity that accumulates state

Agent-Level Admission Control

CONCUR adds a lightweight control layer above the LLM serving engine. It does not replace the backend cache manager. Instead, it regulates agent admission so the aggregate active-agent pressure stays below the point where cache efficiency collapses.

The design borrows the spirit of congestion control. The KV cache is treated as a shared bottleneck resource, and the number of concurrently active agents becomes the control window. When runtime cache signals indicate the system is healthy, CONCUR increases concurrency to use more capacity. When the signals show congestion, it backs off before thrashing takes over.

This is closer to AIMD-style control than static batching. Additive increase lets the system cautiously probe for more parallelism. Multiplicative decrease reacts quickly when cache pressure becomes dangerous. The important detail is that the control unit is an agent, not a request. Pausing admission of new agents preserves execution continuity for already-admitted agents and avoids repeatedly evicting the histories they will soon reuse.

This proactive control also preserves compatibility. Existing LLM serving systems can continue to manage request scheduling and KV placement internally. CONCUR only decides how many agents should be allowed into the active set based on cache-aware feedback.

Design shape. CONCUR is intentionally a layer above the serving engine: the backend still manages requests and KV placement, while CONCUR adjusts the active-agent window using cache-aware congestion signals.

Why It Works

Mid-phase thrashing is caused by cumulative state pressure, so the solution has to act before the cache reaches the thrashing regime. By bounding active agents, CONCUR reduces the number of long-lived contexts competing for KV cache at once. The system may run fewer agents concurrently, but each active agent experiences less eviction and recomputation, so useful generation throughput improves.

The paper reports that CONCUR prevents mid-phase thrashing across large models and real-world agent workloads, improving batch inference throughput by up to 4.09x on Qwen3-32B and 1.9x on DeepSeek-V3.

What CONCUR buys.

It prevents the mid-phase collapse caused by eviction/recomputation feedback.
It improves batch inference throughput by up to 4.09x on Qwen3-32B.
It improves throughput by 1.9x on DeepSeek-V3.

The lesson is simple but easy to miss: for agentic inference, the right scheduling object is not always the request. Sometimes the request is only a symptom. The agent is the entity accumulating state, consuming cache over time, and returning to the same history again and again. CONCUR makes that entity visible to the serving system.

Paper: CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control
Preprint: arXiv:2601.22705

ASTRAEA: Fairness Is More Than Counting GPUs

Sun, 17 May 2026 13:00:00 +0800

Fairness sounds simple until a GPU cluster starts running real deep learning workloads.

In a shared research or production cluster, different tenants submit jobs with very different shapes. Some jobs need one GPU for a quick debugging run. Others need many GPUs and run for days. A scheduler that only optimizes utilization may let long jobs dominate the cluster. A scheduler that aggressively favors short jobs may make large training jobs wait forever. Both users can reasonably say the system is unfair.

ASTRAEA was built around this problem: how should a multi-tenant GPU cluster enforce fairness without wasting expensive accelerators?

TL;DR. ASTRAEA makes fairness measurable in the unit the cluster actually spends: long-term GPU-time. It then uses that signal at both tenant and job levels, so fairness accounts for how many GPUs a job occupies and how long it occupies them.

Why Existing Fairness Breaks

Traditional cluster schedulers often think in terms of instantaneous resource fairness. If two users share a cluster, each should receive a fair share of resources at the current moment. This works well for many big-data workloads, where tasks are easier to split, migrate, and rebalance.

Deep learning training is less flexible. Jobs usually require gang scheduling: all requested GPUs must be allocated together. Communication-heavy jobs are sensitive to GPU topology. Preemption is also costly because model state must be checkpointed, moved, and restored. If a scheduler tries to enforce fairness by frequently reshuffling GPUs, it can destroy the performance it was meant to protect.

Another approach is finish-time fairness, where the scheduler asks whether a job would finish no later than it would in a private fair-share cluster. That is useful, but incomplete. It focuses on time and can miss the spatial side of fairness: a job that asks for more GPUs consumes more cluster capacity per unit time. Treating a 1-GPU job and an 8-GPU job only through finish time can create incentives to overclaim resources.

ASTRAEA’s core idea is to measure what the cluster is actually spending: GPU-time.

Key observation. In GPU clusters, fairness has both a spatial dimension and a temporal dimension. Counting only GPUs ignores time; counting only finish time ignores how much cluster capacity a job consumed.

Long-Term GPU-Time Fairness

ASTRAEA introduces Long-Term GPU-Time Fairness, or LTGF. Instead of asking only “how many GPUs does a tenant have right now?” or “when will this job finish?”, LTGF asks how much GPU service a tenant or job has received over a period of time compared with how much it deserves.

This captures both dimensions of allocation:

temporal impact: how long the job runs;
spatial impact: how many GPUs it occupies while running.

At the tenant level, LTGF distributes GPU-time according to tenant weights, such as budget or quota. At the job level, it distributes GPU-time fairly among concurrent jobs inside a tenant. This two-level view is important because a fair cluster should protect both the organization sharing contract and the individual jobs waiting inside each tenant’s queue.

The metric also avoids relying on fragile remaining-time prediction. In real clusters, users cancel jobs, jobs fail, and training throughput changes with placement. ASTRAEA can evaluate fairness from past allocation history, then use that signal to decide who should receive service next.

Fairness becomes easier to reason about once the scheduler measures service in GPU-time instead of only instantaneous allocation or completion time.

How ASTRAEA Schedules

ASTRAEA uses a two-phase scheduling algorithm.

First, it selects the tenant with the lowest tenant-level fairness index. In plain language: the scheduler finds the tenant that has received the least GPU-time relative to what it should have received. If that tenant has pending jobs and the cluster can place one of them, ASTRAEA grants resources to it.

Second, ASTRAEA selects a job within that tenant using the job-level fairness index. This keeps one tenant’s internal queue from becoming unfair even when the tenant as a whole is being treated fairly. Job-level policies can still incorporate practical priorities, but they are constrained by the fairness signal.

The scheduler is lease-based. Instead of preempting whenever fairness changes, ASTRAEA gives a running job a lease term. At lease boundaries, the scheduler can rearrange execution order to repair fairness. This is a practical compromise: short leases improve fairness response, but too-short leases increase preemption overhead and hurt job completion time. ASTRAEA chooses a lease length that balances those forces for deep learning training.

Scheduling layer	Fairness signal	Scheduling decision
Tenant level	Tenant-level LTGF index	Pick the tenant that has received the least service relative to its share
Job level	Job-level LTGF index	Pick a job inside that tenant without making the tenant’s queue unfair
Lease boundary	Updated allocation history	Repair fairness while avoiding constant preemption

What It Buys

ASTRAEA was evaluated with large-scale simulations on real GPU cluster traces, including SenseTime’s Venus trace and Microsoft’s Philly trace. The paper reports that ASTRAEA improves tenant-level fairness by up to 9.42x and job-level fairness by up to 10.3x compared with state-of-the-art schedulers, without sacrificing average job completion time.

What ASTRAEA buys.

It measures fairness in long-term GPU-time, combining space and time.
It improves tenant-level fairness by up to 9.42x.
It improves job-level fairness by up to 10.3x without sacrificing average job completion time.

The important lesson is that fairness in GPU clusters is not just a policy preference. It is a measurement problem. If the metric ignores GPU count, users can overclaim. If it ignores time, long-running jobs can be starved. If it ignores tenants, the cluster violates sharing agreements. If it ignores jobs, individual users still experience unfairness.

ASTRAEA’s contribution is to make fairness measurable in the unit that matters most for deep learning clusters: long-term GPU-time.

Paper: ASTRAEA: A Fair Deep Learning Scheduler for Multi-tenant GPU Clusters
Code: Astraea Artifacts

Hydro: Squeezing Hyperparameter Tuning into Pipeline Bubbles

Sun, 17 May 2026 12:00:00 +0800

Hyperparameter tuning used to feel like a tolerable tax. Train a model many times, sweep a few learning rates and batch sizes, keep the winner. It was expensive, but still part of the normal engineering rhythm.

Then models became large enough that this mental model quietly broke.

If training one model already occupies a large slice of a GPU cluster, a conventional hyperparameter sweep becomes almost absurd: the system asks us to train many near-identical models, most of which exist only to be discarded. Worse, existing tuning frameworks usually see only the resources granted to the tuning job. They do not understand that the cluster around them may contain idle GPU fragments, heterogeneous accelerators, or long-running pipeline-parallel jobs with periodic bubbles.

Hydro started from a simple question: can we make hyperparameter tuning behave less like brute force and more like a systems problem?

TL;DR. This post is mainly about Bubble Squeezer: the Hydro component that turns pipeline-parallel training bubbles into safe, temporary execution slots for hyperparameter tuning trials. Its job is not just to “use idle GPUs”; it must detect when a bubble is actually safe, fit the right amount of surrogate work into that window, and pause before the primary training job notices.

Hydro has two sides. At the job level, it makes each trial cheaper by tuning a smaller surrogate model. At the cluster level, it asks a more interesting question: can the datacenter run those cheap trials in GPU time that is currently wasted?

First, Make Trials Small Enough

Hydro’s first move is to avoid tuning the target model directly whenever possible. It shrinks the model, tunes the smaller version, and transfers the discovered hyperparameters back to the original model. The danger is that naive shrinking changes training dynamics. A learning rate that works for a narrow model may fail badly for a wider one, so a cheap search can produce misleading answers.

Hydro makes this idea practical through parametrization, specifically a system adaptation of maximal update parametrization. Instead of only changing layer widths, Hydro adjusts initialization and optimizer behavior layer by layer so that models of different widths preserve comparable update scales during training. In more practical terms, the surrogate and the target model are encouraged to agree on which hyperparameter configurations are good.

The implementation is intentionally service-oriented. Model Shrinker traces the PyTorch model with torch.fx, scales eligible layers, applies the parametrization rules, and runs a lightweight correctness check before the tuning job proceeds. Trial Binder then fuses many small surrogate trials into one batched execution unit through grouped hydro.nn modules. This matters because a single surrogate trial may be too small to keep an A100 busy; fusion turns many tiny trials into a better-shaped GPU workload.

These pieces are important, but in this post I want to focus on the part that feels most datacenter-native: Bubble Squeezer.

Background component	What it gives Bubble Squeezer
Model Shrinker	Surrogate trials small enough to fit into short bubble windows
Trial Binder	Fused trial bundles whose size can be adjusted to match leftover memory
`hydro.nn` modules	Hook points where Bubble Squeezer can pause and resume execution

The Cluster Is Part of the Tuning System

Most tuning frameworks treat the cluster scheduler as a resource vending machine. Hydro treats the cluster as part of the optimization surface.

The Hydro Coordinator adds this cluster-level view. Its most distinctive component is Bubble Squeezer, which targets long-running pipeline-parallel training jobs. Pipeline parallelism is common for large models because the model is split into stages placed across multiple GPUs or nodes. In the widely used 1F1B schedule, each worker alternates forward and backward microbatches, but the schedule is not perfectly dense. A stage may finish the forward pass for one microbatch and then wait for another stage to produce the corresponding backward work. That waiting interval is a pipeline bubble.

For the large training job, bubbles are awkward. They are short, appear repeatedly, and are mixed with communication. During a bubble, the only active kernel may be NCCL communication, so SM activity can be extremely low even though the GPU is technically allocated. For a normal training job, this is not enough room to run safely. For Hydro, it is an opening.

Bubble Squeezer’s role. It looks inside an already allocated pipeline-parallel job and extracts a new kind of resource: short, repeated, stage-local GPU windows where compute is low but memory may still be available.

HydroTrials are unusually suitable for bubble execution for three reasons. First, they are throughput-tolerant: slowing down one candidate trial is acceptable as long as the tuning job as a whole progresses. Second, they are profiled: Hydro knows the memory and compute footprint of each fused trial before placing it near a large model. Third, they are elastic: the fusion count can be adjusted so that a trial bundle fits the leftover memory and time budget of a bubble.

How Bubble Squeezer Works

Bubble Squeezer turns pipeline bubbles into ephemeral resources. When a pipeline-parallel large-model job is running, Hydro coordinates with the datacenter scheduler to acquire these temporary opportunities and tags the corresponding GPUs as usable only during bubbles. The goal is deliberately narrow: run tuning work without slowing down the primary large-model training job.

The hard part is not finding idle-looking time. The hard part is making that time usable without turning colocation into interference.

The control loop has three responsibilities:

Detect the opening. Hydro modifies the DeepSpeed-based execution path to report pipeline progress and resource consumption. This tells Bubble Squeezer when a worker is entering a bubble and how much memory is still available.
Check that the opening is safe. Hydro watches CUDA stream status for NCCL kernels so it can distinguish communication-heavy waiting time from compute time.
Control the guest work. Bubble Squeezer registers hooks on hydro.nn modules so a HydroTrial can pause and resume at fine granularity, including inside forward and backward passes.

That second responsibility is crucial: direct colocation would let two workloads compete blindly, and the paper reports about 12% slowdown to the large model under direct colocation.

At the start of a bubble, Hydro resumes a set of fused surrogate trials. At the end of the bubble, it pauses them again before the large model needs the GPU. The implementation uses Linux signals for pause and resume control, while the scheduling decision is guided by the profiled trial footprint and the currently available memory.

Figure 1. Bubble Squeezer interleaves HydroTrials into idle intervals of a pipeline-parallel large-model training job.

The fusion count is not fixed. If a pipeline stage has more spare memory, Hydro can run a larger fused HydroTrial. If the stage is tighter, Hydro can reduce the fusion count or skip that bubble. This is the small but important connection between the job-level and cluster-level parts of Hydro: surrogate scaling makes each trial small, trial fusion shapes the work, and Bubble Squeezer chooses how much of that shaped work can fit into a specific bubble.

Bubble Squeezer decision	Signal or mechanism	Why it matters
When can a HydroTrial run?	Pipeline progress reports from the large-model job	Bubbles are short and stage-local, so timing must follow the pipeline schedule
How much work should run?	Profiled trial footprint plus available memory	The fusion count can be increased, reduced, or skipped per bubble
When should work stop?	Fine-grained pause/resume hooks in fused HydroTrials	Guest work must yield before the large model returns to compute

The ideal case is a long-running pipeline-parallel foundation-model job with multiple stages across multiple servers. More stages usually mean more bubbles and more ephemeral execution slots. Multi-fidelity tuning also fits especially well, because many unpromising trials can be advanced or eliminated using bubble resources while the strongest trials later receive exclusive resources.

What the Bubbles Buy

The evaluation gives a useful sense of scale. Hydro interleaves ResNet-18 HydroTrials with a large GPT training job running on 32 A100 GPUs across 4 pipeline stages. In the original GPT training trace, SM activity inside bubbles is about 2%. With Bubble Squeezer, Hydro raises that bubble-period SM utilization to about 50% without evident slowdown to the GPT job.

Figure 2. Interleaving HydroTrials increases useful GPU activity during bubbles while keeping the large-model training timeline stable.

The tuning work does not run as fast as it would on an exclusive GPU, and that is expected. In the experiment, a HydroTrial with fusion count 16 obtains about 15% of its exclusive throughput while living inside bubbles. But the resource is effectively reclaimed from otherwise idle intervals. In a simulated end-to-end setting where the tuning job has only 1 exclusive GPU because the large model occupies most of the cluster, Bubble Squeezer reduces tuning makespan by 2.7x.

What Bubble Squeezer buys.

It raises bubble-period SM activity from about 2% to about 50%.
It lets a fused HydroTrial make progress at about 15% of exclusive throughput inside bubbles.
It avoids the blind-colocation failure mode, where the paper reports about 12% slowdown to the large model.
It reduces HPO makespan by 2.7x in the constrained end-to-end setting.

This is the part of Hydro I find most interesting. A scheduler usually sees a GPU assigned to a large training job as unavailable. Bubble Squeezer looks inside that allocation and finds repeatable, bounded, low-interference windows where small, profiled, pauseable work can make progress.

The broader Hydro system still matters: surrogate scaling makes trials cheap, fusion shapes them into efficient bundles, and Bubble Squeezer places those bundles into pipeline bubbles. Together, these pieces turn HPO from a brute-force outer loop into a datacenter-aware service.

There are limits. Parametrization is most effective for hyperparameters that control initialization and training dynamics, such as learning rate, batch size, schedulers, and momentum. Regularization-related choices like dropout and weight decay are harder because they depend more directly on model and data scale. Some architectures may also require tailored analysis. Hydro does not claim that every hyperparameter can be transferred for every model.

But the central lesson is durable: once model training becomes datacenter-scale, hyperparameter search must understand the model, the runtime, and the cluster. Hydro is one attempt to make that full stack visible.

Paper: Hydro: Surrogate-Based Hyperparameter Tuning Service in Datacenters
Code: S-Lab-System-Group/Hydro

GPU Cluster Scheduling: A Map for Deep Learning Workloads

Sat, 16 May 2026 14:30:00 +0800

GPU cluster scheduling is easy to underestimate. At first glance, it looks like a familiar resource allocation problem: jobs arrive, GPUs are free or busy, and the scheduler decides who runs next.

Deep learning breaks that simplicity.

Training jobs can run for days, need gangs of GPUs, and care deeply about placement topology. Inference services are online, latency-sensitive, and often underutilize a GPU unless requests are batched or colocated. Hyperparameter tuning launches many similar trials, most of which are meant to be discarded. LLM workloads add model parallelism, massive memory footprints, long contexts, and bursty development patterns.

Our survey, Deep Learning Workload Scheduling in GPU Datacenters, tries to organize this messy design space. The most useful way to read the field is not as a list of schedulers, but as a set of tensions: speed versus cost, utilization versus isolation, fairness versus efficiency, and online latency versus cluster-wide throughput.

Why DL Scheduling Is Different

Traditional HPC and big-data schedulers provide useful starting points, but DL workloads have their own physics.

Training jobs are often gang-scheduled. A distributed job needs all requested GPUs at the same time, so GPUs are not easily divisible like CPU slots. Placement matters because communication-heavy jobs may run much faster when GPUs are packed within a node or connected by NVLink rather than scattered across weaker links. Preemption is expensive because model and optimizer states are large. At the same time, training is iterative, so a few profiled iterations can often reveal throughput, memory behavior, and placement sensitivity.

Inference has nearly opposite pressure. Each request is small compared with a training job, but the service has latency SLOs. Batching improves GPU utilization, yet waiting too long to form a batch hurts latency. Colocation improves throughput, yet interference can violate tail latency. The scheduler has to trade average efficiency against worst-case user experience.

This is why GPU cluster scheduling is not one problem. It is a family of related problems whose correct answer depends on the workload.

Training: Efficiency, Fairness, Deadlines

For training workloads, the survey groups scheduling objectives into three broad categories.

The first is efficiency. Some schedulers reduce job completion time through priority rules, such as least attained service or progress-aware variants. Others use profiling or learning-based methods to predict job duration, speed, placement sensitivity, or future resource needs. Placement is a core part of efficiency: a scheduler can have enough GPUs in aggregate but still produce poor performance if it fragments the cluster and cannot satisfy locality.

The second is fairness. Fairness is subtle because GPUs are indivisible in common gang-scheduling settings, and heterogeneous GPUs do not provide equal value to every job. Finish-time fairness, long-term GPU-time fairness, and heterogeneity-aware fairness all try to answer a version of the same question: how much service did this job or tenant deserve, and how much did it actually receive?

The third is deadline guarantee. Deadline-aware training is less explored, but important for production workflows. A best-effort job can tolerate delay; an SLO job cannot. Systems in this direction need to predict whether a job can finish before its deadline under different placements and resource allocations, then decide how to mix deadline jobs with normal jobs.

Training: How GPUs Are Used

Objectives are only half the taxonomy. The other half is how a scheduler uses resources.

Heterogeneous resource scheduling recognizes that “a GPU” is not a uniform unit. Different model architectures benefit differently from newer GPU generations, CPU allocation, memory, network bandwidth, and storage. A cost-effective scheduler should place jobs where their bottlenecks match the available hardware, not blindly send every job to the newest device.

GPU sharing attacks the underutilization problem. Many training jobs cannot saturate a modern GPU. Packing multiple jobs onto one device through MPS, MIG, virtualization, time sharing, or framework-level co-execution can improve utilization. The risk is interference: the scheduler must know when sharing helps and when it silently slows everything down.

Elastic training changes the number of GPUs assigned to a job over time. This can reduce queueing and improve utilization, especially when demand fluctuates. But elasticity is not free. Resource changes may require checkpointing, reinitialization, or batch-size adaptation. If batch size changes affect convergence, a scheduler may improve system throughput while quietly changing model behavior.

The broad lesson is that training schedulers increasingly need to be co-designed with training frameworks. The scheduler wants fine-grained control, but the framework knows whether a job can safely pause, resize, share, or change batch size.

Inference: Latency, Cost, Throughput

Inference scheduling is shaped by a different triangle: latency, cost, and accuracy.

Latency is usually the first-class constraint. A model serving system can improve throughput by batching requests, but a request waiting in a queue is still user-visible latency. A practical scheduler often uses dynamic batching: increase batch size when the service is healthy, shrink it when latency approaches the SLO.

Cost enters through cloud instance choice, autoscaling, and heterogeneous hardware. Some workloads are cheaper on CPU, some need GPU, and some become cost-efficient only when batching is large enough. The scheduler has to decide not only where to run a model, but how many replicas and which instance types are worth paying for.

Accuracy adds another axis. Some systems choose among model variants, ensembles, or modalities. A smaller model may be cheap and fast but less accurate; a larger model may be slower but better. This turns inference scheduling into a policy problem: what accuracy loss is acceptable for a given latency or cost budget?

Throughput techniques include batching, caching, model residency, and colocation. But inference colocation is more dangerous than training colocation because SLO violations are immediate. A scheduler needs interference models, isolation mechanisms, or hardware partitioning to make sharing safe.

Beyond Training and Inference

Some workloads deserve their own category.

Hyperparameter optimization is technically training, but operationally different. It launches many similar trials, prunes weak ones, and shifts resources toward promising configurations. This structure creates opportunities for early stopping, elastic trial allocation, trial packing, model fusion, and surrogate-based tuning. Our Hydro work is one example: it uses model scaling, trial fusion, and cluster-level interleaving to make HPO less brute-force.

Mixed training and inference workloads are another frontier. Inference clusters are often overprovisioned for bursts, leaving idle GPUs during low-traffic periods. Training jobs can sometimes borrow that capacity if the system can preempt or resize them quickly when inference demand returns. The challenge is respecting online SLOs while reclaiming otherwise wasted capacity.

These cases point to a larger trend: future schedulers will be more workload-aware. A generic GPU queue is too blunt for the diversity of DL development.

Where the Field Is Going

The survey ends with three research directions that still feel current.

First, emerging workloads will keep changing scheduler design. LLM pretraining, fine-tuning, serving, agentic inference, and HPO all expose different bottlenecks. The scheduler must understand more than GPU count; it must understand memory pressure, communication structure, context length, trial similarity, and elasticity.

Second, scheduling decisions need better intelligence. Heuristics are robust and deployable, mathematical optimization can be principled but slow, and ML/RL-based schedulers can capture complex patterns but are hard to trust and benchmark. A practical scheduler may combine all three: heuristics for the fast path, profiling for calibration, and optimization or learning for difficult decisions.

Third, hardware heterogeneity is becoming unavoidable. A production cluster may contain multiple GPU generations, specialized interconnects, CPUs, storage tiers, and accelerators. Heterogeneity creates opportunities for better cost-performance, but it also complicates fairness. Allocating an old GPU and a new GPU for the same amount of wall-clock time is rarely equal service.

The simplest summary is this: GPU scheduling is no longer just about filling empty slots. It is about matching workload structure to hardware structure under user-visible objectives.

That is what makes the area interesting. The best scheduler is not merely the one with the shortest queue. It is the one that understands what kind of deep learning work is in front of it, what resources it truly needs, and what trade-off the cluster is willing to make.

Paper: Deep Learning Workload Scheduling in GPU Datacenters: A Survey
Project: Awesome DL Scheduling Papers

GPU Pause, Resume, and Migration: The Missing Primitive in Cluster Scheduling

Fri, 15 May 2026 15:00:00 +0800

GPU cluster scheduling would be much easier if a running GPU job behaved like an ordinary CPU process. Pause it. Move it. Resume it somewhere else. Reclaim the device when a higher-priority job arrives. Repair fragmentation without killing user work.

In practice, this is exactly where GPU scheduling gets stuck.

A CPU process can be checkpointed by saving its address space, file descriptors, and kernel-visible state. A GPU task has an extra half of its life outside the normal process abstraction: CUDA contexts, device allocations, streams, events, library handles, kernels in flight, and data resident in GPU memory. The operating system does not naturally know how to serialize that state. The scheduler can stop the host process, but that is not the same thing as having a correct, portable checkpoint of the GPU computation.

FlowGPU is about turning GPU checkpoint/restore into a system primitive. Before FlowGPU became a full system, I wrote cudaw as the first version of the codebase: a CUDA wrapper prototype for interposing on runtime calls, tracking GPU objects, translating application-visible addresses, and making pause/resume/migration possible above an unmodified CUDA application.

Why Schedulers Want This Primitive

Pause/resume and migration change what a scheduler can do.

Without GPU checkpoint/restore, preemption is often blunt. A scheduler can kill a job, ask the framework to checkpoint at a pre-defined training boundary, or wait until the user code cooperates. That is acceptable for some training loops, but it is poorly aligned with cluster events. A high-priority job may arrive now. A GPU may fail now. A fragmented placement may need repair now. Framework-level checkpoints are usually placed for application convenience, not scheduler control.

With a transparent GPU checkpoint, the scheduler gets stronger operations:

pause a job and release its GPU memory;
resume it later on the same GPU;
migrate it to another GPU or node;
checkpoint periodically for fault tolerance;
defragment the cluster by moving jobs away from awkward placements;
support elastic scaling and priority scheduling with less user code involvement.

This is the missing link between scheduling policy and GPU execution. A scheduler may know the right decision, but without a safe migration primitive, it cannot act on that decision cheaply.

The CUDA Wrapper View

The basic idea behind my cudaw prototype is to place a wrapper between the application and CUDA runtime. Instead of letting the application talk directly to libcudart, the wrapper intercepts CUDA calls such as allocation, memory copy, and kernel launch. From the scheduler’s perspective, this creates an execution log and a shadow view of GPU state.

This wrapper layer can record which device memory regions exist, what host-side pointers correspond to them, how data moves between CPU and GPU, and which kernels are launched with which arguments. It can also maintain virtual GPU addresses: the application sees stable logical addresses, while the wrapper maps them to real CUDA allocations underneath. That indirection is what makes restore and migration plausible, because the restored task may receive different physical GPU addresses on the target device.

In a simplified checkpoint flow, the wrapper reaches a safe point, synchronizes GPU work, copies live GPU memory into a checkpoint image, saves enough metadata to reconstruct CUDA state, and releases the device. Restore reverses the process: allocate memory on the target GPU, rebuild mappings, copy data back, replay necessary CUDA setup calls, and continue execution.

This early prototype captured the central intuition that later shaped FlowGPU. GPU migration is not magic; it is state reconstruction. The hard part is making the reconstructed world indistinguishable from the original one.

Where Wrapper-Only Designs Struggle

The wrapper idea is powerful, but the edge cases are brutal.

First, CUDA state is larger than cudaMalloc and cudaMemcpy. Real applications use streams, events, cuBLAS, cuDNN, NCCL, memory pools, unified memory, graph execution, and framework allocators. Many of these objects are opaque: CUDA exposes handles, not serializable internals. A checkpoint system must record and replay the operations that created or mutated them.

Second, address identity matters. A pointer value may be stored inside application data structures, kernel arguments, framework metadata, or library state. If restore gives the program a different GPU virtual address, the application can become subtly wrong even if the bytes were copied correctly.

Third, deep learning frameworks hide memory behavior. PyTorch and TensorFlow often reserve large GPU memory blocks and keep them for reuse. Much of that reserved memory may be inactive at a given moment. A naive checkpoint that saves everything allocated by the runtime can produce enormous checkpoint images, even when the useful live state is much smaller.

Fourth, distributed training is a synchronization problem. A consistent checkpoint of a multi-GPU job requires pausing all participating ranks safely. With NCCL communication, pausing one side of a blocking send/receive pair while the other side waits can deadlock the checkpoint protocol itself.

These are the problems FlowGPU is designed to handle systematically.

FlowGPU’s Core Move

FlowGPU’s key insight is that prior system-level GPU checkpoint/restore designs coupled C/R with API forwarding. In API forwarding, all GPU operations pass through a privileged central process. That makes interception and state separation easier, but it imposes runtime overhead, creates GPU address conflicts under sharing, and blocks some GPU features.

FlowGPU decouples checkpoint/restore from virtualization.

During normal execution, each task uses a per-task intercept library. GPU operations stay private to that task and go directly to the GPU, avoiding the IPC overhead of a central forwarding process. When checkpointing is needed, FlowGPU creates a ghost process. The ghost process temporarily takes over GPU state, while the original process becomes a conventional CPU process that can be checkpointed with CRIU. GPU state and CPU state are saved in parallel, then recombined during restore.

This design keeps the useful part of interception without forcing every GPU operation through a virtualization server during normal execution.

Making Checkpoints Small and Correct

FlowGPU adds several mechanisms that are especially important for deep learning workloads.

Active memory identification avoids saving the whole framework-reserved memory pool. FlowGPU inserts a memory stub at stable DL framework backend allocation/free interfaces, tracking the memory regions that are actually active. It can also wait briefly for active memory to reach a low point in the training iteration before checkpointing. This matters because active memory in training can fluctuate dramatically between the end of an iteration and the activation-heavy middle of forward/backward execution.

Virtual memory management preserves GPU address identity. FlowGPU intercepts GPU allocations and uses CUDA VMM APIs such as cuMemAddressReserve, cuMemCreate, and cuMemMap to reserve and remap the same virtual addresses on restore. That removes a major source of correctness bugs for pointer-rich GPU applications.

Record/replay handles opaque runtime objects. Since CUDA streams, events, contexts, and library handles cannot simply be read out as bytes, FlowGPU records operations that create or modify them and replays those operations during recovery.

The pause mechanism is refined for distributed tasks. FlowGPU coordinates pausing across ranks, but avoids a known NCCL deadlock pattern by resuming all instances after a timeout if a complete pause cannot be achieved. This is a small detail with a large consequence: checkpointing must not introduce a failure mode worse than the one it tries to solve.

For multi-GPU tasks, FlowGPU also performs fine-grained deduplication. Replicated model parameters may appear on multiple GPUs, but runtime memory blocks rarely match exactly. FlowGPU deduplicates fixed-size regions, reducing checkpoint image size for distributed jobs.

What This Means for Scheduling

Once GPU pause/resume becomes practical, several scheduling policies become more realistic.

Priority scheduling can preempt a low-priority GPU job without throwing away all its progress. Fairness scheduling can redistribute service over time with lower disruption. Fragmentation-aware schedulers can migrate jobs to rebuild contiguous placements for gang-scheduled workloads. Fault-tolerance systems can checkpoint at scheduler-controlled intervals instead of relying only on framework checkpoints. Elastic schedulers can shrink, expand, or relocate jobs with a clearer recovery path.

The primitive also changes the economics of GPU sharing. If a job can be paused and restored quickly, a cluster can take more aggressive actions under bursty demand. Online inference, training, and HPO workloads no longer need to live in completely isolated resource islands; the scheduler has a better way to move work when priorities change.

FlowGPU’s evaluation shows why the details matter. It reports no runtime overhead during normal single-GPU execution because tasks can access the GPU directly without API forwarding. For DL tasks, it reduces checkpoint pause time by 6.2x to 15x over POS and up to 10.4x over Singularity. Restore time drops by 12x to 18x over POS and up to 4.1x over Singularity. For migration, FlowGPU outperforms Singularity by up to 2.1x and PyTorch framework-level checkpointing by 1.7x to 4.5x.

Those numbers are not only checkpointing results. They are scheduling-enablement results. A slow checkpoint is a policy that the scheduler cannot afford to use often. A fast, transparent checkpoint becomes a real control knob.

The Takeaway

GPU scheduling is often discussed in terms of algorithms: fairness metrics, placement heuristics, bin packing, elastic allocation, and priority queues. But the scheduler is only as powerful as the execution primitives beneath it.

cudaw was my first working cut at the wrapper-level intuition: interpose on CUDA, virtualize what the application sees, and reconstruct GPU state when needed. FlowGPU pushes that intuition into a more complete system design: per-task interception for low overhead, ghost processes for state separation, active-memory tracking for small images, VMM for address correctness, and distributed pause logic for multi-GPU workloads.

The result is a cleaner boundary between policy and mechanism. The scheduler decides when a job should pause, resume, or move. The checkpoint/restore layer makes that decision safe enough to execute.

Paper: FlowGPU: Transparent and Efficient GPU Checkpointing and Restore
Early codebase: yzs981130/cudaw

Optimizations and Services

Mon, 30 Aug 2021 12:10:58 +0800

Optimizations applied to this site

Building process

The whole site is make up with a few static pages, generated by hugo. It is hard for a Golang user to comment on whether is right to use go modules to manage blog themes for hugo or not, along with the Golang environment requirements for hugo bin to execute. At least the building process is very fast. Similarly with many other hugo powered sites, the building process of yezhisheng.me is also triggered by Github actions on push to the repo. Then content is delivered by CDN to end users like you.

CDN

Cloudflare pages for global serving.
Upyun for serving https://yezhisheng.com.cn.

After compared with Vercel with Cloudflare, I finally chose Cloudflare for wide IPv6 support and fast response time in CERNET, in mainland China. However, Vercel has more advantages in building pipeline (less restrictions compared with cloudflare pages) and higher quality for other ISPs in mainland China. Vercel also provides automatic SSL cert renewal by LetsEncrypt, which is also convenient for setting up a CAA record. Cloudflare is quite familiar with me for continous usage for over 5 years. It is also very easy to leverage CNAME accelartion by simple click.

Another important thing needs to mention is Cloudflare pages is also enabled to directly mirroring html files from the original repo of Github pages, which should amortize the back-to-origin overhead to sub-milliseconds.

Meanwhile, things may be a little different in mainland China. Either the OSS or the CDN with HTTPS is not free for almost all cloud providers, including Aliyun, Tencent Cloud and Qiniu Cloud, not to mention AWS China and Azure China. I finally chose Upyun for free HTTPS CDN along with http3 support, in the compromise of less CDN hosts and no https rewrites. The origin server is set to Github pages directly.

Current service

Benefit from the great developing experience and easy usage of Cloudflare, serveral services are gradually coming to live with no additional cost, supported by Cloudflare CDN and Cloudflare workers.

Posts | 木叶吟

Helix: Automating Communication-Computation Overlap with Graph Scheduling

Why Manual Overlap Does Not Scale

The Scheduling Objective

Tiling: Create Overlap Opportunities

Reordering: Make Overlap Safe

Bucketing: Recover Kernel Efficiency

The Simulator Is the Control Loop

What It Buys

ResiHP: Surviving LLM Training Failures with Dynamic Hybrid Parallelism

Why Failure Detection Is Hard

FLOPs-Aware Detection

Why Hybrid Parallelism Makes Recovery Tricky

Dynamic Hybrid Parallelism

Executor Support

What ResiHP Buys

CONCUR: Controlling Mid-Phase Thrashing in Agentic Batch Inference

What Is Mid-Phase Thrashing?

Why Request-Level Control Is Too Late

Agent-Level Admission Control

Why It Works

ASTRAEA: Fairness Is More Than Counting GPUs

Why Existing Fairness Breaks

Long-Term GPU-Time Fairness

How ASTRAEA Schedules

What It Buys

Hydro: Squeezing Hyperparameter Tuning into Pipeline Bubbles

First, Make Trials Small Enough

The Cluster Is Part of the Tuning System

How Bubble Squeezer Works

What the Bubbles Buy

GPU Cluster Scheduling: A Map for Deep Learning Workloads

Why DL Scheduling Is Different

Training: Efficiency, Fairness, Deadlines

Training: How GPUs Are Used

Inference: Latency, Cost, Throughput

Beyond Training and Inference

Where the Field Is Going

GPU Pause, Resume, and Migration: The Missing Primitive in Cluster Scheduling

Why Schedulers Want This Primitive

The CUDA Wrapper View

Where Wrapper-Only Designs Struggle

FlowGPU’s Core Move

Making Checkpoints Small and Correct

What This Means for Scheduling

The Takeaway

Optimizations and Services

Optimizations applied to this site

Building process

CDN

Theme related optimizations

Current service

Reverse proxy

Other