Deep Learning Systems | 木叶吟

ASTRAEA: Fairness Is More Than Counting GPUs

Sun, 17 May 2026 13:00:00 +0800

Fairness sounds simple until a GPU cluster starts running real deep learning workloads.

In a shared research or production cluster, different tenants submit jobs with very different shapes. Some jobs need one GPU for a quick debugging run. Others need many GPUs and run for days. A scheduler that only optimizes utilization may let long jobs dominate the cluster. A scheduler that aggressively favors short jobs may make large training jobs wait forever. Both users can reasonably say the system is unfair.

ASTRAEA was built around this problem: how should a multi-tenant GPU cluster enforce fairness without wasting expensive accelerators?

TL;DR. ASTRAEA makes fairness measurable in the unit the cluster actually spends: long-term GPU-time. It then uses that signal at both tenant and job levels, so fairness accounts for how many GPUs a job occupies and how long it occupies them.

Why Existing Fairness Breaks

Traditional cluster schedulers often think in terms of instantaneous resource fairness. If two users share a cluster, each should receive a fair share of resources at the current moment. This works well for many big-data workloads, where tasks are easier to split, migrate, and rebalance.

Deep learning training is less flexible. Jobs usually require gang scheduling: all requested GPUs must be allocated together. Communication-heavy jobs are sensitive to GPU topology. Preemption is also costly because model state must be checkpointed, moved, and restored. If a scheduler tries to enforce fairness by frequently reshuffling GPUs, it can destroy the performance it was meant to protect.

Another approach is finish-time fairness, where the scheduler asks whether a job would finish no later than it would in a private fair-share cluster. That is useful, but incomplete. It focuses on time and can miss the spatial side of fairness: a job that asks for more GPUs consumes more cluster capacity per unit time. Treating a 1-GPU job and an 8-GPU job only through finish time can create incentives to overclaim resources.

ASTRAEA’s core idea is to measure what the cluster is actually spending: GPU-time.

Key observation. In GPU clusters, fairness has both a spatial dimension and a temporal dimension. Counting only GPUs ignores time; counting only finish time ignores how much cluster capacity a job consumed.

Long-Term GPU-Time Fairness

ASTRAEA introduces Long-Term GPU-Time Fairness, or LTGF. Instead of asking only “how many GPUs does a tenant have right now?” or “when will this job finish?”, LTGF asks how much GPU service a tenant or job has received over a period of time compared with how much it deserves.

This captures both dimensions of allocation:

temporal impact: how long the job runs;
spatial impact: how many GPUs it occupies while running.

At the tenant level, LTGF distributes GPU-time according to tenant weights, such as budget or quota. At the job level, it distributes GPU-time fairly among concurrent jobs inside a tenant. This two-level view is important because a fair cluster should protect both the organization sharing contract and the individual jobs waiting inside each tenant’s queue.

The metric also avoids relying on fragile remaining-time prediction. In real clusters, users cancel jobs, jobs fail, and training throughput changes with placement. ASTRAEA can evaluate fairness from past allocation history, then use that signal to decide who should receive service next.

Fairness becomes easier to reason about once the scheduler measures service in GPU-time instead of only instantaneous allocation or completion time.

How ASTRAEA Schedules

ASTRAEA uses a two-phase scheduling algorithm.

First, it selects the tenant with the lowest tenant-level fairness index. In plain language: the scheduler finds the tenant that has received the least GPU-time relative to what it should have received. If that tenant has pending jobs and the cluster can place one of them, ASTRAEA grants resources to it.

Second, ASTRAEA selects a job within that tenant using the job-level fairness index. This keeps one tenant’s internal queue from becoming unfair even when the tenant as a whole is being treated fairly. Job-level policies can still incorporate practical priorities, but they are constrained by the fairness signal.

The scheduler is lease-based. Instead of preempting whenever fairness changes, ASTRAEA gives a running job a lease term. At lease boundaries, the scheduler can rearrange execution order to repair fairness. This is a practical compromise: short leases improve fairness response, but too-short leases increase preemption overhead and hurt job completion time. ASTRAEA chooses a lease length that balances those forces for deep learning training.

Scheduling layer	Fairness signal	Scheduling decision
Tenant level	Tenant-level LTGF index	Pick the tenant that has received the least service relative to its share
Job level	Job-level LTGF index	Pick a job inside that tenant without making the tenant’s queue unfair
Lease boundary	Updated allocation history	Repair fairness while avoiding constant preemption

What It Buys

ASTRAEA was evaluated with large-scale simulations on real GPU cluster traces, including SenseTime’s Venus trace and Microsoft’s Philly trace. The paper reports that ASTRAEA improves tenant-level fairness by up to 9.42x and job-level fairness by up to 10.3x compared with state-of-the-art schedulers, without sacrificing average job completion time.

What ASTRAEA buys.

It measures fairness in long-term GPU-time, combining space and time.
It improves tenant-level fairness by up to 9.42x.
It improves job-level fairness by up to 10.3x without sacrificing average job completion time.

The important lesson is that fairness in GPU clusters is not just a policy preference. It is a measurement problem. If the metric ignores GPU count, users can overclaim. If it ignores time, long-running jobs can be starved. If it ignores tenants, the cluster violates sharing agreements. If it ignores jobs, individual users still experience unfairness.

ASTRAEA’s contribution is to make fairness measurable in the unit that matters most for deep learning clusters: long-term GPU-time.

Paper: ASTRAEA: A Fair Deep Learning Scheduler for Multi-tenant GPU Clusters
Code: Astraea Artifacts

ASTRAEA：GPU 集群里的公平，不只是分到几张卡

Sun, 17 May 2026 13:00:00 +0800

公平性听起来很简单，直到一个 GPU 集群真的开始承载各种深度学习任务。

在共享的科研或生产集群里，不同租户提交的任务形态差异很大。有些任务只需要一张 GPU，跑几分钟做调试；有些训练需要很多张 GPU，并且连续运行好几天。只追求利用率的调度器，可能让长任务长期占据集群；过度偏向短任务的调度器，又可能让大型训练一直排队。两类用户都可以很合理地说：这个系统不公平。

ASTRAEA 关注的正是这个问题：多租户 GPU 集群如何在不浪费昂贵加速器的前提下，把公平性真正落到调度决策里？

TL;DR. ASTRAEA 把公平性度量放到集群真正消耗的单位上：长期 GPU-time。它同时在租户层和任务层使用这个信号，让公平性既考虑任务占了多少 GPU，也考虑占用了多久。

为什么已有公平性会失效

传统集群调度器经常从“瞬时资源公平”的角度思考问题。比如两个用户共享一个集群，那么在某个时刻，每个人都应该拿到自己的公平份额。对很多大数据任务来说，这个思路很自然，因为任务更容易切分、迁移和重新平衡。

深度学习训练没有这么灵活。训练任务通常需要 gang scheduling：请求的所有 GPU 必须同时分配到位。通信密集型任务对 GPU 拓扑很敏感。抢占也很贵，因为模型状态和优化器状态需要 checkpoint、移动和恢复。如果调度器为了追求公平而频繁重排 GPU，反而可能破坏原本想保护的性能。

另一类方法是 finish-time fairness，也就是判断一个任务是否不晚于它在私有 fair-share cluster 里的完成时间。这个目标有用，但不完整。它强调时间，却容易忽略公平性的空间维度：一个请求更多 GPU 的任务，在单位时间内消耗了更多集群容量。只用完成时间看待 1-GPU 任务和 8-GPU 任务，可能鼓励用户多报资源。

ASTRAEA 的核心想法是直接度量集群真正付出的东西：GPU-time。

关键观察。 GPU 集群中的公平性同时有空间维度和时间维度。只数 GPU 会忽略时间，只看完成时间又会忽略一个任务消耗了多少集群容量。

长期 GPU-Time 公平性

ASTRAEA 提出了 Long-Term GPU-Time Fairness，简称 LTGF。它不只问“这个租户现在有几张 GPU”，也不只问“这个任务什么时候完成”，而是问：在一段时间内，某个租户或任务实际获得了多少 GPU service，相比它应得的份额是否公平。

这个指标同时捕捉了分配的两个维度：

时间维度：任务运行了多久；
空间维度：任务运行时占用了多少 GPU。

在租户层面，LTGF 按照预算或 quota 这类权重分配 GPU-time。在任务层面，它在同一个租户内的并发任务之间公平分配 GPU-time。这个两级视角很重要，因为公平集群既要保护组织层面的共享契约，也要照顾每个租户队列里正在等待的具体任务。

这个指标还避免了对剩余时间预测的过度依赖。真实集群里，用户会取消任务，任务会失败，训练吞吐也会随 placement 变化。ASTRAEA 可以根据历史分配记录评估公平性，再用这个信号决定下一步该服务谁。

当调度器用 GPU-time 而不是瞬时 allocation 或完成时间来衡量 service 时，公平性才更容易被推理和修复。

ASTRAEA 如何调度

ASTRAEA 使用两阶段调度算法。

第一阶段，它选择 tenant-level fairness index 最低的租户。直白地说，调度器会找到那个相对自己应得份额而言，过去获得 GPU-time 最少的租户。如果这个租户有等待中的任务，并且集群可以放下其中某个任务，ASTRAEA 就会把资源分给它。

第二阶段，ASTRAEA 在该租户内部用 job-level fairness index 选择具体任务。这样即使一个租户整体上被公平对待，它内部的任务队列也不会变得很不公平。任务级策略仍然可以加入实际优先级，但需要受到公平性信号约束。

ASTRAEA 采用 lease-based 调度。它不会在公平性发生变化时立刻抢占，而是给运行中的任务一个租约周期。在租约边界，调度器可以重新安排执行顺序来修复公平性。这是一个务实折中：短租约能更快响应公平性变化，但太短又会增加抢占开销，拉长任务完成时间。ASTRAEA 会为深度学习训练选择一个平衡这些因素的租约长度。

调度层次	公平性信号	调度决策
租户层	Tenant-level LTGF index	选择相对自身份额获得 service 最少的租户
任务层	Job-level LTGF index	在该租户内部选择任务，避免队列内部不公平
租约边界	更新后的 allocation history	在避免频繁抢占的同时修复公平性

它带来了什么

ASTRAEA 在真实 GPU 集群 trace 上做了大规模仿真评估，包括 SenseTime 的 Venus trace 和 Microsoft 的 Philly trace。论文报告称，相比已有先进调度器，ASTRAEA 将租户级公平性最多提升 9.42x，将任务级公平性最多提升 10.3x，同时没有牺牲平均任务完成时间。

ASTRAEA 换来了什么。

用 long-term GPU-time 衡量公平性，同时覆盖空间和时间。
将 tenant-level fairness 最多提升 9.42x。
将 job-level fairness 最多提升 10.3x，同时不牺牲平均任务完成时间。

这里重要的经验是：GPU 集群里的公平性不只是政策偏好，它首先是一个测量问题。如果指标忽略 GPU 数量，用户就可能 overclaim；如果指标忽略时间，长任务就可能被饿死；如果指标忽略租户，集群就会违反共享契约；如果指标忽略任务，个体用户仍然会感受到不公平。

ASTRAEA 的贡献，是把深度学习集群中最关键的资源服务单位“长期 GPU-time”变成了可测量、可调度的公平性目标。

Paper: ASTRAEA: A Fair Deep Learning Scheduler for Multi-tenant GPU Clusters
Code: Astraea Artifacts

GPU Cluster Scheduling: A Map for Deep Learning Workloads

Sat, 16 May 2026 14:30:00 +0800

GPU cluster scheduling is easy to underestimate. At first glance, it looks like a familiar resource allocation problem: jobs arrive, GPUs are free or busy, and the scheduler decides who runs next.

Deep learning breaks that simplicity.

Training jobs can run for days, need gangs of GPUs, and care deeply about placement topology. Inference services are online, latency-sensitive, and often underutilize a GPU unless requests are batched or colocated. Hyperparameter tuning launches many similar trials, most of which are meant to be discarded. LLM workloads add model parallelism, massive memory footprints, long contexts, and bursty development patterns.

Our survey, Deep Learning Workload Scheduling in GPU Datacenters, tries to organize this messy design space. The most useful way to read the field is not as a list of schedulers, but as a set of tensions: speed versus cost, utilization versus isolation, fairness versus efficiency, and online latency versus cluster-wide throughput.

Why DL Scheduling Is Different

Traditional HPC and big-data schedulers provide useful starting points, but DL workloads have their own physics.

Training jobs are often gang-scheduled. A distributed job needs all requested GPUs at the same time, so GPUs are not easily divisible like CPU slots. Placement matters because communication-heavy jobs may run much faster when GPUs are packed within a node or connected by NVLink rather than scattered across weaker links. Preemption is expensive because model and optimizer states are large. At the same time, training is iterative, so a few profiled iterations can often reveal throughput, memory behavior, and placement sensitivity.

Inference has nearly opposite pressure. Each request is small compared with a training job, but the service has latency SLOs. Batching improves GPU utilization, yet waiting too long to form a batch hurts latency. Colocation improves throughput, yet interference can violate tail latency. The scheduler has to trade average efficiency against worst-case user experience.

This is why GPU cluster scheduling is not one problem. It is a family of related problems whose correct answer depends on the workload.

Training: Efficiency, Fairness, Deadlines

For training workloads, the survey groups scheduling objectives into three broad categories.

The first is efficiency. Some schedulers reduce job completion time through priority rules, such as least attained service or progress-aware variants. Others use profiling or learning-based methods to predict job duration, speed, placement sensitivity, or future resource needs. Placement is a core part of efficiency: a scheduler can have enough GPUs in aggregate but still produce poor performance if it fragments the cluster and cannot satisfy locality.

The second is fairness. Fairness is subtle because GPUs are indivisible in common gang-scheduling settings, and heterogeneous GPUs do not provide equal value to every job. Finish-time fairness, long-term GPU-time fairness, and heterogeneity-aware fairness all try to answer a version of the same question: how much service did this job or tenant deserve, and how much did it actually receive?

The third is deadline guarantee. Deadline-aware training is less explored, but important for production workflows. A best-effort job can tolerate delay; an SLO job cannot. Systems in this direction need to predict whether a job can finish before its deadline under different placements and resource allocations, then decide how to mix deadline jobs with normal jobs.

Training: How GPUs Are Used

Objectives are only half the taxonomy. The other half is how a scheduler uses resources.

Heterogeneous resource scheduling recognizes that “a GPU” is not a uniform unit. Different model architectures benefit differently from newer GPU generations, CPU allocation, memory, network bandwidth, and storage. A cost-effective scheduler should place jobs where their bottlenecks match the available hardware, not blindly send every job to the newest device.

GPU sharing attacks the underutilization problem. Many training jobs cannot saturate a modern GPU. Packing multiple jobs onto one device through MPS, MIG, virtualization, time sharing, or framework-level co-execution can improve utilization. The risk is interference: the scheduler must know when sharing helps and when it silently slows everything down.

Elastic training changes the number of GPUs assigned to a job over time. This can reduce queueing and improve utilization, especially when demand fluctuates. But elasticity is not free. Resource changes may require checkpointing, reinitialization, or batch-size adaptation. If batch size changes affect convergence, a scheduler may improve system throughput while quietly changing model behavior.

The broad lesson is that training schedulers increasingly need to be co-designed with training frameworks. The scheduler wants fine-grained control, but the framework knows whether a job can safely pause, resize, share, or change batch size.

Inference: Latency, Cost, Throughput

Inference scheduling is shaped by a different triangle: latency, cost, and accuracy.

Latency is usually the first-class constraint. A model serving system can improve throughput by batching requests, but a request waiting in a queue is still user-visible latency. A practical scheduler often uses dynamic batching: increase batch size when the service is healthy, shrink it when latency approaches the SLO.

Cost enters through cloud instance choice, autoscaling, and heterogeneous hardware. Some workloads are cheaper on CPU, some need GPU, and some become cost-efficient only when batching is large enough. The scheduler has to decide not only where to run a model, but how many replicas and which instance types are worth paying for.

Accuracy adds another axis. Some systems choose among model variants, ensembles, or modalities. A smaller model may be cheap and fast but less accurate; a larger model may be slower but better. This turns inference scheduling into a policy problem: what accuracy loss is acceptable for a given latency or cost budget?

Throughput techniques include batching, caching, model residency, and colocation. But inference colocation is more dangerous than training colocation because SLO violations are immediate. A scheduler needs interference models, isolation mechanisms, or hardware partitioning to make sharing safe.

Beyond Training and Inference

Some workloads deserve their own category.

Hyperparameter optimization is technically training, but operationally different. It launches many similar trials, prunes weak ones, and shifts resources toward promising configurations. This structure creates opportunities for early stopping, elastic trial allocation, trial packing, model fusion, and surrogate-based tuning. Our Hydro work is one example: it uses model scaling, trial fusion, and cluster-level interleaving to make HPO less brute-force.

Mixed training and inference workloads are another frontier. Inference clusters are often overprovisioned for bursts, leaving idle GPUs during low-traffic periods. Training jobs can sometimes borrow that capacity if the system can preempt or resize them quickly when inference demand returns. The challenge is respecting online SLOs while reclaiming otherwise wasted capacity.

These cases point to a larger trend: future schedulers will be more workload-aware. A generic GPU queue is too blunt for the diversity of DL development.

Where the Field Is Going

The survey ends with three research directions that still feel current.

First, emerging workloads will keep changing scheduler design. LLM pretraining, fine-tuning, serving, agentic inference, and HPO all expose different bottlenecks. The scheduler must understand more than GPU count; it must understand memory pressure, communication structure, context length, trial similarity, and elasticity.

Second, scheduling decisions need better intelligence. Heuristics are robust and deployable, mathematical optimization can be principled but slow, and ML/RL-based schedulers can capture complex patterns but are hard to trust and benchmark. A practical scheduler may combine all three: heuristics for the fast path, profiling for calibration, and optimization or learning for difficult decisions.

Third, hardware heterogeneity is becoming unavoidable. A production cluster may contain multiple GPU generations, specialized interconnects, CPUs, storage tiers, and accelerators. Heterogeneity creates opportunities for better cost-performance, but it also complicates fairness. Allocating an old GPU and a new GPU for the same amount of wall-clock time is rarely equal service.

The simplest summary is this: GPU scheduling is no longer just about filling empty slots. It is about matching workload structure to hardware structure under user-visible objectives.

That is what makes the area interesting. The best scheduler is not merely the one with the shortest queue. It is the one that understands what kind of deep learning work is in front of it, what resources it truly needs, and what trade-off the cluster is willing to make.

Paper: Deep Learning Workload Scheduling in GPU Datacenters: A Survey
Project: Awesome DL Scheduling Papers

GPU Pause, Resume, and Migration: The Missing Primitive in Cluster Scheduling

Fri, 15 May 2026 15:00:00 +0800

GPU cluster scheduling would be much easier if a running GPU job behaved like an ordinary CPU process. Pause it. Move it. Resume it somewhere else. Reclaim the device when a higher-priority job arrives. Repair fragmentation without killing user work.

In practice, this is exactly where GPU scheduling gets stuck.

A CPU process can be checkpointed by saving its address space, file descriptors, and kernel-visible state. A GPU task has an extra half of its life outside the normal process abstraction: CUDA contexts, device allocations, streams, events, library handles, kernels in flight, and data resident in GPU memory. The operating system does not naturally know how to serialize that state. The scheduler can stop the host process, but that is not the same thing as having a correct, portable checkpoint of the GPU computation.

FlowGPU is about turning GPU checkpoint/restore into a system primitive. Before FlowGPU became a full system, I wrote cudaw as the first version of the codebase: a CUDA wrapper prototype for interposing on runtime calls, tracking GPU objects, translating application-visible addresses, and making pause/resume/migration possible above an unmodified CUDA application.

Why Schedulers Want This Primitive

Pause/resume and migration change what a scheduler can do.

Without GPU checkpoint/restore, preemption is often blunt. A scheduler can kill a job, ask the framework to checkpoint at a pre-defined training boundary, or wait until the user code cooperates. That is acceptable for some training loops, but it is poorly aligned with cluster events. A high-priority job may arrive now. A GPU may fail now. A fragmented placement may need repair now. Framework-level checkpoints are usually placed for application convenience, not scheduler control.

With a transparent GPU checkpoint, the scheduler gets stronger operations:

pause a job and release its GPU memory;
resume it later on the same GPU;
migrate it to another GPU or node;
checkpoint periodically for fault tolerance;
defragment the cluster by moving jobs away from awkward placements;
support elastic scaling and priority scheduling with less user code involvement.

This is the missing link between scheduling policy and GPU execution. A scheduler may know the right decision, but without a safe migration primitive, it cannot act on that decision cheaply.

The CUDA Wrapper View

The basic idea behind my cudaw prototype is to place a wrapper between the application and CUDA runtime. Instead of letting the application talk directly to libcudart, the wrapper intercepts CUDA calls such as allocation, memory copy, and kernel launch. From the scheduler’s perspective, this creates an execution log and a shadow view of GPU state.

This wrapper layer can record which device memory regions exist, what host-side pointers correspond to them, how data moves between CPU and GPU, and which kernels are launched with which arguments. It can also maintain virtual GPU addresses: the application sees stable logical addresses, while the wrapper maps them to real CUDA allocations underneath. That indirection is what makes restore and migration plausible, because the restored task may receive different physical GPU addresses on the target device.

In a simplified checkpoint flow, the wrapper reaches a safe point, synchronizes GPU work, copies live GPU memory into a checkpoint image, saves enough metadata to reconstruct CUDA state, and releases the device. Restore reverses the process: allocate memory on the target GPU, rebuild mappings, copy data back, replay necessary CUDA setup calls, and continue execution.

This early prototype captured the central intuition that later shaped FlowGPU. GPU migration is not magic; it is state reconstruction. The hard part is making the reconstructed world indistinguishable from the original one.

Where Wrapper-Only Designs Struggle

The wrapper idea is powerful, but the edge cases are brutal.

First, CUDA state is larger than cudaMalloc and cudaMemcpy. Real applications use streams, events, cuBLAS, cuDNN, NCCL, memory pools, unified memory, graph execution, and framework allocators. Many of these objects are opaque: CUDA exposes handles, not serializable internals. A checkpoint system must record and replay the operations that created or mutated them.

Second, address identity matters. A pointer value may be stored inside application data structures, kernel arguments, framework metadata, or library state. If restore gives the program a different GPU virtual address, the application can become subtly wrong even if the bytes were copied correctly.

Third, deep learning frameworks hide memory behavior. PyTorch and TensorFlow often reserve large GPU memory blocks and keep them for reuse. Much of that reserved memory may be inactive at a given moment. A naive checkpoint that saves everything allocated by the runtime can produce enormous checkpoint images, even when the useful live state is much smaller.

Fourth, distributed training is a synchronization problem. A consistent checkpoint of a multi-GPU job requires pausing all participating ranks safely. With NCCL communication, pausing one side of a blocking send/receive pair while the other side waits can deadlock the checkpoint protocol itself.

These are the problems FlowGPU is designed to handle systematically.

FlowGPU’s Core Move

FlowGPU’s key insight is that prior system-level GPU checkpoint/restore designs coupled C/R with API forwarding. In API forwarding, all GPU operations pass through a privileged central process. That makes interception and state separation easier, but it imposes runtime overhead, creates GPU address conflicts under sharing, and blocks some GPU features.

FlowGPU decouples checkpoint/restore from virtualization.

During normal execution, each task uses a per-task intercept library. GPU operations stay private to that task and go directly to the GPU, avoiding the IPC overhead of a central forwarding process. When checkpointing is needed, FlowGPU creates a ghost process. The ghost process temporarily takes over GPU state, while the original process becomes a conventional CPU process that can be checkpointed with CRIU. GPU state and CPU state are saved in parallel, then recombined during restore.

This design keeps the useful part of interception without forcing every GPU operation through a virtualization server during normal execution.

Making Checkpoints Small and Correct

FlowGPU adds several mechanisms that are especially important for deep learning workloads.

Active memory identification avoids saving the whole framework-reserved memory pool. FlowGPU inserts a memory stub at stable DL framework backend allocation/free interfaces, tracking the memory regions that are actually active. It can also wait briefly for active memory to reach a low point in the training iteration before checkpointing. This matters because active memory in training can fluctuate dramatically between the end of an iteration and the activation-heavy middle of forward/backward execution.

Virtual memory management preserves GPU address identity. FlowGPU intercepts GPU allocations and uses CUDA VMM APIs such as cuMemAddressReserve, cuMemCreate, and cuMemMap to reserve and remap the same virtual addresses on restore. That removes a major source of correctness bugs for pointer-rich GPU applications.

Record/replay handles opaque runtime objects. Since CUDA streams, events, contexts, and library handles cannot simply be read out as bytes, FlowGPU records operations that create or modify them and replays those operations during recovery.

The pause mechanism is refined for distributed tasks. FlowGPU coordinates pausing across ranks, but avoids a known NCCL deadlock pattern by resuming all instances after a timeout if a complete pause cannot be achieved. This is a small detail with a large consequence: checkpointing must not introduce a failure mode worse than the one it tries to solve.

For multi-GPU tasks, FlowGPU also performs fine-grained deduplication. Replicated model parameters may appear on multiple GPUs, but runtime memory blocks rarely match exactly. FlowGPU deduplicates fixed-size regions, reducing checkpoint image size for distributed jobs.

What This Means for Scheduling

Once GPU pause/resume becomes practical, several scheduling policies become more realistic.

Priority scheduling can preempt a low-priority GPU job without throwing away all its progress. Fairness scheduling can redistribute service over time with lower disruption. Fragmentation-aware schedulers can migrate jobs to rebuild contiguous placements for gang-scheduled workloads. Fault-tolerance systems can checkpoint at scheduler-controlled intervals instead of relying only on framework checkpoints. Elastic schedulers can shrink, expand, or relocate jobs with a clearer recovery path.

The primitive also changes the economics of GPU sharing. If a job can be paused and restored quickly, a cluster can take more aggressive actions under bursty demand. Online inference, training, and HPO workloads no longer need to live in completely isolated resource islands; the scheduler has a better way to move work when priorities change.

FlowGPU’s evaluation shows why the details matter. It reports no runtime overhead during normal single-GPU execution because tasks can access the GPU directly without API forwarding. For DL tasks, it reduces checkpoint pause time by 6.2x to 15x over POS and up to 10.4x over Singularity. Restore time drops by 12x to 18x over POS and up to 4.1x over Singularity. For migration, FlowGPU outperforms Singularity by up to 2.1x and PyTorch framework-level checkpointing by 1.7x to 4.5x.

Those numbers are not only checkpointing results. They are scheduling-enablement results. A slow checkpoint is a policy that the scheduler cannot afford to use often. A fast, transparent checkpoint becomes a real control knob.

The Takeaway

GPU scheduling is often discussed in terms of algorithms: fairness metrics, placement heuristics, bin packing, elastic allocation, and priority queues. But the scheduler is only as powerful as the execution primitives beneath it.

cudaw was my first working cut at the wrapper-level intuition: interpose on CUDA, virtualize what the application sees, and reconstruct GPU state when needed. FlowGPU pushes that intuition into a more complete system design: per-task interception for low overhead, ghost processes for state separation, active-memory tracking for small images, VMM for address correctness, and distributed pause logic for multi-GPU workloads.

The result is a cleaner boundary between policy and mechanism. The scheduler decides when a job should pause, resume, or move. The checkpoint/restore layer makes that decision safe enough to execute.

Paper: FlowGPU: Transparent and Efficient GPU Checkpointing and Restore
Early codebase: yzs981130/cudaw