Smaller Model State Did Not Establish Faster Cold Requests for Diffio on Modal

Diffio’s deployment experiments ran from July through September 2026 on our audio-restoration service. Diffio 2.0 uses ahead-of-time compiled model packages; Diffio 3.5 also has an autoregressive language-model path and a diffusion component. The general next-token diffusion architecture explains why the autoregressive loop and diffusion head offer different optimization boundaries.

A cold request includes work that a GPU profiler cannot see

Our restoration workflow includes source preparation, transcription, dispatch, platform waiting, worker startup, model execution, publication, and a completion join that waits for all required outputs. Some stages overlap, so adding their elapsed times can double-count the same wall-clock interval.

A restore flag alone cannot classify a request as cold: the second request in that process still inherits a worker that originally restored from a snapshot. We required distinct worker identities, evidence that the previous worker was gone, and platform restore events. Snapshot-creation attempts remained in the attempt journal but outside the restored cohort.

The boundary labeled “physical start” also needed verification. The final harness preferred Modal’s restore-start system log; it was not invariably an operating-system process-creation timestamp. Its interval to the start of application finalization is therefore a restore-to-finalization-start proxy, not an isolated measurement of checkpoint copying. Submission to that event can include placement, image preparation, retries, and other platform work.

GPU snapshots avoided package loading and ScoreQ initialization

GPU snapshots become valuable when restoring initialized state costs less than repeating the computation that constructed it. In the July probe, application finalization after a CPU snapshot took about 22.5 seconds, compared with about 0.4 seconds after a GPU-plus-CPU snapshot. That small historical comparison measured finalization, not total request latency, and used separate groups of six CPU-snapshot and eight GPU-snapshot observations.

Our early 2.0 worker still loaded its compiled packages on the first restored request, despite preparing their files during image construction. That first control trace spent 6.97 seconds loading the packages.

ScoreQ, the learned audio-quality scorer used by the service, had an even more expensive first-use path. Its first production-shape ONNX inference took roughly 35–36 seconds, overlapping word-to-audio alignment enough to make that stage look slow. Warming alignment alone did not remove the delay. Exercising ScoreQ’s actual input shape before capture moved its convolution-search and initialization work out of the first restored request.

After restoration, the worker created fresh credentials, network clients, runtime threads, and random identity. Retaining reusable computation while refreshing externally meaningful state avoided reusing captured connections or a creator’s random stream.

Modal’s current snapshot documentation places pre-capture work in @modal.enter(snap=True) and describes GPU snapshots as an alpha feature. Snapshots use the same distributed storage system as Images and Volumes, so storage-bandwidth-limited weight loading may gain little.

Snapshot creation still has a cost. Our candidate spent longer preparing its initial reusable state, and creation can recur when code, configuration, or platform conditions change.

The earlier bundle lowered median completion but delayed the 3.5 preview

The September combined optimization changed several components together: compiled-package preload, ScoreQ warmup, redundant model copies, allocator state, CPU memory reservation, and the 3.5 diffusion execution path. Its 60-second-input comparison used ten matched restored pairs per model. This bundled design does not isolate any individual change’s contribution.

The 3.5 mean stayed flat despite its lower median because capacity-related outliers remained. Median time to an advertised HLS (HTTP Live Streaming) playlist or segment increased from 35.892 to 46.096 seconds; this measured advertisement, not successful download and decoding. The preview subset contained eight observations per arm. The preview and completion medians use different subsets, so their relative ordering is not a per-request ordering.

Repository pruning did not measure snapshot shrinkage

The model repository was our versioned release source, not a download repeated on every restored request. We published immutable Hugging Face revisions, verified their file hashes, and synchronized them into append-only Modal Volume directories. Image construction prepared compatible executables and caches. Restored trials ran with Hugging Face downloads disabled.

Removing unused legacy GPU-architecture packages saved about 4.04 GB in the 2.0 repository, but those files were never active state on the tested GPU. Pruning an unused semantic tokenizer and connector from 3.5 removed about 694 MB of parameters while leaving retained tensors unchanged. These studies measured repository, compiled-constant, or device-memory reductions; they did not measure serialized snapshot size.

Immutable revisions also protect restores and rollbacks. Hugging Face’s download API supports explicit commit revisions. Modal documents that changing a Volume does not invalidate its snapshots, and deleting files needed by a restore can break it. We therefore preserved old revision directories while live or rollback deployments could still depend on them.

The release identity combined source, model revision, image, and configuration. Development deployment and fixture checks preceded any promotion decision.

Compiled packages failed after the server passed health checks

Ahead-of-time compilation, which produces executables before serving, adds compatibility requirements beyond tensor shapes. Our packages targeted SM120, NVIDIA’s architecture identifier for the tested Blackwell GPU. A fallback GPU needed its own compatible package; an existing package for another NVIDIA architecture was not evidence of compatibility.

Three packaging failures only became visible when we exercised the real model:

• System-library compatibility. The compiled shared libraries required GLIBC_2.38; the older image supplied glibc 2.36.
• Executable-stack flags. A newer image’s hardened loader rejected flags on compiled wrapper libraries. We cleared them in image-local copies, rebuilt and checked the archives, and left pinned source artifacts intact.
• Execution-provider selection. Dependencies could install both CPU and GPU ONNX Runtime distributions. The final build installed the pinned GPU distribution last and required CUDAExecutionProvider, the backend that runs supported ONNX operations on CUDA.

The canary therefore loaded every required package and ran real audio. A successful import, HTTP health check, or tensor allocation could miss all three failures.

CUDA graphs reduced diffusion runtime; broader compilation added first-use work

A CUDA graph records a GPU execution sequence for replay with stable memory addresses, reducing repeated launch overhead. We captured the deterministic 3.5 diffusion trajectory, generated its initial random noise outside capture, and kept graph inputs private. Moving decoder cache indices to the CPU also avoided repeated GPU-to-CPU lookups.

The integrated service exposed a failure that isolated graph code missed: alignment performed CUDA work on another thread during global capture. Thread-local capture restricted unsafe-operation error checks to the capturing thread, resolving this observed capture failure while preserving a single denoising scheduler and private buffers. It was validated as a complete configuration; changing the capture mode alone does not make shared buffers safe. PyTorch’s capture-mode documentation describes the different cross-thread error checks.

The selected graph matched baseline decoded PCM, the waveform’s numerical samples, across all 232 evaluation clips with frozen transcripts. Comparing WAV file bytes would have produced false differences because metadata timestamps changed. For a minute of input, median Diffio 3.5 local-runtime request time fell from 16.445 to 12.916 seconds. This included local transcription, alignment, and generation, and excluded the deployed storage, scoring, and gateway path.

Broader compilation did not produce the same result. Compiling the whole language-model path introduced roughly 117 seconds of first-use work and 62 seconds on a new shape. The tested whole-Whisper compilation also added substantial compilation overhead on first use and on new input shapes. Compilation cost alone would not reject a configuration whose compilation could finish before capture; these tested configurations did not establish a reduction in complete-request latency.

Quantized storage, successful reload, and acceptable audio are separate gates

Post-training quantization replaces trained weights with lower-precision representations, optionally using calibration recordings to choose the mapping, without fine-tuning the model.

Our INT4 export initially failed to preserve packed storage. Tensor metadata could describe a low-bit object while copying into an ordinary floating-point destination reconstructed a larger representation. We inspected the physical packed backing storage and compiled constants, then reloaded each candidate in a fresh process. Earlier serialized candidates that reconstructed the expected tensor class had still failed output parity.

Even correct packing did not establish an audio-quality pass. We compared frozen inputs and transcript conditioning, first against the current deployed model and then against the original BF16 model to measure cumulative change. The full evaluation contained 232 clips; the final acoustic-INT4 candidate also passed a fresh 32-recording confirmation set.

Quality decisions used UTMOS and WVMOS, learned predictors of perceived speech quality, averaged into a dataset-balanced blend, together with ScoreQ as a separate learned score. The provisional limits allowed at most a 0.02 drop in each aggregate and a 0.05 drop on any dataset. An improved aggregate could not override a failed dataset threshold. Those automated gates did not establish perceptual identity; no human listening verdict was recorded. Recognition-based checks likewise could flag numerical-word disagreements but could not resolve them without checking what was actually spoken.

MLP refers to feed-forward multilayer perceptron sublayers. HQQ, GPTQ, and AWQ-lite name the tested quantization recipes; NVFP4 is a four-bit floating-point format.

Positive deltas below favor the candidate.

These are component- and recording-specific results, not a ranking of quantization methods.

The final smaller-state candidates did not meet the cold-latency gate

The final study compared candidates against controls with the earlier deployment improvements already in place. Acceptance required the specified median completion improvement, a paired interval supporting improvement, passing quality, and no clear first-playable regression.

Pairs used the same frozen input and workflow settings, interleaving control and candidate and reversing their order on alternating pairs. The allocator candidate asked ONNX Runtime to release unused ScoreQ arena memory while retaining its session and caches. The acoustic candidate quantized the original BF16 weights directly, avoiding requantization of the lossy INT8 representation.

Changes are candidate minus matched control; negative favors the candidate. The intervals describe the paired median and come from resampling whole matched pairs. The acoustic candidate’s interval describes a sample stopped under a futility rule introduced during collection; it does not have guaranteed fixed-sample coverage. Creator counters repeated after a restore are frozen metadata rather than new memory observations.

The allocator result was inconclusive. Its mean completion change was slower by 11.66 seconds, and its median restore-to-finalization-start proxy changed by only −0.096 seconds. The favorable completion median did not establish that less retained memory restored faster.

Acoustic-INT4 completion was slower in six of seven pairs, and median first-playable time also worsened. We stopped further dispatches when the planned ten-pair median could no longer improve, even with arbitrarily good remaining pairs. The best possible median was still +4.54 seconds. One request already in flight finished and remained unmatched, outside the paired statistics.

Neither candidate qualified for the planned additional cold tests at other input durations or a cold-latency release.

The acoustic candidate’s local warm slowdown was much smaller than its complete request difference. The alternating-process RunPod diagnostic measured model execution, excluding transcription, ScoreQ, Modal restoration, transport, and output-file writing.

The diagnostic made 48 calls. Excluding the first call per duration in each process left 32 calls, with four observations per table cell.

The compiled model processes fixed 120-second windows, padding shorter inputs. The three shorter fixtures therefore each execute one window; the longest executes three, explaining the nearly flat short-input timings.

We retained the candidate’s artifacts so its lower storage requirement and higher execution time could be reproduced. Attributing the entire cold-request regression to low-bit kernels would conflate the local compute penalty with platform and workflow variability.

Waiting before restoration dominated the largest observed delay

One allocator-control request took about 845 seconds to complete, with roughly 776 seconds between GPU submission and the explicit restore-start log. Transcription had already completed while Modal showed a pending call with no worker task binding. After the restore event, the observed interval to finalization start was about six seconds.

The timing places most of that delay before the recorded restore event without identifying which platform operation consumed it. The tested state reductions did not establish an improvement in that waiting interval. Both arms experienced long waits, and we retained them in the paired sample.

An earlier region-pinned arm had worse scheduling observations and was stopped. A cheaper GPU fallback offered no measured advantage and introduced a different architecture to support. Reducing the CPU memory reservation was eventually viable for the changed bundle, but an earlier comparison had favored the larger reservation’s scheduling. Initialization peaks, steady memory, placement, and cost were separate constraints.

Independent transcription let Diffio 2.0 release the GPU before the job finished

Moving speech recognition off the GPU removed a separate ASR and alignment package, about 1.89 GB per model. ASR means automatic speech recognition; the acoustic Whisper encoders used by restoration still remained necessary. Single-run GPU profiles showed less kernel execution after offload, but paired warm service measurements showed slower completion because the worker waited for remote transcripts.

Each mean uses two subsequent calls in the same process; the kernel profiles were separate single observations, not a performance distribution. Moving work to another service changed which resource performed it, without automatically removing it from the completion dependency graph.

Diffio 2.0 allowed restoration and transcription to execute independently because its restored audio does not require the external transcript. We dispatched audio restoration independently, let it publish and release its GPU invocation, and completed the job on the CPU after both audio and transcript were valid. Tests that deliberately held back transcript-provider responses verified that ordering for short and long inputs, rather than measuring a new latency distribution. Diffio 3.5 still depended on canonical transcript conditioning, so it could not inherit the same independence. An earlier evaluation using external transcripts also failed a dataset quality threshold.

The CPU completion coordinator checked that audio and transcript belonged to the same operation before marking the job successful. It prevented uncertain dispatch from duplicating GPU work, respected cancellation and later transcript edits, and charged usage once. Audio availability and full-job completion became separate states.

CPU preparation could remain slow even on a reused instance. One roughly ten-second extraction/upload interval occurred after that instance had handled 157 earlier requests. It did not support attributing the delay to an empty container’s initialization. Phase telemetry was needed to distinguish media work, storage round trips, and time between stages.

Retention and testing budgets depend on reuse and information gained

Keeping a container warm allows reuse if another request arrives before shutdown. In a small production sample of nine events, the shortest observed gap exceeded 22 minutes. That incomplete sample gave no evidence that retaining workers for a few additional minutes would often avoid a restore.

Modal’s autoscaler controls expose minimum warm containers, an active buffer, and an idle scale-down window.

Keeping that allocation for five extra minutes would cost about $0.31; actual metered usage and billing policy determine the invoice.

We used the already allocated RunPod GPU for calibration, packing, fresh reload, audio quality, profiling, and duration checks. Modal calls were reserved for the behavior that local processes could not reproduce: actual snapshot creation/restoration, platform waiting, and the complete deployed workflow. Interleaved controls, frozen revisions, explicit exclusions, and preserved failures made those paid calls interpretable.

The final three-app Modal billing checkpoint for the cold-quantization study was $4.43. It could lag usage and excluded RunPod, the separate CPU workflow, and external ASR.

Sources

• Diffio deployment measurements: original study methods and quality gates, anonymized final paired timings, initialization and packaging evidence, the earlier bundled comparison, warm-duration and transcription results, and the cold-study billing checkpoint. The experiments are dated; they are not a benchmark of every Modal configuration.
• Modal Memory Snapshots: lifecycle hooks, CPU/GPU snapshot behavior, storage-bandwidth limitations, captured randomness, and Volume dependencies in the snapshot and artifact sections.
• PyTorch 2.9 CUDA graph API: replay and capture-mode semantics in the CUDA graph section.
• ONNX Runtime CUDA execution provider: CUDA provider selection and convolution-algorithm search underlying the ScoreQ and packaging sections.
• Hugging Face Hub downloads: immutable revision selection in the artifact-release section.
• Modal scaling and pricing: retention controls and the dated resource-cost calculation.

Related Diffio Posts

• VibeVoice: Frame Rate Is the Context Budget for Long-Form TTS: the general autoregressive-plus-diffusion architecture referenced above.