Inference - Diffio AI Blog

ML

Overlapping I/O With Inference Around One Shared Model, Not Batching, Raised Mac Mini Throughput on I/O-Bound Restores

Diffio 4.5 restores speech on a fleet of Mac minis (M4 Pro, 24 GB of unified memory). Each mini runs one worker, which can serve our production environment, our dev environment or both. The worker takes jobs from each enabled environment's edge scheduler (a Cloudflare Durable Object

ML

Partitioning MLPs Along the Intermediate Dimension Let the Apple Neural Engine Run Two of Three Guidance Branches

How Diffio 4.0 reached 7.8x (pro) and 10.7x (flash) realtime on an M4 Pro Mac mini by splitting guidance branches across the GPU, the CPU's SME units and the Neural Engine, and what memory, precision and scheduling taught us.

ML

Smaller Model State Did Not Establish Faster Cold Requests for Diffio on Modal

What Diffio’s Modal experiments revealed about GPU snapshots, initialization, quantization, CUDA graphs, transcription dependencies, and cold-request latency.