Inference - Diffio AI Blog

ML

Overlapping Per-Job I/O With One Shared Model, Not Batching, Raised Mac Mini Serving Throughput

Diffio 4.5 restores speech on a fleet of Mac minis (M4 Pro, 24 GB of unified memory). Each mini runs one worker, which supports our production and dev environments; today the fleet serves dev only. The worker takes jobs from each enabled environment's edge scheduler (a Cloudflare

ML

Partitioning MLPs Along the Intermediate Dimension Let the Apple Neural Engine Run Two of Three Guidance Branches

How Diffio 4.0 reached 7.8x (pro) and 10.7x (flash) realtime on an M4 Pro Mac mini by splitting guidance branches across the GPU, the CPU's SME units and the Neural Engine, and what memory, precision and scheduling taught us.

ML

Smaller Model State Did Not Establish Faster Cold Requests for Diffio on Modal

What Diffio’s Modal experiments revealed about GPU snapshots, initialization, quantization, CUDA graphs, transcription dependencies, and cold-request latency.