>10X speed-ups or nothing

Frank Belletti, PhD

CTO @ Stealth · Formerly Google DeepMind (Veo, Genie) · PhD, UC Berkeley (RISELab / BAIR)

Below an order of magnitude, a speed-up lowers the bill. Past it, the model gets new jobs. A video generator fast enough to steer becomes a world model, and a world model fast enough to sit inside a control loop can test a drone policy before the drone leaves the ground.

01 · The arithmetic

A diffusion sampler pays per network call. Fifty denoising steps with classifier-free guidance cost two forward passes each, so one sample costs 100 network function evaluations (NFE). A student distilled to four steps, with guidance folded into its weights, costs 4 NFE.

  • Teacher: 50 steps × 2 passes (guidance) = 100 network calls per sample

  • Distilled student: 4 steps × 1 pass (guidance baked in) = 4 network calls per sample

  • Speed-up: 100 ÷ 4 = 25×

That is 25× before anyone touches a kernel, and it compounds with every systems optimization that comes after. The teacher integrates the probability-flow ODE in small steps. The student is trained to land where the teacher lands, in a handful of long strides.

02 · Custom step distillation

Published distillation recipes are tuned for someone else's model, resolution and step count. I treat distillation as an engineering problem specific to each product: decide the operating point you will ship, then distill to exactly that point and measure there.

  • Start from the latency budget. The step count falls out of the frame budget on the target hardware. The schedule is built backward from that number.

  • Bake guidance into the student. Folding classifier-free guidance into the weights makes every step one forward pass. That halves the cost before a single step is removed.

  • Evaluate where you ship. Quality is measured against the teacher at the exact steps, resolution and clip length that go to production. A student that wins at 8 steps can lose at 4.

  • Leave the base model alone. The speed-up should be separable from the model it accelerates. That requirement is what led to Turbo-LoRA.

03 · Turbo-Peft, early 2024

In early 2024, I pioneered the pattern the diffusion community now calls Turbo-LoRA: carry the entire step distillation inside a low-rank adapter and leave the base weights frozen.

W(fast) = W + BA

The frozen base weights W keep full-quality behavior. The low-rank update BA, with rank r much smaller than the layer width, holds the few-step behavior.

  • One base, two modes. Adapter on: few-step sampling at interactive speed. Adapter off: full-quality sampling from the same weights in memory.

  • Ships like a patch. A low-rank adapter is a small fraction of the model's size, so it trains, versions and deploys like a patch, with no second checkpoint to maintain.

  • Composes with fine-tunes. The speed adapter stacks with style and domain adapters, so a fine-tune inherits the speed-up without being distilled again.

  • Zero overhead when merged. Merged into W at load time, the adapter adds no inference cost. The 10× stays a 10×.

Today, few-step "Turbo" adapters are one of the standard ways fast diffusion models are released and run.

04 · A record of order-of-magnitude bets

The same instinct, before diffusion: find where the system can go ten times faster or ten times bigger, then build that. Full and current list on Google Scholar.

Scaling the data

Synthetic and expanded datasets, generated at the scale models need.

  • 2022 · Kubric: A Scalable Dataset Generator. CVPR 2022. Greff, Belletti, Beyer, Doersch, et al.

  • 2019 · Scaling Up Collaborative Filtering Data Sets through Randomized Fractal Expansions. arXiv. Belletti, Lakshmanan, Krichene, Mayoraz, Chen, Anderson, et al.

  • 2019 · Scalable Realistic Recommendation Datasets through Fractal Expansions. arXiv. Belletti, Lakshmanan, Krichene, Chen, Anderson.

Finance on accelerators

Moving Monte Carlo pricing and risk sensitivities onto TPUs and TensorFlow.

  • 2020 · Tensor Processing Units for Financial Monte Carlo. SIAM Conference on Parallel Processing for Scientific Computing. Belletti, King, Yang, Nelet, Shafi, Chen, Anderson.

  • 2020 · Sensitivity Analysis in the Dupire Local Volatility Model with TensorFlow. arXiv. Belletti, King, Lottes, Chen, Anderson.

Sequential models and RL at scale

Long-range dependence, recurrent architectures and off-policy learning for production recommenders and control.

  • 2019 · Top-K Off-Policy Correction for a REINFORCE Recommender System. WSDM 2019. Chen, Beutel, Covington, Jain, Belletti, Chi.

  • 2019 · Towards Neural Mixture Recommender for Long Range Dependent User Sequences. WWW 2019. Tang, Belletti, Jain, Chen, Beutel, Xu, Chi.

  • 2019 · Quantifying Long Range Dependence in Language and User Behavior to Improve RNNs. KDD 2019. Belletti, Chen, Chi.

  • 2018 · Factorized Recurrent Neural Architectures for Longer Range Dependence. AISTATS 2018. Belletti, Beutel, Jain, Chi.

  • 2018 · Categorical-Attributes-Based Item Classification for Recommender Systems. RecSys 2018. Zhao, Chen, Chen, Jain, Beutel, Belletti, Chi.

  • 2018 · Expert Level Control of Ramp Metering Based on Multi-Task Deep Reinforcement Learning. IEEE Transactions on Intelligent Transportation Systems. Belletti, Haziza, Gomes, Bayen.

Scalable time series

PhD work at Berkeley: making analysis of large, irregular time series embarrassingly parallel.

  • 2017 · Alternate Representations for Scalable Analysis and Control of Heterogeneous Time Series. PhD thesis, UC Berkeley.

  • 2017 · Random Projection Design for Scalable Implicit Smoothing of Randomly Observed Stochastic Processes. AISTATS 2017. Belletti, Sparks, Bayen, Gonzalez.

  • 2016 · Scalable Linear Causal Inference for Irregularly Sampled Time Series with Long Range Dependencies. arXiv. Belletti, Sparks, Franklin, Bayen, Gonzalez.

  • 2015 · Embarrassingly Parallel Time Series Analysis for Large Scale Weak Memory Systems. arXiv. Belletti, Sparks, Franklin, Bayen.

Control and optimization

Traffic flow and energy systems, solved with distributed methods.

  • 2016 · Characterization of the Convective Instability of the Aw-Rascle-Zhang Model via Spectral Analysis.American Control Conference 2016. Belletti, Huo, Litrico, Bayen.

  • 2015 · Distributed Optimal Charging of Electric Vehicles for Demand Response and Load Shaping. IEEE CDC 2015. Le Floch, Belletti, Saxena, Bayen, Moura.

  • 2015 · Privacy-Preserving Dual Splitting Distributed Optimization with Application to Load Flattening in California. IEEE CDC 2015. Le Floch, Belletti, Moura, Bayen.