Key Takeaways
- The bottleneck: Large language models need more active memory than a phone can reserve while the OS, camera, and background apps are running.
- The architecture: Weights stay on NAND flash. Each inference step pulls only the slice required into DRAM, then clears it.
- The techniques: Windowing reuses recent parameters in RAM. Row-column bundling reads larger sequential chunks from flash instead of byte-sized fetches.
Apple’s LLM in a Flash research reports models up to twice the size of available DRAM, with 4–5× faster CPU inference and 20–25× faster GPU inference versus naive flash loading.
