When Will 400B Parameter Language Models Run On Smartphones?

As mobile technology advances rapidly, one exciting prospect on the horizon is running large language models (LLMs) with 400 billion parameters directly on our smartphones and tablets. Meta has just released its Llama 3.1 model with a 400 billion parameter version, and it's looking like it's contented to top GPT4o. 

This would allow us to have incredibly powerful natural language processing, generation, and understanding capabilities at our fingertips without relying on cloud servers or internet connectivity.

The applications could be game-changing, from real-time speech translation to intelligent virtual assistants to creative writing aids. However, squeezing such massive AI models onto the limited hardware of mobile devices presents some formidable technical challenges that will require significant innovations to overcome. In this post, I'll explore the current state of ultra-large language models, the hurdles standing in the way of getting them onto phones, and the technological breakthroughs needed to make it happen.

Current State of LLMs

Over the past few years, language models have achieved new heights in scale and capability. State-of-the-art models like GPT-4o, PaLM, Chinchilla, and Megatron-Turing NLG have surpassed 100 billion parameters, displaying impressive language understanding and generation abilities. However, these behemoths require enormous computational resources to train and run. For example, GPT-3 was trained on 45 terabytes of text data using over a thousand A100 GPUs from NVIDIA. Running the full model for inference requires at least dozens of gigabytes of memory and customized hardware accelerators to achieve reasonable speeds and costs.

On the open source side, projects like EleutherAI's GPT-Neo and GPT-J have aimed to replicate the capabilities of GPT-3.5 using publicly available data and code. The largest of these, GPT-J-6B, has 6 billion parameters. While much more accessible than its larger cousins, it still has a sizeable footprint of around 22 GB. Simply storing the model weights would overwhelm the 4-6 GB of RAM found in most modern smartphones, let alone the additional scratchpad memory needed to make predictions.

So, in summary, the current crop of ultra-large language models require data centres full of specialized AI accelerator chips to train and run them cost- and energy-efficiently. A 400 billion parameter model like the one we're dreaming of would likely be 10-20x more demanding than GPT-3.5 regarding computational and memory requirements. Running that continuously on a battery-powered handheld device is unimaginable with today's technology.

Challenges of Running LLMs on Mobile Phones

Several compounding challenges make the prospect of running 400 billion parameter LLMs on smartphones exceedingly difficult:

• Limited computational power: While mobile SoCs have advanced by leaps and bounds, they are still orders of magnitude slower than the beefy GPUs and specialized AI accelerators used in data centres and HPC environments. The latest A16 Bionic chip in the iPhone 14 Pro has just 16 billion transistors. NVIDIA's A100 GPU, a standard workhorse for AI, has 54 billion. The A100 is explicitly designed to accelerate the linear algebra operations that dominate neural network workloads. Smartphone chips must split their transistor budget between the CPU, GPU, AI accelerators, ISP, DSP, modem, etc.
• Limited memory capacity: As mentioned above, flagship phones today have 8 GB of RAM at the high end, shared across all apps and system functions. The storage capacity is more adequate, with 128 GB as a common starting configuration, but accessing flash storage is much slower than RAM. Some Android phones are pushing up to 16 GB (e.g., ASUS ROG Phone 6), but that's still minuscule compared to a server with 1 TB of RAM.
• Energy efficiency: Perhaps the biggest blocker is power consumption. Datacenters can dedicate 250-400 W and elaborate cooling to a single high-end GPU. In contrast, when pushed to the max, the A16 Bionic uses just 3-4 watts. And it has to share power and thermal headroom with the display, modem, and other components. Running a 400B parameter model on a battery would deplete a 5000 mAh pack in minutes if you could even cram the model in memory. As a point of reference, the 20B parameter BLOOM model costs about $3 per 100-token generation when run on GPT-J-6B-level hardware.
• Latency sensitivity: On smartphones, AI models are typically used for latency-sensitive applications like speech recognition, machine vision, AR, etc. They must produce results in tens of milliseconds to provide a responsive user experience. LLMs are often used in a more batch-oriented fashion, where slightly higher latencies are acceptable. To make LLMs useful on phones, inference speed must improve by 10-100x.

Technological Advancements Needed

To run 400 billion parameter models on a smartphone, we'll need revolutionary breakthroughs on multiple fronts:

• Model compression: Finding ways to prune, quantize, distil, and otherwise compress these models without losing quality is an active area of research. For example, techniques like DistilBERT and Q-BERT have shown promising results in shrinking BERT-like models by 30-40% while retaining most of their accuracy. However, we likely need compression ratios of 100x or more to make ultra-large LLMs viable on smartphones.
• Sparsity and hardware support: Another way to reduce computational and memory footprint is to take advantage of neural networks' natural sparsity. By avoiding the storage and arithmetic of weights and activations close to zero, it may be possible to cut resource usage by 10x or more. However, this requires hardware that can efficiently operate on sparse data structures. Numeric formats like block floating point (BFP) can help somewhat. Chips that directly implement sparse linear algebra primitives would be ideal.
• Novel architectures: Moving beyond simple transformer stacks to more efficient and representationally powerful architectures will also be key. We've seen how models like Chinchilla achieve higher quality than GPT-3.5 with 4x fewer parameters through better architecture and training. Techniques like Low-rank adaptation could allow training gigantic shared "foundation" models and then rapidly specializing lightweight "head" models for mobile deployment. Dynamic models like Pathways could adaptively activate only the relevant parts of the network for a given input, saving compute.
• Better hardware: While algorithmic innovations will help, the hardware will need to level up to massively enable 400B models on smartphones. We need to push performance per watt and per mm2 of silicon by one or two orders of magnitude. More capable AI accelerators purpose-built for sparse inference, in-memory computing, analog neural networks, and optical interconnects—these types of moonshot technologies will be essential to packing datacenter-scale AI onto a palm-sized slab.

Timeline for Implementation

So when can we expect 400 billion parameter LLMs to run locally on smartphones? This is a difficult question to answer precisely, as it depends on the pace of progress in multiple fields. However, here is my rough timeline:

• 3-5 years: Continued model size scaling on the cloud side, exceeding 1 trillion parameters. Smartphones reach 16-32 GB RAM capacities. Initial practical successes in extreme model compression (100x+) and new efficient architectures. New AI accelerator designs head towards production.
• 5-10 years: Cloud models hit 10 trillion parameters. 100-500 billion parameter models are widely deployed through APIs. Model compression, sparsity, and novel architectures mature and proliferate. Dedicated AI chips bring sparse computing into the mainstream on mobile. Models in the 1-10 billion parameter range can run on flagship phones.
• 10-15 years: Convergence of scaled-up hardware performance and scaled-down model innovations enables 100-400 billion parameter models on high-end mobile SoCs. AI-first SoC designs shake up the smartphone landscape. Seamless hybrid local/cloud processing of giant foundation models becomes prevalent.
• 15-20 years: Running LLMs with 100s of billions up to 1 trillion parameters locally on smartphones will become commonplace. Compute and memory will be essentially free. AI will dominate the mobile experience.

Of course, these are just my educated guesses based on the current state of the field and reasonable projections. The actual path of this technology will likely surprise us in both promising and challenging ways. These developments will almost certainly be unevenly distributed, with the most advanced capabilities initially limited to select high-end flagship devices before eventually trickling down to the mass market.

Conclusion

The prospect of running 400 billion parameter language models locally on smartphones is one of the most exciting and transformative developments for mobile technology. The ability to carry around human-level language understanding and generation capabilities in our pockets, untethered from the cloud, would be a monumental milestone in computing and AI.

However, bridging the massive gap between the data centre-scale resources required by today's LLMs and the stringent constraints of mobile hardware is a herculean challenge. Significant breakthroughs will be needed on both the software and hardware fronts - compressing models by 100x+ without losing quality, crafting vastly more efficient architectures, inventing new chips to efficiently process ultra-sparse models, perhaps shifting to wholly novel substrates like analog or optical computing.

None of these will be easy. But the immense potential of putting LLMs into the hands of billions - for education, health, productivity, accessibility, entertainment, and more - makes it a challenge well worth undertaking. With focused research and investment, backed by the relentless advancement of hardware, I believe we will not only achieve 400B models on smartphones but perhaps even 1T models or beyond. And that will truly change the game for mobile and AI.