Perplexity may have a way to run LLMs on consumer hardware: How it works

Perplexity's experiment with a 35-billion-parameter model shows how inference engines, quantisation and memory optimisation can unlock more performance from existing hardware

As artificial intelligence (AI) companies try to run increasingly large models directly on phones and laptops instead of in the cloud, the conversation has mostly been about hardware. A 35-billion-parameter AI model is not something you would expect to run on a laptop. Yet US-based AI company Perplexity managed to get one running on a MacBook. It then made it faster, not by using better hardware, but by rewriting the software around how that specific model uses the chip inside the machine.

 

Perplexity's experiment points to something that gets less attention: The software sitting between the model and the chip can decide how much of that hardware's power actually gets used.

What is inference and why does it matter

When people talk about running an AI model, what they usually mean is inference. Inference is different from training, which is the process of teaching a model using huge amounts of data. Training happens mostly in data centres because of the massive amount of data and computing power it requires. Inference is what happens every time you use the model to generate a response. While training happens in data centres, inference is possible on edge devices as long as the required models are available on them.

 

Inference itself has two stages. The first, called prefill, is when the model reads and processes everything you have typed. The second, called decode, is when it generates the reply, one word at a time. These two stages behave differently on hardware. Prefill can be processed in parallel because the model already has the entire prompt in front of it. Decode is slower by nature because each new word depends on the words generated just before it, so the model keeps going back to memory to fetch the information it needs at every step.

 

The piece of software that manages all of this, deciding how the model's calculations are split up and run on a chip, is called an inference engine. 

How an on-device AI model processes a prompt, from prefill and inference to generating the final response (AI-generated)

What Perplexity actually did

Perplexity built its own inference engine, called Lily, designed for one model, China’s Qwen3.6-35B-A3B, running on Apple's own chips, rather than as a general-purpose engine.

 

The company compared Lily with MLX-LM, a general-purpose framework built by Apple for running AI models on its chips. On a Mac powered by an M5 Max chip, Lily processed prompts up to 1.23 times faster and generated responses up to 1.32 times faster than MLX-LM, averaged across different prompt lengths, according to Perplexity.

 

To understand why, it helps to know that Qwen3.6-35B-A3B is what is called a mixture-of-experts model. Instead of using its entire 35 billion parameters for every single word it generates, it activates only a small, relevant subset of them, called experts. This makes the model cheaper to run than its size suggests, but it also creates unpredictable computing patterns that a general-purpose engine finds harder to optimise.

 

The model also uses two different kinds of internal layers. Some layers use a method called full attention, which looks back at everything said so far in a conversation. Others use a newer method called Gated DeltaNet, which keeps a running summary of past context instead of storing everything, and is generally faster and lighter on memory. Handling both types adds further complexity.

 

Because Lily was built for this one model, Perplexity's engineers could make decisions that a general-purpose tool cannot. Two examples stood out.

 

The first involves a technique called quantisation, which shrinks a model, similar to how a large photograph can be compressed into a smaller file. Perplexity compressed the model from about 70GB to roughly 19.4GB using 4-bit quantisation, making it small enough to fit in a laptop's memory.

 

This approach is not new. It has been experimented with, as was showcased by US-based startup PrismML, which used quantisation to make China-based Alibaba's Qwen3.6 27B AI model smaller and more efficient without rebuilding it from scratch. The company said it retained the original architecture while significantly reducing the model's memory footprint, allowing it to run on consumer hardware that could not previously accommodate a model of this size.

 

But a compressed model still needs to be decompressed before the chip can perform calculations with it, and that decompression step usually creates extra work by writing the expanded numbers back into memory before using them. This is where the novelty comes in with regard to Perplexity’s Lily. It decompresses each small piece at the moment it is needed, inside the chip itself, without first writing the expanded version to memory. Perplexity said this change alone made prompt processing up to 77.4 per cent faster in one of its tests.

 

The second involves memory movement. Apple's chips use what is called unified memory, where the central processing unit (CPU) and graphics processing unit (GPU) share the same pool of memory instead of having separate ones. This avoids some of the delays involved in copying data back and forth, but it does not remove the problem of a chip constantly waiting for data to arrive.

 

Perplexity's team reduced how often the model needed to move data in and out of memory during the response-generation stage, partly by keeping more calculations on the GPU rather than sending data back to the CPU unnecessarily.

 

None of this changed the underlying M5 Max chip. What changed was how completely the software used it.

Beyond Perplexity

This pattern is not unique to Perplexity. As noted above, US-based AI startup PrismML managed to run a compressed version of a 27-billion-parameter AI model locally on consumer devices without relying on cloud infrastructure.

 

Apple's own machine-learning research team published a similar case in 2024. It took Llama 3.1, an open-source model with 8 billion parameters, and ran it on an older M1 Max chip, reaching about 33 tokens of output per second. Apple got there using two techniques: shrinking the model with 4-bit quantisation and reusing previously calculated information instead of recalculating it at every step, a method known as a stateful cache. Neither required new hardware.

 

Microsoft has taken a different approach with Phi Silica, a small AI model built into Windows Copilot+ PCs. Because these laptops include a dedicated AI chip called a neural processing unit (NPU), which is more limited than a full graphics chip but far more power-efficient, Microsoft designed Phi Silica specifically around that constraint.

 

It uses a technique called speculative decoding, where a smaller, faster model guesses the next few words and a larger model quickly checks whether those guesses are right, rather than having the larger model generate every word from scratch. This lets the NPU produce responses faster without needing to be more powerful.

 

Each of these cases shows that the software layer sitting on top of the chip, deciding how a specific model's calculations map on to that specific hardware, determines how much of the chip's potential is actually usable.

Do we have to build for every model?

This creates an obvious practical concern. If getting the best performance means writing a custom inference engine for every model and every chip, running AI locally could become extremely difficult to maintain.

 

This is where open-source projects matter. Frameworks like MLX-LM or llama.cpp, another widely used open-source project that runs AI models across many different kinds of hardware, exist precisely because most developers cannot afford to write specialised software for every model and chip combination. They trade some performance for the ability to work across a wider range of hardware. Perplexity has also open-sourced Lily. Some of the specific techniques it uses may now filter into more general tools over time.

The bigger picture

Perplexity's own testing showed that not every optimisation helped. Some changes improved individual steps without meaningfully speeding up the full response. The company also found that certain parts of the process were already running close to the chip's limits, leaving little room for improvement.

 

That is the underlying point. A model's design decides what kind of calculations need to happen. Quantisation decides how much data has to move. The inference engine decides how that work gets scheduled. And the chip's memory design decides how quickly the required data can arrive.

 

As more AI moves from the cloud on to laptops and phones, chip specifications will keep improving, but they will tell only part of the story. Increasingly, how fast an AI model feels to use will depend just as much on the software quietly deciding how to run it.