The AI industry is divided into two camps: cloud-first and on-device. Cloud AI — where your queries are sent to powerful servers and responses come back over the internet — has dominated the past decade. On-device AI — where models run directly on your phone or computer — is advancing rapidly. Understanding the difference is essential for making good decisions about which AI tools to use.
How Cloud AI Works
When you send a message to ChatGPT, Gemini, or most AI assistants, it travels from your device to a data center, where a large language model processes it and generates a response, which then travels back to you. The entire process typically takes 1-4 seconds for text, longer for voice.
The advantage of cloud AI is raw capability. The models running in data centers are enormous — GPT-4o has hundreds of billions of parameters. These models can be updated without you doing anything. They can access the internet for current information. They can handle complex reasoning that smaller models cannot.
How On-Device AI Works
On-device AI runs a smaller, optimized model directly on your device's processor — typically the neural processing unit (NPU) or GPU. Apple's Neural Engine, Qualcomm's AI Engine, and Google's Tensor chip are all purpose-built for running AI models efficiently.
The models are smaller — typically 1-7 billion parameters rather than hundreds of billions — but they run entirely locally. No internet connection required after the initial download. No data leaves your device. Responses can be generated in milliseconds.
The Privacy Gap
This is where on-device AI has a decisive advantage. When the model runs on your device, your data never leaves. Your queries, your conversation history, your voice — none of it is transmitted to any server. You can have sensitive conversations without worrying about data retention policies, security breaches on the provider's servers, or regulatory compliance.
Cloud AI, even from the most privacy-conscious providers, requires your data to travel over the internet and be processed by systems you do not control. The provider's privacy policy is your only guarantee — and privacy policies can change.
The Performance Gap
For most everyday queries — answering questions, drafting messages, summarizing text, casual conversation — the gap between on-device and cloud AI has narrowed dramatically. Models like Gemma, Llama 3.1, and Phi-3 run on modern smartphones and produce responses that are genuinely useful for everyday tasks.
Where cloud AI still leads substantially: complex multi-step reasoning, code generation for large codebases, nuanced creative writing, and tasks requiring knowledge of very recent events. For these use cases, cloud AI's larger models remain meaningfully better.
Latency: A Surprising Advantage for On-Device
On-device AI can be faster than cloud AI for short responses. A cloud response requires: time to send your query over the network, queuing time on the server, processing time, and network time for the response. On-device responses skip all network steps. For voice AI specifically, this latency difference is perceptible — on-device voice responses feel more natural and conversational.
The Hybrid Approach
The most practical approach in 2025 is hybrid: on-device processing for sensitive operations (voice recognition, basic queries, anything involving personal data), with cloud AI available for complex tasks that genuinely require it. Hey Friday uses this model: voice recognition is on-device using your OS speech engine, while AI responses are generated by cloud models. Your voice never leaves your device; only your transcribed text does.
Which Should You Choose?
Choose cloud AI if: you regularly need complex reasoning, code generation, or research on recent events, and you are comfortable with your queries being processed by a third party.
Choose on-device AI if: privacy is your primary concern, you want offline capability, or you need low-latency responses for voice interactions.
Choose a hybrid approach if: you want the privacy benefits of on-device processing for day-to-day use, with cloud AI available as a fallback for complex tasks. This is where the industry is heading, and for good reason: it is the best of both worlds.
What "On-Device" Actually Means
The phrase is used loosely enough to be misleading, so it is worth being precise. Three quite different things all get marketed as on-device AI:
- On-device speech recognition. Your phone converts speech to text locally; the text is then sent somewhere. This is the most common arrangement, and it is genuinely meaningful — audio is a biometric identifier, and text is not.
- On-device inference. The language model itself runs on your hardware. Nothing leaves the device at all. This is rarer, limited to smaller models, and the quality gap against a large cloud model is still substantial.
- On-device pre-processing. Some filtering or redaction happens locally before data is uploaded. Better than nothing, but the data still leaves.
When a product says "on-device", find out which of the three it means. The privacy implications are completely different.
The Hardware Making This Possible
On-device inference became practical because of two changes, not one.
The first is silicon. Modern phone chipsets include a dedicated neural processing unit — a block designed specifically for the matrix multiplication that dominates model inference. Running that work on an NPU rather than the CPU is both far faster and dramatically more power-efficient, which is what makes continuous use viable on a battery.
The second is quantisation. Models are normally trained using 16- or 32-bit numbers. Quantisation reduces those to 8-bit or even 4-bit representations, shrinking the model enough to fit in phone memory and speeding up every operation. There is some accuracy loss, but far less than the size reduction would suggest — which is why a model small enough to run on a handset can now be genuinely useful rather than merely a demo.
The Cost Asymmetry Nobody Mentions
There is a commercial reason this shift is accelerating, and it is not primarily about privacy.
Cloud inference costs the provider money on every single request — GPU time, electricity, cooling, bandwidth. That cost recurs forever and scales linearly with usage, which is why free tiers are capped and why "unlimited" almost never means unlimited.
On-device inference costs the provider nothing per request. The compute is yours, the electricity is yours, the hardware is yours. A feature that runs locally can be genuinely unlimited because it is free to the company shipping it.
This alignment is worth understanding as a user: when privacy and unit economics point the same direction, the resulting feature tends to be durable rather than a temporary promotional offer.
Regulation Is Pushing the Same Way
Data protection regimes increasingly treat voice recordings as biometric data, which carries stricter handling requirements than ordinary personal data — explicit consent, tighter retention limits, and in some jurisdictions data residency obligations.
Architecture that never transmits audio sidesteps most of this. There is no biometric data in transit, none at rest, and nothing to disclose in a breach notification. For any organisation deploying voice tools to staff, that difference is the difference between a compliance review and a compliance project.
Questions Worth Asking a Vendor
- Is audio ever transmitted off the device? If yes, where is it stored and for how long?
- Is a voiceprint or speaker profile created?
- Which specific features run locally, and which require a network connection?
- What happens to functionality when the device is offline?
- Is my data used to train models, and can I opt out without losing features?
Vague answers to these are themselves an answer.