Last reviewed on July 17, 2026. Product availability and platform requirements can change, so check the linked documentation before you design around a specific API.
The phrase "on-device AI" now appears in marketing for phones, laptops, chips, and developer tools. But it often leaves the critical question unanswered: what processing stays on the device?
A photo classifier can run locally from start to finish. A writing assistant may summarize a short note on the device, then send a longer document to a server. A laptop may have an NPU while the app you use still calls a cloud model. These are different architectures with different costs, privacy boundaries, and failure modes.
This guide explains the boundary in practical terms. It covers current options on Apple platforms, Android, and Windows, along with the cases where local inference helps or falls short. It also shows how to test an architecture before putting it on the product roadmap.
On-device, edge, cloud, and hybrid AI are not the same
On-device AI runs inference on the user's phone, computer, camera, vehicle, or other client hardware. The model may use a CPU, GPU, NPU, or a mix of them. Once the required model and runtime are installed, a network connection may not be needed for the inference itself.
Edge AI is a wider category. It can mean on-device processing, but it also includes nearby gateways and local servers. A model running on a computer inside a factory is at the edge of the network, even if a sensor sends data to that computer.
Cloud AI sends the request to infrastructure controlled by the product vendor or its provider. The server can run a larger model and access current or shared information, but the feature depends on a network path and a remote data-processing boundary.
Hybrid AI uses more than one of those locations. A product might handle common requests locally and reserve a cloud model for long context, complex reasoning, fresh information, or unsupported hardware. In practice, this is often the useful comparison: not local versus cloud, but which part belongs in each place.
What runs locally in 2026?
There is no single catalog of on-device features. Availability depends on the operating system, hardware, language, region, model, and app implementation. The platform documentation does show which APIs developers can use.
Apple platforms
Core ML lets an app run compatible models through the CPU, GPU, and Neural Engine. It is suited to tasks such as classification, object detection, speech, and other purpose-built machine learning features. Apple also provides the Foundation Models framework. Eligible apps can use the on-device Apple Intelligence language model for summarization, extraction, structured output, and tool calling.
The word "on-device" does not remove version risk. Apple's 2026 Foundation Models updates note that the system model changes with operating system releases and tell developers to retest prompts. That is a useful warning for every platform: the runtime may be managed for you, but the behavior is still a moving dependency.
For apps that ship their own neural networks, Apple has introduced Core AI to build, run, debug, and profile models on Apple silicon. As of July 17, 2026, Apple labels Core AI as Beta Software, so its APIs and behavior are preliminary and should be tested with final operating system software. Core ML remains available for Core ML models, including converted models and non-neural model types.
Android
Google's current Android AI guide separates local and cloud options instead of treating them as interchangeable. Gemini Nano runs through AICore on supported devices. ML Kit GenAI provides APIs for summarization, proofreading, rewriting, image description, speech recognition, and custom prompting. Device and language support vary by API, and release status must be checked API by API; for example, the Summarization API remains beta and is not covered by an SLA or deprecation policy. Developers can also deploy custom models with LiteRT.
The caveat is hardware reach. Android spans many chipsets and device classes. LiteRT's current NPU documentation presents a unified API, but actual support still depends on the SoC, model and operator compatibility, runtime libraries, and device configuration. The current documentation lists Qualcomm and MediaTek support, while Google Tensor access is experimental. Therefore, "runs on Android" and "uses the NPU on every Android device" are not equivalent claims.
Windows
Windows ML is Microsoft's local inference framework built on ONNX Runtime. It can route compatible models to an NPU, GPU, or CPU through hardware-specific execution providers. Microsoft also allows the runtime and models to be distributed separately from an app, which can reduce duplicate downloads and make updates easier.
Copilot+ PCs are one visible hardware tier. Microsoft defines them around an NPU capable of at least 40 TOPS, along with other system requirements. That number is a qualification threshold, not a promise that every AI feature will be local or that two machines with similar TOPS will deliver identical results. Microsoft documents the current requirements and examples in its Copilot+ PC comparison.
Four good reasons to run AI on the device
1. The feature needs to work offline
Local inference can keep a feature usable in airplane mode, in a basement, or on an unreliable connection. This is valuable for transcription, camera processing, input assistance, industrial inspection, and other tasks where a network round trip would make the feature fragile.
Test the whole path before calling a feature offline. Some apps download a model after installation, fetch language packs on demand, or require a server for authentication. The inference may be local while first-run setup is not.
2. You want to reduce data leaving the device
If the input never crosses the network, you eliminate a network transfer and one place where another party could process or store the data. This can be a strong design choice for private notes, message drafts, photos, audio, and business data.
It is not a complete privacy guarantee. An app can run the model locally and still upload prompts through analytics, crash reports, sync, backups, or a fallback service. Review the actual network behavior and data lifecycle, not the marketing label.
3. The interaction has a tight latency budget
Removing a network hop can make repeated or real-time interactions feel more responsive. Local processing is especially attractive when the model is small, the input is bounded, and the user's hardware has an efficient accelerator.
Local is not automatically faster. A larger cloud accelerator may finish a difficult request sooner than an entry-level device. Compare cold start, warm latency, and long-session behavior on the slowest supported hardware. Do not set the product requirement from an average measured on a flagship phone.
4. Per-request cloud inference is a poor fit
Moving a high-volume, narrow task to client hardware can remove a server inference charge for each request. That can make costs easier to forecast.
The cost does not disappear. Teams still pay for model development, evaluation, packaging, downloads, support across hardware tiers, monitoring, and updates. Users also pay indirectly through storage, memory, battery use, and device requirements. Compare total cost per successful task, not the cloud bill with an assumed zero for local inference.
When the cloud is still the better tool
A local model is usually smaller and has less memory available than a server model. Cloud inference remains a reasonable choice when the feature needs:
- long documents or a large conversation history;
- strong reasoning on complex, open-ended tasks;
- fresh web, company, or account data;
- large shared indexes or cross-user collaboration;
- one consistent model across a broad device base;
- central deployment of urgent model or safety updates.
Cloud processing also makes sense when a local model would consume too much storage or exclude too many users. Forcing a 4 GB model into an app is not a privacy win if most people cannot use it.
Why hybrid architecture is common
A hybrid product can use a small local model for immediate, private work and offer a cloud path only when the task exceeds that model's limits. Google's Android guidance explicitly presents hybrid as an option. Apple provides a concrete system example: Apple Intelligence processes requests on the device when possible and can use Private Cloud Compute for more demanding requests.
A useful hybrid design answers four questions:
- What triggers the cloud path? Define it in testable terms, such as input length, unsupported language, missing local model, or a requested server-only tool.
- What data is sent? Send the minimum required fields, not the full local context by default.
- Can the user tell? If moving processing to the cloud changes privacy, cost, or connectivity, tell the user clearly.
- What happens when the network fails? Show a clear degraded state. Do not let a local feature appear frozen while a hidden cloud fallback times out.
Local processing helps privacy, but it does not certify compliance
Keeping data on a device can reduce disclosure to remote recipients, but it does not by itself establish data minimization. Under Article 5(1)(c) of the GDPR, personal data processed must be adequate, relevant, and limited to what is necessary for the purposes of the processing. Local inference may be one data-protection-by-design measure, but compliance still depends on the complete processing operation.
You still need to consider purpose, lawful basis, transparency, retention, security, user rights, sensitive data, and every service that receives telemetry or derived output. Local models can also expose information through notifications, logs, cached files, or insecure storage on the device. Treat the processing location as one control inside a wider privacy review.
Engineering constraints to plan for
- Hardware fragmentation: build a support matrix for CPU, GPU, NPU, memory, operating system, and driver combinations.
- Model delivery: decide whether the model ships with the app or downloads later. Measure install size, update size, and failure recovery.
- Memory and heat: a benchmark that passes once can still throttle during a sustained session.
- Quality drift: operating system and runtime updates can change model behavior. Version your evaluation set and rerun it.
- Observability: collect enough aggregate performance data to maintain the feature without uploading the private content you moved on-device to protect.
- Fallback safety: make local failure visible and avoid silently sending data to the cloud as a recovery step.
A decision checklist for product teams
Before choosing a local, cloud, or hybrid design, write down the answers to these questions:
- What exact user task are we improving?
- Must it work without a connection?
- Does the input contain personal, confidential, or regulated data?
- What is the latency target, and what is the slowest supported device?
- How much memory, storage, battery, and heat can the feature use?
- How often does the model need fresh knowledge?
- What measurable quality level must the local model reach?
- Which devices, languages, and regions will not support it?
- What is the cloud fallback, and what will the user see before data is sent?
- What is the total cost per successful task for each architecture?
If several answers are unknown, the next step is a measured prototype, not a platform commitment.
How to test an on-device AI prototype
Use real target devices and a fixed evaluation set. Include ordinary cases, edge cases, unsafe inputs, multiple languages if supported, and examples the feature should refuse or hand off.
Record at least:
- cold-start and warm latency, including p50 and p95;
- task success against a written scoring rubric;
- peak memory, model size, battery impact, and thermal throttling;
- behavior in airplane mode and after a clean installation;
- network traffic during setup, inference, analytics, and fallback;
- results on the oldest or least capable device you plan to support;
- behavior after model, runtime, and operating system updates.
Publish the test date and device list in internal documentation. "Runs locally" is easy to claim. A reproducible matrix of what ran, where it ran, and how well it worked is much more useful.
Frequently asked questions
Is on-device AI always private?
No. Local inference can keep inputs off a model server, but the app may still transmit data through sync, telemetry, backups, or fallback features. Verify storage and network behavior end to end.
Does an NPU mean every AI feature runs locally?
No. An NPU is available hardware. The app, model format, runtime, and hardware-specific provider must use it. Some features on an AI-branded device can still be cloud services.
Does on-device AI need internet access?
The inference may work offline once the model and runtime are present. Installation, model downloads, account checks, fresh information, or cloud fallback can still require a connection.
Will local models replace cloud models?
Not across the board. Local models are a strong fit for bounded, frequent, private, or latency-sensitive tasks. Cloud models retain advantages for large context, current knowledge, centralized updates, and demanding workloads. Many products will combine them.
The bottom line
On-device AI is a useful architecture, not a quality badge. It can improve offline behavior, reduce data transfer, lower latency, and change unit economics. It can also create support gaps, battery problems, stale models, and privacy claims that the rest of the product does not honor.
Start with the task and the boundary. Decide what must stay local, what genuinely benefits from the cloud, and what users need to know when the system switches between them. Then test that design on real hardware. The right architecture is the one your test matrix supports.
