Edge AI or Cloud Inference: Where Your Model Should Actually Run

The case for edge inference is usually made with a latency number, and the number is real: a round trip to a regional cloud endpoint typically costs somewhere between 40 and 200 milliseconds depending on the network, while a small quantised model on a modern microcontroller or NPU returns in single-digit milliseconds. For anything in a control loop, that difference is the whole argument. A safety cutoff that waits on a network round trip is not a safety cutoff.

But latency is the easiest factor to reason about and rarely the one that decides the architecture. Bandwidth economics tend to matter more. A camera streaming continuously to the cloud for analysis consumes bandwidth whether or not anything is happening, and in most deployments almost nothing is happening almost all of the time. Running detection on the device and uploading only the events, or only the frames around them, changes the traffic profile by orders of magnitude. On a cellular link with metered data, that is often the difference between a viable product and one whose connectivity cost exceeds its margin.

Offline behaviour is the factor teams underestimate most consistently. A device that depends on cloud inference has made its core function conditional on connectivity, which means every network outage becomes a product outage. Whether that is acceptable is a product question rather than a technical one, but it should be answered deliberately rather than discovered during the first regional outage.

Privacy and regulation increasingly push the same direction. Inference performed on the device means raw sensor data, which is to say audio, video and location, need never leave it. That materially simplifies compliance conversations and it is far easier to explain to a customer than a data processing agreement. When the sensitive input never crosses the network boundary, an entire class of risk stops applying.

The cloud still wins in several situations, and pretending otherwise leads to bad architecture. Large models simply do not fit on constrained hardware, and quantising until they do can degrade accuracy past the point of usefulness. Workloads that need to be updated frequently are far easier to iterate centrally than to push across a fleet. Anything requiring aggregation across devices, where the inference depends on what other devices are seeing, is a cloud workload by definition. And when per-device compute is expensive relative to inference volume, centralising is cheaper: an accelerator sitting idle in every unit is capital spent on capacity nobody uses.

The most robust production designs are rarely purely one or the other. A common and durable pattern is a small model on the device handling the common case and the time-critical path, with escalation to a larger cloud model for ambiguous inputs. The device decides quickly and conservatively, the cloud adjudicates the hard cases, and the system degrades to device-only behaviour when connectivity fails rather than stopping altogether.

If the architecture is still undecided, the practical way to resolve it is to measure rather than argue. Instrument what fraction of inputs are genuinely ambiguous, what the real cost per inference is on each side including bandwidth, and what the product does during a two-hour outage. Those three numbers settle the question faster than any general principle about edge computing, and they are specific to the deployment in a way that no benchmark from a vendor can be.