A GPU at every edge site is a non-starter at scale. Multiply $5,000+ per site by 100+ sites and the economics collapse. We chose a harder path: production-grade inference on $400 commodity gateways. Here's how it works.
The model side
Three constraints drive the architecture. First, models must be quantized to int8 or smaller. Second, they must run on CPU. Third, they must fit in 4GB of RAM. The model architecture follows.
- Vision: MobileNet-v3 backbone with custom heads per use case.
- Anomaly: lightweight autoencoders with bounded reconstruction error.
- Forecasting: ESN (echo-state networks) and gradient-boosted trees over deep models.
- Object detection: int8 YOLOv8n family fine-tuned per site.
The runtime side
Inference latency on commodity x86 (Atom-class) runs 30–80ms for our vision models. Three patterns get us there:
Pattern 1: Frame skipping
For continuous vision, we don't need to infer every frame. We infer key frames and interpolate. A 15-FPS camera becomes effectively 5-FPS inference — well within budget.
Pattern 2: Cascaded inference
A cheap "is something happening?" detector runs always-on. The expensive classifier only fires when the cheap detector triggers. The expensive model effectively runs at 1-FPS.
Pattern 3: Edge-local caching
Most decisions repeat. An LRU cache on the inference output catches the common case in microseconds. Cache hit rates on retail RFID workloads run 70%+.
The honest answer
You don't get GPU-class performance on CPU. You get good-enough performance for 90% of edge workloads at 1/10th the per-site cost. That trade-off is what makes the math work.