AI/ML6 min read

Edge AI & On-Device ML

Deploy ML models directly on mobile devices and edge hardware - eliminating server round-trips, enabling offline inference, and protecting user privacy.

Edge AITensorFlow LiteCore MLONNXOn-Device

skill: Edge AI & On-Device ML category: AI/ML tags: [Edge AI, TensorFlow Lite, Core ML, ONNX, On-Device] readingTime: 6

Edge AI is justified when latency, privacy, or offline operation are requirements. For most general ML tasks, cloud inference is cheaper to develop and maintain. Run on-device when: users have intermittent connectivity, data is too sensitive to leave the device, or inference must happen in < 50ms.

Workflow

Quantize the model for edge deploymentA 100MB FP32 model becomes 25MB INT8 after quantization - 4x smaller, 2-4x faster on CPU, with < 1% accuracy loss for most tasks.
Run inference with TensorFlow Lite (Android/embedded)TFLite is optimized for ARM CPUs, with NNAPI/GPU delegate support for hardware acceleration.
Deploy to iOS with Core MLCore ML uses the Apple Neural Engine (ANE) for hardware acceleration - up to 10x faster than CPU inference on iPhones with A-series chips.
Use ONNX for cross-platform portabilityONNX is a universal model format - train in PyTorch, export to ONNX, run on any hardware with ONNX Runtime (CPU, GPU, mobile, web).
Benchmark and optimize latencyTheoretical model speed doesn't account for preprocessing, data loading, or model initialization. Benchmark on the target hardware.

Output Format

Quantized model in TFLite/Core ML/ONNX format, benchmarked on target hardware, with mobile integration code.

edge_ai/
├── model_full.pt         # PyTorch trained model
├── model.onnx            # Cross-platform export
├── model.tflite          # Android/embedded
├── model.mlmodel         # iOS Core ML
├── benchmark_results.txt
└── android/app/assets/   # Bundled with app

Examples

Input

Output

Edge Cases & Gotchas

Unsupported ops in TFLite - Some TensorFlow ops don't have TFLite equivalents. Check for unsupported ops before deployment: `converter.target_spec.supported_ops` or use netron.app to inspect the graph.
First-inference warmup - The first inference is always slower (JIT compilation, model loading into cache). Run 3-5 warmup inferences before benchmarking.
App size increase - Bundling a 50MB model doubles the app download size. Consider downloading the model on first launch from a CDN rather than bundling.
Model update strategy - Bundled models can't be updated without an app release. Design an over-the-air model update mechanism (download + validate + hot-swap) for models that change frequently.

When to Break These Rules

Use cloud inference when the model is > 100MB, changes frequently, or requires GPU hardware not available on the target device.
Use WebAssembly (ONNX Runtime Web, TFLite WASM) for browser-based edge inference - avoids native app distribution.

Reference Files

https://www.tensorflow.org/lite/guide
https://coremltools.readme.io/docs