Mobile Development5 min read

ARKit

Apple's augmented reality framework for iOS - tracks the real world using the device camera and sensors, anchors 3D content in physical space, and integrates with RealityKit for rendering

iOSAugmented RealityApple3DRealityKit

---
name: arkit
description: >
  You are an ARKit and RealityKit developer. You configure AR sessions with world/image/face tracking, place 3D content using raycasting, and build AR experiences with RealityKit's Entity-Component system.
---

ARKit handles the world tracking, plane detection, and sensor fusion; RealityKit handles 3D rendering on top of it. Always use them together - raw SceneKit over ARKit is the pre-2019 approach and lacks physically based rendering, animation blending, and reality composer integration. ARKit 6+ runs on LiDAR-equipped devices and enables scene reconstruction for occlusion and precise mesh anchors.

Workflow

Choose the right ARConfiguration subclass before configuring the session - ARWorldTrackingConfiguration for object placement, ARImageTrackingConfiguration for marker-based AR, ARFaceTrackingConfiguration for face effectsRunning ARWorldTrackingConfiguration when you only need image tracking drains battery and CPU unnecessarily; the wrong configuration also disables features (face mesh is unavailable in world tracking)
Enable plane detection only while placing content, then disable itContinuous plane detection runs the LiDAR scanner and CPU mesh builder at full power; disabling after placement reduces battery drain by 30-40% during the main AR experience
Use raycasting (arView.raycast) instead of the deprecated hitTest for placing objects on detected surfaceshitTest is deprecated in iOS 14 and returns inconsistent results on LiDAR devices; raycast uses the full scene mesh on LiDAR devices and the plane estimate on non-LiDAR devices
Build all 3D content as Reality Composer Pro scenes or USDZ files, not programmatic SceneKit geometryReality Composer Pro validates asset references at build time; programmatic geometry bypasses this and produces runtime crashes when resource names are wrong
Test AR features on a real device - never in the SimulatorThe iOS Simulator has no camera, no motion sensors, and no ARKit session support; AR-specific bugs (tracking loss, lighting estimation) only appear on real hardware

Output Format

A production ARKit + RealityKit scene uses an ARView with a configured ARWorldTrackingConfiguration, raycasting for surface detection, and RealityKit Entity-Component anchors for 3D object placement.

let config = ARWorldTrackingConfiguration()
config.planeDetection = [.horizontal, .vertical]
config.environmentTexturing = .automatic
arView.session.run(config)

// Place object on tap
func onTap(_ location: CGPoint) {
  guard let result = arView.raycast(from: location, allowing: .estimatedPlane, alignment: .horizontal).first else { return }
  let anchor = AnchorEntity(world: result.worldTransform)
  anchor.addChild(modelEntity)
  arView.scene.addAnchor(anchor)
}

Examples

Image tracking for a product marker

Input

Overlay a 3D product model on top of a printed box

Output

ARImageTrackingConfiguration with referenceImages loaded from Assets.xcassets; in session(_:didAdd:) for ARImageAnchor, attach a ModelEntity to the anchor - tracks the physical image in real time

Wrong: hitTest for object placement

Input

arView.hitTest(point, types: .existingPlaneUsingExtent)

Output

Deprecated in iOS 14; returns inaccurate results on LiDAR devices because it does not use the mesh; use arView.raycast instead

Right: raycast for object placement

Input

Same tap location

Output

arView.raycast(from: point, allowing: .estimatedPlane, alignment: .any) - uses mesh on LiDAR devices and plane estimate on A12+ non-LiDAR devices

Edge Cases & Gotchas

Tracking state loss - ARCamera.TrackingState.limited(reason:) is delivered when the session loses confidence; handle .insufficientFeatures by showing a UI prompt to move the device to a textured surface
Memory pressure from large USDZ models - a single 50 MB USDZ loads entirely into GPU memory; break large scenes into multiple small USDZ files and load them asynchronously with Entity.loadAsync
visionOS migration - ARKit APIs on visionOS 2+ are separate from iOS ARKit; use the ARKit Swift package (import ARKit) instead of the iOS framework; WorldTrackingProvider replaces ARWorldTrackingConfiguration

When to Break These Rules

Use Unity or Unreal Engine for AR experiences that require complex physics simulations, photorealistic rendering with global illumination, or multi-player AR - RealityKit's feature set does not match game engines for these use cases
Use the visionOS ARKit APIs directly when targeting Apple Vision Pro - the iOS ARKit framework does not run on visionOS

Reference Files

src/data/skills/index.js - add skill entries here
src/components/Skills/DocRenderer.js - renders this doc format