Contact

ARKit Image Tracking Tutorial: Build an AR Newspaper App That Plays Video

12 minutes to read

Point your iPhone camera at a printed photo and have a video play on top of it, like the moving newspaper in Harry Potter. That is exactly what ARKit's image tracking feature does. This tutorial builds a working "AR Newspaper" app in Swift using ARImageTrackingConfiguration.

What Is ARKit Image Tracking?

Since iOS 12, ARKit has shipped an API called ARImageTrackingConfiguration. It uses the back-facing camera to detect a specific 2D image (a photo, a logo, a page of a magazine) and then track its position and movement in real time. Once ARKit recognizes the image, your app can anchor 3D content, video, or animation directly on top of it, and that content will move naturally as the image (or the camera) moves.

This is different from ARWorldTrackingConfiguration, which maps the broader environment. Image tracking configuration is focused on one job: find a known image, then track it precisely.

Common real-world uses:

  • Interactive print media (magazines, business cards, product packaging)
  • Museum exhibits that animate when you point a phone at a placard
  • Restaurant menus that show a 3D preview of a dish
  • Marketing campaigns that turn a poster into a video ad

What You'll Need Before You Start

  • Xcode with the Augmented Reality project template
  • An image file (JPG/PNG) you want the app to recognize
  • A video file (MP4) to play once that image is detected
  • Optionally, a printed copy of the image (A4 paper works well) for real-world testing, or you can just display it on an iPad or Mac screen

Step 1: Create the ARKit Project

Start a new Xcode project using the Augmented Reality template, set the Content Technology to SceneKit, and name it something like ARNewspaper.

The template ships with a demo 3D spaceship and a default ARWorldTrackingConfiguration, neither of which you need. Strip them out:

// Replace the default scene loader in viewDidLoad()
let scene = SCNScene()

// Clear out the default world-tracking setup in viewWillAppear
override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)
}

You'll rebuild viewWillAppear in Step 3 with the image tracking configuration instead.

Step 2: Add Your Reference Image the Right Way

Image tracking requires a special asset setup. It's not the same as dropping a picture into your normal asset catalog.

  1. Open Assets.xcassets, right-click, and choose AR and Textures → New AR Resource Group.
  2. Drag your image (e.g., snowboarding.jpg) into that group.
  3. Select the image and open the Attributes Inspector.
  4. Set the physical size of the image in real-world units. If you're printing on A4 paper at roughly 9cm wide, set the width to 9. Xcode calculates the height automatically.

This physical size matters more than it looks like it should: ARKit uses it to judge distance and scale accurately. If you print the image larger or smaller, or display it on a screen of a different size, update this value to match.

You can add multiple images to the same resource group if your app needs to recognize more than one.

Step 3: Configure Image Tracking

With the reference image in place, wire up ARImageTrackingConfiguration:

override func viewWillAppear(_ animated: Bool) {
    super.viewWillAppear(animated)

    let configuration = ARImageTrackingConfiguration()

    guard let referenceImages = ARReferenceImage.referenceImages(
        inGroupNamed: "AR Resources",
        bundle: Bundle.main
    ) else {
        fatalError("Failed to load the reference images")
    }

    configuration.trackingImages = referenceImages
    sceneView.session.run(configuration)
}

ARReferenceImage.referenceImages(inGroupNamed:bundle:) loads every image in the "AR Resources" group, along with the physical size data you set in Step 2. That collection is assigned to configuration.trackingImages, and sceneView.session.run(configuration) starts the camera session and begins scanning for a match.

Step 4: Overlay a Video When the Image Is Found

When ARKit spots your reference image, it automatically creates an ARImageAnchor. Your job is to react to that anchor and place video content on top of it.

First, set up an AVPlayer for the video:

let videoPlayer: AVPlayer = {
    guard let url = Bundle.main.url(forResource: "snowboarding", withExtension: "mp4") else {
        return AVPlayer()
    }
    return AVPlayer(url: url)
}()

Then implement the ARSCNViewDelegate method that fires when a new anchor is added:

func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) {
    guard let imageAnchor = anchor as? ARImageAnchor else { return }

    let plane = SCNPlane(
        width: imageAnchor.referenceImage.physicalSize.width,
        height: imageAnchor.referenceImage.physicalSize.height
    )

    plane.firstMaterial?.diffuse.contents = self.videoPlayer

    let planeNode = SCNNode(geometry: plane)
    planeNode.eulerAngles.x = -.pi / 2
    node.addChildNode(planeNode)

    self.videoPlayer.play()
}

A few details worth understanding:

  • The plane is sized to match the physical image, using imageAnchor.referenceImage.physicalSize, so the video overlay lines up exactly with the printed photo instead of floating off to one side.
  • The video is set as the plane's material, so instead of a static color or texture, the plane literally is the video.
  • The rotation (eulerAngles.x = -.pi / 2) matters. SCNPlane defaults to a vertical orientation, but a printed image lying on a table or held up to the camera is horizontal. This rotation aligns the two.

Deploy to a real device, point the camera at your printed (or displayed) image, and the video should appear directly over it, tracking as the image moves.

Step 5: Make the Video Loop and Restart on Re-Detection

The video plays once and stops, and it will not restart if the image leaves and re-enters the frame. Two things to fix:

Loop the video by listening for playback completion and seeking back to the start:

NotificationCenter.default.addObserver(
    forName: .AVPlayerItemDidPlayToEndTime,
    object: self.videoPlayer.currentItem,
    queue: .main
) { [weak self] _ in
    self?.videoPlayer.seek(to: CMTime.zero)
}

Restart playback when the image comes back into view using the didUpdate delegate method, which ARKit calls continuously as it tracks the anchor:

func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) {
    guard let pointOfView = sceneView.pointOfView else { return }

    let isVisible = sceneView.isNode(node, insideFrustumOf: pointOfView)

    if isVisible && videoPlayer.rate == 0.0 {
        videoPlayer.play()
    }
}

isNode(_:insideFrustumOf:) checks whether the anchored plane is currently within the camera's visible field of view. If it is, and the player isn't already running (rate == 0.0), playback resumes automatically.

Step 6: Clean Up the Interface

Hide the debug statistics bar (frame rate, timing info) that Xcode's AR template shows by default:

sceneView.showsStatistics = false

Make the scene view fill the whole screen. By default the generated storyboard constrains the scene view to the safe area, leaving a white bar at the bottom. In the storyboard, select the scene view's bottom constraint, change its second item to Superview.Bottom, and set the constant to 0.

Testing Your AR Newspaper App

You need a physical device to test image tracking because the simulator cannot access the camera. Print your reference image at the physical size you configured (A4 paper is a convenient size), or display it full-screen on an iPad or Mac. Point your iPhone's camera at it, and within a second or two the video snaps into place and starts playing.

Summary

ARImageTrackingConfiguration gives you a way to overlay video, 3D models, or animation the moment ARKit recognizes a printed image. The Harry Potter newspaper is the most memorable example; the same technique works for interactive packaging, museum placards, business cards, and marketing collateral. Once you understand the four pieces (the reference image group, the tracking configuration, the anchor-detection delegate method, and the video or content overlay), you can adapt this pattern to almost any "scan this, see that" AR experience.

Frequently Asked Questions

What is the difference between ARImageTrackingConfiguration and ARWorldTrackingConfiguration? ARWorldTrackingConfiguration maps and tracks the broader 3D environment around the device, which is ideal for placing objects in a room. ARImageTrackingConfiguration is purpose-built to detect and follow a single known 2D image, independent of the surrounding environment.

Do I need internet access for image tracking to work? No. Reference images are bundled with your app and processed entirely on-device.

Can I track more than one image at a time? Yes. Add multiple images to the same AR Resource Group, and ARKit generates a separate ARImageAnchor for each one it detects.

Why does the physical size of the reference image matter so much? ARKit uses the real-world dimensions you provide to judge distance and scale accurately. If the printed or displayed image does not match the size you configured, tracking accuracy and content alignment will be noticeably off.

All articles