Contact

Core ML Tutorial: Build a Real-Time Image Recognition App

12 minutes to read

Point a camera at something and have the app identify it, entirely on-device. Core ML makes that possible with no server round-trip, no internet dependency, and surprisingly little code. This tutorial builds a real-time image recognition app using a ready-made model, walking through camera capture, image conversion, and live predictions in Swift.

What Is Core ML?

Core ML is Apple's framework for integrating trained machine learning models directly into iOS apps. Machine learning itself is a branch of AI where a program learns patterns from historical data and uses them to make predictions on new data. A trained model is the output of running a learning algorithm against a large training dataset: for example, thousands of labeled photos teaching a model what a "happy" versus "angry" facial expression looks like, so it can later predict emotion from a new, unseen photo.

Before Core ML existed, getting a trained model running inside an iOS app was difficult. Core ML solves that: convert a trained model into its format, drop it into your project, and call it with a few lines of code. It supports many model types, including deep learning with 30+ layer types, tree ensembles, SVMs, and generalized linear models, and because it is built on Metal and Accelerate, it automatically uses the CPU and GPU. Inference runs entirely on-device, so image data never has to leave the phone.

What You'll Build

A real-time image recognition app that captures live camera frames, feeds them into a pre-trained model, and displays its best guess of what's in frame, updating continuously as you point the camera around.

What You'll Need

  • Xcode and a physical iOS device (camera access is required, so the Simulator won't work for testing this)
  • A starter project with a working camera capture UI already in place (or your own equivalent)
  • A pre-trained Core ML model

Step 1: Choose a Pre-Trained Model

Training your own model from scratch is a separate skill (see: Create ML). For this tutorial, you'll use one of Apple's ready-to-use models, available from Apple's Machine Learning models page. A few of the most common options:

  • MobileNetV2: detects the dominant object in an image across 1,000 categories (animals, vehicles, food, people, and more). Lightweight and fast, which makes it a strong default for real-time, on-device use.
  • SqueezeNet: serves a similar purpose to MobileNetV2, also across 1,000 categories, with a different accuracy/size tradeoff.
  • ResNet50: also classifies across 1,000 categories, generally with higher accuracy at the cost of a larger model size.
  • DeepLabV3: segments every pixel in an image into a predefined set of classes, rather than producing a single label.
  • YOLOv3: classifies 80 different object types, commonly used for object detection tasks.

This tutorial uses MobileNetV2, whose speed and accuracy suit a real-time camera use case well. Download the .mlmodel file from Apple's site.

Step 2: Import the Model Into Xcode

Drag the MobileNetV2.mlmodel file into your Xcode project navigator (organizing it under a dedicated group like MLModel keeps things tidy). Selecting the file in Xcode reveals its metadata:

  • Model type, author, description, license
  • Model Class: the auto-generated Swift class name you'll use to instantiate the model:
var mlModel = try! MobileNetV2(configuration: MLModelConfiguration())
  • Model Evaluation Parameters: the model's expected input and output. MobileNetV2 expects a 224×224 image as input, and returns two outputs: the single most likely category label, and a full dictionary of every possible category paired with its predicted probability.

That 224x224 input size shapes most of the implementation work ahead: every camera frame has to be resized to match it before the model can use it.

Step 3: Capture Video Frames

If your starter project only has camera preview wired up, you need to add frame output so you can actually process individual frames. In your capture session setup, after adding the camera input, add a video data output:

let videoDataOutput = AVCaptureVideoDataOutput()

videoDataOutput.setSampleBufferDelegate(self, queue: DispatchQueue(label: "imageRecognition.queue"))

videoDataOutput.alwaysDiscardsLateVideoFrames = true

captureSession.addOutput(videoDataOutput)

This creates an AVCaptureVideoDataOutput, sets a delegate to receive each captured frame, and adds it to the running capture session. alwaysDiscardsLateVideoFrames = true is a deliberate choice for real-time use: if processing falls behind, it drops stale frames rather than queuing them up, keeping the displayed prediction close to real-time instead of lagging further and further behind.

Conform to the required delegate protocol via an extension:

extension CameraController: AVCaptureVideoDataOutputSampleBufferDelegate {

    func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

        connection.videoOrientation = .portrait

        // Frame processing goes here

    }

}

captureOutput(_:didOutput:from:) fires every time a new frame arrives from the camera. This is where the rest of the pipeline lives.

Step 4: Resize Each Frame to 224×224

The model can't accept a raw camera frame at its native resolution. It needs exactly 224×224 pixels. Inside the delegate method:

guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

let ciImage = CIImage(cvPixelBuffer: imageBuffer)

let image = UIImage(ciImage: ciImage)

UIGraphicsBeginImageContext(CGSize(width: 224, height: 224))

image.draw(in: CGRect(x: 0, y: 0, width: 224, height: 224))

let resizedImage = UIGraphicsGetImageFromCurrentImageContext()!

UIGraphicsEndImageContext()

This extracts the raw pixel buffer from the sample buffer, converts it to a UIImage, then draws that image into a 224×224 graphics context, effectively resizing (and cropping, depending on aspect ratio) it to exactly what the model expects.

Step 5: Convert the Image to CVPixelBuffer

There's one more format mismatch to resolve. MobileNetV2's prediction(image:) method expects a CVPixelBuffer, not a UIImage:

func prediction(image: CVPixelBuffer) throws -> MobileNetV2Output {
    let input_ = MobileNetV2Input(image: image)
    return try self.prediction(input: input_)
}

So the resized UIImage needs one more conversion step before it can be handed to the model:

let attrs = [
    kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
    kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue
] as CFDictionary

var pixelBuffer: CVPixelBuffer?

let status = CVPixelBufferCreate(
    kCFAllocatorDefault,
    Int(resizedImage.size.width),
    Int(resizedImage.size.height),
    kCVPixelFormatType_32ARGB,
    attrs,
    &pixelBuffer
)

guard status == kCVReturnSuccess else { return }

CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)

let rgbColorSpace = CGColorSpaceCreateDeviceRGB()

let context = CGContext(
    data: pixelData,
    width: Int(resizedImage.size.width),
    height: Int(resizedImage.size.height),
    bitsPerComponent: 8,
    bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!),
    space: rgbColorSpace,
    bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue
)

context?.translateBy(x: 0, y: resizedImage.size.height)

context?.scaleBy(x: 1.0, y: -1.0)

UIGraphicsPushContext(context!)

resizedImage.draw(in: CGRect(x: 0, y: 0, width: resizedImage.size.width, height: resizedImage.size.height))

UIGraphicsPopContext()

CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

This allocates a pixel buffer matching the image dimensions, draws the resized image into that buffer's backing graphics context, and hands back a CVPixelBuffer ready for the model. The translateBy/scaleBy calls flip the coordinate system, since Core Graphics and UIKit disagree on which direction is "up."

Step 6: Run the Prediction and Display the Result

With a properly formatted pixel buffer ready, the actual prediction call is short:

if let pixelBuffer = pixelBuffer,
    let output = try? mlModel.prediction(image: pixelBuffer) {
    DispatchQueue.main.async {
        self.descriptionLabel.text = output.classLabel
    }

}

mlModel.prediction(image:) runs inference on the current frame and returns an output object containing classLabel, the model's single best guess. Since UI updates must happen on the main thread and frame processing happens on a background queue, the label update is wrapped in DispatchQueue.main.async.

Add the necessary import at the top of the file, and declare the model instance as a property on the class:

import CoreML

var mlModel = try! MobileNetV2(configuration: MLModelConfiguration())

The app is now functional: build it, deploy to a physical device, point the camera at objects, and watch the label update live. Accuracy won't be perfect (MobileNetV2 only recognizes 1,000 predefined categories), but the core pipeline of capture, resize, convert, predict, and display is what powers production real-time recognition features.

Bonus: Show All Predictions, Not Just the Top Guess

The model does not only return a single label. classLabelProbs exposes every candidate category alongside its probability. To inspect the full distribution (useful for debugging or for building a "top 5 guesses" UI), add this inside the same DispatchQueue.main.async block:

for (key, value) in output.classLabelProbs {
    print("\(key) = \(value)")
}

Running the app again prints every category the model considered, each with its associated confidence score, giving you visibility into not just what the model predicted, but how confident (or uncertain) it actually was.

Summary

Real-time image recognition with Core ML comes down to five stages: capture a live video frame, resize it to the model's required dimensions, convert it into the pixel buffer format the model expects, run the prediction, and display the result. Once that pipeline exists, swapping in a different pre-trained model (or one you have trained yourself with Create ML) is largely a drop-in change. This is the same foundation production features like live object detection, AR content triggers, and accessibility tools are built on.

Frequently Asked Questions

Why does the image need to be resized to exactly 224x224? That is the fixed input size MobileNetV2 was trained on. Every Core ML image model specifies its required input dimensions under Model Evaluation Parameters in Xcode. Feeding it a different size will fail or produce meaningless results.

Do I need an internet connection for predictions to work? No. Once the model is bundled into the app, all inference happens on-device using the CPU and GPU. No network call is made.

Why convert to CVPixelBuffer instead of just passing a UIImage? Because that is the input type the generated model class expects. CVPixelBuffer is the standard low-level image format that Core ML and AVFoundation share, optimized for the direct memory access that ML inference needs.

Can I swap in a different model, like ResNet50 or SqueezeNet, without rewriting the whole pipeline? Largely yes. The capture, resize, and pixel buffer conversion logic stays the same. You would mainly need to update the model class name and confirm the new model's expected input size matches (some models use different dimensions than 224x224).

All articles