Vision Framework API Reference

Comprehensive reference for Vision framework computer vision: subject segmentation, hand/body pose detection, person detection, face analysis, text recognition (OCR), barcode detection, and document scanning.

When to Use This Reference

Implementing subject lifting using VisionKit or Vision
Detecting hand/body poses for gesture recognition or fitness apps
Segmenting people from backgrounds or separating multiple individuals
Face detection and landmarks for AR effects or authentication
Combining Vision APIs to solve complex computer vision problems
Looking up specific API signatures and parameter meanings
Recognizing text in images (OCR) with VNRecognizeTextRequest
Detecting barcodes and QR codes with VNDetectBarcodesRequest
Building live scanners with DataScannerViewController
Scanning documents with VNDocumentCameraViewController
Extracting structured document data with RecognizeDocumentsRequest (iOS 26+)

Related skills: See axiom-vision for decision trees and patterns, axiom-vision-diag for troubleshooting

Vision Framework Overview

Vision provides computer vision algorithms for still images and video:

Core workflow:

Create request (e.g., VNDetectHumanHandPoseRequest())
Create handler with image (VNImageRequestHandler(cgImage: image))
Perform request (try handler.perform([request]))
Access observations from request.results

Coordinate system: Lower-left origin, normalized (0.0-1.0) coordinates

Performance: Run on background queue - resource intensive, blocks UI if on main thread

Request Handlers

Vision provides two request handlers for different scenarios.

VNImageRequestHandler

Analyzes a single image. Initialize with the image, perform requests against it, discard.

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request1, request2])  // Multiple requests, one image

Initialize with: CGImage, CIImage, CVPixelBuffer, Data, or URL

Rule: One handler per image. Reusing a handler with a different image is unsupported.

VNSequenceRequestHandler

Analyzes a sequence of frames (video, camera feed). Initialize empty, pass each frame to perform(). Maintains inter-frame state for temporal smoothing.

let sequenceHandler = VNSequenceRequestHandler()

// In your camera/video frame callback:
func processFrame(_ pixelBuffer: CVPixelBuffer) throws {
    try sequenceHandler.perform([request], on: pixelBuffer)
}

Rule: Create once, reuse across frames. The handler tracks state between calls.

When to Use Which

Use Case	Handler
Single photo or screenshot	`VNImageRequestHandler`
Video stream or camera frames	`VNSequenceRequestHandler`
Temporal smoothing (pose, segmentation)	`VNSequenceRequestHandler`
One-off analysis of a CVPixelBuffer	`VNImageRequestHandler`

Requests That Benefit from Sequence Handling

These requests use inter-frame state when run through VNSequenceRequestHandler:

VNDetectHumanBodyPoseRequest — Smoother joint tracking
VNDetectHumanHandPoseRequest — Smoother landmark tracking
VNGeneratePersonSegmentationRequest — Temporally consistent masks
VNGeneratePersonInstanceMaskRequest — Stable person identity across frames
VNDetectDocumentSegmentationRequest — Stable document edges
Any VNStatefulRequest subclass — Designed for sequences

Common Mistake

Creating a new VNImageRequestHandler per video frame discards temporal context. Pose landmarks jitter, segmentation masks flicker, and you lose the smoothing that sequence handling provides.

// Wrong — loses temporal context every frame
func processFrame(_ buffer: CVPixelBuffer) throws {
    let handler = VNImageRequestHandler(cvPixelBuffer: buffer)
    try handler.perform([poseRequest])
}

// Right — maintains inter-frame state
let sequenceHandler = VNSequenceRequestHandler()
func processFrame(_ buffer: CVPixelBuffer) throws {
    try sequenceHandler.perform([poseRequest], on: buffer)
}

Subject Segmentation APIs

VNGenerateForegroundInstanceMaskRequest

Availability: iOS 17+, macOS 14+, tvOS 17+, axiom-visionOS 1+

Generates class-agnostic instance mask of foreground objects (people, pets, buildings, food, shoes, etc.)

Basic Usage

let request = VNGenerateForegroundInstanceMaskRequest()
let handler = VNImageRequestHandler(cgImage: image)

try handler.perform([request])

guard let observation = request.results?.first as? VNInstanceMaskObservation else {
    return
}

InstanceMaskObservation

allInstances: IndexSet containing all foreground instance indices (excludes background 0)

instanceMask: CVPixelBuffer with UInt8 labels (0 = background, 1+ = instance indices)

instanceAtPoint(_:): Returns instance index at normalized point

let point = CGPoint(x: 0.5, y: 0.5)  // Center of image
let instance = observation.instanceAtPoint(point)

if instance == 0 {
    print("Background tapped")
} else {
    print("Instance \(instance) tapped")
}

Generating Masks

createScaledMask(for:croppedToInstancesContent:)

Parameters:

for: IndexSet of instances to include
croppedToInstancesContent:
- false = Output matches input resolution (for compositing)
- true = Tight crop around selected instances

Returns: Single-channel floating-point CVPixelBuffer (soft segmentation mask)

// All instances, full resolution
let mask = try observation.createScaledMask(
    for: observation.allInstances,
    croppedToInstancesContent: false
)

// Single instance, cropped
let instances = IndexSet(integer: 1)
let croppedMask = try observation.createScaledMask(
    for: instances,
    croppedToInstancesContent: true
)

Instance Mask Hit Testing

Access raw pixel buffer to map tap coordinates to instance labels:

let instanceMask = observation.instanceMask

CVPixelBufferLockBaseAddress(instanceMask, .readOnly)
defer { CVPixelBufferUnlockBaseAddress(instanceMask, .readOnly) }

let baseAddress = CVPixelBufferGetBaseAddress(instanceMask)
let width = CVPixelBufferGetWidth(instanceMask)
let bytesPerRow = CVPixelBufferGetBytesPerRow(instanceMask)

// Convert normalized tap to pixel coordinates
let pixelPoint = VNImagePointForNormalizedPoint(
    CGPoint(x: normalizedX, y: normalizedY),
    width: imageWidth,
    height: imageHeight
)

// Calculate byte offset
let offset = Int(pixelPoint.y) * bytesPerRow + Int(pixelPoint.x)

// Read instance label
let label = UnsafeRawPointer(baseAddress!).load(
    fromByteOffset: offset,
    as: UInt8.self
)

let instances = label == 0 ? observation.allInstances : IndexSet(integer: Int(label))

VisionKit Subject Lifting

ImageAnalysisInteraction (iOS)

Availability: iOS 16+, iPadOS 16+

Adds system-like subject lifting UI to views:

let interaction = ImageAnalysisInteraction()
interaction.preferredInteractionTypes = .imageSubject  // Or .automatic
imageView.addInteraction(interaction)

Interaction types:

.automatic: Subject lifting + Live Text + data detectors
.imageSubject: Subject lifting only (no interactive text)

ImageAnalysisOverlayView (macOS)

Availability: macOS 13+

let overlayView = ImageAnalysisOverlayView()
overlayView.preferredInteractionTypes = .imageSubject
nsView.addSubview(overlayView)

Programmatic Access

ImageAnalyzer

let analyzer = ImageAnalyzer()
let configuration = ImageAnalyzer.Configuration([.text, .visualLookUp])

let analysis = try await analyzer.analyze(image, configuration: configuration)

ImageAnalysis

subjects: [Subject] - All subjects in image

highlightedSubjects: Set<Subject> - Currently highlighted (user long-pressed)

subject(at:): Async lookup of subject at normalized point (returns nil if none)

// Get all subjects
let subjects = analysis.subjects

// Look up subject at tap
if let subject = try await analysis.subject(at: tapPoint) {
    // Process subject
}

// Change highlight state
analysis.highlightedSubjects = Set([subjects[0], subjects[1]])

Subject Struct

image: UIImage/NSImage - Extracted subject with transparency

bounds: CGRect - Subject boundaries in image coordinates

// Single subject image
let subjectImage = subject.image

// Composite multiple subjects
let compositeImage = try await analysis.image(for: [subject1, subject2])

Out-of-process: VisionKit analysis happens out-of-process (performance benefit, image size limited)

Person Segmentation APIs

VNGeneratePersonSegmentationRequest

Availability: iOS 15+, macOS 12+

Returns single mask containing all people in image:

let request = VNGeneratePersonSegmentationRequest()
// Configure quality level if needed
try handler.perform([request])

guard let observation = request.results?.first as? VNPixelBufferObservation else {
    return
}

let personMask = observation.pixelBuffer  // CVPixelBuffer

VNGeneratePersonInstanceMaskRequest

Availability: iOS 17+, macOS 14+

Returns separate masks for up to 4 people:

let request = VNGeneratePersonInstanceMaskRequest()
try handler.perform([request])

guard let observation = request.results?.first as? VNInstanceMaskObservation else {
    return
}

// Same InstanceMaskObservation API as foreground instance masks
let allPeople = observation.allInstances  // Up to 4 people (1-4)

// Get mask for person 1
let person1Mask = try observation.createScaledMask(
    for: IndexSet(integer: 1),
    croppedToInstancesContent: false
)

Limitations:

Segments up to 4 people
With >4 people: may miss people or combine them (typically background people)
Use VNDetectFaceRectanglesRequest to count faces if you need to handle crowded scenes

Hand Pose Detection

VNDetectHumanHandPoseRequest

Availability: iOS 14+, macOS 11+

Detects 21 hand landmarks per hand:

let request = VNDetectHumanHandPoseRequest()
request.maximumHandCount = 2  // Default: 2, increase if needed

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

for observation in request.results as? [VNHumanHandPoseObservation] ?? [] {
    // Process each hand
}

Performance note: maximumHandCount affects latency. Pose computed only for hands ≤ maximum. Set to lowest acceptable value.

Hand Landmarks (21 points)

Wrist: 1 landmark

Thumb (4 landmarks):

.thumbTip
.thumbIP (interphalangeal joint)
.thumbMP (metacarpophalangeal joint)
.thumbCMC (carpometacarpal joint)

Fingers (4 landmarks each):

Tip (.indexTip, .middleTip, .ringTip, .littleTip)
DIP (distal interphalangeal joint)
PIP (proximal interphalangeal joint)
MCP (metacarpophalangeal joint)

Group Keys

Access landmark groups:

Group Key	Points
`.all`	All 21 landmarks
`.thumb`	4 thumb joints
`.indexFinger`	4 index finger joints
`.middleFinger`	4 middle finger joints
`.ringFinger`	4 ring finger joints
`.littleFinger`	4 little finger joints

// Get all points
let allPoints = try observation.recognizedPoints(.all)

// Get index finger points only
let indexPoints = try observation.recognizedPoints(.indexFinger)

// Get specific point
let thumbTip = try observation.recognizedPoint(.thumbTip)
let indexTip = try observation.recognizedPoint(.indexTip)

// Check confidence
guard thumbTip.confidence > 0.5 else { return }

// Access location (normalized coordinates, lower-left origin)
let location = thumbTip.location  // CGPoint

Gesture Recognition Example (Pinch)

let thumbTip = try observation.recognizedPoint(.thumbTip)
let indexTip = try observation.recognizedPoint(.indexTip)

guard thumbTip.confidence > 0.5, indexTip.confidence > 0.5 else {
    return
}

let distance = hypot(
    thumbTip.location.x - indexTip.location.x,
    thumbTip.location.y - indexTip.location.y
)

let isPinching = distance < 0.05  // Normalized threshold

Chirality (Handedness)

let chirality = observation.chirality  // .left or .right or .unknown

Body Pose Detection

VNDetectHumanBodyPoseRequest (2D)

Availability: iOS 14+, macOS 11+

Detects 18 body landmarks (2D normalized coordinates):

let request = VNDetectHumanBodyPoseRequest()
try handler.perform([request])

for observation in request.results as? [VNHumanBodyPoseObservation] ?? [] {
    // Process each person
}

Body Landmarks (18 points)

Face (5 landmarks):

.nose, .leftEye, .rightEye, .leftEar, .rightEar

Arms (6 landmarks):

Left: .leftShoulder, .leftElbow, .leftWrist
Right: .rightShoulder, .rightElbow, .rightWrist

Torso (7 landmarks):

.neck (between shoulders)
.leftShoulder, .rightShoulder (also in arm groups)
.leftHip, .rightHip
.root (between hips)

Legs (6 landmarks):

Left: .leftHip, .leftKnee, .leftAnkle
Right: .rightHip, .rightKnee, .rightAnkle

Note: Shoulders and hips appear in multiple groups

Group Keys (Body)

Group Key	Points
`.all`	All 18 landmarks
`.face`	5 face landmarks
`.leftArm`	shoulder, elbow, wrist
`.rightArm`	shoulder, elbow, wrist
`.torso`	neck, shoulders, hips, root
`.leftLeg`	hip, knee, ankle
`.rightLeg`	hip, knee, ankle

// Get all body points
let allPoints = try observation.recognizedPoints(.all)

// Get left arm only
let leftArmPoints = try observation.recognizedPoints(.leftArm)

// Get specific joint
let leftWrist = try observation.recognizedPoint(.leftWrist)

VNDetectHumanBodyPose3DRequest (3D)

Availability: iOS 17+, macOS 14+

Returns 3D skeleton with 17 joints in meters (real-world coordinates):

let request = VNDetectHumanBodyPose3DRequest()
try handler.perform([request])

guard let observation = request.results?.first as? VNHumanBodyPose3DObservation else {
    return
}

// Get 3D joint position
let leftWrist = try observation.recognizedPoint(.leftWrist)
let position = leftWrist.position  // simd_float4x4 matrix
let localPosition = leftWrist.localPosition  // Relative to parent joint

3D Body Landmarks (17 points): Same as 2D except no ears (15 vs 18 2D landmarks)

3D Observation Properties

bodyHeight: Estimated height in meters

With depth data: Measured height
Without depth data: Reference height (1.8m)

heightEstimation: .measured or .reference

cameraOriginMatrix: simd_float4x4 camera position/orientation relative to subject

pointInImage(_:): Project 3D joint back to 2D image coordinates

let wrist2D = try observation.pointInImage(leftWrist)

3D Point Classes

VNPoint3D: Base class with simd_float4x4 position matrix

VNRecognizedPoint3D: Adds identifier (joint name)

VNHumanBodyRecognizedPoint3D: Adds localPosition and parentJoint

// Position relative to skeleton root (center of hip)
let modelPosition = leftWrist.position

// Position relative to parent joint (left elbow)
let relativePosition = leftWrist.localPosition

Depth Input

Vision accepts depth data alongside images:

// From AVDepthData
let handler = VNImageRequestHandler(
    cvPixelBuffer: imageBuffer,
    depthData: depthData,
    orientation: orientation
)

// From file (automatic depth extraction)
let handler = VNImageRequestHandler(url: imageURL)  // Depth auto-fetched

Depth formats: Disparity or Depth (interchangeable via AVFoundation)

LiDAR: Use in live capture sessions for accurate scale/measurement

Face Detection & Landmarks

VNDetectFaceRectanglesRequest

Availability: iOS 11+

Detects face bounding boxes:

let request = VNDetectFaceRectanglesRequest()
try handler.perform([request])

for observation in request.results as? [VNFaceObservation] ?? [] {
    let faceBounds = observation.boundingBox  // Normalized rect
}

VNDetectFaceLandmarksRequest

Availability: iOS 11+

Detects face with detailed landmarks:

let request = VNDetectFaceLandmarksRequest()
try handler.perform([request])

for observation in request.results as? [VNFaceObservation] ?? [] {
    if let landmarks = observation.landmarks {
        let leftEye = landmarks.leftEye
        let nose = landmarks.nose
        let leftPupil = landmarks.leftPupil  // Revision 2+
    }
}

Revisions:

Revision 1: Basic landmarks
Revision 2: Detects upside-down faces
Revision 3+: Pupil locations

Person Detection

VNDetectHumanRectanglesRequest

Availability: iOS 13+

Detects human bounding boxes (torso detection):

let request = VNDetectHumanRectanglesRequest()
try handler.perform([request])

for observation in request.results as? [VNHumanObservation] ?? [] {
    let humanBounds = observation.boundingBox  // Normalized rect
}

Use case: Faster than pose detection when you only need location

CoreImage Integration

CIBlendWithMask Filter

Composite subject on new background using Vision mask:

// 1. Get mask from Vision
let observation = request.results?.first as? VNInstanceMaskObservation
let visionMask = try observation.createScaledMask(
    for: observation.allInstances,
    croppedToInstancesContent: false
)

// 2. Convert to CIImage
let maskImage = CIImage(cvPixelBuffer: axiom-visionMask)

// 3. Apply filter
let filter = CIFilter(name: "CIBlendWithMask")!
filter.setValue(sourceImage, forKey: kCIInputImageKey)
filter.setValue(maskImage, forKey: kCIInputMaskImageKey)
filter.setValue(newBackground, forKey: kCIInputBackgroundImageKey)

let output = filter.outputImage  // Composited result

Parameters:

Input image: Original image to mask
Mask image: Vision's soft segmentation mask
Background image: New background (or empty image for transparency)

HDR preservation: CoreImage preserves high dynamic range from input (Vision/VisionKit output is SDR)

Text Recognition APIs

VNRecognizeTextRequest

Availability: iOS 13+, macOS 10.15+

Recognizes text in images with configurable accuracy/speed trade-off.

Basic Usage

let request = VNRecognizeTextRequest()
request.recognitionLevel = .accurate  // Or .fast
request.recognitionLanguages = ["en-US", "de-DE"]  // Order matters
request.usesLanguageCorrection = true

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

for observation in request.results as? [VNRecognizedTextObservation] ?? [] {
    // Get top candidates
    let candidates = observation.topCandidates(3)
    let bestText = candidates.first?.string ?? ""
}

Recognition Levels

Level	Performance	Accuracy	Best For
`.fast`	Real-time	Good	Camera feed, large text, signs
`.accurate`	Slower	Excellent	Documents, receipts, handwriting

Fast path: Character-by-character recognition (Neural Network → Character Detection)

Accurate path: Full-line ML recognition (Neural Network → Line/Word Recognition)

Properties

Property	Type	Description
`recognitionLevel`	`VNRequestTextRecognitionLevel`	`.fast` or `.accurate`
`recognitionLanguages`	`[String]`	BCP 47 language codes, order = priority
`usesLanguageCorrection`	`Bool`	Use language model for correction
`customWords`	`[String]`	Domain-specific vocabulary
`automaticallyDetectsLanguage`	`Bool`	Auto-detect language (iOS 16+)
`minimumTextHeight`	`Float`	Min text height as fraction of image (0-1)
`revision`	`Int`	API version (affects supported languages)

Language Support

// Check supported languages for current settings
let languages = try VNRecognizeTextRequest.supportedRecognitionLanguages(
    for: .accurate,
    revision: VNRecognizeTextRequestRevision3
)

Language correction: Improves accuracy but takes processing time. Disable for codes/serial numbers.

Custom words: Add domain-specific vocabulary for better recognition (medical terms, product codes).

VNRecognizedTextObservation

boundingBox: Normalized rect containing recognized text

topCandidates(_:): Returns [VNRecognizedText] ordered by confidence

VNRecognizedText

Property	Type	Description
`string`	`String`	Recognized text
`confidence`	`VNConfidence`	0.0-1.0
`boundingBox(for:)`	`VNRectangleObservation?`	Box for substring range

// Get bounding box for substring
let text = candidate.string
if let range = text.range(of: "invoice") {
    let box = try candidate.boundingBox(for: range)
}

Barcode Detection APIs

VNDetectBarcodesRequest

Availability: iOS 11+, macOS 10.13+

Detects and decodes barcodes and QR codes.

Basic Usage

let request = VNDetectBarcodesRequest()
request.symbologies = [.qr, .ean13, .code128]  // Specific codes

let handler = VNImageRequestHandler(cgImage: image)
try handler.perform([request])

for barcode in request.results as? [VNBarcodeObservation] ?? [] {
    let payload = barcode.payloadStringValue
    let type = barcode.symbology
    let bounds = barcode.boundingBox
}

Symbologies

1D Barcodes:

.codabar (iOS 15+)
.code39, .code39Checksum, .code39FullASCII, .code39FullASCIIChecksum
.code93, .code93i
.code128
.ean8, .ean13
.gs1DataBar, .gs1DataBarExpanded, .gs1DataBarLimited (iOS 15+)
.i2of5, .i2of5Checksum
.itf14
.upce

2D Codes:

.aztec
.dataMatrix
.microPDF417 (iOS 15+)
.microQR (iOS 15+)
.pdf417
.qr

Performance: Specifying fewer symbologies = faster detection

Revisions

Revision	iOS	Features
1	11+	Basic detection, one code at a time
2	15+	Codabar, GS1, MicroPDF, MicroQR, better ROI
3	16+	ML-based, multiple codes, better bounding boxes

VNBarcodeObservation

Property	Type	Description
`payloadStringValue`	`String?`	Decoded content
`symbology`	`VNBarcodeSymbology`	Barcode type
`boundingBox`	`CGRect`	Normalized bounds
`topLeft/topRight/bottomLeft/bottomRight`	`CGPoint`	Corner points

VisionKit Scanner APIs

DataScannerViewController

Availability: iOS 16+

Camera-based live scanner with built-in UI for text and barcodes.

Check Availability

// Hardware support
DataScannerViewController.isSupported

// Runtime availability (camera access, parental controls)
DataScannerViewController.isAvailable

Configuration

import VisionKit

let dataTypes: Set<DataScannerViewController.RecognizedDataType> = [
    .barcode(symbologies: [.qr, .ean13]),
    .text(textContentType: .URL),  // Or nil for all text
    // .text(languages: ["ja"])  // Filter by language
]

let scanner = DataScannerViewController(
    recognizedDataTypes: dataTypes,
    qualityLevel: .balanced,  // .fast, .balanced, .accurate
    recognizesMultipleItems: true,
    isHighFrameRateTrackingEnabled: true,
    isPinchToZoomEnabled: true,
    isGuidanceEnabled: true,
    isHighlightingEnabled: true
)

scanner.delegate = self
present(scanner, animated: true) {
    try? scanner.startScanning()
}

RecognizedDataType

Type	Description
`.barcode(symbologies:)`	Specific barcode types
`.text()`	All text
`.text(languages:)`	Text filtered by language
`.text(textContentType:)`	Text filtered by type (URL, phone, email)

Delegate Protocol

protocol DataScannerViewControllerDelegate {
    func dataScanner(_ dataScanner: DataScannerViewController,
                     didTapOn item: RecognizedItem)

    func dataScanner(_ dataScanner: DataScannerViewController,
                     didAdd addedItems: [RecognizedItem],
                     allItems: [RecognizedItem])

    func dataScanner(_ dataScanner: DataScannerViewController,
                     didUpdate updatedItems: [RecognizedItem],
                     allItems: [RecognizedItem])

    func dataScanner(_ dataScanner: DataScannerViewController,
                     didRemove removedItems: [RecognizedItem],
                     allItems: [RecognizedItem])

    func dataScanner(_ dataScanner: DataScannerViewController,
                     becameUnavailableWithError error: DataScannerViewController.ScanningUnavailable)
}

RecognizedItem

enum RecognizedItem {
    case text(RecognizedItem.Text)
    case barcode(RecognizedItem.Barcode)

    var id: UUID { get }
    var bounds: RecognizedItem.Bounds { get }
}

// Text item
struct Text {
    let transcript: String
}

// Barcode item
struct Barcode {
    let payloadStringValue: String?
    let observation: VNBarcodeObservation
}

Async Stream

// Alternative to delegate
for await items in scanner.recognizedItems {
    // Current recognized items
}

Custom Highlights

// Add custom views over recognized items
scanner.overlayContainerView.addSubview(customHighlight)

// Capture still photo
let photo = try await scanner.capturePhoto()

VNDocumentCameraViewController

Availability: iOS 13+

Document scanning with automatic edge detection, perspective correction, and lighting adjustment.

Basic Usage

import VisionKit

let camera = VNDocumentCameraViewController()
camera.delegate = self
present(camera, animated: true)

Delegate Protocol

protocol VNDocumentCameraViewControllerDelegate {
    func documentCameraViewController(_ controller: VNDocumentCameraViewController,
                                       didFinishWith scan: VNDocumentCameraScan)

    func documentCameraViewControllerDidCancel(_ controller: VNDocumentCameraViewController)

    func documentCameraViewController(_ controller: VNDocumentCameraViewController,
                                       didFailWithError error: Error)
}

VNDocumentCameraScan

Property	Type	Description
`pageCount`	`Int`	Number of scanned pages
`imageOfPage(at:)`	`UIImage`	Get page image at index
`title`	`String`	User-editable title

func documentCameraViewController(_ controller: VNDocumentCameraViewController,
                                   didFinishWith scan: VNDocumentCameraScan) {
    controller.dismiss(animated: true)

    for i in 0..<scan.pageCount {
        let pageImage = scan.imageOfPage(at: i)
        // Process with VNRecognizeTextRequest
    }
}

Document Analysis APIs

VNDetectDocumentSegmentationRequest

Availability: iOS 15+, macOS 12+

Detects document boundaries for custom camera UIs or post-processing.

let request = VNDetectDocumentSegmentationRequest()
let handler = VNImageRequestHandler(ciImage: image)
try handler.perform([request])

guard let observation = request.results?.first as? VNRectangleObservation else {
    return  // No document found
}

// Get corner points (normalized)
let corners = [
    observation.topLeft,
    observation.topRight,
    observation.bottomLeft,
    observation.bottomRight
]

vs VNDetectRectanglesRequest:

Document: ML-based, trained specifically on documents
Rectangle: Edge-based, finds any quadrilateral

RecognizeDocumentsRequest (iOS 26+)

Availability: iOS 26+, macOS 26+

Structured document understanding with semantic parsing.

Basic Usage

let request = RecognizeDocumentsRequest()
let observations = try await request.perform(on: imageData)

guard let document = observations.first?.document else {
    return
}

DocumentObservation Hierarchy

DocumentObservation
└── document: DocumentObservation.Document
    ├── text: TextObservation
    ├── tables: [Container.Table]
    ├── lists: [Container.List]
    └── barcodes: [Container.Barcode]

Table Extraction

for table in document.tables {
    for row in table.rows {
        for cell in row {
            let text = cell.content.text.transcript
            let detectedData = cell.content.text.detectedData
        }
    }
}

Detected Data Types

for data in document.text.detectedData {
    switch data.match.details {
    case .emailAddress(let email):
        let address = email.emailAddress
    case .phoneNumber(let phone):
        let number = phone.phoneNumber
    case .link(let url):
        let link = url
    case .address(let address):
        let components = address
    case .date(let date):
        let dateValue = date
    default:
        break
    }
}

TextObservation Hierarchy

TextObservation
├── transcript: String
├── lines: [TextObservation.Line]
├── paragraphs: [TextObservation.Paragraph]
├── words: [TextObservation.Word]
└── detectedData: [DetectedDataObservation]

API Quick Reference

Subject Segmentation

API	Platform	Purpose
`VNGenerateForegroundInstanceMaskRequest`	iOS 17+	Class-agnostic subject instances
`VNGeneratePersonInstanceMaskRequest`	iOS 17+	Up to 4 people separately
`VNGeneratePersonSegmentationRequest`	iOS 15+	All people (single mask)
`ImageAnalysisInteraction` (VisionKit)	iOS 16+	UI for subject lifting

Pose Detection

API	Platform	Landmarks	Coordinates
`VNDetectHumanHandPoseRequest`	iOS 14+	21 per hand	2D normalized
`VNDetectHumanBodyPoseRequest`	iOS 14+	18 body joints	2D normalized
`VNDetectHumanBodyPose3DRequest`	iOS 17+	17 body joints	3D meters

Face & Person Detection

API	Platform	Purpose
`VNDetectFaceRectanglesRequest`	iOS 11+	Face bounding boxes
`VNDetectFaceLandmarksRequest`	iOS 11+	Face with detailed landmarks
`VNDetectHumanRectanglesRequest`	iOS 13+	Human torso bounding boxes

Text & Barcode

API	Platform	Purpose
`VNRecognizeTextRequest`	iOS 13+	Text recognition (OCR)
`VNDetectBarcodesRequest`	iOS 11+	Barcode/QR detection
`DataScannerViewController`	iOS 16+	Live camera scanner (text + barcodes)
`VNDocumentCameraViewController`	iOS 13+	Document scanning with perspective correction
`VNDetectDocumentSegmentationRequest`	iOS 15+	Programmatic document edge detection
`RecognizeDocumentsRequest`	iOS 26+	Structured document extraction

Observation Types

Observation	Returned By
`VNInstanceMaskObservation`	Foreground/person instance masks
`VNPixelBufferObservation`	Person segmentation (single mask)
`VNHumanHandPoseObservation`	Hand pose
`VNHumanBodyPoseObservation`	Body pose (2D)
`VNHumanBodyPose3DObservation`	Body pose (3D)
`VNFaceObservation`	Face detection/landmarks
`VNHumanObservation`	Human rectangles
`VNRecognizedTextObservation`	Text recognition
`VNBarcodeObservation`	Barcode detection
`VNRectangleObservation`	Document segmentation
`DocumentObservation`	Structured document (iOS 26+)

Resources

WWDC: 2019-234, 2021-10041, 2022-10024, 2022-10025, 2025-272, 2023-10176, 2023-111241, 2023-10048, 2020-10653, 2020-10043, 2020-10099

Docs: /vision, /visionkit, /vision/vnrecognizetextrequest, /vision/vndetectbarcodesrequest

Skills: axiom-vision, axiom-vision-diag

axiom-vision-ref

Vision Framework API Reference

When to Use This Reference

Vision Framework Overview

Request Handlers

VNImageRequestHandler

VNSequenceRequestHandler

When to Use Which

Requests That Benefit from Sequence Handling

Common Mistake

Subject Segmentation APIs

VNGenerateForegroundInstanceMaskRequest

Basic Usage

InstanceMaskObservation

Generating Masks

Instance Mask Hit Testing

VisionKit Subject Lifting

ImageAnalysisInteraction (iOS)

ImageAnalysisOverlayView (macOS)

Programmatic Access

ImageAnalyzer

ImageAnalysis

Subject Struct

Person Segmentation APIs

VNGeneratePersonSegmentationRequest

VNGeneratePersonInstanceMaskRequest

Hand Pose Detection

VNDetectHumanHandPoseRequest

Hand Landmarks (21 points)

Group Keys

Gesture Recognition Example (Pinch)

Chirality (Handedness)

Body Pose Detection

VNDetectHumanBodyPoseRequest (2D)

Body Landmarks (18 points)

Group Keys (Body)

VNDetectHumanBodyPose3DRequest (3D)

3D Observation Properties

3D Point Classes

Depth Input

Face Detection & Landmarks

VNDetectFaceRectanglesRequest

VNDetectFaceLandmarksRequest

Person Detection

VNDetectHumanRectanglesRequest

CoreImage Integration

CIBlendWithMask Filter

Text Recognition APIs

VNRecognizeTextRequest

Basic Usage

Recognition Levels

Properties

Language Support

VNRecognizedTextObservation

VNRecognizedText

Barcode Detection APIs

VNDetectBarcodesRequest

Basic Usage

Symbologies

Revisions

VNBarcodeObservation

VisionKit Scanner APIs

DataScannerViewController

Check Availability

Configuration

RecognizedDataType

Delegate Protocol

RecognizedItem

Async Stream

Custom Highlights

VNDocumentCameraViewController

Basic Usage

Delegate Protocol

VNDocumentCameraScan

Document Analysis APIs

VNDetectDocumentSegmentationRequest

RecognizeDocumentsRequest (iOS 26+)

Basic Usage

DocumentObservation Hierarchy

Table Extraction