January 27, 2014 Sam Siewert

Computer and Machine

Vision

Lecture Week 3

Part-1

Outline of Week 3

Processing Images and Moving Pictures –

High Level View and Computer

Architecture for it

Linux Platforms for Computer/Machine

Vision

I/O, Memory and Processing Challenges

Sam Siewert 2

Old School Moving Picture Media

and Cameras NTSC OTA (1941, 1953 color, 2009 dead)

– Analog, Interlaced, Continuous Broadcast Transmission or CCTV (Closed Circuit TV)

– Coax Cable or Tuner with Immediate CRT Display

– No Buffers, No Routing, No De-mux

– No Compression

Analog Cable

AM/FM OTA

Film Projectors

Sam Siewert 3

Modern Digital Cameras Camera Link – High Frame Rates

– High Data Rates and Resolutions

– Industry Standard for Machine Vision Automation

– E.g. Inspection Systems

– E.g. – Sony, IDT, National Instruments

SD-SDI and HD-SDI – Standard and High Definition Synchronous Digital Interface

– Standard for Studios, Broadcast

Digital Cinema – Red Camera

– 1080p, 2K, 4K Resolutions and Much Higher

– Automated Digital Delivery and Projection

Webcams and Mobile Phone Cameras – Very Low Cost

– Proprietary

– Performance Varies Dramatically

Sam Siewert 4

Differences Analog vs Digital Encoding for Transmission

– Digital Allows for Image Processing

– Adds Latency

– Requires Compression for Packet Switched Networks and Storage

Routed (Diversely), Buffered

Compressed (MPEG, JPEG) to Lower Bit-rates

Multiplexed (Shares Transmission Carrier for Audio, Video, Channels)

Transported by IP (Large Packets)

Continuous Transmission – Analog or Constant Bit-Rate / Frame-Rate

Sam Siewert 5

E.g. UAV Latency and Jitter

Verification of Video Frame Latency Telemetry for UAV Systems Using A Secondary Optical Method, Sam Siewert, Muhammad Ahmad, Kevin Yao

Sam Siewert 6

NTSC (Analog TV)

Sam Siewert 7

AM Video to CRT

FM Audio

Chroma Added Later

Odd/Even Lines (Interlaced)

29.97 FPS (30 before color)

Vertical Blanking (CRT Retrace Time, Closed Captioning)

525 Lines, 262.5 per Field, 60 Fields per Second

http://en.wikipedia.org/wiki/File:Ntsc_channel.svg

Linux in Computer Vision

Embedded Solutions – Texas Instruments OMAP (Beagle xM, Bone)

– Numerous ARM SoCs (NVIDIA, Qualcomm, Broadcomm, …)

Scalable Solutions – Multi-Core (Xeon Phi)

– Vector Processing – CUDA, OpenCL GPU and GP-GPU

Computer and Machine Vision is I/O, Memory and Processing Intensive

Sam Siewert 8

Camera Interfaces

CCD (Charge Coupled Device) or CMOS (Common Metal Oxide Substrate) Detector – Integration Time for Photo-sensitive Elements in Array (to Build

up Charge)

– Read-out Time to Sample Elements in Array

Luminance and Chroma Analog to Digital Conversion

Double Buffer for Read-out + Processing

Frame Capture – http://www.cse.uaa.alaska.edu/~ssiewert/a485_doc/Frame-

Capture-Chips/

– Host Interface over PCI Bus or USB

Sam Siewert 9

Digital Video Transport QoS Latency – To Tune in a Program, Turn-on

– To Deliver a Video Frame or Audio PCM Sample

– To Start, FF, REW, Start-Over, Pause

Bandwidth – Resolution, Lossy/Lossless Compression, High Motion

– Pixel Encoding for Color

– Frame Rate

– Constant Bit-rate Transport?

– Variable Bit-rate Transport and Encoding?

Jitter – Decode and Presentation Rates

– Elasticity in Decode to Presentation Buffering Necessary

Sam Siewert 10

January 27, 2014 Sam Siewert

Linux System Options

(Linux for Image Processing, Camera

Interfacing and Computer Vision)

Sam Siewert 12

Processing Outline

Many-Core Linux Host(s)

– Intel Atom

– ARM

– Xeon

GP-GPU Vector Processing PCI-E Co-Processors

NVIDIA Tesla/Fermi

AMD ATI

NPTL – Native POSIX Threads Library

NPTL Example Code Walkthrough

Sam Siewert 13

Conceptual View of RT Resources Three-Space View of Utilization Requirements

– CPU Margin?

– IO Latency (and Bandwidth) Margin?

– Memory Capacity (and Latency) Margin?

Upper Right Front Corner – Low-Margin

Origin – High-Margin

Mobile – Must Consider Battery Life Too (Power)

CPU-Utility

IO-Utility

Memory-Utility

Processing – Initial Focus Processing and Scaling Frame Transformation, Encode, Decode is Critical

Memory for Buffering (Frame Transformations, CPU Integrated or GPU Offloaded – e.g. Linux VDPAU)

I/O for Networking (Transport)

I/O for Storage (On-Demand, Post, Non-Linear Editing)

Sam Siewert 14

Flynn’s Computer Architecture

Taxonomy Single Instruction Multiple Instruction

Single Data SISD (Traditional Uni-

processor)

MISD (Voting schemes

and active-active

controllers)

Multiple Data SIMD (e.g. SSE 4.2, GP-

GPU, Vector Processing)

MIMD (Distributed

systems (MPMD),

Clusters with MPI/PVM

(SPMD), AMP/SMP)

Sam Siewert 15

GPC has gone MIMD with SIMD Instruction Sets and SIMD Offload

(GP-GPU) NUMA vs. UMA (Trend away from UMA to NUMA or MCH vs. IOH)

SMP with One OS (Shared Memory, CPU-balanced Interrupt Handling,

Process Load Balancing, Mutli-User, Multi-Application, CPU Affinity

Possible)

MIMD - Single Program Multi-Data vs. Multi-Program Multi-Data

Computer and Machine Vision

Treated as a Real-time and/or Interactive

System

– Requires Predictable Response (By Deadline)

– Rate Monotonic

– Earliest Deadline First

– Least Laxity First

Covered in Depth in OS and Digital Media

(Just be Aware for Computer Vision) Sam Siewert 16

Sam Siewert 17

CPU Scheduling Taxonomy Execution Scheduling

Global-MP Local-Uniprocessor

Distributed Asymmetric

(AMP )

Symmetric

(SMP OS)

Preemptive Non-Preemptive

Fixed-Priority

Hybrid

Dynamic-Priority Cooperative

Batch

FCFS SJN

Co-Routine Continuation

Function

Heuristic EDF/LLF RR Timeslice

(desktop)

Multi-Frequency

Executives

Static Dynamic

Rate

Monotonic

Deadline

Monotonic

Dataflow

(Preemptive, Non-Preemptive Subtree

Under Each Global-MP Leaf)

SMT

(Micro-Paralell)

Sam Siewert 18

Response Latency Ci WCET

Input/Output Latency

Interference Time

Event

Sensed Interrupt Dispatch Preemption Dispatch

Interference

Completion

(IO Queued)

Actuation

(IO Completion)

Input-Latency

Dispatch-Latency

Execution Execution

Output-Latency

Time

Response Time = TimeActuation – TimeSensed

(From Release to Response)

Resource Scaling

Processing – Co-Processors – GPU – CUDA, OpenCL

– Many-Core – E.g. Intel Xeon Phi MICA

– FPGA – E.g. Altera Stratix

– Ideally Camera Interface

I/O – High Rate Transport – HD-SDI, Camera Link, GigE/10GE

Memory – SSD, PCIe Nand, NVM – FusionIO, Micron, Intel

– Memristor (Future)

Sam Siewert 19