Embedded Video Analytics

DSP Algorithms forDetection, Tracking and Recognition

http://synesis.ru/

Media and Internet

Face detection and recognition servers

Intelligent Video Surveillance

Intelligent cameras, encoders and DVRs

Digital TVDVB receivers,

STBs, PVRs,media centres

HD Intelligent Network Video

Efficient video surveillance (1)

Accurateevent recognition• correct classification• false positives and

false negatives• response time• documentation

?

Efficient video surveillance (2)

Widespread infrastructure• Cross-correlation of

events captured by multiple cameras and other sensors

• Alert prioritization • Distributed attacks

(multiple point intrusions)

Efficient video surveillance (3)

Operator productivity

• Keep attention focused• Reduce subjectivism• Increase response time

Efficient video surveillance (4)

Cost of ownership

• Deployment• Maintenance• Telecom service charges• Minimum team size• Training• Upgrade

(Investment protection)

What is video analytics?

X, Y, Z

Functions of video analytics

1. Anti-tampering and operability monitoring2. Operational alerts

– Automatic priorities

3. Automatic PTZ-camera targeting4. Event recording for instant forensic analysis5. Optimal usage of

network bandwidth and storage memory

Solution: embedded video analytics• Edge device transmits video and

metadata (object and its behaviour description)

VIDEO

METADATAEVENT

DATABASE

Zone 5intrusiondetected

EVENT RULES

Embedded vs server analytics

camera orencoder

video management system or DVR

compressedvideo & audiocodecs video-

analytics

video management system or DVR

ip-cameraor encoder

video and audiocodecs

videoanalytics

metadata

Embedded(front-end)analytics

Server(back-end)analytics

BOTTLENECK

Video signal sources

1. Analoguestandard definition cameras(PAL/NTSC)

2. Network cameras(standard and highdefinition)

3. Thermal cameras

Network cameraAxis 211A

Thermal cameraTitan-14

Wide angle perimeter surveillance(multiple tripwire alert levels)

Fence crossing detector

Apartment housing event recording

Directional detector

Running behaviour recognition

Time-based loitering behaviour recognition

Split target /abandon luggage detection

Group people tracking

Tampering and malfunction detectors• Loss of signal• Obstruction• Out of focus and lens

dusting• Blackout and overexposure • AE failure• Lighting

failure

Upon a suspicious event…• PTZ-targeting• System notification

over IP network to VMS– Sound and visual alarms, SMS etc

• ‘Dry contact’ signal• High quality recording to local

or remote storage (NAS)• Analogue output to legacy

systems (matrix or DVR)

Digital image stabiliser (antishaker)• Eliminates video shaking

caused by wind and industrial vibrations • Essential for analytics performance• Differentiates the camera movements

from scene background/foreground movements

Video analytics components

Detection

Tracking

Recognition

Sterile zone Public spaces

Rare appearance Occasional appearance People flow

perimeter security,strategic

infrastructure

apartment housing, petrol stations, office

buildings

airports,railway stations,

underground

Object tracker complexity

complexity

Dynamic texture of the real world

Dynamic texture modelling

• 4D-pyramid• Feature

probability cloud• α-channel (mask) for

each object

BACKGROUND OBJECT HAAR FEATURES

People group tracking (Q4 2010)

• Feature cloud enables object tracking under partial visibility

• Z-buffer to identify object occlusions

Rule based behaviour recognitionEach zone is configured independently

Zone entrance

Zone exist

Zone loitering:Staying overpredefined period of time

Zone running:Exceeding a predefined speed

Directional move within zone

Metadata sent over IP network / ONVIF• Event type, data and time• Zone or tripwire number• 2D object feature:

– Position, size, area, speed• Real 3D features

– Estimated from 2D featuresusing calibration data

• JPEG frame image withobject trajectory annotation

Videoanalytics calibration

• Two human figures define scale & angle

• Drag’n’drop calibration

• Tracking region

• 2D to 3D coordinate transform

Video analytics parameters1. Service detectors2. Antishaker3. Object tracker

1. Contrast sensitivity2. Special sensitivity3. Min. stabilisation time

4. Object filters1. Maximum object speed2. Min and max areas

1

2

3

4

Video analytics evaluationMethods and results

Video analytics public testsOrganisation Videoanalytics tests

Home Office Scientific Development Branch (HOSDB), UK

• Imagery library for intelligent detection systems (i-LIDS)

National Institute of Standards and Technology (NIST), USA

• AVSS 2009 Multi-Camera Tracking Challenge (based on i-LIDS)

• Face Recognition Vendor Test (FRVT)

Institute of Electrical and Electronics Engineers (IEEE), USA

• Performance Evaluation of Tracking and Surveillance (PETS)

• International Workshop on Performance Evaluation of Tracking and Surveillance

• International Conference on Advanced Video and Signal Based Surveillance

Sterile Zone Performance38 hours, PAL (720 x 576 x 25 fps), M-JPEG, 40 MbpsNumber of true positive alarms: a = 432

False positives alarms (type I error): b = 2

False negatives alarms (type II error): с = 0

Role Recall bias Recall rate Precision Weighted average

Operating alert 0.65 1.00 1.00 1.00

Event recording 75.00 1.00 1.00 1.00

Resolution vs width field of view (FoV)

7-12 m

12-23 m

27-37 m

Maximum response time

• People walking and running–2 seconds

• People moving slowly(e.g. crawling)–10 seconds

Causes of false negatives(simple motion detectors)

• Unstable background decreases sensitivity of an adaptive detector

DYNAMIC TEXTURE MODELING ALGORITHMSENABLE ROBUST OBJECT DETECTION IN A CHALENGING ENVIROMENT

Causes of false positives(basic motion detectors)

• Variable lighting– Shadows from moving clouds and sun– Moving trees, bushes and water

• Camera shaking• Animals, birds and insects• Object trajectory split and double detection• Snow, rain, fog

Examples of false positives(simple motion detectors)

INSECT RABBIT

CAMERA SHAKING

VIDEO ANALYTICS PREVENTS FALSE ALARMS CAUSED BY THESE FACTORS

BIRD

Object trackingwhilst tree shadows moving

Performance estimation by3D security modeling

• 3D modeling– building infrastructure– control zones of cameras

and third-party detectors– treats (in space-time)

• Estimation of detection probabilities under variable external conditions– day/night, fog, snow

• Video presentation

ORIGINAL BUILDING

3D MODEL OF BUILDNG

Hardware reference designsMultifunctional video services and HD cameras

with embedded analytics

System-on-chip video analytics

Videoanalytics

HD H.264 codec

Linux Videofilters

1080p

Dual channel video analytics encoder

04/11/2023 44

• Two analogue inputs (BNC)• Two managed outputs (BNC)

and digital video over IP• H.264 & MJPEG encoding• Embedded video & audio analytics• POE+ and backup power• ONVIF 1.01 support• - 40⁰...+50⁰ С• Lightning guard

ANALOG + IPHYBRID TECHNOLOGY

Dual channel video analytics encoder

Interfaces

LAN USB I/OAUDIO OUT

POWER BATTERY

AUDIO INRESET

HD video analytics camera

MJPEG vs H.264 compression

HD 1080i HD 720p D1 480p0

5

10

15

20

25

30

35

40

H.264MJPEG

DA

TAF

LO

W, M

BP

S

RESOLUTION

H.264 MJPEGHD 1080i 2.3 34.1HD 720p 1.8 19.6D1 480p 1.5 3.4

Applications and use-casesVideo analytics encoder

Self-contained intelligence for perimeter security

Integrated solution:1. Embedded video analytics2. Automatic PTZ targeting3. Unlimited, multizone sensor

integration (I/O, RS485)4. Active illumination5. Two-way intercom6. Backup power &

battery management MB

Sophisticated landscape

Strategic infrastructure

Cost-effective upgrade oflegacy analogue infrastructure

• No cable or camera replacement required• Increase storage efficiency by 10-100 times• Automatic operational alerts• Intelligent search using recorder events• Future proof network surveillance via ONVIF

Local/backup storage• Detachable video storage

– USB 2.5” hard drive or flash memory• Accurate timestamp (NTP sync)• Backup storage if NAS not available• Portable player, video can be played on any PC

Unique selling position1. Fully embedded (DSP) implementation

– Real-time processing of uncompressed video– HD/Megapixel resolution– Highly scalable

2. Unmatched performance in harsh environment– dynamic texture engine

3. Wide interoperability– ONVIF compliance

Example of customization

1. Custom user interface2. Custom network and serial protocols3. Overlay text (POS, industrial etc)4. Custom DaVinci codecs (e.g. H.264 SVC)5. Custom video analytics

Future of video surveillanceMultiple camera tracking using 3D model

Segmentation problemand object occlusions

‘Single camera’video analytics

AB

C

A

‘Multiple camera’video analytics

i-LIDS multiple camera tracking scenario

2 3 4

11/04/2023 www.synesis.ru 60

1 2

Камера 1 Камера 2

3D model of a buildingand camera controlzones

Video analytics + 3D modeling

11/04/2023 61

OBJECT UNIQUE ID PRESERVED WHEN TRACKING FROM CAMERA TO CAMERA

3D trajectory reconstructed frommultiple video sources