Big Data and Intel® Intelligent Systems Solution for Intelligent transportation

Big Data and Intel® Intelligent Systems Solution for Intelligent Transportation

EMBS001

Xiao Dong Wang, Manager, Big Data Solution Team, Intel Robin Wang, Platform Solution Architect, Intel Albert Hu, Solution Architect, Intel

2

Agenda • Intelligent Transportation System (ITS) landscape in

China • Blueprint for ITS • Big Data overview and benefit for ITS • Intel® Architecture based products for Big Data on

ITS • ITS case study in China

The PDF for this Session presentation is available from our Technical Session Catalog at the end of the day at: intel.com/go/idfsessionsBJ

URL is on top of Session Agenda Pages in Pocket Guide

3




4

China Environment Government Objective • Government 12-5Y Plan/Social Harmony • Determine to lead in Internet of Things (IoT)

‒ Setting international standards for new technology

‒ $0.8Bn government funds

Mega Trend Challenges & Government approach • Urbanization

‒ 690Mn now to 900Mn by ‘25

• IoT/Smart City is one way to solve the challenges ‒ IoT market size $80~120Bn by ‘15 ‒ 90+ smart cities plans underway ‒ Gaps: Core technology, standards, immature ecosystem, deployment

model

“The biggest development potential lies in the process of urbanization.”

-- China New Premier Li Keqiang, ‘12

“Sensing China” (IoT) strategy -- China Premier Wen Jiabao, ‘09

5

-200

0

200

400

600

800

1000

1200

1400

1600

-2% 0% 2% 4% 6% 8% 10%

PRC Transportation Infrastructure Landscape

Highway

Urban Public

Railway Waterway

2011-2015 CAGR (in terms of length)

2011

Inv

estm

ent

(Bn

RM

B)

4,500,000km

126,000km

120,000km

394,000km

Total Number of Vehicles will exceed 200 Million by 2020

Infrastructure build-out trending to be stabilized. Key challenges due to large scale of the infrastructure network, growing number of

vehicles, yet still higher traveler, vehicle/infrastructure density: Safety, Infrastructure/Traveler’s efficiency, Environment.

Source: China Ministry of Transportation’s 12th 5 year plan Source: ISH* Research Report

1

1

2

2

6

Big Data Source From Transportation

3%

13%

7%

3%

7%

21%

4% 3%

37%

2%

2016 Banking & Finance

Casinos & Gaming

City Surveillance

Commercial

Education

Government

Industrial, Manufacturing, &UtilitiesRetail

Transport

Other

Source: IMS Enterprise and IP Storage used for Video Surveillance – World-2012

Worldwide Enterprise and IP Storage used for Video Surveillance by End User (2016)

0.3PB ~ 6.7PB/Day Video Data generated for Smart City

Environment

How to effectively collect, aggregate,

manage and analyze data to help Intelligent

Transportation System (ITS) application?

7




8

Goals of Intelligent Transportation System • Traffic Management

– Enforcing traffic regulations – Transportation planning support – Adaptive traffic control – Case investigation for police

• Traveler information system – Real-time road condition

Speed & congestion Historical camera images & statistics

– Travel time information Available to various terminals Proactive travel plan

• Commercial vehicle systems – Commercial vehicle management,

tracking, administration

• Public security – Video surveillance (remote video

streaming & video searching)

9

Edge Server

Intelligent Transportation System (ITS) End-to-End Solution

Collect, store, transform, analyze

and mine • Embedded • Cloud service • Proprietary • High-performance

Data Center Morphology

Data Center Solutions

Distributed Filesystem – HDFS*

Distributed data analysis – Hadoop*

Distributed real-time database – HBase*

Terminal Device Abundant data visualization Data analysis and cache

NVR/DVR/Hybrid NVR

Decoder

Edge Video Analytic

10

Intelligent Transportation System (ITS) Cross Region Deployment

11




12

Scale Up or Scale Out Intelligent

Transportation System (ITS)

Data Burst

Relational Database shift left

Distributional Database shift right

13

Intelligent Transportation System (ITS) Software Architecture

Sqoop*

Data Integration

HBase*

Distributed Database

for texts & images

RDB

MapReduce Offline analysis

Online/Interactive Applications Data Mining

Legacy Applications

Aggregated results

14

Big Data Intel® Distribution for Apache Hadoop* Software

Optimized Software Stack • Stable, enterprise-ready Hadoop* • Optimized for Intel® Architecture • Bring “Real-time” analysis to Hadoop by HBase*

enhancements • Enhanced features to Hadoop for vertical

segments

Hive* 0.9.0 Data Warehouse

Sqo

op*

1.4.

1

RD

B D

ata

Col

lect

or

Flum

e* 1

.3.0

Lo

g D

ata

Col

lect

or

Intel® Manager for Apache Hadoop Software 2.3 Deployment, Configuration, Monitoring, Alerting and Security

ZooK

eepe

r*

3.4.

5 Coo

rdin

atio

n

Pig* 0.9.2 Data Manipulation

Mahout* 0.7 Data Mining

HBase 0.94.1 Real-time Distributed Big Table

MapReduce 1.0.3 Distributed Processing Framework

HDFS* 1.0.3 Hadoop Distributed File System

R - statistics Data Manipulation

Oozie* 3.3.0 Workflow Scheduler

15

Intel® Distribution for Apache Hadoop* Software Enhancements for Intelligent Transportation System

Enhancement Benefit for ITS Cross-site Big Table for HBase* • Data are stored in different region data center

with a global virtual view • Each data center is the live backup to provide

data access high availability

SQL Layer on top of HBase • Real-time statistics on the big mount of traffic data

• The interactive query and offline statistic share the same set of data

Full-text indexing and near-real-time search for HBase

• Provide the full text search capability on the structured data in distribution database system

• Build in index make sure that the traffic data always synchronize with the index

Efficiently Big Object Storage in HBase

• Increase the traffic image store performance with the standard HBase interface

R language statistics support to Hadoop*

• Brings the mature R language library to the MapReduce, HDFS* and HBase

• Reduce the effort to develop the complex data mining logic

16




17

Value for Edge Analytics

Private Cloud Public Cloud

Edge Client (Video capture)

Video Indexer/Analyzer/Transcoder (Image extraction & Metadata Creation)

Video Storage (Edge or

Centralized)

Data Center/ Cloud (Private/Public)

Data Services (VSaaS, VAaaS)

Checkpoint District City Province PRC Sm

art

Camera

Police Car

Management System

Video created Video analyzed Video Cold storage Video metadata stored

Edge VA’s Value • Real-time intelligence (into metadata)

1/6 of video data • Reduce the footage to be 1/8 ~ 1/12

of its original size • Resolve the bandwidth issue and

backend storage capacity constrain

76 PB Metadata Per Day

By end of 2017 457 PB raw video

for traffic generated per day

People, cars, license plates

By end of 2017 1

Source: Internal Team Analysis based on IHS* Research Report 1

18

Enhanced NVR (Network Video Recorder) Key Features for Intelligent Transportation System (ITS) 闯红灯// Run the red light

车牌颜色识别 // Plate colour recognition

逆行 // Retrogradation

车身颜色识别 // Vehicle colour recognition

车牌识别 // Plate Recognition

交通拥堵 // Traffic jam

车流统计// Vehicle counting

行使缓慢 // Run slowly

禁停// No parking 行使超速 // Speeding 禁左禁右 // No turn right or left

行人横穿//Jaywalk

占道(不按规定车道行使) 车标识别//Auto logo recognition

变道 // Lane Change

机动车抓拍 // Abnormality quick shot

压线 // Line crossing

超速 // Speeding

Crossing vehicle capture

Vehicle features recognition

19

Intel® Architecture Base NVR

3rd Generation Intel® Core™ Processor Family - 4 x 8G DDR3 Memory - 2 x 10M/100M/1000M Base-T LAN - 8 x SATA3.0 or 16 x SATA3.0 - 1 x MSATA - 2U or 3U rack-mounted Chassis

Intel® Atom™ Processor D2550, Intel® NM10 Express Chipset - 1 x 4G DDR3 Memory - 2 x 100M/1000M LAN - 8 x SATA3.0; - 1 x MSATA; - 2U rack-mounted Chassis

Intel® Xeon® Processor E3 Family, Intel® C216 Chipset -Up to 32G DDR3 Memory -4 x 10M/100M/1000M Base-T LAN -16 x SATA3.0 or 24 x SATA3.0 -2 x MSATA; -3U or 4U rack-mounted Chassis

*

*

*

20

InfiniBand* and Ethernet Switches

Intel® Server Board S2600GZ “Grizzly Pass” Intel® Server System R2000 “Big Horn Peak” HDFS* Data Node • Large Storage Capacity • Large Memory Capacity • Extreme Power Efficiency • Extreme FDR InfiniBand • Extensive I/O • Optional SSD or PCI Express* SSD

Intel Server Board S2600JF “Jefferson Pass” Intel Server System H2000 “Bobcat Peak”

HDFS Name Node • High Density Form Factor • High Memory Bandwidth • Extreme FDR InfiniBand • Optional SSD or PCI Express SSD Can be data node for compute-intensive Big Data applications

Big Data Appliance Reference Design from Intel

21

Big Data Appliance Reference Design: Turnkey Platforms for ISV/SI/LOEM

Easy to Use • Easy to deploy • Easy to scale-out • Easy to manage • Rapid deployment in days • Quickly isolate root cause

between appliance and application

Quality • Integrated validation of all

components • OS and device drivers • Big Data software packages • BIOS, firmware, etc. • Embedded acceptance test • Disk health monitoring

Power Efficiency • Spread core design • Cold Redundant Power Supply • Intelligent disk spin-up/off • ACPI* S3/S4 support • DCM integrated at rack

Performance • 10GbE, InfiniBand* • Advanced storage controller • SSD and PCI Express* SSD • SW tuning: block size, # of

reducers, etc.

Big Data ISV/SI/LOEM all look forward to a total solution

22

Big Data Appliances Reference Design from Intel® Performance & Power Advantages

• 10GbE & InfiniBand* FDR • Network protocol advances • SSD, PCI Express* SSD, Hybrid • Advanced storage controller • Balance oriented optimization

Low Power Technology Sources

Power Supply: 80 PLUS Platinum HW

Power Supply: “cold” redundancy HW

Spread-core server board layout HW

ACPI S3 support HW-SW

ACPI S4 support HW-SW

Staggered disk spin up HW-FW

Intelligent disk spin off control HW-SW

Data center, rack, and node level power monitoring and limiting HW-FW-SW

Load (%)

Uti

lizat

ion

(%

)

Threshold (40%)

CRPS (5-10% up)

1 module works

2 modules work

Normal PSU

Cold Redundancy Power Supply (CRPS)

80 PLUS Certification

230V Internal Redundant

% of Rated Load 20% 50% 100% 80 PLUS Bronze 81% 85% 81% 80 PLUS Silver 85% 89% 85% 80 PLUS Gold 88% 92% 88% 80 PLUS Platinum 90% 94% 91%

23




24

Case Study: Intelligent Monitoring and Recording System

HBase*

MapReduce

HDFS*

Hive*

ETL

3G

Real-time Vehicle License Analysis

Vehicle History

Behavior

Traffic Flow Analysis

Edge Video Analytic (Enhanced NVR)

25

Case Study: Intelligent Transportation System (ITS) Solution Traffic Management • Real-time road conditions report • Over speed vehicles detection in

road segment • Fake plate number detection Public Security • Tracking vehicle in real-time • Alerts and alarms based on

blacklist Traveler Guide • Real-time road condition by

getting latest camera images and traffic flow statistics

• Travel time estimation for road segments in the city

26

Case Study: Intelligent Transportation System (ITS) Result

违法车辆追踪效率提升

恶性交通事故死亡人数减少

道路拥堵率下降

通过海量数据实时分析处理功能能将违法车辆数据定位时间由小时级缩减为分钟级甚至秒级

通过浮动车监控系统收集车辆信息并且实时分析，能够对事故高发车辆（如工程货车）进行行为监控，降低恶性事故率。

通过路况监控设备收集路况信息并实时处理，能够精确绘制道路拥堵线圈，提供交管部门快速处理突发事故，并提供给大众平台供驾驶员参考从而疏导车流

Illegal vehicle tracking efficiency

Deaths in bad traffic accidents

Road congestion rate

Through the massive data real-time analysis function, the illegal vehicle location data time is by the hour Level reduced to minutes or even seconds.

Through the floating vehicle monitoring system to collect vehicle information and real-time analysis to monitor high Service Vehicles (such as engineering truck ) behavior and reduce the accident rate.

Through the traffic monitoring equipment to collect traffic information and real-time processing, the road congestion coil can be drawn accurately, emergency can be routed to traffic management departments rapidly and traffic drivers can be diverted accordingly.

27

Summary

• Intelligent Transportation System (ITS) is Intel® global focus now and future

• Intel®’s end-to-end analytics architecture

fits ITS solution development • Intel® has rich resources to help developers

for ITS related application development

28

Additional Sources of Information: Other Sessions

• EMBS002 ‒ Real Time Cloud Infrastructure and Virtualized Data Plane Design with Intel®

Architecture: April 10, 14:30 in Room 307A • EMBL001

‒ Hands-on Lab: Next Generation Firewall and Deep Packet Inspection on Intel® Platforms: April 10, 13:15 in Room 306A

• EMBL001R ‒ Repeat of Hands-on Lab: Next Generation Firewall and Deep Packet Inspection on

Intel® Platforms: April 10, 15:45 in Room 306A • EMBS003

‒ Telecommunication Platforms: Streaming Media Processing on Intel® Architecture: April 11, 15:45 in Room 307A

• EMBS004 ‒ Create Intelligent Retail Solutions that Deliver Engaging User Experiences: April 11,

15:00 in Room 307A

Demos in the showcase ‒ Booth No.: E20 “Hikvision* demo” ‒ Booth No.: E40 “Intel® Server Solutions”

More web based info: ‒ DSS web link: http://www.intel.com/info/dss ‒ Server Edge: www.IntelServerEdge.com

http://www.intel.com/info/dss

http://www.intelserveredge.com/

29

Legal Disclaimer INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. • A "Mission Critical Application" is any application in which failure of the Intel® Product could result, directly or indirectly, in

personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

• Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information.

• The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.

• Intel® product plans in this presentation do not constitute Intel plan of record product roadmaps. Please contact your Intel representative to obtain Intel's current plan of record product roadmaps.

• Intel® processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. Go to: http://www.intel.com/products/processor_number.

• Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. • Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be

obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm • Grizzly Pass, Big Horn Peak, Jefferson Pass, Bobcat Peak and other code names featured are used internally within Intel to

identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user

• Intel, the Intel logo, Intel Atom, Intel Atom Inside, Intel Core, Core Inside, Intel Sponsors of Tomorrow., the Intel Sponsors of Tomorrow. logo, Xeon and Xeon Inside are trademarks of Intel Corporation in the United States and/or other countries.

• *Other names and brands may be claimed as the property of others. • Copyright © 2013 Intel Corporation.

http://www.intel.com/design/literature.htm

30

Risk Factors The above statements and any others in this document that refer to plans and expectations for the first quarter, the year and the future are forward-looking statements that involve a number of risks and uncertainties. Words such as “anticipates,” “expects,” “intends,” “plans,” “believes,” “seeks,” “estimates,” “may,” “will,” “should” and their variations identify forward-looking statements. Statements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements. Many factors could affect Intel’s actual results, and variances from Intel’s current expectations regarding such factors could cause actual results to differ materially from those expressed in these forward-looking statements. Intel presently considers the following to be the important factors that could cause actual results to differ materially from the company’s expectations. Demand could be different from Intel's expectations due to factors including changes in business and economic conditions; customer acceptance of Intel’s and competitors’ products; supply constraints and other disruptions affecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Uncertainty in global economic and financial conditions poses a risk that consumers and businesses may defer purchases in response to negative financial events, which could negatively affect product demand and other related matters. Intel operates in intensely competitive industries that are characterized by a high percentage of costs that are fixed or difficult to reduce in the short term and product demand that is highly variable and difficult to forecast. Revenue and the gross margin percentage are affected by the timing of Intel® product introductions and the demand for and market acceptance of Intel's products; actions taken by Intel's competitors, including product offerings and introductions, marketing programs and pricing pressures and Intel’s response to such actions; and Intel’s ability to respond quickly to technological developments and to incorporate new features into its products. The gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; start-up costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; product manufacturing quality/yields; and impairments of long-lived assets, including manufacturing, assembly/test and intangible assets. Intel's results could be affected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Expenses, particularly certain marketing and compensation expenses, as well as restructuring and asset impairment charges, vary depending on the level of demand for Intel's products and the level of revenue and profits. Intel’s results could be affected by the timing of closing of acquisitions and divestitures. Intel’s current chief executive officer plans to retire in May 2013 and the Board of Directors is working to choose a successor. The succession and transition process may have a direct and/or indirect effect on the business and operations of the company. In connection with the appointment of the new CEO, the company will seek to retain our executive management team (some of whom are being considered for the CEO position), and keep employees focused on achieving the company’s strategic goals and objectives. Intel's results could be affected by adverse effects associated with product defects and errata (deviations from published specifications), and by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues, such as the litigation and regulatory matters described in Intel's SEC reports. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel’s ability to design its products, or requiring other remedies such as compulsory licensing of intellectual property. A detailed discussion of these and other factors that could affect Intel’s results is included in Intel’s SEC filings, including the company’s most recent Form 10-Q, report on Form 10-K and earnings release. Rev. 1/17/13

31

Backup

32

Intelligent Transportation System (ITS) benefits from Interactive Hive Query

68 98

63

159

0.2 0.2 18 28

0

50

100

150

200

Query 1 Query 2 Query 3 Query 4

Hive 0.9.0 (M/R) (sec)

Interactive Hive (sec)

100 million records over a 8-node cluster

User Scenario Query Calculate each day’s internet traffic of a specific user

SELECT sum(down+up) FROM cdr201209 WHERE number = '13300000000' GROUP BY day;

Get the 10 most heavily called numbers for a specific user

SELECT TOP(10) tonumber, sum(call_length) len FROM cdr_201209 WHERE number = '13300032810' GROUP BY tonumber ORDER BY len DESC

Get the top 1000 call length from all user phone calls

SELECT TOP(1000) number, call_length FROM cdr_201209 ORDER BY call_length DESC

Get the top 1000 users having highest total monthly charge

SELECT TOP(1000) number, sum(fee) f FROM cdr_201209 GROUP BY number order by f DESC

Intel® Distribution for Apache Hadoop* Software Enhancement

*

33

1. Global Table View 2. Data are physically stored

in geo-distributed data centers

3. Higher availability 4. Better locality 5. Distributed aggregation

removes data transfer

Virtual Big Table

Async Replication

Data Center A

Data Center B

Data Center C

Intelligent Transportation System (ITS) benefits form Cross-site Big Table Two deployment models:

1. In transportation system, each district has a DC, one can connect to any DC and view all of the data

2. In banks, provincial branch has its own DC. Central bank can view all of the data, but branches can not see each other.


34

Intelligent Transportation System (ITS) Benefits From HBase* Big Object Storage

0

50

100

150

200

250

0s 100s 200s 300s 400s 500s

hbase(no presplit)hbase lob

Insertion Performance(500KB/record) records/second

Insertion Performance(Single Client, No pre-split)

Test setup: (intel-01 cluster, 6 machines, E5-2620, 24core, 48G memory). No client cache, No WAL. For HBase* (no split), after insertion, the region count is 20.

Insertion Delay(Pre-split 32 regions, 6 Client Nodes)

Insert performance increase 200%，insert latency reduces 90%


*

0

20

40

60

80

100

120

0s 200s 400s 600s 800s 1000s 1200s

hbase lob delay/s

hbase delay/s

Insertion

Delay:

*

Big Data and Intel® Intelligent Systems Solution for Intelligent transportation

Technology

Transcript of Big Data and Intel® Intelligent Systems Solution for Intelligent transportation