University of Southern California · Data Social Sci. Power Engr. Facility Mgmt. Univ. Staff Extern...

University of Southern California

USC Center for Energy Informatics cei.usc.edu

Microsoft Cloud Futures Workshop 2


Consumers

Information Technology

Energy Technology



INFR

AST

RU

CTU

RE

SOC

IAL

CYBER SPACE PHYSICAL SPACE

Compute & Data Cyber

Infrastructure

Physical Microgrid

Infrastructure

Online Social Networks

Physical Social Interactions

H H

H

H

H H

H

Sensor

Actuator



“The Smart Grid Explained, The Hype and The Promise”, WESCO, FMEA/FMPA Conference 2009



US EIA’s Annual Energy Outlook 2011

Source: GDF Suez



Software Platform

Energy System

Consumer Behavior



Pillcrow Graph Analytics

Cryptonite

Learn from the Microgrid. Scale to the Mega-city grid.

Security & Privacy

Scalable Machine Learning

Consumer Behavior

Model

Building Curtailment

Model

Demand Forecasting DR Policy Engine

Trigger

Select

Initiate

Campus

Complex Event Pattern

Detection Observe & Adapt

Observe & Ingest

Information Repository

Mine & Model

Track

SCEPter CEP Engine

OpenPLANET MapReduce

CRAFT Causal Models

Floe Dataflow Engine

Stream Provenance

Portal & Apps


Microsoft Cloud Futures Workshop

smartgrid.usc.edu


Data Social Sci.

Power Engr.

Facility Mgmt.

Univ. Staff

Extern

Cons. Survey O R* - - -

Staff Schedule R* R* - O -

Consumption R R O - -

Weather R R O R W Campus Power Forecast

R W O R -

Shared Data Repository

Social Scientist

Power Engineering Researcher

Facility Managemen

t Student

Staff

Building Sensors

13


17c96af0-02a2-4ed6-8b45-8b436e6ea79d

Filename: usc-upc/eeb/kwh-2011- 09-15.csv

... data...

fa9457ef-66bc-4bc4-88a5-472b5c2ac75b

e3256660-8e27-4d0c-aca0-4ce33a104c06

Repository Service

OWNER

WR

ITER R

EAD

ER U

NA

UTH

OR

IZED



F = encrypt(D, Ksymencr)

SF = sign(F, Kpri

sign)

M = <UUID, SUUID, Kpubsign, Uowner>

SM = sign(M, Kpri

owner)

SB =

encrypt(L, Ksymencr)

broad-encrypt(Ksymencr, Kshared

r-encr)

broad-encrypt(Kprisign, Kshared

w-encr)

Uowner IV-StrongBox

sign(SB, Kpriowner)

File Handle Init. Vector

UUID-1 IV-1

UUID-2 IV-2

…

L =

Data File

Strong Box

Designing a Secure Storage Repository for Sharing Scientific Datasets using Public

Clouds, A. Kumbhare, Y. Simmhan & V. Prasanna , DataCloud, 2011



Cryptonite Data Repository Service

File Manager

Audit Manager

Azure Web Role VM Azure BLOB Store

Azure Table Store Cryptonite Audit Table

WINDOWS AZURE CLOUD PLATFORM DESKTOP CLIENT

Cryptonite Client Library

File Manager

Audit Manager

1a 1b

1c

StrongBox Processor

Crypt Layer

2a

DFile Processor

3 4 Upload Processor

Crypt Layer

Download Processor

5 6a, 12a

8,14

7a, 13a 9a,15a

7b, 13b

9b, 15b

10 Cryptonite DFiles

Cryptonite Strongbox Files

Local Disk Plain Text File

11

2b

6b, 12b

Trusted PKI Server

Operations • GET • PUT • POST



Owner

Cryptonite

FM AM

Request StrongBox

a)Encrypt & Sign File b)Generate UUID c) Add and Sign Security

Metadata Encrypted file with security

metadata

Verify Signature and Store in CSS

Add Audit Entry

Signed Confirmation Containing File Hash

Verify Owner and Store in CSS

Add Audit Entry

Signed Confirmation Containing StrongBox Hash

Upload StrongBox

a)Update StrongBox b)BEr+w(Ke), BEw(Ks) c) Encrypt StrongBox



E S T F R V U

E T

F R V U 3 2 1 S T

U 3 2 1

H H

3 2 1

E T

F R V U 3 2 1 S T

U 3 2 1

H H

3

2

1

U U

R U S T

U

3

2

1

H H

3

2

1

U U

F E

3

2

1

E E

3 2 1

T

3

2

1

T T

V 3 2 1 H

Desktop Client Service BLOB Store Internet

(a)

(b)

(c)

(d)

Cryptonite: A Secure and Performant Data Repository on Public Clouds, A. Kumbhare, Y. Simmhan & V. Prasanna , IEEE CLOUD, 2012



Brandes, U.: A faster algorithm for betweenness centrality. J. Mathematical Sociology, 25(2), 163–177 (2001)



Performance Analysis of Vertex-centric Graph Algorithms on the Azure Cloud Platform, M. Redekopp, Y. Simmhan & V. Prasanna, ParGraph, 2011

Web

Instance Manager

Instance

Workers

Results (Blob Storage)

Requests

Notification

Task Queue

Result Notification Queue

Graphs (Blob Storage)



0

2

4

6

8

10

12

14

16

4 workers 8 workers 16 workers

Spee

du

p R

elat

ive

to 1

VM

Number of BC Worker Instances

Speedup Relative to 1-VM

10,000 vertices

100,000 vertices

0

1

2

3

4

5

6

7

8

1 worker 4 workers 8 workers 16 workersSp

eed

up

Rel

ativ

e to

1 L

oca

l Co

reNumber of BC Worker Instances

Speedup Relative to 1 Local Core

1000 vertices

10,000 vertices

100,000 vertices

For large graphs performance is

scalable over number of workers For large graphs and 16 workers we get about 8x

speedup compared to a single local core



0%

20%

40%

60%

80%

100%

-12% -8% -4% 0% 4% 8% 12%

Task Completion Time Normalized to the Average over all Tasks

Task Size 1000

Task Size 500

Task Size 100

For 105 vertex graph, all tasks finish with

in 12% variation from the average

Set timeout to average task time+12% Analyze performance penalties for different MTBF

More tasks per worker (smaller task size) helps to mitigate performance impact of failures



Pregel: a system for large-scale graph processing, G. Malewicz, et al., SIGMOD 2010

Superstep 2

Superstep 1

Superstep 3

…

32


Socket

Conne-

ctions

Web Instance

Manager Instance

Workers

Results (Blob Storage)

Requests

Notification

Step Queue

Barrier Queue

Graphs (Blob Storage)

1

1 1

1

Superstep 2

Workers Superstep 1

Superstep 3

…

Azure

Desktop

Pregel.NET Worker Role Implementation on Azure



GetThreads (1 per remote worker connection)

ComputeThreads

Local Vertices Message Queues for next Superstep

Local Vertices Message Queues for current Superstep

Put Buffers (Remote Messages)

PutThreads (1 per remote worker connection)



0

200000

400000

600000

800000

1000000

1200000

1400000

1600000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19N

um

be

r o

f Me

ssag

es

Ge

ne

rate

d p

er

BFS

SuperStep



0

20000

40000

60000

80000

100000

120000

140000

1 2 3 4 5 6 7 8 9 10 11 12 13

0

20000

40000

60000

80000

100000

120000

140000

160000

180000

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Message traffic from 4 swaths of BFS launched at an interval of 2 supersteps

Cumulative message traffic for the 4 swaths



(TBA)



This material is based upon work supported by the National Science Foundation and Microsoft under Award Number CCF-1048311 of the Computing in the Cloud initiative, and by the Department of Energy under Award Number DEOE0000192.

The views and opinions expressed herein do not necessarily state or reflect those of the US Government or any agency thereof.


University of Southern California · Data Social Sci. Power Engr. Facility Mgmt. Univ. Staff Extern...

Documents

Transcript of University of Southern California · Data Social Sci. Power Engr. Facility Mgmt. Univ. Staff Extern...