Cellular Networks and Mobile Computing COMS 6998-8, Spring 2012
description
Transcript of Cellular Networks and Mobile Computing COMS 6998-8, Spring 2012
Cellular Networks and Mobile ComputingCOMS 6998-8, Spring 2012
Instructor: Li Erran Li ([email protected])
http://www.cs.columbia.edu/~coms6998-8/3/5/2012: Mobile Cloud Platform Services
Cellular Networks and Mobile Computing (COMS 6998-8)
2
Announcements
• iOS assignment 2 and Android assignment 1 are due March 19th
• More mobile security papers in syllabus – Please email me if you want to present one of
them instead of the originally assigned• Windows Phones are available for project use– On loan from Microsoft, please take good care of
them
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
3
Mobile Cloud Platform Services
• Syncing and storage service (iCloud)• Proxy service (Kindle Split Browser)• Speech to text/text to speech service• Natural language processing service (open Siri
API for 3rd party applications in the future)• Push notification service• Track service (supporting location based
services)
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
4
Outline
• Speech to text service demo• Push notification service– Apple push notification service– Google C2DM (not covered in this lecture)– Thialfi: reliable push notification system
• Track service
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
5
Apple Push Notification Architecture Overview
• iOS device maintains a persistent TCP connection to a Apple Push Notification Server(APNS)
3/5/12
A push notification from a provider to a client application
Multi-providers to multiple devices
Cellular Networks and Mobile Computing (COMS 6998-8)
6
Apple Push Notification Architecture Overview (Cont’d)
• What if devices uninstalled the app?– Feedback service
• Providers poll to obtain list of device tokens for their applications
• What if devices are offline?– QoS service
• QoS stores the notification• It retains only the last notification received from a provider• When the offline device reconnects, the QoS forwards the
stored notification to the device• QoS retains a notification for a limited period before deleting it
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
7
Push Notification
• Push notification– Delivery is best effort and is not guaranteed– Max size is 256 bytes– Providers compose a JSON dictionary object • This dictionary must contain another dictionary
identified by the key aps– Action:• An alert message to display to the user• A number to badge the application icon with• A sound to play
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
8
Device Token• Device token is analogous to a phone number
– Contains information that enables APNs to locate the device– Client app needs to provide the token to its provider– Device token should be requested and passed to providers every time your application
launches
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
9
Apple Push Notification Programming Example
• Provisioning: https://developer.apple.com/ios/manage/provisioningprofiles/howto.action– Generate Certification Signing Request (CSR) using Keychain
Access• Save to disk: PushChat.certSigningRequest• Export the private key as “PushChatKey.p12” and enter a passphrase
– Make an App ID in iOS Provisioning Portal• Check the Enable for Apple Push Notification service box• click on the Configure button for the Development Push SSL Certificate• click Download to get the certificate – it is named
“aps_development.cer”3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
10
Apple Push Notification Programming Example (Cont’d)
• Client code1. - (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary
*)launchOptions2. {3. // Let the device know we want to receive push notifications4. [[UIApplication sharedApplication] registerForRemoteNotificationTypes:5. (UIRemoteNotificationTypeBadge | UIRemoteNotificationTypeSound |
UIRemoteNotificationTypeAlert)];6. 7. return YES;8. }
9. - (void)application:(UIApplication*)application didReceiveRemoteNotification:(NSDictionary*)userInfo10. {//userInfo contains the notification11. NSLog(@"Received notification: %@", userInfo);12. }
13. - (void)application:(UIApplication*)application didRegisterForRemoteNotificationsWithDeviceToken:(NSData*)deviceToken
14. {15. NSLog(@"My token is: %@", deviceToken);16. }
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
11
Apple Push Notification Programming Example (Cont’d)
• Server code1. $devicetoken ='f05571e4be60a4e11524d76e4366862128f430522fb470c46fc6810fffb07af7’;2. // Put your private key's passphrase here:3. $passphrase = 'PushChat';4. // Put your alert message here:5. $message = 'Erran: my first push notification!';
6. $ctx = stream_context_create();7. Stream_context_set_option($ctx, 'ssl', 'local_cert', 'ck.pem');8. stream_context_set_option($ctx, 'ssl', 'passphrase', $passphrase);
9. // Open a connection to the APNS server10. $fp = stream_socket_client(11. 'ssl://gateway.sandbox.push.apple.com:2195', $err,12. $errstr, 60, STREAM_CLIENT_CONNECT|STREAM_CLIENT_PERSISTENT, $ctx);
13. if (!$fp)14. exit("Failed to connect: $err $errstr" . PHP_EOL);
15. echo 'Connected to APNS' . PHP_EOL;
16. // Create the payload body17. $body['aps'] = array(18. 'alert' => $message,19. 'sound' => 'default'20. );
21. // Encode the payload as JSON22. $payload = json_encode($body);
23. // Build the binary notification24. $msg = chr(0) . pack('n', 32) . pack('H*', $deviceToken) . pack('n', strlen($payload)) . $payload;
25. // Send it to the server26. $result = fwrite($fp, $msg, strlen($msg));
27. if (!$result)28. echo 'Message not delivered' . PHP_EOL;29. else30. echo 'Message successfully delivered' . PHP_EOL;
31. // Close the connection to the server32. fclose($fp);
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
12
Push Notification Programming Example (Cont’d)
• Demo
3/5/12
Thialfi: A Client Notification Servicefor Internet-Scale Applications
Atul Adya, Gregory Cooper, Daniel Myers, Michael Piatek
Google Seattle
Cellular Networks and Mobile Computing (COMS 6998-8)
14
A Case for NotificationsProblem: Ensuring cached data is fresh across
users and devices
Courtesy: Adya et al.3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
15
Common Application Patterns
• Clients poll to detect changes– Simple and reliable, but slow and inefficient
• Push updates to the client– Fast but complex– Add backup polling to get reliability– Tail latencies can be high: masks bugs– Application-specific protocol
sacrifice reliability
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
16
Solution: Thialfi
• Scalable: tracks millions of clients and objects• Fast: notifies clients in less than a second• Reliable: even when entire data centers fail• Easy to use: deployed in Chrome Sync, Contacts,
Google Plus
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
17
Thialfi Outline
• Thialfi’s abstraction: reliable signaling
• Delivering notifications in the common case
• Detecting and recovering from failures
• Evaluation and experience
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
18
Thialfi Overview
Thialfi client library
Register X Notify X
ClientData center
X: C1, C2
Client C1 Client C2
Thialfi Service
Update XRegister
Register
Update XApplication backend
Notify X Notify X
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
19
Thialfi Abstraction
• Objects have unique IDs and version numbers, monotonically increasing on every update
• Delivery guarantee– Registered clients learn latest version number– Reliable signal only: cached object ID X at version Y
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
20
Why Signal, Not Data?
• Developers want reliable, in-order data delivery
• Adds complexity to Thialfi and application, e.g.,– Hard state, arbitrary buffering– Offline applications flooded with data on wakeup
• For most applications, reliable signal is enough– Invoke polling path on signal: simplifies integration
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
21
API Without Failure Recovery
Thialfi Service Publish(objectId, version)
ClientLibrary
Register(objectId)Unregister(objectId)
Notify(objectId, version)
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
22
Thialfi Outline
• Thialfi’s abstraction: reliable signaling
• Delivering notifications in the common case
• Detecting and recovering from failures
• Evaluation and experience
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
23
Architecture
ClientBigtable
• Matcher: Object ID registered clients, version• Registrar: Client ID registered objects, notifications
Client
Registrar
MatcherObjectBigtable
Data center
Notifications Application Backend
Registrations, notifications,acknowledgments
Client library
3/5/12 Courtesy: Adya et al.
• Each server handles a contiguous range of keys, • Each server maintains an in-memory version• Bigtable: log structured, fast write
Cellular Networks and Mobile Computing (COMS 6998-8)
24
C1: x, v7C2: x, v7C1: x, v5C2: x,
x: v5; C1, C2x: v7; C1, C2x: v7; C1, C2
x
Life of a Notification
ClientBigtable
C1: x, v7
C2: x, v7
Notify: x, v7
Client C2
MatcherObjectBigtable
Data center
Publish(x, v7)x, v7
Ack: x, v7
Registrar
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
25
Thialfi Outline
• Thialfi’s abstraction: reliable signaling
• Delivering notifications in the common case
• Detecting and recovering from failures
• Evaluation and experience
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
26
Data center lossServer state loss/schema migrationPartial storage unavailability
Possible Failures
ClientLibrary
ClientBigtable Registrar
MatcherObjectBigtable
ClientBigtable Registrar
MatcherObjectBigtable
. . .
Data center 1 Data center nThialfi Service
ClientStore
Client restartClient state loss
Publish Feed
Network failures
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
27
Failures Addressed by Thialfi
• Client restart• Client state loss• Network failures• Partial storage unavailability• Server state loss / schema migration• Publish feed loss• Data center outage
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
28
Main Principle: No Hard State
• Thialfi remains correct even if all state is lost– All registrations– All object versions
• Detect and reconstruct after failures using:– ReissueRegistrations() client event– Registration Sync Protocol– NotifyUnknown() client event
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
29
Recovering Client Registrations
Registrar
MatcherObjectBigtable
x
y
x yReissueRegistrations()
Register(x); Register(y)
ReissueRegistrations: Not a burden for applications– Application stores objects in its cache, or – Object list is implicit, e.g., bookmarks for user X
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
30
Registrar
MatcherObjectBigtable
Register: x, y
Syncing Client Registrations
x
y
Hash(x, y)x y
• Goal: Keep client-registrar registration state in sync• Every message contains hash of registered objects• Registrar initiates protocol when detects out-of-sync• Allows simpler reasoning of registration state
Reg syncHash(x, y)
3/5/12 Courtesy: Adya et al.
Merkle tree for syncing large number of objects
Cellular Networks and Mobile Computing (COMS 6998-8)
31
Recovering From Lost Versions
• Versions may be lost, e.g. schema migration
• Refreshing from backend requires tight coupling
• Inform client with NotifyUnknown(objectId) – Client must refresh, regardless of its current state
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
32
Thialfi Outline
• Thialfi’s abstraction: reliable signaling
• Delivering notifications in the common case
• Detecting and recovering from failures
• Evaluation and experience
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
33
Notification Latency Breakdown
Notification latency (ms)0
100
200
300
Matcher to Registrar RPC (Batched)
Matcher Bigtable Read
Matcher Bigtable Write (Batched)
Bridge to Matcher RPC (Batched)
App Backend to Bridge
Batching accounts for significant fraction of latency3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
34
Thialfi Usage by ApplicationsApplication Language Network
ChannelClient Lines of Code(Semi-colons)
Chrome Sync C++ XMPP 535Contacts JavaScript Hanging GET 40
Google+ JavaScript Hanging GET 80Android Application Java C2DM +
Standard GET300
Google BlackBerry Java RPC 340
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
35
Some Lessons Learned
• Add complexity at the server, not the client– Deploy at server: minutes. Upgrade clients: years+
• Asynchronous events, not callbacks– Spontaneous events occur: need to handle them
• Initial applications have few objects per client– Earlier use of polling forces such a model
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
36
Thialfi Summary
• Fast, scalable notification service• Reliable even when data centers fail• Two key ideas simplify failure handling– Deliver a reliable signal, not data– No hard state: reconstruct after failure
• Deployed in Chrome Sync, Contacts, Google+
3/5/12 Courtesy: Adya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
37
Outline
• Speech to text service demo• Push notification service– Apple push notification service– Google C2DM(not covered in this lecture)– Thialfi: reliable push notification system
• Track service
3/5/12
Cellular Networks and Mobile Computing (COMS 6998-8)
38
Location-Based Applications• Many phones already have the ability to
determine their own location– GPS, cell tower triangulation, or proximity to WiFi
hotspots
• Many mobile applications use location information
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
39
TrackTime-ordered sequence of location readings
Latitude: 37.4013Longitude: -122.0730Time: 07/08/10 08:46:45.125
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
40
Application: Personalized Driving Directions
Goal: Find directions to new gymTake US-101 North
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
41
A Taxonomy of ApplicationsPersonal Social
Current location
Driving directions, Nearby restaurants
Friend finder, Crowd scenes
Past locations
Personal travel journal, Geocoded photos
Post-it notes, Recommendations
Tracks Personalized Driving Directions, Track-Based Search
Ride sharing, Discovery, Urban sensing
Class of applications enabled by StarTrack
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
42
StarTrack System
ST Client
Insertion Application
Location Manager
• Retrieval• Manipulation• Comparison …
Application
ST Client
• InsertionST Server
ST Server
ST Server
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
43
System Challenges
1. Handling error-prone tracks
2. Flexible programming interface
3. Efficient implementation of operations on tracks
4. Scalability and fault tolerance3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
44
Challenges of Using Raw Tracks
Advantages of Canonicalization:– More efficient retrieval and comparison operations– Enables StarTrack to maintain a list of non-duplicate tracks
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
45
StarTrack API
Track Collections (TC): Abstract grouping of tracks– Programming Convenience– Implementation Efficiency• Prevent unnecessary client-server message exchanges− Enable delayed evaluation− Enable caching and use of in-memory data structures
Pre-filter tracks Manipulate tracks Fetch tracks
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
46
StarTrack API: Track Collections
TC JoinTrackCollections (TC tCs[], bool removeDuplicates) TC SortTracks (TC tC, SortAttribute attr) TC TakeTracks(TC tC, int count) TC GetSimilarTracks (TC tC, Track refTrack, float simThreshold) TC GetPassByTracks (TC tC, Area[] areas) TC GetCommonSegments(TC tC, float freqThreshold)
Track[] GetTracks (TC tC, int start, int count)
Manipulation
Retrieval
Creation TC MakeCollection(GroupCriteria criteria, bool removeDuplicates)
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
47
API Usage: Ride-Sharing Application// get user’s most popular track in the morning
TC myTC = MakeCollection(“name = Maya”, [0800 1000], true);TC myPopTC = SortTracks(myTC, FREQ);Track track = GetTracks(myPopTC, 0, 1);
// find tracks of all fellow employeesTC msTC = MakeCollection(“name.Employer = MS”, [0800 1000], true);
// pick tracks from the community most similar to user’s popular trackTC similarTC = GetSimilarTracks(msTC, track, 0.8);Track[] similarTracks = GetTracks(similarTC, 0, 20);
// Verify if each track is frequently traveled by its respective ownerUser[] result = FindOwnersOfFrequentTracks(similarTracks);
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
48
Efficient Implementation of Operations
• StarTrack exploits redundancy in tracks for efficient retrieval from database– Set of non-duplicate tracks per user– Separate table of unique coordinates
• StarTrack builds specialized in-memory data-structures to accelerate the evaluation of some operations– Quad-Trees for geographic range searches– Track Trees for similarity searches
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
49
S1-4
S5 S6-7
Track Similarity
Tracks A, B
s1
s2
s3
s4
s5
Track D
s8
s9
Track C
s6 s7
Limited database support for computing track similarity
SIM (A ,C )=¿S1−4∨ ¿|S1−4|+|S5|+¿S6−7∨¿¿
¿
SIM (A , B)=¿S1−5∨ ¿|S1−5|
¿
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
50
Track Tree
s1 s2 s3 s4 s5 s6 s7 s8 s9
S1-2 S6-7 S8-9
S1-3
S1-4
S1-5Tracks A, B
s1
s2
s3
s4
s5
Track Ds8
s9
Track Cs6 s7 1) Create leaf nodes for all
segments
2) Merge nodes based on # of tracks that go through adjacent segments
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
51
Evaluation• Performance of our Track Tree approach• Performance of 2 sample applications– Ride-sharing– Personalized Driving Directions
• Configuration– Synthetically generated tracks– Up to 9 StarTrack Servers + 3 Database Servers– Server Configuration:
• 2.6 GHz AMD Opteron Quad-Core Processors• 16 GB RAM
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
52
Evaluation: Track Tree• Evaluation of GetSimilarTracks
• Alternative approaches:– Database filteringPre-filter tracks that intersect ref track at database– In-memory filteringPre-filter tracks that intersect ref track in memory – In-memory brute forceCompute similarity between each track and ref track in memory
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
53
Get Similar Tracks – Query Time
0 10 20 30 40 50 60 70 80 90 1000.1
1
10
100
1000
10000
Number of tracks (thousands)
Que
ry T
ime
(ms)
Track Tree
In-Memory Filtering
In-Memory Brute Force
Database Filtering
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
54
Track Tree Construction Costs
0 20 40 60 80 1000
30
60
90
120
150
180
Number of Tracks (thousands)
Time (s)
Memory (MB)
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
55
Performance of ApplicationsRide Sharing
- Track Collection on multiple users- Calls to GetSimilarTracks- 30 requests/s at about 170 ms
Personalized Driving Directions
- Track Collection for single user at a time- Calls to GetCommonSegments- 30 requests/s at about 100 ms (uncached)- 250 requests/s at about 55 ms (cached)
0 10 20 30 400
100200300400500600
Request Rate (per second)
Resp
onse
Tim
e (m
s)
150 175 200 225 2500
20406080
100120
Request Rate (per second)
Resp
onse
Tim
e (m
s)
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
56
Related Work• Management of tracks has been studied by the database
community– Storage of tracks as 3-dimensional objects – Specialized indexing schemes (Quad-Trees, R-Trees, etc.)
• CarTel Project (MIT) – Provides an infrastructure for collecting traces, relying on a relational database using spatial queries
• Access and sharing of data in StarTrack is similar to that provided by social networks, where users’ data is shared by applications; Similar access control policies could be employed to ensure privacy in StarTrack.
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
57
Summary
• StarTrack is a scalable service designed to manage tracks and facilitate the construction of track-based applications
• Important Design Features– Canonicalization of Tracks– API based on Track Collections– Use of Novel Data Structures
3/5/12 Courtesy: Maya et al.
Cellular Networks and Mobile Computing (COMS 6998-8)
58
Questions?
3/5/12