Urban Airship and Android Integration for Push Notification and In-App Notification
Using CoreML to Push the Limits of your App
-
Upload
mugunth-kumar -
Category
Software
-
view
41 -
download
0
Transcript of Using CoreML to Push the Limits of your App
Core ML and your AppHow to use Core ML to push the limits
Mugunth Kumar, Itty Bitty Apps, Melbourne
Agenda
This talk explains how aspiring app developers/companies can take full advantage of Core ML to push their apps to the next level.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Agenda
This talk explains how aspiring app developers/companies can take full advantage of Core ML to push their apps to the next level.
I'll introduce you to - The fundamentals of Core ML - What is possible with machine learning in iOS 11 - What is not possible with machine learning in iOS 11
Mugunth Kumar, Itty Bitty Apps, Melbourne
Agenda
This talk explains how aspiring app developers/companies can take full advantage of Core ML to push their apps to the next level.
I'll introduce you to - The fundamentals of Core ML - What is possible with machine learning in iOS 11 - What is not possible with machine learning in iOS 11
Lastly, What could potentially be possible in a future release of iOS (12 and above
!
)
Mugunth Kumar, Itty Bitty Apps, Melbourne
Agenda
Mugunth Kumar, Itty Bitty Apps, Melbourne
Agenda
» What is Core ML
Mugunth Kumar, Itty Bitty Apps, Melbourne
Agenda
» What is Core ML
» Machine Learning in iOS 11
» Vision
» Natural Language Processing
Mugunth Kumar, Itty Bitty Apps, Melbourne
Agenda
» What is Core ML
» Machine Learning in iOS 11
» Vision
» Natural Language Processing
» Features not yet available
» Retraining models (iOS 12
!
)
Mugunth Kumar, Itty Bitty Apps, Melbourne
Agenda
» What is Core ML
» Machine Learning in iOS 11
» Vision
» Natural Language Processing
» Features not yet available
» Retraining models (iOS 12
!
)
» Questions
Mugunth Kumar, Itty Bitty Apps, Melbourne
Core ML
What is Core ML
Mugunth Kumar, Itty Bitty Apps, Melbourne
Core ML
What is Core ML
» Core ML is a framework newly introduced in iOS 11 SDK
Mugunth Kumar, Itty Bitty Apps, Melbourne
Core ML
What is Core ML
» Core ML is a framework newly introduced in iOS 11 SDK
» Once considered complicated
Mugunth Kumar, Itty Bitty Apps, Melbourne
Core ML
What is Core ML
» Core ML is a framework newly introduced in iOS 11 SDK
» Once considered complicated
» Once considered CPU intensive
Mugunth Kumar, Itty Bitty Apps, Melbourne
Core ML
What is Core ML
» Core ML is a framework newly introduced in iOS 11 SDK
» Once considered complicated
» Once considered CPU intensive
» Once considered impossible to be done on the client
Mugunth Kumar, Itty Bitty Apps, Melbourne
Core ML
What is Core ML
» Core ML is a framework newly introduced in iOS 11 SDK
» Once considered complicated
» Once considered CPU intensive
» Once considered impossible to be done on the client
» Made easy in iOS 11
!
Mugunth Kumar, Itty Bitty Apps, Melbourne
Features available today in iOS 11
Mugunth Kumar, Itty Bitty Apps, Melbourne
Features available today in iOS 11
» Vision
» Object Tagging and Recognition
Mugunth Kumar, Itty Bitty Apps, Melbourne
Features available today in iOS 11
» Vision
» Object Tagging and Recognition
» Natural Language Processing
» Sentiment Analysis
» Language Tagging
» Parts of Speech Analysis
Mugunth Kumar, Itty Bitty Apps, Melbourne
This talk
This talk is exclusively on NLP and text processing for two reasons.
Mugunth Kumar, Itty Bitty Apps, Melbourne
This talk
This talk is exclusively on NLP and text processing for two reasons.
» Plenty of libraries and blogs already exist on vision and Object Tagging.
Mugunth Kumar, Itty Bitty Apps, Melbourne
This talk
This talk is exclusively on NLP and text processing for two reasons.
» Plenty of libraries and blogs already exist on vision and Object Tagging.
» Vision tagging model files are large (30MB to 200+ MB) and the "side" benefit you get using Core ML gets defeated.
Mugunth Kumar, Itty Bitty Apps, Melbourne
This talk
This talk is exclusively on NLP and text processing for two reasons.
» Plenty of libraries and blogs already exist on vision and Object Tagging.
» Vision tagging model files are large (30MB to 200+ MB) and the "side" benefit you get using Core ML gets defeated.
» As of iOS 11, you can't on the fly download a Core ML model on the fly like on-demand resource.
Mugunth Kumar, Itty Bitty Apps, Melbourne
This talk
With Core ML and NLP, your app could do
Mugunth Kumar, Itty Bitty Apps, Melbourne
This talk
With Core ML and NLP, your app could do
» Sentiment Analysis
Mugunth Kumar, Itty Bitty Apps, Melbourne
This talk
With Core ML and NLP, your app could do
» Sentiment Analysis
» Parts of Speech Analysis/Language Tagging
Mugunth Kumar, Itty Bitty Apps, Melbourne
This talk
With Core ML and NLP, your app could do
» Sentiment Analysis
» Parts of Speech Analysis/Language Tagging
» Combining Both
Mugunth Kumar, Itty Bitty Apps, Melbourne
Who can be benefitted by Sentiment Analysis
Mugunth Kumar, Itty Bitty Apps, Melbourne
Who can be benefitted by Sentiment Analysis
» Apps that show reviews
Mugunth Kumar, Itty Bitty Apps, Melbourne
Who can be benefitted by Sentiment Analysis
» Apps that show reviews
» Social media account classifiers
Mugunth Kumar, Itty Bitty Apps, Melbourne
Who can be benefitted by Sentiment Analysis
» Apps that show reviews
» Social media account classifiers
» Pretty much any app that deals with lots of text from e-book readers to magazine apps
Mugunth Kumar, Itty Bitty Apps, Melbourne
Getting Started
You need a Core ML Model. You can download from
Or you can convert using coremltools
Mugunth Kumar, Itty Bitty Apps, Melbourne
Getting Started
You need a Core ML Model. You can download from
» https://developer.apple.com/machine-learning/
Or you can convert using coremltools
Mugunth Kumar, Itty Bitty Apps, Melbourne
Getting Started
You need a Core ML Model. You can download from
» https://developer.apple.com/machine-learning/
» https://github.com/likedan/Awesome-CoreML-Models
Or you can convert using coremltools
Mugunth Kumar, Itty Bitty Apps, Melbourne
Getting Started
You need a Core ML Model. You can download from
» https://developer.apple.com/machine-learning/
» https://github.com/likedan/Awesome-CoreML-Models
Or you can convert using coremltools
» https://pypi.python.org/pypi/coremltools
Mugunth Kumar, Itty Bitty Apps, Melbourne
Sentiment Analysis Demo
Mugunth Kumar, Itty Bitty Apps, Melbourne
Sentiment Analysis Demo
» For the demo, I'll be using one such model from the Awesome-CoreML-Models repo.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Sentiment Analysis Demo
» For the demo, I'll be using one such model from the Awesome-CoreML-Models repo.
» Specifically SentimentPolarity.mlmodel by Vadym Markov.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Sentiment Analysis Demo
» For the demo, I'll be using one such model from the Awesome-CoreML-Models repo.
» Specifically SentimentPolarity.mlmodel by Vadym Markov.
» It's a very light weight model weighing in at 275Kb.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Sentiment Analysis Demo
Mugunth Kumar, Itty Bitty Apps, Melbourne
The Code!
func predictSentiment(from text: String) -> Sentiment { do { var inputFeatures = features(from: text) guard inputFeatures.count > 1 else { throw Error.featuresMissing }
let output = try model.prediction(input: inputFeatures)
let positiveProb = output.classProbability["Pos"]! let negativeProb = output.classProbability["Neg"]!
return Sentiment(positiveProbablity: positiveProb, negativeProbablity: negativeProb) } catch { preconditionFailure("Error occurred. Model file might be corrupted.") } }
Mugunth Kumar, Itty Bitty Apps, Melbourne
More code!
func features(from text: String) -> [String: Double] { var wordCounts = [String: Double]()
tagger.string = text let range = NSRange(location: 0, length: text.utf16.count)
tagger.enumerateTags(in: range, scheme: .nameType, options: options) { _, tokenRange, _, _ in let token = (text as NSString).substring(with: tokenRange).lowercased()
guard token.count >= 2 else { return }
if let value = wordCounts[token] { wordCounts[token] = value + 1.0 } else { wordCounts[token] = 1.0 } }
return wordCounts }
Mugunth Kumar, Itty Bitty Apps, Melbourne
Performance
Mugunth Kumar, Itty Bitty Apps, Melbourne
Performance
» Reasonably fast.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Performance
» Reasonably fast.
» With more text, performance can take a hit
Mugunth Kumar, Itty Bitty Apps, Melbourne
Performance
» Reasonably fast.
» With more text, performance can take a hit
» As long as you avoid the main thread, you should be good to go
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
» Parts of speech analysis (and Language Tagging) was introduced since iOS 5.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
» Parts of speech analysis (and Language Tagging) was introduced since iOS 5.
» All those calendar apps you saw were using this feature to convert your natural language input into a calendar or a reminder event.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
» Parts of speech analysis (and Language Tagging) was introduced since iOS 5.
» All those calendar apps you saw were using this feature to convert your natural language input into a calendar or a reminder event.
» You don't need a model file to do this. Everything you need is built into the SDK.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
» Parts of speech analysis (and Language Tagging) was introduced since iOS 5.
» All those calendar apps you saw were using this feature to convert your natural language input into a calendar or a reminder event.
» You don't need a model file to do this. Everything you need is built into the SDK.
» So why is this section in this talk?
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
» In iOS 11, Parts of Speech Analysis was completely reimplemented on top of Core ML framework
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
» In iOS 11, Parts of Speech Analysis was completely reimplemented on top of Core ML framework
» Language Tagging wasn't very accurate with some Eastern European and Asian languages.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis
» In iOS 11, Parts of Speech Analysis was completely reimplemented on top of Core ML framework
» Language Tagging wasn't very accurate with some Eastern European and Asian languages.
» This is mostly fixed with Core ML in iOS 11
Mugunth Kumar, Itty Bitty Apps, Melbourne
Lets get our hands dirty!
func tags(from text: String) -> [[String: String]] { var tagsToReturn = [[String: String]]()
tagger.string = text let range = NSRange(location: 0, length: text.utf16.count)
// Tokenize and count the sentence tagger.enumerateTags(in: range, scheme: .lexicalClass, options: options) { tag, tokenRange, _, _ in let token = (text as NSString).substring(with: tokenRange).lowercased() let tag = (tag?.rawValue ?? "") tagsToReturn.append([token : tag]) } return tagsToReturn }
Mugunth Kumar, Itty Bitty Apps, Melbourne
Parts of Speech Analysis Demo
Mugunth Kumar, Itty Bitty Apps, Melbourne
Mixing them all up
There are plenty of research articles that argues sentiment analysis is better done by emphasising adverbs and adjectives.
Words like beautiful, useless, hopeless, gigantic convey more "sentiment" information compared to objects in a sentence like "Australia", "iPhone" or subjects like "I", "You".
Ex: I can clearly say, Australia is one of the most beautiful countries on the planet.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Lets get our hands dirty!
func featuresUsingTaggingAlgorithm(from text: String) -> [String: Double] {
let parsedTags = tags(from: text) var wordCounts = [String: Double]()
for tag in parsedTags { let (token, tag) = (tag.keys.first!, tag.values.first!)
var valueToAdd = Double(1.0)
if tag == "Adverb" { valueToAdd = Double(30.0) } if tag == "Adjective" { valueToAdd = Double(35.0) }
if let value = wordCounts[token] { wordCounts[token] = value + valueToAdd } else { wordCounts[token] = valueToAdd }
} return wordCounts }
Mugunth Kumar, Itty Bitty Apps, Melbourne
MIX UP ALGORITHM DEMO
Mugunth Kumar, Itty Bitty Apps, Melbourne
Features not yet available
Mugunth Kumar, Itty Bitty Apps, Melbourne
Features not yet available
» Retraining models is not possible (iOS 12 !
)
» Face Recognition can't be performed as of now
» Face Recognition first involves detection of faces (available since iOS 6)
» Pass these as training data to a classifier
» Face Recognition is however available for the native Photos app. So it is only a matter of time.
Mugunth Kumar, Itty Bitty Apps, Melbourne
Features not yet available
SARCASM DETECTION!
Mugunth Kumar, Itty Bitty Apps, Melbourne
What you should know by now
Mugunth Kumar, Itty Bitty Apps, Melbourne
What you should know by now» Apple has very elegantly abstracted out the inner
workings that, most of Core ML is fairly straight forward (codewise: under 10 lines) if you know the foundations.
Mugunth Kumar, Itty Bitty Apps, Melbourne
What you should know by now» Apple has very elegantly abstracted out the inner
workings that, most of Core ML is fairly straight forward (codewise: under 10 lines) if you know the foundations.
» Understanding the foundations will let you apply Core ML in your app easily.
Mugunth Kumar, Itty Bitty Apps, Melbourne
What you should know by now» Apple has very elegantly abstracted out the inner
workings that, most of Core ML is fairly straight forward (codewise: under 10 lines) if you know the foundations.
» Understanding the foundations will let you apply Core ML in your app easily.
» Using CoreML innovatively will help make your app stand out from the crowd.
Mugunth Kumar, Itty Bitty Apps, Melbourne
That's ME!
I'm a
Mugunth Kumar, Itty Bitty Apps, Melbourne
That's ME!
I'm a
» Developer
Mugunth Kumar, Itty Bitty Apps, Melbourne
That's ME!
I'm a
» Developer
» Author of iOS Programming Pushing the Limits
Mugunth Kumar, Itty Bitty Apps, Melbourne
That's ME!
I'm a
» Developer
» Author of iOS Programming Pushing the Limits
» Currently baking some cool apps @ Itty Bitty Apps, Melbourne
Mugunth Kumar, Itty Bitty Apps, Melbourne
Questions?Mugunth Kumar, Itty Bitty Apps, Melbourne