Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source...
Transcript of Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source...
![Page 1: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/1.jpg)
Big Data and Arabic Sentiment Analysis
Presented by:Dr Jassim HajiDirector of IT, Gulf Air
![Page 2: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/2.jpg)
gulfair.com
� Billions of users in social media/networks
� Never in history had we this amount of opinionated data
� Word of mouth can make or break your business
Social Media Explosion
28-May-2013 2
![Page 3: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/3.jpg)
gulfair.com
� Crucial to monitor your customer feedback and public perception
� When you know how your customer feel & think:
� Take corrective/proactive action if the customer is upset
� Exploit satisfied customers and good publicity
Social Media Importance to Business
28-May-2013 3
![Page 4: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/4.jpg)
gulfair.com
� You can monitor your competitors
� You can track market situation
� The tool to achieve this is sentiment analysis and opinion mining
Social Media Importance to Business
28-May-2013 4
![Page 5: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/5.jpg)
gulfair.com
� Big Data is playing a critical role in social media monitoring
� Automates the feed, makes it structure for processing
� Scalable to handle huge amount of data processing
� Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)
� Gulf Air developed the skills required for this implementation internally without relying on external parties
Big Data & Social Media Monitoring
28-May-2013 5
![Page 6: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/6.jpg)
gulfair.com
� Determining feelings from text: (attitudes, opinions, emotions)
� Complex field, subset within Natural Language Processing
� With social media, sentiment analysis has gained importance
� Very subjective, depends on topics/language/ region/context, no single solution to fit all
Sentiment Analysis/Opinion Mining
28-May-2013 6
![Page 7: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/7.jpg)
gulfair.com
� Arabic: most prevalent language in the Middle East and North Africa
� 422 million native speakers
� Rich but complex language andgrammar, with formal, colloquial and dialect varieties.
The Arabic Language
28-May-2013 7
![Page 8: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/8.jpg)
gulfair.com
� More than 80% of posts in social mediaabout Gulf Air are in Arabic
� Very few providers offer Arabic Sentiment Analysis, with limitations and inaccuracies
� Available solutions don’t satisfy Gulf Air’s needs
Arabic Social Media
28-May-2013 8
![Page 9: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/9.jpg)
gulfair.com
� Gulf Air adopted Lexicon-based approach
� A language is comprised of:
� Lexicon (list of words)
� Rules for combining words to form meaning
� Lexicon and rules between words were defined to determine meaning and sentiment
Arabic Sentiment Analysis Approach
28-May-2013 9
![Page 10: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/10.jpg)
gulfair.com
� Formal Arabic
� Local dialects of the region
� Informal and colloquial words and phrases frequently used
� English Translation
� Arabic tweets (Twitter) are currently being analyzed (short and concise text)
Scope of Arabic Sentiment Analysis
28-May-2013 10
![Page 11: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/11.jpg)
gulfair.com
� Gulf Air has rebuilt the Arabic language in the sentiment analysis logic
� Just like a child learning a language, this logic has to learn and mature to increase accuracy of results
Arabic Sentiment Analysis Maturation
28-May-2013 11
![Page 12: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/12.jpg)
gulfair.com
� Test and learn approach used:
� Continuously adding new words
� Improving/changing the rules
� Results are regularly reviewed to find opportunities for improvements
� Accuracy achieved as of now is 84%
Arabic Sentiment Analysis Maturation
28-May-2013 12
![Page 13: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/13.jpg)
gulfair.com
� Timely reporting of hot topics and areas of controversy about Gulf Air
� Public/customers’ sentiment reporting
� Quick corrective action on negative situations to increase customer satisfaction
� Trend analysis of sentiments and topics over time
� Quick reaction to competition and market/demand shifts
Business Benefits to Gulf Air
28-May-2013 13
![Page 14: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/14.jpg)
gulfair.com
� Extend the Sentiment Analysis Logic to handle the English language
� Expand to other forms of social media, like Facebook
� Cover newspapers’ articles and reviews in travel sites (complex text structure)
Gulf Air’s Future Plans
28-May-2013 14
![Page 15: Presented by: Dr Jassim Haji Director of IT, Gulf Air · Gulf Air is using Cloudera’s open-source software on virtual servers (leveraging existing Gulf Air’s cloud environment)](https://reader030.fdocuments.in/reader030/viewer/2022041109/5f0e12587e708231d43d79d6/html5/thumbnails/15.jpg)
Q & A سين
جيم
Gulf Air is the 1st airline and one of few organizations to develop and use Arabic Sentiment Analysis with Big Data