From Big Data to AI
-
Upload
maloy-manna-pmp -
Category
Technology
-
view
190 -
download
0
Transcript of From Big Data to AI
FromBigDatatoAIBuildingMachineLearningApplications
MaloyManna
Abstract
• ThenewestbuzzwordafterBigDataisAI.FromGooglesearchtoFacebookmessengerbots,AIisalsoeverywhere.
• Machinelearninghasgonemainstream.OrganizationsaretryingtobuildcompetitiveadvantagewithAIandBigData.
• But,whatdoesittaketobuildMachineLearningapplications?BeyondtheunicorndatascientistsandPhDs,howdoyoubuildonyourbigdataarchitectureandapplyMachineLearningtowhatyoudo?
• Thistalkwilldiscusstechnicaloptionstoimplementmachinelearningonbigdataarchitecturesandhowtomoveforward.
Yourpresenter
MaloyMannaDataengineeringPM,AXADataInnovationLab
Ibuilddataproducts&services.- Solutionarchitecture|Design|Lean-Agile
linkedin.com/in/maloymaloymanna.github.io
Aboutyou
Howmanyofyouare:
• Business?• Datascientists?• Dataanalysts?• DevOpsorIT?• Dataengineering?
Whatwe’lltalkabouttoday
• EvolutionofDataAnalytics… andproductstrategies• BigDataarchitectures• MachineLearning• MLUseCases• DeepLearning• APIs&Chatbots• Challenges&Takeaways
Search
MessengerBots
GoogleAssistant
TeslaAutopilot
Homeautomationplatform
Don’tlookatAIinisolation
DataAnalyticsEvolution
• Automatedatafeeds|ETL• Structuredanalytics|Datawarehousing&BI• Bigdatacapabilities|Datalake• Real-timeinsights|Streamingdataanalytics• AI|Machinelearning
DataScience
Datasciencetobuilddataproducts
• Whatistheproblemwearetryingtosolve?• Howdoweknowwhenwe’vewon?(keysuccessmetrics)• Isituseful?
• Whatdatadowehave?• Whatdatashouldwehave?• WhichassumptionscanIverify?• Askquestions
• https://www.datasciencecentral.com/profiles/blogs/the-data-science-project-lifecycle
Wherewillyouintegratethisprediction?
• Strongbusinessobjectivesandgoals• Specificproblems• Targetedbenefits
Datalakearchitecturecomponents
• Platform• Dataacquisition• Dataprocessing• Dataaccess• Security
Influences:• Objectives:Agile/Fail-fastandopen-source• ITstrategy- infra• Interplayoffactors
Dataplatform
Dataacquisition Dataprocessing Dataaccess
Security&Governance
Streamingusecases
• Clickstreamanalysis• Processpoint-of-saletransactions• Mobilegaming• IoT data
MachineLearning
Howmanyofyouhave:
• Usedanopen-sourceML?• Built/plantobuildyourownML?• UsedMLinthecloud?• Use/plantouseMLon-prem?
MachineLearning
MachineLearning
• Atitscore,asetofalgorithms• Canlearnfromandmakepredictionsbasedonrecordeddata• MLmodelisasetofexplanationsoftherelationshipsbetweeninputfeaturesandoutputpredictions• deployedwhereexplicitprogramingistoorigidorisimpractical
https://biguru.wordpress.com/2015/02/05/a-gentle-introduction-to-machine-learning/
MachineLearning
• Atitscore,asetofalgorithms• Canlearnfromandmakepredictionsbasedonrecordeddata• MLmodelisasetofexplanationsoftherelationshipsbetweeninputfeaturesandoutputpredictions• Predictsfutureoutcomesbasedonpasttrendsandtransaction• Deployedwhereexplicitprogramingisimpactical
https://biguru.wordpress.com/2015/02/05/a-gentle-introduction-to-machine-learning/
MachineLearningUseCases• Healthcare– Virtualassistants,wearables,lifestylemanagementandmonitoring,ER&surgery,drugdiscovery,mentalhealth,patientdataandriskanalytics,hospitalmanagement
• Fintech– creditscoring/directlending,regulatorycompliance,frauddetection,personalfinanceassistants,robo-advisors,insurance
• Security– cybersecurity,datasecurity
• Marketing&Advertising
• Facialrecognition– Image&Videoclassification/Segmentation• Speechrecognition• NaturalLanguageUnderstanding– NLP• Personalization- Recommendationengines
DeepLearning
• SubfieldofMLthatusesseverallayersofalgorithmscalledneuralnetworks(algorithmsthatmimicthehumanbrain)• Algorithmsnotlimitedtocreateanexplainablesetofrelationships• ConvolutionalNNsaregoodforimagerecognition• Longshort-termmemorynetworksaregoodforspeechrecognition• Steeplearningcurvebecauseoflow-levelAPIs
DeepLearning
Neuralnetworks
Neuralnetworkarchitectures
https://medium.com/@xenonstack/overview-of-artificial-neural-networks-and-its-applications-2525c1addff7
Usingdeeplearning
Whatwouldinterestyoumore:
• In-depthtutorialsondeeplearning• HowtouseMLframeworks• HowtouseMLAPIsinpractice
APIs
• Vision• Speech• Language
MLframeworks
• MLLib – ApacheSparkMLlibrary• Scikit-Learn(Pythonmathandscience)• Tensorflow - dataflowgraphs,neuralnetworks• ApacheMXNet• Caffe – deeplearning• H20.ai– extendtoHadoopandSpark(Sparklingwater)
• MicrosoftCognitiveToolkit(vision,speech,language,searchapis)• Amazonservices• Googleservices
Spark&Deeplearningpipelines
https://databricks.com/blog/2017/06/06/databricks-vision-simplify-large-scale-deep-learning.html
Spark&Deeplearningpipelines
https://databricks.com/blog/2017/06/06/databricks-vision-simplify-large-scale-deep-learning.html
Chatbots
• Botframeworks– MicrosoftBotFramework,Api.ai,Facebookbotengine• Botplatforms– Chatfuel,BeepBoop,Botengine.ai
AIinproductstrategy
AIacrossindustries
Takeaways
• ProductstrategywithAIinyourproduct• Specificbusinessgoalsandsuccessmetrics• Agilemodelnotalwayssuitablefordata/AIproducts• Peopleandprocessasimportantastechnology• Investinuser-centricdesign