Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news...

20
DataCamp Natural Language Processing Fundamentals in Python Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON Katharine Jarmul Founder, kjamistan

Transcript of Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news...

Page 1: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

ClassifyingfakenewsusingsupervisedlearningwithNLP

NATURALLANGUAGEPROCESSINGFUNDAMENTALSINPYTHON

KatharineJarmulFounder,kjamistan

Page 2: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

Whatissupervisedlearning?Formofmachinelearning

ProblemhaspredefinedtrainingdataThisdatahasalabel(oroutcome)youwantthemodeltolearnClassificationproblemGoal:Makegoodhypothesesaboutthespeciesbasedongeometricfeatures

SepalLength SepalWidth PetalLength PetalWidth Species

5.1 3.5 1.4 0.2 I.setosa

7.0 3.2 4.77 1.4 I.versicolor

6.3 3.3 6.0 2.5 I.virginica

Page 3: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

SupervisedlearningwithNLPNeedtouselanguageinsteadofgeometricfeaturesscikit-learn:Powerfulopen-sourcelibrary

Howtocreatesupervisedlearningdatafromtext?Usebag-of-wordsmodelsortf-idfasfeatures

Page 4: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

IMDBMovieDatasetPlot Sci-Fi Action

Inapost-apocalypticworldinhumandecay,a... 1 0

Moheiisawanderingswordsman.Hearrivesin... 0 1

#137isaSCI/FIthrilleraboutagirl,Marla,... 1 0

Goal:PredictmoviegenrebasedonplotsummaryCategoricalfeaturesgeneratedusingpreprocessing

Page 5: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

SupervisedlearningstepsCollectandpreprocessourdataDeterminealabel(Example:Moviegenre)SplitdataintotrainingandtestsetsExtractfeaturesfromthetexttohelppredictthelabel

Bag-of-wordsvectorbuiltintoscikit-learn

Evaluatetrainedmodelusingthetestset

Page 6: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

Let'spractice!

NATURALLANGUAGEPROCESSINGFUNDAMENTALSINPYTHON

Page 7: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

Buildingwordcountvectorswithscikit-

learn

NATURALLANGUAGEPROCESSINGFUNDAMENTALSINPYTHON

KatharineJarmulFounder,kjamistan

Page 8: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

PredictingmoviegenreDatasetconsistingofmovieplotsandcorrespondinggenreGoal:Createbag-of-wordvectorsforthemovieplots

Canwepredictgenrebasedonthewordsusedintheplotsummary?

Page 9: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

CountVectorizerwithPythonIn[1]:importpandasaspd

In[2]:fromsklearn.model_selectionimporttrain_test_split

In[3}:fromsklearn.feature_extraction.textimportCountVectorizer

In[4]:df=...#LoaddataintoDataFrame

In[5]:y=df['Sci-Fi']

In[6]:X_train,X_test,y_train,y_test=train_test_split(df['plot'],y,test_size=0.33,random_state=53)

In[7]:count_vectorizer=CountVectorizer(stop_words='english')

In[8]:count_train=count_vectorizer.fit_transform(X_train.values)

In[9]:count_test=count_vectorizer.transform(X_test.values)

Page 10: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

Let'spractice!

NATURALLANGUAGEPROCESSINGFUNDAMENTALSINPYTHON

Page 11: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

Trainingandtestingaclassificationmodelwithscikit-learn

NATURALLANGUAGEPROCESSINGFUNDAMENTALSINPYTHON

KatharineJarmulFounder,kjamistan

Page 12: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

NaiveBayesclassifierNaiveBayesModel

CommonlyusedfortestingNLPclassificationproblemsBasisinprobability

Givenaparticularpieceofdata,howlikelyisaparticularoutcome?Examples:

Iftheplothasaspaceship,howlikelyisittobesci-fi?Givenaspaceshipandanalien,howlikelynowisitsci-fi?

EachwordfromCountVectorizeractsasafeature

NaiveBayes:Simpleandeffective

Page 13: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

NaiveBayeswithscikit-learnIn[10]:fromsklearn.naive_bayesimportMultinomialNB

In[11]:fromsklearnimportmetrics

In[12]:nb_classifier=MultinomialNB()

In[13]:nb_classifier.fit(count_train,y_train)

In[14]:pred=nb_classifier.predict(count_test)

In[15]:metrics.accuracy_score(y_test,pred)Out[15]:0.85841849389820424

Page 14: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

ConfusionMatrix

Action Sci-Fi

Action 6410 563

Sci-Fi 864 2242

In[16]:metrics.confusion_matrix(y_test,pred,labels=[0,1])Out[16]:array([[6410,563],[864,2242]])

Page 15: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

Let'spractice!

NATURALLANGUAGEPROCESSINGFUNDAMENTALSINPYTHON

Page 16: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

SimpleNLP,ComplexProblems

NATURALLANGUAGEPROCESSINGFUNDAMENTALSINPYTHON

KatharineJarmulFounder,kjamistan

Page 17: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

Translation

(source: )https://twitter.com/Lupintweets/status/865533182455685121

https://twitter.com/Lupintweets/status/865533182455685121
Page 18: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

SentimentAnalysis

(source: )https://nlp.stanford.edu/projects/socialsent/

https://nlp.stanford.edu/projects/socialsent/
Page 19: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

LanguageBiases

(relatedtalk: )https://www.youtube.com/watch?v=j7FwpZB1hWc

https://www.youtube.com/watch?v=j7FwpZB1hWc
Page 20: Classifying fake news using supervised learning with NLP · 2017-08-14 · Classifying fake news using supervised learning with NLP NATURAL LANGUAGE PROCESSING FUNDAMENTALS IN PYTHON

DataCamp NaturalLanguageProcessingFundamentalsinPython

Let'spractice!

NATURALLANGUAGEPROCESSINGFUNDAMENTALSINPYTHON


CS11-711 Advanced NLP NLP Experimental Design

CS11-711 Advanced NLP NLP Experimental Design

NLP Practitioner Notes · NLP Practitioner Notes NLP Practitioner Course presented by Micheal Goodman NLP Master Practitioner (ABNLP) and NLP Trainer (NLPEA) 2018

NLP Practitioner Notes · NLP Practitioner Notes NLP Practitioner Course presented by Micheal Goodman NLP Master Practitioner (ABNLP) and NLP Trainer (NLPEA) 2018

Classifying Movie Scripts by Genre with a MEMM Using NLP ...nlp.stanford.edu/courses/cs224n/2008/reports/06.pdf · conventions, that vast majority ... The transformation of a movie

Classifying Movie Scripts by Genre with a MEMM Using NLP ...nlp.stanford.edu/courses/cs224n/2008/reports/06.pdf · conventions, that vast majority ... The transformation of a movie

NLP mastering nlp coaching skills.pdf

NLP mastering nlp coaching skills.pdf

Fake coin detection. Fake coins Suppose we have a number of coins, at most one is fake (i.e. either one is fake or none is fake). You have a pair of scales.

Fake coin detection. Fake coins Suppose we have a number of coins, at most one is fake (i.e. either one is fake or none is fake). You have a pair of scales.

Robert Kiyosaki: Fake: Fake Money, Fake Teachers, Fake ...bestbookbits.com/wp-content/uploads/2019/10/Fake-by-Robert-Kiyo… · Robert Kiyosaki: Fake: Fake Money, Fake Teachers, Fake

Robert Kiyosaki: Fake: Fake Money, Fake Teachers, Fake ...bestbookbits.com/wp-content/uploads/2019/10/Fake-by-Robert-Kiyo… · Robert Kiyosaki: Fake: Fake Money, Fake Teachers, Fake

Life Changing NLP Training - Home - Dr Bridget NLP · Life Changing NLP Training ... Certified Master Practitioner of NLP (Neuro Linguistic Programming), ... NLP PRACTITIONER COURSE

Life Changing NLP Training - Home - Dr Bridget NLP · Life Changing NLP Training ... Certified Master Practitioner of NLP (Neuro Linguistic Programming), ... NLP PRACTITIONER COURSE

The Linguistic Features of Fake News Headlines and Statements · sify fake news, the accuracy being less than chance. This demonstrates the scope of improvement regarding this classifying

The Linguistic Features of Fake News Headlines and Statements · sify fake news, the accuracy being less than chance. This demonstrates the scope of improvement regarding this classifying

FAKE BOOKS...FAKE BOOKS 39 BEST FAKE BOOK EVER – 3RD EDITION FAKE BOOKSAnother excellent selection of songs compiled into one convenient fake book! 1,075 songs! Features: Guitar

FAKE BOOKS...FAKE BOOKS 39 BEST FAKE BOOK EVER – 3RD EDITION FAKE BOOKSAnother excellent selection of songs compiled into one convenient fake book! 1,075 songs! Features: Guitar

NLP Classic Code & New Code NLP Practitioner 2014holisticdirections.com/.../2014/...Winnipeg_NEW_flyer_2014_10_Aug1.pdf · Jacquie Nagy NLP Classic Code & New Code NLP Practitioner

NLP Classic Code & New Code NLP Practitioner 2014holisticdirections.com/.../2014/...Winnipeg_NEW_flyer_2014_10_Aug1.pdf · Jacquie Nagy NLP Classic Code & New Code NLP Practitioner

Bringing NLP to Life NLP Coaching Onlineapi.ning.com/.../NLPOnlineNLPContentCompleted15thJuly.pdf · 2016-10-21 · Bringing NLP to Life NLP Coaching Online Course Manual Pam Rigden

Bringing NLP to Life NLP Coaching Onlineapi.ning.com/.../NLPOnlineNLPContentCompleted15thJuly.pdf · 2016-10-21 · Bringing NLP to Life NLP Coaching Online Course Manual Pam Rigden

NLP Practitioner Certification Training - ubermind.coubermind.co/assets/uber-mind-nlp-flyer-2017-1213-03.pdf · grates both traditional (behavioral) NLP along with NLP New Code. NLP

NLP Practitioner Certification Training - ubermind.coubermind.co/assets/uber-mind-nlp-flyer-2017-1213-03.pdf · grates both traditional (behavioral) NLP along with NLP New Code. NLP

(Nlp) Nlp Secrets

(Nlp) Nlp Secrets

FINAL PROJECT REPORT On EVALUATING AND ENHANCING ...twang1/studentProjects/NLP...detection, most of which has been focused on classifying online reviews and publicly available social

FINAL PROJECT REPORT On EVALUATING AND ENHANCING ...twang1/studentProjects/NLP...detection, most of which has been focused on classifying online reviews and publicly available social

Introduction to NLP - ils.albany.eduWhat is NLP? 2.Some application areas of NLP 3.A brief history of NLP 4.Famous NLP systems 5. Ambiguity and NLP 6.Overcoming ambiguity – Brief

Introduction to NLP - ils.albany.eduWhat is NLP? 2.Some application areas of NLP 3.A brief history of NLP 4.Famous NLP systems 5. Ambiguity and NLP 6.Overcoming ambiguity – Brief

FAKE NEWS DETECTION USING NLP

FAKE NEWS DETECTION USING NLP

NLP Practitioner Heart of NLP - NLP Courses

NLP Practitioner Heart of NLP - NLP Courses

[Nlp ebook] anne linden - mindworks - nlp tools

[Nlp ebook] anne linden - mindworks - nlp tools

Fake degree&fake diploma template,buy fake university transcript

Fake degree&fake diploma template,buy fake university transcript

NLP Training - NLP Certification

NLP Training - NLP Certification