Towards Automatic Detection of Misinformation in Social Media

Symeon (Akis) Papadopoulos - @sympap

Information Technologies Institute (ITI) / Centre for Research and Technology Hellas (CERTH)

Workshop on Tools for Video Discovery & Verification in Social Media Dec 14, 2017 @ Thessaloniki, Greece

Real or fake?

Real or fake?

Real or fake?

Real or fake?

Types of misinformation

Towards computational verification

The Tweet Verification Assistant

A web-based service for marking an input tweet as “real” or “fake”

2012 first ideas and experiments (SocialSensor)

2013-2016 main research, development and validation (REVEAL)

2016-now incremental refinements and testing (InVID)

User interface

http://reveal-mklab.iti.gr/reveal/fake/

Credibility cues (aka features)

Building the classification model

Tweet Verification Corpus

• 53 events or hoaxes involving false and/or real imagery and videos

• 257 cases of “fake” content, 261 of “real”

• 10,634 tweets sharing “fake” content, 7,223 tweets sharing “real” content

• Examples events: • Hurricane Sandy

• Boston Marathon bombing

• Sochi Olympics

• MA Flight 370

• Nepal Earthquake…

https://github.com/MKLab-ITI/image-verification-corpus

The “Verifying Multimedia Use” Task

•VMU: Organized in 2015 and 2016 as part of the MediaEval benchmarking initiative

•Goal: compare automated approaches for fake tweet detection

•Outcomes: several methods from different research groups across the globe were tested and compared

Experimental validation

92.5% accuracy in identifying misleading posts

88-98% accuracy depending on language

(major languages tested: en, fr, es, nl)

New features, bagging and agreement-based retraining led to significant improvements! One of the top performing methods in the VMU 2015 & 2016 tasks!

Context Analysis and Aggregation

• Available at: http://caa.iti.gr

• YouTube, Facebook and Twitter videos

• metadata from APIs

• mentioned locations

• “verification”-related comments

• thumbnails for near-duplicate search

• weather at time and location of video

• video sharing on Twitter

Tip in comment led to debunking

A comment points to second 23 of the video where suddenly the snake appears out of nowhere

# verification comments too high

1550 verification-related comments out of 4219 total number of comments

Tweets sharing video are flagged

37 out of 43 tweets sharing the video are classified as fake

Video verification experiments

• 117 fake videos and 110 real videos

• The dataset covers different types of manipulation: • staged videos,

• videos misrepresenting the depicted event,

• videos of past events claimed to be captured now,

• digitally manipulated videos.

• A supervised learning approach using credibility features extracted from video comments and video metadata managed to achieve promising accuracy:

P=72%, R=86%, F=79%

Limitations

• Models are still based on aged training data (could be affected by concept drift…)

• Results not always easy to justify or explain to end users

• A well-informed adversary can easily fool the model by emulating “credible-looking” posts

• Journalists are still expected to make the final decision!

The future of misinformation

Acknowledgements

• Christina Boididou (Feature extraction, model building, initial REST API development)

• Olga Papadopoulou (Validation, code refactoring, REST API refinement and support)

• Lazaros Apostolidis (UI/UX)

• Markos Zampoglou (Evaluation)

• Yiannis Kompatsiaris (PI)

Thank you!

http://reveal-mklab.iti.gr/reveal/fake/

http://caa.iti.gr

Get in touch!

Akis Papadopoulos papadop@iti.gr / @sympap

References

• Boididou, C., Papadopoulos, S., Kompatsiaris, Y., Schifferes, S., & Newman, N. (2014, April). Challenges of computational verification in social multimedia. In Proceedings of the 23rd International Conference on World Wide Web (pp. 743-748). ACM

• Boididou, C., Middleton, S. E., Jin, Z., Papadopoulos, S., Dang-Nguyen, D. T., Boato, G., & Kompatsiaris, Y. (2017). Verifying information with multimedia content on twitter. Multimedia Tools and Applications, 1-27

• Boididou, C., Papadopoulos, S., Apostolidis, L., & Kompatsiaris, Y. (2017, June). Learning to Detect Misleading Content on Twitter. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval (pp. 278-286). ACM

• Castillo, C., Mendoza, M., & Poblete, B. (2011, March). Information credibility on twitter. In Proceedings of the 20th international conference on World Wide Web (pp. 675-684). ACM

• Liu, M. Y., Breuel, T., & Kautz, J. (2017). Unsupervised Image-to-Image Translation Networks. arXiv preprint arXiv:1703.00848

• Papadopoulou, O., Zampoglou, M., Papadopoulos, S., & Kompatsiaris, Y. (2017, June). Web Video Verification using Contextual Cues. In Proceedings of the 2nd International Workshop on Multimedia Forensics and Security (pp. 6-10). ACM

• Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. arXiv preprint arXiv:1703.10593.