Making Predictions with Textual Contents Making Predictions with Textual Contents Indira Gandi...

download Making Predictions with Textual Contents Making Predictions with Textual Contents Indira Gandi Mascarenhas

of 67

  • date post

    24-May-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of Making Predictions with Textual Contents Making Predictions with Textual Contents Indira Gandi...

  • Making Predictions with Textual Contents

    Indira Gandi Mascarenhas de Brito

    Thesis to obtain the Master of Science Degree in

    Information Systems and Computer Engineering

    Supervisor: Prof. Bruno Emanuel da Graça Martins

    Examination Committee

    Chairperson: Prof. Mário Jorge Costa Gaspar da Silva Supervisor: Prof. Bruno Emanuel da Graça Martins

    Member of the Committee: Prof. David Manuel Martins de Matos

    April 2014

  • Abstract

    Forecasting real-world quantities with basis on information from textual descriptions has re-cently attracted significant interest as a research problem, although previous studies have focused on applications involving only the English language.

    This document presents an experimental study on the subject of making predictions with tex-

    tual contents written in Portuguese, using documents from three distinct domains. I specifically

    report on experiments using different types of regression models, using state-of-the-art feature

    weighting schemes, and using features derived from cluster-based word representations.

    Through controlled experiments, I have shown that prediction models using the textual infor-

    mation achieve better results than simple baselines such as taking the average value over the

    training data, and that richer document representations (i.e., using Brown clusters and the Delta-

    TF-IDF feature weighting scheme) result in slight performance improvements.

    Keywords: Text-Driven Forecasting, Supervised Learning of Regression Models, Word Clus-

    tering, Feature Engineering for NLP Applications

    i

  • Sumário

    A previsão de quantidades do mundo real com base em informação proviniente de descrições

    textuais atraiu recentemente um interesse significativo enquanto problema de investigação,

    embora os estudos anteriores na área se tenham concentrado em aplicações que envolvem

    apenas textos no idioma Inglês.

    Este documento apresenta um estudo experimental sobre a realização de previsões com base

    em conteúdos textuais escritos em Português, envolvendo o uso de documentos associados a

    três domı́nios distintos. Eu relato especificamente experiências utilizando diferentes tipos de

    modelos de regressão, usando esquemas de pesagem para as caracterı́sticas do actual estado

    da arte, e usando caracterı́sticas derivadas de representações para as palavras baseadas no

    agrupamento automático das mesmas.

    Através de experiências controladas, desmonstrei que modelos preditores usando informação

    textual atingem melhores resultados, quando comparados com abordagens simples tais como

    realizar as previsões com base no valor médio dos dados de treino. Demonstrei ainda que

    as representações de documentos mais ricas (ou seja, usando o algoritmo de Brown para o

    agrupamento automático de palavras, e o esquema de pesagem das caracterı́sticas denomindo

    Delta-TF-IDF) resultam em ligeiras melhorias no desempenho.

    Palavras Chave: Previsões com Base em Texto, Modelos de Regressão, Agrupamento Au-

    tomático de Palavras Semelhantes, Engenharia de Caracteristicas para Aplicações em PLN

    iii

  • Acknowledgements

    This work was supported by Fundação para a Ciência e a Tecnologia (FCT), through theproject grant with reference UTA-Est/MAI/0006/2009 (REACTION), as well as through the INESC-ID multi-annual funding from the PIDDAC programme (PEst-OE/EEI/LA0021/2013).

    This work would not have been possible without the enormous contribution of all friends and

    colleagues that were present during the development of this project. I would like to express my

    thanks to these persons, for the friendship, encouragement, understanding, and wisdom that I

    had the privilege of receiving.

    Firstly, I want to thank God for giving me life and health. Many thanks also to my supervisor, Dr.

    Bruno Martins, who read through my numerous revisions and helped in the course of the project.

    To my parents Francisco Brito and Maria da Conceição Mascarenhas, for the emotional and

    financial support. Thanks for always believing in me.

    To my brothers Diva Brito and Djeivy Brito, for their love and support. To my boyfriend, Gelton

    Delgado, my partner of all time. To my uncles, Manuel and Fatima, who have helped me since I

    arrived in Portugal. Finally, to all my Portuguese and Cape Verdean friends, specifically Aurea,

    Any, Elio, Elizangela, Lenise, Tatiana, Lilian, Rodney, Dirce, and Helton.

    I would also like to express my sincere gratitude to the colleagues from the aforementioned

    REACTION project, for their assistance and insightful comments.

    Thanks to all.

    v

  • Contents

    Abstract i

    Sumário iii

    Acknowledgements v

    1 Introduction 1

    1.1 Thesis Proposal and Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

    1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

    1.3 Document Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

    2 Previous and Related Work 7

    2.1 Work in Text-Driven Forecasting from Noah Smith et al. . . . . . . . . . . . . . . . 7

    2.2 Textual Predictors of Congress Bill Survival . . . . . . . . . . . . . . . . . . . . . . 13

    2.3 Characterizing Variation in Well-Being Using Tweets . . . . . . . . . . . . . . . . . 15

    2.4 Predicting News Story Importance Using Their Text . . . . . . . . . . . . . . . . . . 17

    2.5 Exploring Yelp Reviews in Forecasting Tasks . . . . . . . . . . . . . . . . . . . . . 20

    2.6 Summary and Critical Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

    3 Making Predictions with Textual Contents 25

    3.1 The General Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

    3.2 Word Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

    3.3 Feature Weighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

    vii

  • 3.4 Regression Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

    3.4.1 Linear Regression Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

    3.4.2 Ensemble Learning Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 34

    3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

    4 Experimental Validation 37

    4.1 Datasets and Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

    4.2 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

    4.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

    5 Conclusions and Future Work 45

    5.1 Main Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

    5.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

    Reference 49

    viii

  • List of Tables

    4.1 Statistical characterization for the three different datasets. . . . . . . . . . . . . . . 38

    4.2 Results for the first experiment, with a representation based on TF-IDF. . . . . . . 40

    4.3 Results with Elastic Net models using different feature weighting schemes. . . . . 42

    4.4 Results with Random Forest models using different feature weighting schemes. . . 42

    4.5 Results for predicting hotel room prices with different feature sets. . . . . . . . . . 43

    4.6 Results for predicting restaurant prices with different feature sets. . . . . . . . . . . 43

    4.7 Results for predicting movie box-office revenues with different feature sets. . . . . 43

    4.8 Overall results for the different forecasting tasks. . . . . . . . . . . . . . . . . . . . 44

    ix

  • List of Figures

    2.1 Local sentiment and price estimates for two test sentences (Chahuneau et al., 2012). 12

    2.2 Top features for jointly predicting sentiment and price (Chahuneau et al., 2012). . . 13

    2.3 Map of the United States showing life satisfaction as measured using survey data

    and as predicted using the approach proposed by Schwartz et al. (2013). . . . . . 17

    3.4 Text-driven forecasting as a regression task. . . . . . . . . . . . . . . . . . . . . . . 27

    3.5 The class-based bi-gram language model, that supports the word clustering. . . . 28

    3.6 Example binary tree resulting from word clustering. . . . . . . . . . . . . . . . . . . 29

    3.7 Estimating Lasso and Ridge regression. . . . . . . . . . . . . . . . . . . . . . . . . 33

    3.8 Multiple regression trees used on the Random Forest method. . . . . . . . . . . . 35

    4.9 Distribution for the target values, in the hotels, restaurants, and movies datasets. . 38

    4.10 Distribution for the target values, per district in Continental Portugal. . . . . . . . . 39

    4.11 The 20 most important features in the case of predicting hotel room prices. . . . . 41

    4.12 The 20 most important features in the case of predicting meal prices at restaurants,

    or in the case of predicting movie box-office results. . . . . . . . . . . . . . . . . . 41

    4.13 Box-Office revenues versus the number of screens on whi