+ - 0:00:00
Notes for current slide
Notes for next slide

Common pitfalls in sentiment analysis

Martin Müller

R lunchs | June 4, 2019

1 / 33

Goals

  • Intro sentiment analysis/text classification
2 / 33

Goals

  • Intro sentiment analysis/text classification

  • Focus on concepts rather than code

3 / 33

Goals

  • Intro sentiment analysis/text classification

  • Focus on concepts rather than code

  • Explore 3 common pitfalls, 2 methods for text classification

4 / 33
  • You shouldn't have prior knowledge to be able to follow this, but maybe it would be good for me to know what the knowledeg distribution is
  • I'm a Python user and you may ask yourself what the hell are you doing here- and no I'm not going into the whole R. vs. Python. Everything I'm going to talk about can be done in either language

Rule-based sentiment analysis

  • Based on large dictionary of words and their sentiment, derived by experts

            mean  std
    unfair  -2.1  0.83066 [-1, -3, -3, -2, -3, -1, -2, -3, -1, -2]
    great    3.1  0.7     [ 2,  4,  4,  4,  3,  3,  3,  3,  2,  3]
    hate    -2.7  1.00499 [-4, -3, -4, -4, -2, -2, -2, -2, -1, -3]
    :'(     -2.2  0.74833 [-2, -1, -2, -2, -2, -2, -4, -3, -2, -2]
5 / 33

Rule-based sentiment analysis

  • Based on large dictionary of words and their sentiment, derived by experts

            mean  std
    unfair  -2.1  0.83066 [-1, -3, -3, -2, -3, -1, -2, -3, -1, -2]
    great    3.1  0.7     [ 2,  4,  4,  4,  3,  3,  3,  3,  2,  3]
    hate    -2.7  1.00499 [-4, -3, -4, -4, -2, -2, -2, -2, -1, -3]
    :'(     -2.2  0.74833 [-2, -1, -2, -2, -2, -2, -4, -3, -2, -2]
  • Out-of-the-box method to compute sentiment (polarity), subjectivity, emotions (anger, happiness, etc.)
    if (!require("pacman")) install.packages("pacman")
    pacman::p_load(sentimentr, dplyr, magrittr)
    mytext <- c('do you like it?  But I hate really bad dogs',
    'I am the best friend.',
    'Do you really like it?  I\'m not a fan')
    mytext <- get_sentences(mytext)
    sentiment_by(mytext)
    ##    element_id word_count       sd ave_sentiment
    ## 1:          1         10 1.497465    -0.8088680
    ## 2:          2          5       NA     0.5813777
    ## 3:          3          9 0.284605     0.2196345
6 / 33

SentimentR package

  • Averages the polarity of words in a local cluster
  • Amplifiers increase polarity by 1.8 (e.g. very good)
  • De-amplifiers reduce polarity (e.g. slightly, decently, etc.)
  • Negation is factored in as \((-1)^{n}\), with \(n\) as the number of negators
  • Adversative conjunctions (e.g. but, however, although) increase polarity of the following cluster and decreases polarity of the previous cluster
7 / 33

SentimentR package

  • Averages the polarity of words in a local cluster
  • Amplifiers increase polarity by 1.8 (e.g. very good)
  • De-amplifiers reduce polarity (e.g. slightly, decently, etc.)
  • Negation is factored in as \((-1)^{n}\), with \(n\) as the number of negators
  • Adversative conjunctions (e.g. but, however, although) increase polarity of the following cluster and decreases polarity of the previous cluster

→ In practice works relatively well

8 / 33

An emotional timeline of the 9/11 attacks

  • 573,000 pager messages (published by Wikileaks, 2009), 6.4M words

Back, Mitja D., et al., SAGE Psychological Science, 2010

9 / 33

Of the 16,624 instances of anger words, 5,974 (35.9%) was the word critical

10 / 33

Of the 16,624 instances of anger words, 5,974 (35.9%) was the word critical

Reboot NT machine [name] in cabinet [name] at [location]:CRITICAL:[date and time].
11 / 33

Of the 16,624 instances of anger words, 5,974 (35.9%) was the word critical

Reboot NT machine [name] in cabinet [name] at [location]:CRITICAL:[date and time].

Pury, SAGE Psychological Science, 2011

12 / 33

Pitfall #1 Never skip quality control

13 / 33

Hypothesis: What we usually want is stance and not sentiment

14 / 33

USA today's Twitter Election meter

15 / 33

What is stance?

From the SemEval 2016 stance dataset

Example Target Stance Sentiment
If abortion is not wrong, then nothing is wrong. Powerful words from Blessed Mother Teresa. Abortion Against 0.49
16 / 33

What is stance?

From the SemEval 2016 stance dataset

Example Target Stance Sentiment
If abortion is not wrong, then nothing is wrong. Powerful words from Blessed Mother Teresa. Abortion Against 0.49
8 years ago today my son was taken from me. If there's a god, fuck you, fuck you very much. Atheism Favor -0.48
17 / 33

What is stance?

From the SemEval 2016 stance dataset

Example Target Stance Sentiment
If abortion is not wrong, then nothing is wrong. Powerful words from Blessed Mother Teresa. Abortion Against 0.49
8 years ago today my son was taken from me. If there's a god, fuck you, fuck you very much. Atheism Favor -0.48
Hillary Clinton has some strengths and some weaknesses. Hillary Clinton Neutral -0.35
18 / 33

What is stance?

From the SemEval 2016 stance dataset

Example Target Stance Sentiment
If abortion is not wrong, then nothing is wrong. Powerful words from Blessed Mother Teresa. Abortion Against 0.49
8 years ago today my son was taken from me. If there's a god, fuck you, fuck you very much. Atheism Favor -0.48
Hillary Clinton has some strengths and some weaknesses. Hillary Clinton Neutral -0.35
Benghazi must be answered for #Jeb16 Hillary Clinton Against 0
19 / 33

Pitfall #2 Opinion (=stance) is often not the same as sentiment (=tonality)

20 / 33

In search for a better method

Given that we have some annotated data, we can design our own algorithm

21 / 33

In search for a better method

Given that we have some annotated data, we can design our own algorithm

Problem: We are lazy :)

22 / 33

In search for a better method: FastText

A. Joulin et. al, Bag of Tricks for Efficient Text Classification (2017)

  • Requires little to no preprocessing
  • Very suitable for text with slang or lots of spelling mistakes
  • As the name suggests: It trains very fast (<10s for >10k samples)
  • No GPU required
  • Pretrained for 157 languages
  • Model sizes of around 100MB (compressable to around 100kB at similar performance using FastText.zip)
23 / 33

In search for a better method: FastText

  • Each word is split into character n-grams. Example for n=3:

    where = ['<wh', 'whe', 'her', 'ere', 're>', '<where>']

  • Large improvements for languages like German, Turkish and Arabic

24 / 33

The problem with language

25 / 33

Oldest tree in the world (9,500 years old, Sweden)

Source: Wikimedia

26 / 33

Source: Google Compute Vision API

27 / 33

What about historic text

Now Can you write a sentence in English the way it would look now, one hundred years ago, and five hundred years ago?

400 years ago Canst thou write a sentence in English, in the wise that it looketh now, and look’d an hundred years past, and yet five hundred years past?

500 years ago Canst thou writ a sentence yn Englissh, yn such wise as yt seemeth nou, and seemed an hundred yeeres gonne, and fiue hundred yeeres gonne?

600 years ago Canst thou wryt ane sentence yn Englische yn this wise: that as yt semeth nouwe, and hath semed an hundred yeres ygonne, and fiue hundred yeres ygonne?

1000 years ago Meaht þu writan anne cwide in Ænglisc þus, swa he nu biþ, ond swa he hundred geara ær wæs, ond swa fif hundred geara ær?

28 / 33

... but not only that

  • The meaning of words can change
  • Constantly new words are introduced in slang
  • New public figures appear

... and worst of all:

  • Our model of the world and implicit shared knowledge changes constantly
29 / 33

... but not only that

  • The meaning of words can change
  • Constantly new words are introduced in slang
  • New public figures appear

... and worst of all:

  • Our model of the world and implicit shared knowledge changes constantly

    Text classifiers of >5 years ago are likely to be outdated now! (aka. model drift)

30 / 33

Pitfall #3 Share models & annotation data and make your work reproducible

31 / 33

Thank you

32 / 33

One book I specifically recommend on the topic:

Bit by bit: Social Research in the Digital Age

(free to read online)

33 / 33

Goals

  • Intro sentiment analysis/text classification
2 / 33
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow