{"id":549,"date":"2014-02-28T13:19:56","date_gmt":"2014-02-28T18:19:56","guid":{"rendered":"http:\/\/homepages.uc.edu\/~yaozo\/wordpress\/?p=549"},"modified":"2014-02-28T13:19:56","modified_gmt":"2014-02-28T18:19:56","slug":"statistics-meets-rhetoric-a-text-analysis-of-i-have-a-dream-in-r","status":"publish","type":"post","link":"https:\/\/zhuoyao.net\/index.php\/2014\/02\/28\/statistics-meets-rhetoric-a-text-analysis-of-i-have-a-dream-in-r\/","title":{"rendered":"Statistics meets rhetoric: A text analysis of “I Have a Dream” in R"},"content":{"rendered":"
\n
\n
This article was first published on analyze stuff<\/a>. It has been contributed to Anything but R-bitrary as the second article in its introductory series.<\/p>\n

By Max Ghenis<\/i><\/center>
\nToday, we celebrate the would-be 85th birthday of Martin Luther King, Jr., a man remembered for pioneering the civil rights movement through his courage, moral leadership, and oratory prowess. This post focuses on his most famous speech, I Have a Dream<\/a> [YouTube<\/a> | text<\/a>] given on the steps of the Lincoln Memorial to over 250,000 supporters of the March on Washington. While many have analyzed the cultural impact of the speech, few have approached it from a natural language processing perspective. I use R\u2019s text analysis packages and other tools to reveal some of the trends in sentiment, flow (syllables, words, and sentences), and ultimately popularity (Google search volume) manifested in the rhetorical masterpiece.<\/div>\n

Bag-of-words<\/h2>\n
Word clouds are somewhat controversial among data scientists: some see them as overused and cliche, while others find them a useful exploratory tool, particularly for connecting with a less analytical audience. I consider them a fun and useful starting point, so I started off by throwing the speech\u2019s text into Wordle<\/a>.<\/div>\n
\n
\"\"<\/a><\/div>\n<\/div>\n
\nR also has a wordcloud<\/a> package, though it\u2019s hard to beat Wordle on looks.<\/div>\n
\n# Load raw data, stored at textuploader.com\nspeech.raw <- paste(scan(url(\"http:\/\/textuploader.com\/1k0g\/raw\"), \n                         what=\"character\"), collapse=\" \")\n\nlibrary(wordcloud)\nwordcloud(speech.raw) # Also takes other arguments like color<\/pre>\n<\/div>\n
\"\"<\/p>\n
\nCalculating textual metrics<\/h2>\n
The qdap<\/a> package provides functions for text analysis, which I use to split sentences, count syllables and words, and estimate sentiment and readability. I also use the data.table<\/a> package to organize the sentence-level data structure.<\/div>\n
\n
\nlibrary(qdap)\nlibrary(data.table)\n\n# Split into sentences\n# qdap's sentSplit is modeled after dialogue data, so person field is needed\nspeech.df <- data.table(speech=speech.raw, person=\"MLK\")\nsentences <- data.table(sentSplit(speech.df, \"speech\"))\n# Add a sentence counter and remove unnecessary variables\nsentences[, sentence.num := seq(nrow(sentences))]\nsentences[, person := NULL]\nsentences[, tot := NULL]\nsetcolorder(sentences, c(\"sentence.num\", \"speech\"))\n\n# Syllables per sentence\nsentences[, syllables := syllable.sum(speech)]\n# Add cumulative syllable count and percent complete as proxy for progression\nsentences[, syllables.cumsum := cumsum(syllables)]\nsentences[, pct.complete := syllables.cumsum \/ sum(sentences$syllables)]\nsentences[, pct.complete.100 := pct.complete * 100]<\/pre>\n<\/div>\n
qdap\u2019s sentiment analysis is based on a sentence-level formula classifying each word as either positive, negative, neutral, negator or amplifier, per Hu & Liu\u2019s sentiment lexicon.<\/a> The function also provides a word count.<\/div>\n
\n
\npol.df <- polarity(sentences$speech)$all\nsentences[, words := pol.df$wc]\nsentences[, pol := pol.df$polarity]<\/pre>\n<\/div>\n
A scatterplot hints that polarity increases throughout the speech; that is, the sentiment gets more positive.<\/div>\n
\n
\nwith(sentences, plot(pct.complete, pol))<\/pre>\n<\/div>\n
\"\"<\/a><\/div>\n
<\/div>\n
\nCleaning up the plot and adding a LOESS<\/a> smoother clarifies this trend, particularly the peak at the end.<\/div>\n
\n
\nlibrary(ggplot2)\nlibrary(scales)\n\nmy.theme <- \n  theme(plot.background = element_blank(), # Remove background\n        panel.grid.major = element_blank(), # Remove gridlines\n        panel.grid.minor = element_blank(), # Remove more gridlines\n        panel.border = element_blank(), # Remove border\n        panel.background = element_blank(), # Remove more background\n        axis.ticks = element_blank(), # Remove axis ticks\n        axis.text=element_text(size=14), # Enlarge axis text font\n        axis.title=element_text(size=16), # Enlarge axis title font\n        plot.title=element_text(size=24, hjust=0)) # Enlarge, left-align title\n\nCustomScatterPlot <- function(gg)\n  return(gg + geom_point(color=\"grey60\") + # Lighten dots\n           stat_smooth(color=\"royalblue\", fill=\"lightgray\", size=1.4) + \n           xlab(\"Percent complete (by syllable count)\") + \n           scale_x_continuous(labels = percent) + my.theme)\n\nCustomScatterPlot(ggplot(sentences, aes(pct.complete, pol)) +\n                    ylab(\"Sentiment (sentence-level polarity)\") + \n                    ggtitle(\"Sentiment of I Have a Dream speech\"))<\/pre>\n<\/div>\n
\"\"<\/a><\/div>\n
Through the variation, the trendline reveals two troughs (calls to action, if you will) along with the increasing positivity.<\/p>\n
Readability tests<\/a> are typically based on syllables, words, and sentences in order to approximate the grade level required to comprehend a text. qdap offers several of the most popular formulas, of which I chose the Automated Readability Index<\/a>.<\/div>\n
\n
\nsentences[, readability := automated_readability_index(speech, sentence.num)\n          $Automated_Readability_Index]<\/pre>\n<\/div>\n
By graphing similarly to the above polarity chart, I show readability to be mostly constant throughout the speech, though it varies within each section. This makes sense, as one generally avoids too many simple or complex sentences in a row.<\/div>\n
<\/div>\n
\n
\nCustomScatterPlot(ggplot(sentences, aes(pct.complete, readability)) +\n                    ylab(\"Automated Readability Index\") +\n                    ggtitle(\"Readability of I Have a Dream speech\"))<\/pre>\n<\/div>\n
<\/div>\n
\"\"<\/a><\/div>\n
\nScraping Google search hits<\/h2>\n
Google search results can serve as a useful indicator of public opinion, if you know what to look for. Last week I had the pleasure of meeting Seth Stephens-Davidowitz<\/a>, a fellow analyst at Google who has used search data to research several topics, such as quantifying the effect of racism on the 2008 presidential election<\/a> (Obama did worse in states with higher racist query volume). There\u2019s a lot of room for exploring historically difficult topics with this data, so I thought I\u2019d use it to identify the most memorable pieces of I Have a Dream.<\/div>\n
Fortunately, I was able to build off of a function from theBioBucket\u2019s blog post<\/a> to count Google hits for a query.<\/div>\n
\n
\nGoogleHits <- function(query){\n  require(XML)\n  require(RCurl)\n  \n  url <- paste0(\"https:\/\/www.google.com\/search?q=\", gsub(\" \", \"+\", query))\n  \n  CAINFO = paste0(system.file(package=\"RCurl\"), \"\/CurlSSL\/ca-bundle.crt\")\n  script <- getURL(url, followlocation=T, cainfo=CAINFO)\n  doc <- htmlParse(script)\n  res <- xpathSApply(doc, '\/\/*\/div[@id=\"resultStats\"]', xmlValue)\n  return(as.numeric(gsub(\"[^0-9]\", \"\", res)))\n}<\/pre>\n<\/div>\n
From there I needed to pass each sentence to the function, stripped of punctuation and grouped in brackets, and with \u201cmlk\u201d added to ensure it related to the speech.<\/div>\n
\nsentences[, google.hits := GoogleHits(paste0(\"[\", gsub(\"[,;!.]\", \"\", speech), \n                                             \"] mlk\"))]<\/pre>\n<\/div>\n
A quick plot reveals that there\u2019s a huge difference between the most-quoted sentences and the rest of the speech, particularly the top seven (though really six as one is a duplicate). Do these top sentences align with your expectations?<\/div>\n
\n
\nggplot(sentences, aes(pct.complete, google.hits \/ 1e6)) +\n  geom_line(color=\"grey40\") + # Lighten dots\n  xlab(\"Percent complete (by syllable count)\") + \n  scale_x_continuous(labels = percent) + my.theme +\n  ylim(0, max(sentences$google.hits) \/ 1e6) +\n  ylab(\"Sentence memorability (millions of Google hits)\") +\n  ggtitle(\"Memorability of I Have a Dream speech\")<\/pre>\n
\"\"<\/a><\/div>\n
<\/div>\n<\/div>\n
\n
\nhead(sentences[order(-google.hits)]$speech, 7)<\/pre>\n
\n[1] \"free at last!\"\n[2] \"I have a dream today.\"\n[3] \"I have a dream today.\"\n[4] \"This is our hope.\"\n[5] \"And if America is to be a great nation this must become true.\"\n[6] \"I say to you today, my friends, so even though we face the difficulties of today and tomorrow, I still have a dream.\"\n[7] \"We cannot turn back.\"<\/pre>\n<\/div>\n
Plotting Google hits on a log scale reduces skew and allows us to work on a ratio scale.<\/p>\n
\nsentences[, log.google.hits := log(google.hits)]\n\nCustomScatterPlot(ggplot(sentences, aes(pct.complete, log.google.hits)) +\n                    ylab(\"Memorability (log of sentence's Google hits)\") +\n                    ggtitle(\"Memorability of I Have a Dream speech\"))<\/pre>\n
\"\"<\/a><\/div>\n
<\/div>\n
\nWhat makes a passage memorable? A linear regression approach<\/h2>\n
With several metrics for each sentence, along with the natural outcome variable of log(Google hits), I ran a linear regression to determine what makes a sentence memorable. I pruned the regressor list using the stepAIC backward selection technique, which minimizes the Akaike Information Criterion<\/a> and leads to a more parsimonious model. Finally, based on preliminary model results, I added polynomials of readability and excluded word count, syllable count, and syllables per word (readability is largely based on these factors).<\/div>\n
\n
\nlibrary(MASS) # For stepAIC\ngoogle.lm <- stepAIC(lm(log(google.hits) ~ poly(readability, 3) + pol +\n                          pct.complete.100, data=sentences))<\/pre>\n<\/div>\n
stepAIC returns the optimal model, which can be summarized like any lm object.<\/div>\n
\n
\nsummary(google.lm)<\/pre>\n
\nCall:\nlm(formula = log(google.hits) ~ poly(readability, 3) + pct.complete.100, \n    data = sentences)\n\nResiduals:\n    Min      1Q  Median      3Q     Max \n-4.2805 -1.1324 -0.3129  1.1361  6.6748 \n\nCoefficients:\n                        Estimate Std. Error t value Pr(>|t|)    \n(Intercept)            11.444037   0.405247  28.240  < 2e-16 ***\npoly(readability, 3)1 -12.670641   1.729159  -7.328 1.75e-10 ***\npoly(readability, 3)2   8.187941   1.834658   4.463 2.65e-05 ***\npoly(readability, 3)3  -5.681114   1.730662  -3.283  0.00153 ** \npct.complete.100        0.013366   0.006848   1.952  0.05449 .  \n---\nSignif. codes:  0 \u2018***\u2019 0.001 \u2018**\u2019 0.01 \u2018*\u2019 0.05 \u2018.\u2019 0.1 \u2018 \u2019 1\n\nResidual standard error: 1.729 on 79 degrees of freedom\nMultiple R-squared:  0.5564, Adjusted R-squared:  0.534 \nF-statistic: 24.78 on 4 and 79 DF,  p-value: 2.605e-13<\/pre>\n<\/div>\n
Four significant regressors explained 56% of the variance (R2=0.5564): a third degree polynomial of readability, along with pct.complete.100 (where the sentence was in the speech); polarity was not significant.<\/div>\n
<\/div>\n
\nThe effect of pct.complete can be calculated by exponentiating the coefficient, since I log-transformed the outcome variable:<\/div>\n
\n
exp(google.lm$coefficients[\"pct.complete.100\"])<\/pre>\n
pct.complete.100\n        1.013456<\/pre>\n<\/div>\n
This result can be interpreted as the following: a 1%<\/b> increase in the location of a sentence in the speech was associated with a 1.3%<\/b> increase in search hits.
\nInterpreting the effect of readability is not as straightforward, since I included polynomials. Rather than compute an average effect, I graphed predicted Google hits for values of readability’s observed range, holding pct.complete.100 at its mean.<\/p>\n
\nnew.data <- data.frame(readability=seq(min(sentences$readability), \n                                       max(sentences$readability), by=0.1),\n                       pct.complete.100=mean(sentences$pct.complete.100))\n\nnew.data$pred.hits <- predict(google.lm, newdata=new.data)\n\nggplot(new.data, aes(readability, pred.hits)) + \n  geom_line(color=\"royalblue\", size=1.4) + \n  xlab(\"Automated Readability Index\") +\n  ylab(\"Predicted memorability (log Google hits)\") +\n  ggtitle(\"Predicted memorability ~ readability\") +\n  my.theme<\/pre>\n
\"\"<\/a><\/div>\n
This cubic relationship indicates that predicted memorability falls considerably until about grade level 10, at which point it levels off (very few passages have readability exceeding 25).<\/p>\n
\nConclusion<\/h2>\n
R tools from qdap to ggplot2 have uncovered some of MLK\u2019s brilliance in I Have a Dream:<\/div>\n
 <\/p>\n
<\/div>\n