{"id":756,"date":"2015-02-16T00:08:02","date_gmt":"2015-02-16T07:08:02","guid":{"rendered":"http:\/\/homepages.uc.edu\/~yaozo\/wordpress\/?p=756"},"modified":"2015-02-16T00:08:02","modified_gmt":"2015-02-16T07:08:02","slug":"text-mining-the-complete-works-of-william-shakespeare-2","status":"publish","type":"post","link":"https:\/\/zhuoyao.net\/index.php\/2015\/02\/16\/text-mining-the-complete-works-of-william-shakespeare-2\/","title":{"rendered":"Text Mining the Complete Works of William Shakespeare"},"content":{"rendered":"

I am starting a new project that will require some serious text mining. So, in the interests of bringing myself up to speed on the tm package, I thought I would apply it to the Complete Works of William Shakespeare and just see what falls out.<\/p>\n

The first order of business was getting my hands on all that text. Fortunately it is available from a number of sources. I chose to use Project Gutenberg<\/a>.<\/p>\n

\n
\n\n\n\n
\n
\n
> TEXTFILE = <\/code>\"data\/pg100.txt\"<\/code><\/div>\n
> <\/code>if <\/code>(!<\/code>file.exists<\/code>(TEXTFILE)) {<\/code><\/div>\n
+\u00a0\u00a0\u00a0\u00a0 <\/code>download.file<\/code>(<\/code>\"http:\/\/www.gutenberg.org\/cache\/epub\/100\/pg100.txt<\/a>\"<\/code>, destfile = TEXTFILE)<\/code><\/div>\n
+ }<\/code><\/div>\n
> shakespeare = <\/code>readLines<\/code>(TEXTFILE)<\/code><\/div>\n
> <\/code>length<\/code>(shakespeare)<\/code><\/div>\n
[1] 124787<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

That\u2019s quite a solid chunk of data: 124787 lines. Let\u2019s take a closer look.<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>head<\/code>(shakespeare)<\/code><\/div>\n
[1] <\/code>\"The Project Gutenberg EBook of The Complete Works of William Shakespeare, by\"<\/code><\/div>\n
[2] <\/code>\"William Shakespeare\"<\/code><\/div>\n
[3] <\/code>\"\"<\/code><\/div>\n
[4] <\/code>\"This eBook is for the use of anyone anywhere at no cost and with\"<\/code><\/div>\n
[5] <\/code>\"almost no restrictions whatsoever.\u00a0 You may copy it, give it away or\"<\/code><\/div>\n
[6] <\/code>\"re-use it under the terms of the Project Gutenberg License included\"<\/code><\/div>\n
> <\/code>tail<\/code>(shakespeare)<\/code><\/div>\n
[1] <\/code>\"http:\/\/www.gutenberg.org\/2\/4\/6\/8\/24689<\/a>\"<\/code>\u00a0\u00a0\u00a0 \"\"<\/code><\/div>\n
[3] <\/code>\"An alternative method of locating eBooks:\"<\/code> \"http:\/\/www.gutenberg.org\/GUTINDEX.ALL<\/a>\"<\/code><\/div>\n
[5] <\/code>\"\"<\/code>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \"*** END: FULL LICENSE ***\"<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

There seems to be some header and footer text. We will want to get rid of that! Using a text editor I checked to see how many lines were occupied with metadata and then removed them before concatenating all of the lines into a single long, long, long string.<\/p>\n

\n
\n\n\n\n
\n
\n
> shakespeare = shakespeare[-(1:173)]<\/code><\/div>\n
> shakespeare = shakespeare[-(124195:<\/code>length<\/code>(shakespeare))]<\/code><\/div>\n
><\/code><\/div>\n
> shakespeare = <\/code>paste<\/code>(shakespeare, collapse = <\/code>\" \"<\/code>)<\/code><\/div>\n
> <\/code>nchar<\/code>(shakespeare)<\/code><\/div>\n
[1] 5436541<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

While I had the text open in the editor I noticed that sections in the document were separated by the following text:<\/p>\n

\n
\n\n\n\n
\n
\n
<<THIS ELECTRONIC VERSION OF THE COMPLETE WORKS OF WILLIAM<\/code><\/div>\n
SHAKESPEARE IS COPYRIGHT 1990-1993 BY WORLD LIBRARY, INC., AND IS<\/code><\/div>\n
PROVIDED BY PROJECT GUTENBERG ETEXT OF ILLINOIS BENEDICTINE COLLEGE<\/code><\/div>\n
WITH PERMISSION.\u00a0 ELECTRONIC AND MACHINE READABLE COPIES MAY BE<\/code><\/div>\n
DISTRIBUTED SO LONG AS SUCH COPIES (1) ARE FOR YOUR OR OTHERS<\/code><\/div>\n
PERSONAL USE ONLY, AND (2) ARE NOT DISTRIBUTED OR USED<\/code><\/div>\n
COMMERCIALLY.\u00a0 PROHIBITED COMMERCIAL DISTRIBUTION INCLUDES BY ANY<\/code><\/div>\n
SERVICE THAT CHARGES FOR DOWNLOAD TIME OR FOR MEMBERSHIP.>><\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Obviously that is going to taint the analysis. But it also serves as a convenient marker to divide that long, long, long string into separate documents.<\/p>\n

\n
\n\n\n\n
\n
\n
> shakespeare = <\/code>strsplit<\/code>(shakespeare, <\/code>\"<<[^>]*>>\"<\/code>)[[1]]<\/code><\/div>\n
> <\/code>length<\/code>(shakespeare)<\/code><\/div>\n
[1] 218<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

This left me with a list of 218 documents. On further inspection, some of them appeared to be a little on the short side (in my limited experience, the bard is not known for brevity). As it turns out, the short documents were the dramatis personae<\/a> for his plays. I removed them as well.<\/p>\n

\n
\n\n\n\n
\n
\n
> (dramatis.personae <- <\/code>grep<\/code>(<\/code>\"Dramatis Personae\"<\/code>, shakespeare, ignore.case = <\/code>TRUE<\/code>))<\/code><\/div>\n
\u00a0<\/code>[1]\u00a0\u00a0 2\u00a0\u00a0 8\u00a0 11\u00a0 17\u00a0 23\u00a0 28\u00a0 33\u00a0 43\u00a0 49\u00a0 55\u00a0 62\u00a0 68\u00a0 74\u00a0 81\u00a0 87\u00a0 93\u00a0 99 105 111 117 122 126 134 140 146 152 158<\/code><\/div>\n
[28] 164 170 176 182 188 194 200 206 212<\/code><\/div>\n
> <\/code>length<\/code>(shakespeare)<\/code><\/div>\n
[1] 218<\/code><\/div>\n
> shakespeare = shakespeare[-dramatis.personae]<\/code><\/div>\n
> <\/code>length<\/code>(shakespeare)<\/code><\/div>\n
[1] 182<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Down to 182 documents, each of which is a complete work.<\/p>\n

The next task was to convert these documents into a corpus<\/a>.<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>library<\/code>(tm)<\/code><\/div>\n
> <\/code><\/div>\n
> doc.vec <- <\/code>VectorSource<\/code>(shakespeare)<\/code><\/div>\n
> doc.corpus <- <\/code>Corpus<\/code>(doc.vec)<\/code><\/div>\n
> <\/code>summary<\/code>(doc.corpus)<\/code><\/div>\n
A corpus with 182 text documents<\/code><\/div>\n
<\/div>\n
The metadata consists of 2 tag-value pairs and a data frame<\/code><\/div>\n
Available tags are:<\/code><\/div>\n
\u00a0\u00a0<\/code>create_date creator <\/code><\/div>\n
Available variables <\/code>in<\/code> the data frame are:<\/code><\/div>\n
\u00a0\u00a0<\/code>MetaID<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

There is a lot of information in those documents which is not particularly useful for text mining. So before proceeding any further, we will clean things up a bit. First we convert all of the text to lowercase and then remove punctuation, numbers and common English stopwords. Possibly the list of English stop words is not entirely appropriate for Shakespearean English, but it is a reasonable starting point.<\/p>\n

\n
\n\n\n\n
\n
\n
> doc.corpus <- <\/code>tm_map<\/code>(doc.corpus, tolower)<\/code><\/div>\n
> doc.corpus <- <\/code>tm_map<\/code>(doc.corpus, removePunctuation)<\/code><\/div>\n
> doc.corpus <- <\/code>tm_map<\/code>(doc.corpus, removeNumbers)<\/code><\/div>\n
> doc.corpus <- <\/code>tm_map<\/code>(doc.corpus, removeWords, <\/code>stopwords<\/code>(<\/code>\"english\"<\/code>))<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Next we perform stemming<\/a>, which removes affixes from words (so, for example, \u201crun\u201d, \u201cruns\u201d and \u201crunning\u201d all become \u201crun\u201d).<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>library<\/code>(SnowballC)<\/code><\/div>\n
><\/code><\/div>\n
> doc.corpus <- <\/code>tm_map<\/code>(doc.corpus, stemDocument)<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

All of these transformations have resulted in a lot of whitespace, which is then removed.<\/p>\n

\n
\n\n\n\n
\n
\n
> doc.corpus <- <\/code>tm_map<\/code>(doc.corpus, stripWhitespace)<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

If we have a look at what\u2019s left, we find that it\u2019s just the lowercase, stripped down version of the text (which I have truncated here).<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>inspect<\/code>(doc.corpus[8])<\/code><\/div>\n
A corpus with 1 text document<\/code><\/div>\n
<\/div>\n
The metadata consists of 2 tag-value pairs and a data frame<\/code><\/div>\n
Available tags are:<\/code><\/div>\n
\u00a0\u00a0<\/code>create_date creator <\/code><\/div>\n
Available variables <\/code>in<\/code> the data frame are:<\/code><\/div>\n
\u00a0\u00a0<\/code>MetaID <\/code><\/div>\n
<\/div>\n
[[1]]<\/code><\/div>\n
\u00a0<\/code>act ii scene messina pompey hous enter pompey menecr mena warlik manner pompey great god just shall<\/code><\/div>\n
\u00a0<\/code>assist deed justest men menecr know worthi pompey delay deni pompey <\/code>while<\/code> suitor throne decay thing<\/code><\/div>\n
\u00a0<\/code>sue menecr ignor beg often harm wise powr deni us good find profit lose prayer pompey shall well<\/code><\/div>\n
\u00a0<\/code>peopl love sea mine power crescent augur hope say will come th full mark antoni egypt sit dinner<\/code><\/div>\n
\u00a0<\/code>will make war without door caesar get money lose heart lepidus flatter flatterd neither love either<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

This is where things start to get interesting. Next we create a Term Document Matrix<\/a> (TDM) which reflects the number of times each word in the corpus is found in each of the documents.<\/p>\n

\n
\n\n\n\n
\n
\n
> TDM <- <\/code>TermDocumentMatrix<\/code>(doc.corpus)<\/code><\/div>\n
> TDM<\/code><\/div>\n
A term-document <\/code>matrix <\/code>(18651 terms, 182 documents)<\/code><\/div>\n
<\/div>\n
Non-\/sparse entries: 182898\/3211584<\/code><\/div>\n
Sparsity\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 : 95%<\/code><\/div>\n
Maximal term length: 31 <\/code><\/div>\n
Weighting\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 : term <\/code>frequency <\/code>(tf)<\/code><\/div>\n
> <\/code>inspect<\/code>(TDM[1:10,1:10])<\/code><\/div>\n
A term-document <\/code>matrix <\/code>(10 terms, 10 documents)<\/code><\/div>\n
<\/div>\n
Non-\/sparse entries: 1\/99<\/code><\/div>\n
Sparsity\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 : 99%<\/code><\/div>\n
Maximal term length: 9 <\/code><\/div>\n
Weighting\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 : term <\/code>frequency <\/code>(tf)<\/code><\/div>\n
<\/div>\n
\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>Docs<\/code><\/div>\n
Terms\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1 2 3 4 5 6 7 8 9 10<\/code><\/div>\n
\u00a0\u00a0<\/code>aaron\u00a0\u00a0\u00a0\u00a0 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abaissiez 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abandon\u00a0\u00a0 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abandond\u00a0 0 1 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abas\u00a0\u00a0\u00a0\u00a0\u00a0 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abashd\u00a0\u00a0\u00a0 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abat\u00a0\u00a0\u00a0\u00a0\u00a0 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abatfowl\u00a0 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abbess\u00a0\u00a0\u00a0 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>abbey\u00a0\u00a0\u00a0\u00a0 0 0 0 0 0 0 0 0 0\u00a0 0<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

The extract from the TDM shows, for example, that the word \u201cabandond\u201d occurred once in document number 2 but was not present in any of the other first ten documents. We could have generated the transpose of the DTM as well.<\/p>\n

\n
\n\n\n\n
\n
\n
> DTM <- <\/code>DocumentTermMatrix<\/code>(doc.corpus)<\/code><\/div>\n
> <\/code>inspect<\/code>(DTM[1:10,1:10])<\/code><\/div>\n
A document-term <\/code>matrix <\/code>(10 documents, 10 terms)<\/code><\/div>\n
<\/div>\n
Non-\/sparse entries: 1\/99<\/code><\/div>\n
Sparsity\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 : 99%<\/code><\/div>\n
Maximal term length: 9 <\/code><\/div>\n
Weighting\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 : term <\/code>frequency <\/code>(tf)<\/code><\/div>\n
<\/div>\n
\u00a0\u00a0\u00a0\u00a0<\/code>Terms<\/code><\/div>\n
Docs aaron abaissiez abandon abandond abas abashd abat abatfowl abbess abbey<\/code><\/div>\n
\u00a0\u00a0<\/code>1\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>2\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>3\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>4\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>5\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>6\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>7\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>8\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>9\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>10\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 0<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Which of these proves to be most convenient will depend on the relative number of documents and terms in your data.<\/p>\n

Now we can start asking questions like: what are the most frequently occurring terms?<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>findFreqTerms<\/code>(TDM, 2000)<\/code><\/div>\n
\u00a0<\/code>[1] <\/code>\"come\"<\/code>\u00a0 \"enter\"<\/code> \"good\"<\/code>\u00a0 \"king\"<\/code>\u00a0 \"let\"<\/code>\u00a0\u00a0 \"lord\"<\/code>\u00a0 \"love\"<\/code>\u00a0 \"make\"<\/code>\u00a0 \"man\"<\/code>\u00a0\u00a0 \"now\"<\/code>\u00a0\u00a0 \"shall\"<\/code> \"sir\"<\/code>\u00a0\u00a0 \"thee\"<\/code><\/div>\n
[14] <\/code>\"thi\"<\/code>\u00a0\u00a0 \"thou\"<\/code>\u00a0 \"well\"<\/code>\u00a0 \"will\"<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Each of these words occurred more that 2000 times.<\/p>\n

What about associations between words? Let\u2019s have a look at what other words had a high association with \u201clove\u201d.<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>findAssocs<\/code>(TDM, <\/code>\"love\"<\/code>, 0.8)<\/code><\/div>\n
beauti\u00a0\u00a0\u00a0 eye<\/code><\/div>\n
\u00a0\u00a0<\/code>0.83\u00a0\u00a0 0.80<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Well that\u2019s not too surprising!<\/p>\n

From our first look at the TDM we know that there are many terms which do not occur very often. It might make sense to simply remove these sparse terms from the analysis.<\/p>\n

\n
\n\n\n\n
\n
\n
> TDM.common = <\/code>removeSparseTerms<\/code>(TDM, 0.1)<\/code><\/div>\n
> <\/code>dim<\/code>(TDM)<\/code><\/div>\n
[1] 18651\u00a0\u00a0 182<\/code><\/div>\n
> <\/code>dim<\/code>(TDM.common)<\/code><\/div>\n
[1]\u00a0 71 182<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

From the 18651 terms that we started with, we are now left with a TDM which considers on 71 commonly occurring terms.<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>inspect<\/code>(TDM.common[1:10,1:10])<\/code><\/div>\n
A term-document <\/code>matrix <\/code>(10 terms, 10 documents)<\/code><\/div>\n
<\/div>\n
Non-\/sparse entries: 94\/6<\/code><\/div>\n
Sparsity\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 : 6%<\/code><\/div>\n
Maximal term length: 6<\/code><\/div>\n
Weighting\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 : term <\/code>frequency <\/code>(tf)<\/code><\/div>\n
<\/div>\n
\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/code>Docs<\/code><\/div>\n
Terms\u00a0\u00a0\u00a0\u00a0 1 2\u00a0 3\u00a0 4\u00a0 5\u00a0 6\u00a0 7\u00a0 8 9 10<\/code><\/div>\n
\u00a0\u00a0<\/code>act\u00a0\u00a0\u00a0\u00a0 1 4\u00a0 7\u00a0 9\u00a0 6\u00a0 3\u00a0 2 14 1\u00a0 0<\/code><\/div>\n
\u00a0\u00a0<\/code>art\u00a0\u00a0\u00a0 53 0\u00a0 9\u00a0 3\u00a0 5\u00a0 3\u00a0 2 17 0\u00a0 6<\/code><\/div>\n
\u00a0\u00a0<\/code>away\u00a0\u00a0 18 5\u00a0 8\u00a0 4\u00a0 2 10\u00a0 5 13 1\u00a0 7<\/code><\/div>\n
\u00a0\u00a0<\/code>call\u00a0\u00a0 17 1\u00a0 4\u00a0 2\u00a0 2\u00a0 1\u00a0 6 17 3\u00a0 7<\/code><\/div>\n
\u00a0\u00a0<\/code>can\u00a0\u00a0\u00a0 44 8 12\u00a0 5 10\u00a0 6 10 24 1\u00a0 5<\/code><\/div>\n
\u00a0\u00a0<\/code>come\u00a0\u00a0 19 9 16 17 12 15 14 89 9 15<\/code><\/div>\n
\u00a0\u00a0<\/code>day\u00a0\u00a0\u00a0 43 2\u00a0 2\u00a0 4\u00a0 1\u00a0 5\u00a0 3 17 2\u00a0 3<\/code><\/div>\n
\u00a0\u00a0<\/code>enter\u00a0\u00a0 0 7 12 11 10 10 14 87 4\u00a0 6<\/code><\/div>\n
\u00a0\u00a0<\/code>exeunt\u00a0 0 3\u00a0 8\u00a0 8\u00a0 5\u00a0 4\u00a0 7 49 1\u00a0 4<\/code><\/div>\n
\u00a0\u00a0<\/code>exit\u00a0\u00a0\u00a0 0 6\u00a0 8\u00a0 5\u00a0 6\u00a0 5\u00a0 3 31 3\u00a0 2<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

Finally we are going to put together a visualisation. The TDM is stored as asparse matrix<\/a>. This was an apt representation for the initial TDM, but the reduced TDM containing only frequently occurring words is probably better stored as a normal matrix. We\u2019ll make the conversion and see.<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>library<\/code>(slam)<\/code><\/div>\n
><\/code><\/div>\n
> TDM.dense <- <\/code>as.matrix<\/code>(TDM.common)<\/code><\/div>\n
><\/code><\/div>\n
> TDM.dense<\/code><\/div>\n
> <\/code>object.size<\/code>(TDM.common)<\/code><\/div>\n
207872 bytes<\/code><\/div>\n
> <\/code>object.size<\/code>(TDM.dense)<\/code><\/div>\n
112888 bytes<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

So, as it turns out the sparse representation was actually wasting space! (This will generally not be true though: it will only apply for a matrix consisting of just the common terms). Anyway, we need the data as a normal matrix in order to produce the visualisation. The next step is to convert it into a tidy format.<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>library<\/code>(reshape2)<\/code><\/div>\n
><\/code><\/div>\n
> TDM.dense = <\/code>melt<\/code>(TDM.dense, value.name = <\/code>\"count\"<\/code>)<\/code><\/div>\n
> <\/code>head<\/code>(TDM.dense)<\/code><\/div>\n
\u00a0\u00a0<\/code>Terms Docs count<\/code><\/div>\n
1\u00a0\u00a0 act\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0\u00a0 1<\/code><\/div>\n
2\u00a0\u00a0 art\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0 53<\/code><\/div>\n
3\u00a0 away\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0 18<\/code><\/div>\n
4\u00a0 call\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0 17<\/code><\/div>\n
5\u00a0\u00a0 can\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0 44<\/code><\/div>\n
6\u00a0 come\u00a0\u00a0\u00a0 1\u00a0\u00a0\u00a0 19<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

And finally generate the visualisation.<\/p>\n

\n
\n\n\n\n
\n
\n
> <\/code>library<\/code>(ggplot2)<\/code><\/div>\n
><\/code><\/div>\n
> <\/code>ggplot<\/code>(TDM.dense, <\/code>aes<\/code>(x = Docs, y = Terms, fill = <\/code>log10<\/code>(count))) +<\/code><\/div>\n
+\u00a0\u00a0\u00a0\u00a0 <\/code>geom_tile<\/code>(colour = <\/code>\"white\"<\/code>) +<\/code><\/div>\n
+\u00a0\u00a0\u00a0\u00a0 <\/code>scale_fill_gradient<\/code>(high=<\/code>\"#FF0000\"<\/code> , low=<\/code>\"#FFFFFF\"<\/code>)+<\/code><\/div>\n
+\u00a0\u00a0\u00a0\u00a0 <\/code>ylab<\/code>(<\/code>\"\"<\/code>) +<\/code><\/div>\n
+\u00a0\u00a0\u00a0\u00a0 <\/code>theme<\/code>(panel.background = <\/code>element_blank<\/code>()) +<\/code><\/div>\n
+\u00a0\u00a0\u00a0\u00a0 <\/code>theme<\/code>(axis.text.x = <\/code>element_blank<\/code>(), axis.ticks.x = <\/code>element_blank<\/code>())<\/code><\/div>\n<\/div>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n

\"shakespeare-common-tdm\"<\/a><\/p>\n

The colour scale indicates the number of times that each of the terms cropped up in each of the documents. I applied a logarithmic transform to the counts since there was a very large disparity in the numbers across terms and documents. The grey tiles correspond to terms which are not found in the corresponding document.<\/p>\n

One can see that some terms, like \u201cwill\u201d turn up frequently in most documents, while \u201clove\u201d is common in some and rare or absent in others.<\/p>\n

That was interesting. Not sure that I would like to make any conclusions on the basis of the results above (Shakespeare is well outside my field of expertise!), but I now have a pretty good handle on how the tm package works. As always, feedback will be appreciated!<\/p>\n

References<\/h1>\n