{"id":741,"date":"2015-02-13T10:46:59","date_gmt":"2015-02-13T17:46:59","guid":{"rendered":"http:\/\/homepages.uc.edu\/~yaozo\/wordpress\/?p=741"},"modified":"2015-02-13T10:46:59","modified_gmt":"2015-02-13T17:46:59","slug":"pdf-2-text-or-csv-r","status":"publish","type":"post","link":"https:\/\/zhuoyao.net\/index.php\/2015\/02\/13\/pdf-2-text-or-csv-r\/","title":{"rendered":"PDF-2-text-or-CSV.r"},"content":{"rendered":"
# Here are a few methods for getting text from PDF files. Do read through \n<\/span># the instructions carefully! NOte that this code is written for Windows 7,\n# slight adjustments may be needed for other OSs\n# Tell R what folder contains your 1000s of PDFs\ndest<\/span> <-<\/span> \"<\/span>G:\/somehere\/with\/many\/PDFs\"\n<\/span><\/span># make a vector of PDF file names\nmyfiles<\/span> <-<\/span> list.files(path<\/span> =<\/span> dest<\/span>, pattern<\/span> =<\/span> \"<\/span>pdf\"<\/span><\/span>, full.names<\/span> =<\/span> TRUE<\/span>)\n\n# now there are a few options...\n############### PDF (image of text format) to TXT ##########\n# This is for is your PDF is an image of text, this is the case\n# if you open the PDF in a PDF viewer and you cannot select\n# words or lines with your cursor.\n\n##### Wait! #####\n# Before proceeding, make sure you have a copy of Tesseract\n# on your computer! Details & download:\n# https:\/\/code.google.com\/p\/tesseract-ocr\/\n# and a copy of ImageMagick: http:\/\/www.imagemagick.org\/\n# and a copy of pdftoppm on your computer! \n# Download: http:\/\/www.foolabs.com\/xpdf\/download.html\n# And then after installing those three, restart to \n# ensure R can find them on your path. \n# And note that this process can be quite slow...\n# PDF filenames can't have spaces in them for these operations\n# so let's get rid of the spaces in the filenames\nsapply(myfiles<\/span>, FUN<\/span> =<\/span> function<\/span>(i<\/span>){file.rename(from<\/span> =<\/span> i<\/span>, to<\/span> =<\/span> paste0(dirname(i<\/span>), \"<\/span>\/\"<\/span><\/span>, gsub(\"<\/span> \"<\/span><\/span>, \"<\/span>\"<\/span><\/span>, basename(i<\/span>))))})\n# get the PDF file names without spaces\nmyfiles<\/span> <-<\/span> list.files(path<\/span> =<\/span> dest<\/span>, pattern<\/span> =<\/span> \"<\/span>pdf\"<\/span><\/span>, full.names<\/span> =<\/span> TRUE<\/span>)\n# Now we can do the OCR to the renamed PDF files. Don't worry\n# if you get messages like 'Config Error: No display \n# font for...' it's nothing to worry about\nlapply(myfiles<\/span>, function<\/span>(i<\/span>){\n# convert pdf to ppm (an image format), just pages 1-10 of the PDF\n# but you can change that easily, just remove or edit the \n# -f 1 -l 10 bit in the line below\nshell(shQuote(paste0(\"<\/span>pdftoppm \"<\/span><\/span>, i<\/span>, \"<\/span> -f 1 -l 10 -r 600 ocrbook\"<\/span><\/span>)))\n# convert ppm to tif ready for tesseract\nshell(shQuote(paste0(\"<\/span>convert *.ppm \"<\/span><\/span>, i<\/span>, \"<\/span>.tif\"<\/span><\/span>)))\n# convert tif to text file\nshell(shQuote(paste0(\"<\/span>tesseract \"<\/span><\/span>, i<\/span>, \"<\/span>.tif \"<\/span><\/span>, i<\/span>, \"<\/span> -l eng\"<\/span><\/span>)))\n# delete tif file\nfile.remove(paste0(i<\/span>, \"<\/span>.tif\"<\/span><\/span> ))\n})\n# where are the txt files you just made?\ndest<\/span> # in this folder\n<\/span># And now you're ready to do some text mining on the text files\n############### PDF (text format) to TXT ###################\n\n\n##### Wait! #####\n# Before proceeding, make sure you have a copy of pdf2text\n# on your computer! Details: https:\/\/en.wikipedia.org\/wiki\/Pdftotext\n# Download: http:\/\/www.foolabs.com\/xpdf\/download.html\n# If you have a PDF with text, ie you can open the PDF in a \n# PDF viewer and select text with your curser, then use these \n# lines to convert each PDF file that is named in the vector \n# into text file is created in the same directory as the PDFs\n# note that my pdftotext.exe is in a different location to yours\nlapply(myfiles<\/span>, function<\/span>(i<\/span>) system(paste('<\/span>\"C:\/Program Files\/xpdf\/bin64\/pdftotext.exe\"'<\/span><\/span>, paste0('<\/span>\"'<\/span><\/span>, i<\/span>, '<\/span>\"'<\/span><\/span>)), wait<\/span> =<\/span> FALSE<\/span>) )\n\n# where are the txt files you just made?\ndest<\/span> # in this folder\n<\/span># And now you're ready to do some text mining on the text files\n\n############### PDF to CSV (DfR format) ####################\n# or if you want DFR-style csv files...\n# read txt files into R\nmytxtfiles<\/span> <-<\/span> list.files(path<\/span> =<\/span> dest<\/span>, pattern<\/span> =<\/span> \"<\/span>txt\"<\/span><\/span>, full.names<\/span> =<\/span> TRUE<\/span>)\nlibrary(tm<\/span>)\nmycorpus<\/span> <-<\/span> Corpus(DirSource(dest<\/span>, pattern<\/span> =<\/span> \"<\/span>txt\"<\/span><\/span>))\n# warnings may appear after you run the previous line, they\n# can be ignored\nmycorpus<\/span> <-<\/span> tm_map(mycorpus<\/span>, removeNumbers<\/span>)\nmycorpus<\/span> <-<\/span> tm_map(mycorpus<\/span>, removePunctuation<\/span>)\nmycorpus<\/span> <-<\/span> tm_map(mycorpus<\/span>, stripWhitespace<\/span>)\nmydtm<\/span> <-<\/span> DocumentTermMatrix(mycorpus<\/span>)\n# remove some OCR weirdness\n# words with more than 2 consecutive characters\nmydtm<\/span> <-<\/span> mydtm<\/span>[,!<\/span>grepl(\"<\/span>(.)\\\\<\/span>1{2,}\"<\/span><\/span>, mydtm<\/span>$<\/span>dimnames<\/span>$<\/span>Terms<\/span>)]\n\n# get each doc as a csv with words and counts\nfor<\/span>(i<\/span> in<\/span> 1<\/span>:<\/span>nrow(mydtm<\/span>)){\n# get word counts\ncounts<\/span> <-<\/span> as.vector(as.matrix(mydtm<\/span>[1<\/span>,]))\n# get words\nwords<\/span> <-<\/span> mydtm<\/span>$<\/span>dimnames<\/span>$<\/span>Terms\n<\/span># combine into data frame\ndf<\/span> <-<\/span> data.frame<\/span>(word<\/span> =<\/span> words<\/span>, count<\/span> =<\/span> counts<\/span>,stringsAsFactors<\/span> =<\/span> FALSE<\/span>)\n# exclude words with count of zero\ndf<\/span> <-<\/span> df<\/span>[df<\/span>$<\/span>count<\/span> !=<\/span> 0<\/span>,]\n# write to CSV with original txt filename\nwrite.csv(df<\/span>, paste0(mydtm<\/span>$<\/span>dimnames<\/span>$<\/span>Docs<\/span>[i<\/span>],\"<\/span>.csv\"<\/span><\/span>), row.names<\/span> =<\/span> FALSE<\/span>)\n}\n\n# and now you're ready to work with the csv files\n\n############### PDF to TXT (all text between two words) ####\n\n# Below is about splitting the text files at certain characters\n## can be skipped...\n\n# if you just want the abstracts, we can use regex to extract that part of\n# each txt file, Assumes that the abstract is always between the words 'Abstract'\n# and 'Introduction'\nabstracts<\/span> <-<\/span> lapply(mytxtfiles<\/span>, function<\/span>(i<\/span>) {\nj<\/span> <-<\/span> paste0(scan(i<\/span>, what<\/span> =<\/span> character<\/span>()), collapse<\/span> =<\/span> \"<\/span> \"<\/span><\/span>)\nregmatches(j<\/span>, gregexpr(\"<\/span>(?<=Abstract).*?(?=Introduction)\"<\/span><\/span>, j<\/span>, perl<\/span>=<\/span>TRUE<\/span>))\n})# Write abstracts into separate txt files...\n# write abstracts as txt files \n# (or use them in the list for whatever you want to do next)\nlapply(1<\/span>:<\/span>length(abstracts<\/span>), function<\/span>(i<\/span>) write.table(abstracts<\/span>[i<\/span>], file<\/span>=<\/span>paste(mytxtfiles<\/span>[i<\/span>], \"<\/span>abstract\"<\/span><\/span>, \"<\/span>txt\"<\/span><\/span>, sep<\/span>=<\/span>\"<\/span>.\"<\/span><\/span>), quote<\/span> =<\/span> FALSE<\/span>, row.names<\/span> =<\/span> FALSE<\/span>, col.names<\/span> =<\/span> FALSE<\/span>, eol<\/span> =<\/span> \"<\/span> \"<\/span><\/span> ))\n# And now you're ready to do some text mining on the txt \n# originally on http:\/\/stackoverflow.com\/a\/21449040\/1036500<\/pre>\n","protected":false},"excerpt":{"rendered":"
# Here are a few methods for getting text from PDF files. Do read through # the instructions carefully! NOte that this code is written… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[],"class_list":["post-741","post","type-post","status-publish","format-standard","hentry","category-r"],"_links":{"self":[{"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/posts\/741","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/comments?post=741"}],"version-history":[{"count":0,"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/posts\/741\/revisions"}],"wp:attachment":[{"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/media?parent=741"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/categories?post=741"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/zhuoyao.net\/index.php\/wp-json\/wp\/v2\/tags?post=741"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}