Utilizing NLTK on Twitter Data

Purpose

In my attempt to improve my competency with Natural Language Processing, I’ve been working on a mini-project in which I obtain insights from tweets. This will help develop my NLP skillset and enhances my abilities in mining unstructured data and obtain insights as a result.

Process

Just a refresher from my past blog. I obtained tweets from the Twitter API using a query. In this particular example, I queried the most recent tweets containing the key word WTI. This acronym stands for West Texas Intermediate the most common US oil commodity. Restraining the query to such a narrow scope will hopefully allow me to focus in on a few key expressions that will help group like tweets together.

Example of text data available

As you can see, there is a bunch of clutter incorporated within the tweet. Whether it be @ symbols, hashtags, or punctuation. There is also a significant amount of words that have no embedded meaning. These will be referred to as stopwords. There will be a number of transformations needed to clean up this raw data.

These are a few of the methods that I’ll use to clean up the tweets.

First off, “word_tokenize,” this allows us to split up the text so we can look into which strings are good, and which ones we should dump.

For the same tweet, I’m looking at whether the word in the list is in the predefined stopwords group. If it does, we do not include it in the refined list.

The next line of code checks to see if the string is numeric or actual letters. If it is the latter, it will remain in the list.

Looking at the cleaned up product its not perfect, but it is improved significantly. The next improvement that could be made is to apply stemming or lemming normalize words that have similar endings or different parts of speech. For example:

The Porter Stemmer transform tool when applied to various different types of the word reverts it to the root. Sometimes its functionality is a little off, but as long as it stays consistent, the effects on our clustering will be limited. There are few other stemming or lemming transformations that work in differently, but are unnecessary at this time.

Here I’m packaging the previous methods into 1 function so I can apply it to a list of words more easily.

And there we have it!

Leave a Reply

Create your own website, free on WordPress.com.Create your own website.
Start free