Blog

This blog post will explain how to filter out stopwords in a sentence with Python. During text processing stopwords are common words that are filtered out. Those common words have little meaning when trying to understand a sentence. Computational efficiency and analysis can be improved by removing stopwords. That means more meaningful words can be concentrated on. Depending on the NLP task that needs to be accomplished, sometimes it is a good idea to or not to remove stopwords. Text classification uses stopword removal to concentrate on words that contribute to the meaning of a sentence. For machine translation and summarization, stopwords are not removed to preserve the meaning of a sentence.

The stopwords library needs to be imported from nltk to help with filtering out stopwords. The re library needs to be imported so the sub function can be used to eliminate punuaction from the sentence during tokenization.

from nltk.corpus import stopwords
import re

The variable tokens is set a few functions that strip punuaction, and lowercase the string. After that, the string is converted into a list of tokens using the split function. The variable stop_words is set to english stopwords. A variable named filtered_tokens is declared as an empty list. A for loop will traverse the tokens list. If the current token in the loop is not a stopword, it is appended to filtered_tokens. After the loop is done, the function will return the list filtered_tokens.

def preprocess(text):
    tokens = re.sub(r'[^\w\s]', '', text).lower().split(' ')
    stop_words = stopwords.words("english")
    filtered_tokens = []
    for t in tokens:
        if t not in stop_words:
            filtered_tokens.append(t)
    return filtered_tokens

A string named text is declared. It will hold the sentence that gets processed by the preprocess function.

text = "Hello, how is it going?"

The variable r holds the result of the preprocess function.

r = preprocess(text)

The contents of r is printed to the screen.

print(r)

This is what the whole source code looks like.

from nltk.corpus import stopwords
import re

def preprocess(text):
    tokens = re.sub(r'[^\w\s]', '', text).lower().split(' ')
    stop_words = stopwords.words("english")
    filtered_tokens = []
    for t in tokens:
        if t not in stop_words:
            filtered_tokens.append(t)
    return filtered_tokens

text = "Hello, how is it going?"

r = preprocess(text)

print(r)