This blog post will explain how to filter out stopwords in a sentence with Python.

The stopwords library needs to be imported from nltk to help with filtering out stopwords. The re library needs to be imported so the sub function can be used to eliminate punuaction from the sentence during tokenization.

from nltk.corpus import stopwords
import re

The variable tokens is set a few functions that strip punuaction, and lowercase the string. After that, the string is converted into a list of tokens using the split function. The variable stop_words is set to english stopwords. A variable named filtered_tokens is declared as an empty list. A for loop will traverse the tokens list. If the current token in the loop is not a stopword, it is appended to filtered_tokens. After the loop is done, the function will return the list filtered_tokens.

def preprocess(text):
    tokens = re.sub(r'[^\w\s]', '', text).lower().split(' ')
    stop_words = stopwords.words("english")
    filtered_tokens = []
    for t in tokens:
        if t not in stop_words:
            filtered_tokens.append(t)
    return filtered_tokens

A string named text is declared. It will hold the sentence that gets processed by the preprocess function.

text = "Hello, how is it going?"

The variable r holds the result of the preprocess function.

r = preprocess(text)

The contents of r is printed to the screen.

print(r)

This is what the whole source code looks like.

from nltk.corpus import stopwords
import re

def preprocess(text):
    tokens = re.sub(r'[^\w\s]', '', text).lower().split(' ')
    stop_words = stopwords.words("english")
    filtered_tokens = []
    for t in tokens:
        if t not in stop_words:
            filtered_tokens.append(t)
    return filtered_tokens

text = "Hello, how is it going?"

r = preprocess(text)

print(r)