Blog

This blog post will explain how to tokenize a string with Python. Tokenization is a needed process in natural language processing. It means that a string is broken down into a list of tokens. Tokens are usually words, but they can be characters and other types of words also. Those list of words are then processed by an algorithm. Tokens can be inputted into algorithms that generate bigram language models, machine learning models, generating random text, machine translation, and sentiment analysis.

First, the regular expressions library has to be loaded. This is done so that a regular expression can be used to eliminate punctuation from a string.

import re

Then a string variable is declared. This is the string that is going to be tokenized.

s = "How's it going?"

The variable tokens will hold the result of the re.sub function with the functions lower and split appended to it. The re.sub function will strip a string of all punctuation marks. The lower function will turn all characters in the string to lowercase. The split function will convert the string into a list of tokens or words using a space as a delimiter.

tokens = re.sub(r'[^\w\s]', '', s).lower().split(' ')

Then the tokens are printed out.

print(tokens)

This is what the whole code looks like.

import re

s = "How's it going?"
tokens = re.sub(r'[^\w\s]', '', s).lower().split(' ')
print(tokens)