Blog

This blog post will explain how to create a simple teachable chatbot with Python. A chatbot is an application that is meant to converse with a user using text. The concept of chatbots have existed for decades and many simple ones were developed during that time. Chatbots have long been used as virtual assistants and for customer support. However, chatbots have been improved with the concept of deep learning during the 2020s. Generative AI chatbots use large language models to generate replies to queries from a user. The example chatbot that is talked about in this blog post does not use generative AI to generate replies. Only simple rules are used to generate replies.

First, create a file named memory.json and place the contents below in the file. That file will store the rules that generate a reply to a user query.

[
  [
    "Hello, how's it going?",
    [
      "hello",
      "hi",
      "hey"
    ]
  ],
  [
    "I am a few days old.",
    [
      "age",
      "old"
    ]
  ]
]

The re library has to be imported to use a regular expression that helps tokenize user queries. The json library has to be imported to read and save to a json file that holds the rules the chatbot uses to generate replies to a user query. From the nltk library, the stopwords library has to be imported to construct the rules the chatbot uses.

import re
import json
from nltk.corpus import stopwords

The variable tokens is set a few functions that strip punuaction, and lowercase the string. After that, the string is converted into a list of tokens using the split function. The variable stop_words is set to english stopwords. A variable named filtered_tokens is declared as an empty list. A for loop will traverse the tokens list. If the current token in the loop is not a stopword, it is appended to filtered_tokens. After the loop is done, the function will return the list filtered_tokens.

def preprocess(text):
    tokens = re.sub(r'[^\w\s]', '', text).lower().split(' ')
    stop_words = stopwords.words("english")
    filtered_tokens = []
    for t in tokens:
        if t not in stop_words:
            filtered_tokens.append(t)
    return filtered_tokens

There needs to be a function that takes the tokenized sentence as input and searches for a certain keyword within it. There is a loop that traverses the keywords array, each keyword list is processed. The ismember function will take the contents of keyword[1] and compare it to the list of tokens. If there is a match, keyword[0] will be returned by the function.

def query(tokens):
    for keyword in keywords:
        if ismember(tokens,keyword[1]):
            return keyword[0]
    return "unknown"

There needs to be a function defined to search for keywords in a tokenized sentence.

def ismember(tokens,keywords):
    for token in tokens:
        if token in keywords:
            return True
    return False

A json file which serves as the chatbots memory is opened and its contents made equal to the variable keywords.

with open("memory.json", "r") as f:
    keywords = json.load(f)

An infinite loop is declared. Input is taken from the user. If the user inputs exit, the infinite loop breaks. The input has punctuation stripped from it, lowercased, and split into a list of tokens. Then the stop words are filtered out of the list of tokens. The list of tokens is processed by the query function. If no result for the query can be found can be found, there is an option to input a response to the query. That rule is appended to the keywords list. Then the loop is started back over. If there is a result found, it is printed to the screen.

while True:
    sentence = input("QUERY>>>")
    if sentence == "exit":
        break
    tokens = preprocess(sentence)
    action = query(tokens)
    
    if action == "unknown":
        print("How should I answer that query?")
        phrase = input("INPUT>>>")
        keywords.append([phrase,tokens])
        continue

    print(action)

The contents of the list keywords is saved to the json file.

with open("memory.json", "w") as f:
    json.dump(keywords, f, indent=2)

This is what the whole source code looks like.

import re
import json
from nltk.corpus import stopwords

def preprocess(text):
    tokens = re.sub(r'[^\w\s]', '', text).lower().split(' ')
    stop_words = stopwords.words("english")
    filtered_tokens = []
    for t in tokens:
        if t not in stop_words:
            filtered_tokens.append(t)
    return filtered_tokens

def query(tokens):
    for keyword in keywords:
        if ismember(tokens,keyword[1]):
            return keyword[0]
    return "unknown"

def ismember(tokens,keywords):
    for token in tokens:
        if token in keywords:
            return True
    return False

with open("memory.json", "r") as f:
    keywords = json.load(f)

while True:
    sentence = input("QUERY>>>")
    if sentence == "exit":
        break
    tokens = preprocess(sentence)
    action = query(tokens)
    
    if action == "unknown":
        print("How should I answer that query?")
        phrase = input("INPUT>>>")
        keywords.append([phrase,tokens])
        continue

    print(action)
    
with open("memory.json", "w") as f:
    json.dump(keywords, f, indent=2)