Blog

This blog post will explain how to count noun phrases in a list of sentences with Python. This is useful when text needs to be analyzed. Keywords, entities, and concepts can be extracted from noun phrases. Information extracted from noun phrases are the main topics of a document. Counting noun phrases helps identify frequently mentioned information that is useful for text summarization and the extraction of keywords. Search engines use noun phrase counts to index documents using information extracted out of noun phrases. A search engine can be optimized because noun phrases can mean more than individual words. Data analysis is improved because counting noun phrases helps point out how often a certain one appears over time. Named entity recognition is improved by counting noun phrases. NER deals with proper nouns while noun phrase counting extracts important terms from a body of text. It can be combined with NER to make it more effective. Detecting recurring noun phrases in a document helps with tagging and classification of it. Some noun phrases can be used for sentiment analysis also.

From the nltk library, pos_tag, and RegexpParser need to be imported. The re library has to be imported. From the collections library, Counter needs to be imported. They are for part of speech tagging the sample text, breaking down the sample text into a list of words or tokens, and parsing the part of speech tagged sentence. A counter object will count noun phrases extracted from the sample text.

import re
from nltk import pos_tag, RegexpParser
from collections import Counter

A list of sentences is declared.

sentences = [
    "The pizza is ready.",
    "There are two toppings on it.",
    "The pizza was delivered."
]

A grammar rule is declared that parses a tagged sentence into noun phrases.

grammar = r"""
    NP: {<DT>?<JJ.*>*<NN.*>+}
"""

A chunk parser is created that uses a grammar rule that is useful for extracting noun phrases out of a sentence.

chunk_parser = RegexpParser(grammar)

An empty list is declared that will hold all noun phrases that are extracted out of the sentence.

noun_phrases = []

The list of sentences is processed. Each sentence is tokenized and part of speech tagged. After that, the part of speech tagged sentence is inputted into the chunk parser. Then a tree structure is used for extracting noun phrases from each sentence.

for sentence in sentences:
    words = re.sub(r'[^\w\s]', '', sentence).lower().split(' ')
    tagged = pos_tag(words)
    tree = chunk_parser.parse(tagged)

    for subtree in tree.subtrees():
        if subtree.label() == 'NP':
            np = " ".join(word for word, pos in subtree.leaves())
            noun_phrases.append(np)

A counter object named counts is declared that will hold the counts for all of the noun phrases.

counts = Counter(noun_phrases)

The counter object is traversed and each noun phrase and its count are printed to the screen.

for phrase, count in counts.items():
    print(phrase,count)

This is what the whole source code looks like.

import re
from nltk import pos_tag, RegexpParser
from collections import Counter

sentences = [
    "The pizza is ready.",
    "There are two toppings on it.",
    "The pizza was delivered."
]

grammar = r"""
    NP: {<DT>?<JJ.*>*<NN.*>+}
"""

chunk_parser = RegexpParser(grammar)

noun_phrases = []

for sentence in sentences:
    words = re.sub(r'[^\w\s]', '', sentence).lower().split(' ')
    tagged = pos_tag(words)
    tree = chunk_parser.parse(tagged)

    for subtree in tree.subtrees():
        if subtree.label() == 'NP':
            np = " ".join(word for word, pos in subtree.leaves())
            noun_phrases.append(np)

counts = Counter(noun_phrases)

for phrase, count in counts.items():
    print(phrase,count)