Blog

This blog post will explain how to extract noun phrases from a sentence with python. This is useful when text needs to be analyzed. Keywords, entities, and concepts can be extracted from noun phrases. Information extracted from noun phrases are the main topics of a document. Extracting noun phrases helps identify frequently mentioned information that is useful for text summarization and the extraction of keywords. Search engines use noun phrases to index documents using information extracted out of noun phrases. A search engine can be optimized because noun phrases can mean more than individual words. Data analysis is improved because counting noun phrases helps point out how often a certain one appears over time. Named entity recognition is improved by extracting noun phrases. NER deals with proper nouns while noun phrases contain important terms from a body of text. It can be combined with NER to make it more effective. Detecting recurring noun phrases in a document helps with tagging and classification of it. Some noun phrases can be used for sentiment analysis also.

From the nltk library, pos_tag, and RegexpParser need to be imported. The re library has to be imported also. They are for part of speech tagging the sample text, breaking down the sample text into a list of words or tokens, and parsing the part of speech tagged sentence.

import re
from nltk import pos_tag, RegexpParser

This is the sample text that will be used for this example.

sample = "i saw the big dog on the hill"

Tokenize the sample text.

words = re.sub(r'[^\w\s]', '', sample).lower().split(' ')

Determine each words part of speech tag.

tagged = pos_tag(words)

The regexpparser uses a rule constructed using regular expressions.

chunker = RegexpParser("""
    NP: {<DT>?<JJ.*>*<NN.*>+}
""")

Declare an empty list to hold noun phrases extracted from the sample.

noun_phrases = []

Create a tree based on the rules of the regexpparser.

tree = chunker.parse(tagged)

Traverse the tree and construct noun phrases with tag info extracted from the sample.

for subtree in tree.subtrees():
    if subtree.label() == 'NP':
        np = " ".join(word for word, pos in subtree.leaves())
        noun_phrases.append(np)

Output results.

for np in noun_phrases:
    print(np)

This is what the whole source code looks like.

import re
from nltk import pos_tag, RegexpParser
 
sample = "i saw the big dog on the hill"
 
words = re.sub(r'[^\w\s]', '', sample).lower().split(' ')

tagged = pos_tag(words)
 
chunker = RegexpParser("""
    NP: {<DT>?<JJ.*>*<NN.*>+}
""")

noun_phrases = []

tree = chunker.parse(tagged)

for subtree in tree.subtrees():
    if subtree.label() == 'NP':
        np = " ".join(word for word, pos in subtree.leaves())
        noun_phrases.append(np)

for np in noun_phrases:
    print(np)