NLP Linguistic Analysis — Complete Guide

Table of Contents

1 Introduction to NLTK & NLP Levels Overview
2 Level 1 — Lexical Analysis Tokenization · Stop Words · Stemming · Lemmatization
3 Level 2 — Morphological Analysis Stemming vs Lemmatization
4 Level 3 — Syntactic Analysis POS Tagging · Chunking · Constituency & Dependency Parsing
5 Level 4 — Semantic Analysis (NER)
6 Summary Table & Full Pipeline — Cheat Sheet

Section 1

Introduction & Overview

1.1 What is NLTK?

The Natural Language Toolkit (NLTK) is a free, open-source Python package providing interfaces for text processing, linguistic structure analysis, and access to a large collection of corpora. It is the leading platform for building Python programs to work with human language data.

NLTK provides interfaces to over 50 corpora and lexical resources such as WordNet, along with libraries for:

Tokenization & Stop Word Removal
Stemming & Lemmatization
POS Tagging, Parsing & Chunking
Semantic Analysis & Classification

💡

Tip

Python NLTK Demos available at: http://text-processing.com/demo/

1.2 The Four Levels of NLP Analysis

Level	Name	Description	Key Tasks
1	Lexical	Breaks text into words and cleans it	Tokenization, Stop Words, Lowercasing
2	Morphological	Studies word formation and structure	Stemming, Lemmatization
3	Syntactic	Analyzes grammar and sentence structure	POS Tagging, Chunking, Parsing
4	Semantic	Extracts meaning from text	NER, WSD, Sense Detection

1.3 Full NLP Processing Pipeline

📄

Raw Text

→

🔤

Tokens Lexical

→

🌱

Root Forms Morphological

→

🏷️

POS Tags Syntactic

→

🌳

Structure Syntactic

→

💡

Meaning Semantic

Level 1

Lexical Analysis

Tokenization · Stop Words · Stemming · Lemmatization

📖

Definition

Lexical Analysis is the first stage of NLP. It breaks raw text into meaningful units (tokens) and removes noise (stop words, punctuation) to prepare clean data for further processing.

Required Libraries & Setup

Python — Imports & Setup python

import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords

# Required NLTK downloads (run once)
nltk.download('punkt_tab')   # Tokenizer models
nltk.download('stopwords')   # Stop words corpus

Lexical Analysis Pipeline

Raw Text

→

Sentence Split

→

Word Split

→

Lowercase

→

Remove Stop Words

→

✅ Clean Tokens

2.1 Tokenization

Tokenization divides text into smaller parts called tokens. It is the base step for stemming and lemmatization. NLTK provides two main methods:

Method	Function	Returns	Description
Word Tokenize	`word_tokenize()`	List of words	Splits text into individual word tokens
Sentence Tokenize	`sent_tokenize()`	List of sentences	Splits text into individual sentences

word_tokenize() — Example python

import nltk
from nltk.tokenize import word_tokenize
nltk.download('punkt_tab')

text = "Hello there! How are you doing today?"
words = word_tokenize(text)
print(words)

▶ OUTPUT

['Hello', 'there', '!', 'How', 'are', 'you', 'doing', 'today', '?']

sent_tokenize() — Example python

from nltk.tokenize import sent_tokenize
text = "Hello there! How are you doing today? NLTK is very useful."
sentences = sent_tokenize(text)
print(sentences)

▶ OUTPUT

['Hello there!', 'How are you doing today?', 'NLTK is very useful.']

2.2 Stop Words Removal

Stop words are commonly used words (such as "the", "a", "an", "in") that carry little semantic value and are typically ignored in processing. NLTK includes stop words lists for 16 different languages.

📝

Note

You can customize the stop words list by editing nltk_data/corpora/stopwords/english.txt.

Stop Words Removal — Complete Example python

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize

nltk.download('punkt_tab')
nltk.download('stopwords')

sentence = "I went to the university in the morning with my friend."
words = word_tokenize(sentence)

stop_words = set(stopwords.words('english'))
filtered_words = [word for word in words if word.lower() not in stop_words]

print("Original:", sentence)
print("Filtered:", " ".join(filtered_words))

▶ OUTPUT

Original: I went to the university in the morning with my friend.
Filtered: went university morning friend .

2.3 Stemming

📖

Definition

Stemming removes prefixes and suffixes from a word to reduce it to its root/stem form. The result may not always be a real dictionary word.

Stemmer	Class	Aggression	Notes	"running" →
Porter	`PorterStemmer`	Moderate	Most common & widely used	`run`
Lancaster	`LancasterStemmer`	High	Very aggressive, shortest roots	`run`
Snowball	`SnowballStemmer`	Moderate–Low	Multi-language support	`run`

a) Porter Stemmer python

from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
words = ["running", "happiness", "flies", "studying"]
stemmed = [stemmer.stem(word) for word in words]
print(stemmed)

▶ OUTPUT

['run', 'happi', 'fli', 'studi']

b) Lancaster Stemmer python

from nltk.stem import LancasterStemmer
stemmer = LancasterStemmer()
words = ["running", "happiness", "flies", "studying"]
stemmed = [stemmer.stem(word) for word in words]
print(stemmed)

▶ OUTPUT

['run', 'happy', 'fli', 'study']

c) Snowball Stemmer python

from nltk.stem import SnowballStemmer
stemmer = SnowballStemmer("english")
words = ["running", "happiness", "flies", "studying"]
stemmed_words = [stemmer.stem(word) for word in words]
print("Original:", words)
print("Snowball:", stemmed_words)

▶ OUTPUT

Original:  ['running', 'happiness', 'flies', 'studying']
Snowball:  ['run', 'happi', 'fli', 'studi']

2.4 Lemmatization

📖

Definition

Lemmatization returns the canonical (base) form of a word. Unlike stemming, the result is always a real dictionary word. It uses WordNet and considers Part-of-Speech (POS).

POS Tag	Part of Speech	Example
n (default)	Noun	bats → bat, feet → foot
v	Verb	stripes → strip, running → run
a	Adjective	better → good
r	Adverb	quickly → quick

📝

Note

If the pos parameter is omitted, the default is Noun (n).

Required Downloads python

nltk.download('wordnet')    # WordNet lexical database
nltk.download('omw-1.4')    # Open Multilingual WordNet

Example 1 — Basic Lemmatization python

import nltk
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()

text = "studies studying cries cry"
tokenization = nltk.word_tokenize(text)
for w in tokenization:
    print(f"Lemma for {w} is {lemmatizer.lemmatize(w)}")

▶ OUTPUT

Lemma for studies  is study
Lemma for studying is studying
Lemma for cries    is cry
Lemma for cry      is cry

Example 2 — With POS Tags python

lemmatizer = WordNetLemmatizer()
print("rocks  :", lemmatizer.lemmatize("rocks"))           # Noun (default)
print("corpora:", lemmatizer.lemmatize("corpora"))         # Noun (default)
print("better :", lemmatizer.lemmatize("better", pos="a")) # Adjective

▶ OUTPUT

rocks  : rock
corpora: corpus
better : good

Example 3 — Single Word Lemmatization python

print(lemmatizer.lemmatize("bats"))            # → bat
print(lemmatizer.lemmatize("are"))             # → are
print(lemmatizer.lemmatize("feet"))            # → foot
print(lemmatizer.lemmatize("stripes", 'v'))   # → strip
print(lemmatizer.lemmatize("stripes", 'n'))   # → stripe

▶ OUTPUT

bat
are
foot
strip
stripe

Example 4 — Lemmatize a Full Sentence python

from nltk.stem import WordNetLemmatizer
from nltk.tokenize import word_tokenize

sentence = "The striped bats are hanging on their feet for best"
lemmatizer = WordNetLemmatizer()
word_list = word_tokenize(sentence)

lemmatized_output = ' '.join([lemmatizer.lemmatize(w) for w in word_list])
print(lemmatized_output)

▶ OUTPUT

The striped bat are hanging on their foot for best

Example 5 — Tokenization + Lemmatization Together python

from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer

lemmatizer = WordNetLemmatizer()
text = "the boy cries when he goes to the school with the students by the bus."
sent = word_tokenize(text)
for w in sent:
    print(lemmatizer.lemmatize(w))

▶ OUTPUT

the boy cry when he go to the
school with the student by the bus .

🧪

Practice — Find the Output

Practice Questionpython

from nltk.stem import WordNetLemmatizer
from nltk.tokenize import word_tokenize
text = "The children were running and the mice were eating cheese"
lemmatizer = WordNetLemmatizer()
result = [lemmatizer.lemmatize(word) for word in word_tokenize(text)]
print(result)

▶ EXPECTED OUTPUT

['The', 'child', 'were', 'running', 'and', 'the', 'mouse', 'were', 'eating', 'cheese']

Note: "children" → "child" | "mice" → "mouse"

Level 2

Morphological Analysis

Stemming vs Lemmatization — Comparison Summary

📖

Definition

Morphological Analysis studies the internal structure of words. The two main techniques are Stemming (rule-based) and Lemmatization (dictionary-based).

Feature	Stemming	Lemmatization
Method	Rule-based (removes suffixes)	Dictionary-based (WordNet)
Output	May not be a real word	Always a real word (lemma)
Speed	⚡ Faster	Slower (requires lookup)
Accuracy	Lower	Higher
POS Aware?	❌ No	✅ Yes (`pos` parameter)
Example	"studies" → `studi`	"studies" → `study`
Best for	Search engines, IR	NLP, text understanding

Level 3

Syntactic Analysis

POS Tagging · Chunking · Constituency Parsing · Dependency Parsing

📖

Definition

Syntactic Analysis examines the grammatical structure of sentences, assigning roles to words and grouping them into phrases and hierarchical trees.

Syntactic Analysis — Imports & Setup python

import nltk
from nltk.tokenize import word_tokenize
from nltk import pos_tag, RegexpParser
from nltk.parse import DependencyGrammar

nltk.download('punkt_tab')
nltk.download('averaged_perceptron_tagger_eng')   # POS tagger

4.1 Part-of-Speech (POS) Tagging

POS tagging assigns a grammatical category (noun, verb, adjective, etc.) to each word in a sentence.

Tag	Meaning	Examples
NN	Noun, singular	"apple", "school"
NNS	Noun, plural	"apples", "students"
NNP	Proper noun, singular	"Tom", "Google"
NNPS	Proper noun, plural	"Avengers"
VB	Verb, base form	"eat", "run"
VBD	Verb, past tense	"ate", "went"
VBZ	Verb, 3rd person singular	"eats", "goes"
VBG	Verb, gerund/present participle	"eating", "running"
JJ	Adjective	"big", "early"
RB	Adverb	"quickly", "everyday"
DT	Determiner	"the", "a", "an"
IN	Preposition / subordinating conj.	"in", "on", "at"
PRP	Personal pronoun	"I", "he", "she"
CD	Cardinal number	"three", "5.1"
TO	Preposition "to"	"to"
CC	Coordinating conjunction	"and", "but", "or"
WRB	Wh-adverb	"when", "where"

POS Tagging — Example 1 python

import nltk
from nltk import word_tokenize, pos_tag

sentence = "Tom ate an apple at 10:00 am."
tokens = word_tokenize(sentence)
tags = pos_tag(tokens)
print(tags)

▶ OUTPUT

[('Tom', 'NNP'), ('ate', 'VBD'), ('an', 'DT'), ('apple', 'NN'),
 ('at', 'IN'), ('10:00', 'CD'), ('am', 'NN')]

POS Tagging — Example 2 python

text = "I went to the school early morning."
tokens = word_tokenize(text)
tag = nltk.pos_tag(tokens)
print(tokens)
print(tag)

▶ OUTPUT

['I', 'went', 'to', 'the', 'school', 'early', 'morning', '.']
[('I', 'PRP'), ('went', 'VBD'), ('to', 'TO'), ('the', 'DT'),
 ('school', 'NN'), ('early', 'JJ'), ('morning', 'NN'), ('.', '.')]

POS Tagging — Example 5 (News Sentence) python

from nltk.tokenize import word_tokenize
from nltk.tag import pos_tag

ex = 'European authorities fined Google a record $5.1 billion on Wednesday'

def preprocess(sent):
    sent = word_tokenize(sent)
    sent = pos_tag(sent)
    return sent

print(preprocess(ex))

▶ OUTPUT

[('European', 'JJ'), ('authorities', 'NNS'), ('fined', 'VBD'),
 ('Google', 'NNP'), ('a', 'DT'), ('record', 'NN'), ('$', '$'),
 ('5.1', 'CD'), ('billion', 'CD'), ('on', 'IN'), ('Wednesday', 'NNP')]

4.2 Chunking / Shallow Parsing

📖

Definition

Chunking (shallow parsing) groups words into meaningful phrases such as Noun Phrases (NP) and Verb Phrases (VP) using Regular Expression grammar rules.

Symbol	Meaning	Example
`<DT>`	Match determiner tag	"the", "a", "an"
`<JJ>*`	Zero or more adjectives	Optional: "big", "red"
`<NN>+`	One or more nouns	"school", "apple book"
`?`	Optional element	`<DT>?` = optional determiner
`{}`	Chunk pattern	`NP: {<DT>?<JJ>*<NN>}`

NP Chunking — Example 1 python

import nltk
text = "The boy ate the apple quickly"
tokens = nltk.word_tokenize(text)
tag = nltk.pos_tag(tokens)

grammar = "NP: {<DT>?<JJ>*<NN>}"
cp = nltk.RegexpParser(grammar)
result = cp.parse(tag)
print(result)

▶ OUTPUT

(S (NP The/DT boy/NN) ate/VB (NP the/DT apple/NN) quickly/RB)

NP + VP Chunking python

import nltk
from nltk import word_tokenize, pos_tag, RegexpParser

sentence = "Tom ate an apple at 10:00 am."
tokens = word_tokenize(sentence)
tagged = pos_tag(tokens)

grammar = r"""
    NP: {<DT>?<JJ>*<NN|NNP|NNS>+}
    VP: {<VB.*><NP|PP|CLAUSE>*}
"""
parser = RegexpParser(grammar)
tree = parser.parse(tagged)
print(tree)

▶ OUTPUT

(S
  (NP Tom/NNP)
  (VP ate/VBD (NP an/DT apple/NN))
  at/IN 10:00/CD am/NN ./.)

4.3 Constituency Parsing

📖

Definition

Constituency Parsing builds a full hierarchical tree showing how words group into nested phrases. The result.draw() method visualizes the tree in a window.

Constituency Parsing — Pre-tagged Input (Example 3) python

import nltk
sentence = [("the", "DT"), ("little", "JJ"), ("yellow", "JJ"),
            ("dog", "NN"), ("barked", "VBD"), ("at", "IN"),
            ("the", "DT"), ("cat", "NN")]

grammar = "NP: {<DT>?<JJ>*<NN>}"
cp = nltk.RegexpParser(grammar)
result = cp.parse(sentence)
print(result)
result.draw()   # Opens visual tree window

▶ OUTPUT

(S
  (NP the/DT little/JJ yellow/JJ dog/NN)
  barked/VBD
  at/IN
  (NP the/DT cat/NN))

4.4 Dependency Parsing

📖

Definition

Dependency Grammar focuses on how individual words relate to each other (head → dependent relationships), rather than grouping words into phrases. It produces a labeled directed graph.

Term	Description
Head	The governing word in a relationship (usually the verb)
Dependent	A word that modifies or relates to the head
`SBJ`	Subject dependency label
`NMOD`	Noun modifier dependency label
`nsubj`	Nominal subject (modern parsers)
`dobj`	Direct object dependency label

Dependency Parsing — "I shot an elephant in my pajamas" python

import nltk
groucho_dep_grammar = nltk.DependencyGrammar.fromstring("""
    'shot'    -> 'I' | 'elephant' | 'in'
    'elephant' -> 'an' | 'in'
    'in'      -> 'pajamas'
    'pajamas' -> 'my'
""")
print(groucho_dep_grammar)

▶ OUTPUT

Dependency grammar with 7 productions
  'shot'    -> 'I'
  'shot'    -> 'elephant'
  'shot'    -> 'in'
  'elephant' -> 'an'
  'elephant' -> 'in'
  'in'      -> 'pajamas'
  'pajamas' -> 'my'

📝

Important Note

This NLTK example captures bare dependency information (which word depends on which head) without specifying the dependency type (subject, object, modifier, etc.).

Level 4

Semantic Analysis

Named Entity Recognition (NER)

📖

Definition

Semantic Analysis extracts meaning from text — going beyond grammar to understand who, what, when, and where. Named Entity Recognition (NER) is the most common first step in information extraction.

5.1 NER Entity Tags

Tag	Entity Type	Example
PERSON	Person names	"Robert Downey Jr.", "Chris Evans"
ORGANIZATION	Organizations, companies	"IBM", "Marvel Studios"
GPE	Geopolitical entity (location)	"Boston", "Jordan", "USA"
geo	Geographical entity	"Mississippi River", "Alps"
tim	Time indicator	"Wednesday", "2020", "10:00 am"
art	Artifact	"iPhone", "Boeing 747"
eve	Event	"Olympics", "World Cup"
nat	Natural phenomenon	"hurricane", "earthquake"

NER Pipeline

Raw Text

→

Tokenize

→

POS Tag

→

ne_chunk()

→

🏷️ Entities

NER Imports & Setup python

import nltk
from nltk.tokenize import word_tokenize
from nltk import word_tokenize, pos_tag, ne_chunk

nltk.download('words')
nltk.download('punkt_tab')
nltk.download('averaged_perceptron_tagger_eng')
nltk.download('maxent_ne_chunker_tab')   # NER model
nltk.download('wordnet')
nltk.download('stopwords')

💡

Why Capitalization Matters for NER

NER relies heavily on capitalization to detect proper nouns (NNP/NNPS). "boston" would likely NOT be detected as a location, but "Boston" will be detected as (GPE Boston).

ℹ️

Why POS Tagging is Required for NER

ne_chunk() requires POS-tagged input. It uses the POS tags to determine which words are proper nouns (NNP) before classifying them as named entities.

NER Example 3 — Simple One-liner python

from nltk import word_tokenize, pos_tag, ne_chunk

input_str = 'Bill works for IBM so he went to Boston for a conference in 2020.'
print(ne_chunk(pos_tag(word_tokenize(input_str))))

▶ OUTPUT

(S
  (PERSON Bill/NNP)
  works/VBZ
  for/IN
  (ORGANIZATION IBM/NNP)
  so/IN he/PRP went/VBD to/TO
  (GPE Boston/NNP)
  for/IN a/DT conference/NN in/IN 2020/CD ./.)

NER Pipeline — "I live in Jordan" python

from nltk import word_tokenize, pos_tag, ne_chunk

text = "I live in Jordan"
tokens  = word_tokenize(text)   # Step 1
tagged  = pos_tag(tokens)       # Step 2
entities = ne_chunk(tagged)     # Step 3
print(entities)

▶ OUTPUT

Tokenization: ['I', 'live', 'in', 'Jordan']
POS:          [('I', 'PRP'), ('live', 'VBP'), ('in', 'IN'), ('Jordan', 'NNP')]
(S I/PRP live/VBP in/IN (GPE Jordan/NNP))

Section 6

Summary & Cheat Sheet

Complete NLP Levels Reference — Exam Preparation

Level	Task	Library / Module	NLTK Downloads	Output
1 Lexical	Tokenizationword_tokenize sent_tokenize	`nltk.tokenize`	punkt_tab	List of tokens
1 Lexical	Stop Words Removal	`nltk.corpus stopwords`	stopwords	Filtered token list
1 Lexical	Lowercasing	`Python .lower()`	None required	Lowercase tokens
2 Morphological	StemmingPorter / Lancaster Snowball	`nltk.stem`	None (built-in rules)	Stemmed root forms
2 Morphological	LemmatizationWordNetLemmatizer	`nltk.stem`	wordnetomw-1.4	Real dictionary words
3 Syntactic	POS Taggingpos_tag()	`nltk`	punkt_tabaveraged_perceptron_tagger_eng	POS tag tuples (word, TAG)
3 Syntactic	ChunkingRegexpParser	`nltk`	punkt_tabaveraged_perceptron_tagger_eng	Phrase chunks (NP, VP)
3 Syntactic	Constituency ParsingRegexpParser + .draw()	`nltk`	punkt_tabaveraged_perceptron_tagger_eng	Full parse trees
3 Syntactic	Dependency ParsingDependencyGrammar	`nltk`	None required	Head→dependent graph
4 Semantic	Named Entity Recognitionne_chunk()	`nltk`	wordspunkt_tabaveraged_perceptron_tagger_engmaxent_ne_chunker_tab	PERSON · ORG · GPE

Full NLP Processing Pipeline

Process

Input → Output

Tools Used

Lexical Analysis

Raw text → Clean tokens

word_tokenize · stopwords

Morphological Processing

Tokens → Root forms

PorterStemmer · WordNetLemmatizer

POS Tagging

Tokens → Tagged tuples

pos_tag() · averaged_perceptron_tagger

Chunking / Parsing

POS tags → Phrase trees

RegexpParser · DependencyGrammar

NER (Semantic)

POS tags → Named entities

ne_chunk() · maxent_ne_chunker

Meaning Extraction

Entities → Structured info

Application-specific processing

Quick Reference — NLTK Downloads

nltk.download('punkt_tab')

Used for: Tokenization

word_tokenize() · sent_tokenize()

nltk.download('stopwords')

Used for: Stop Words

stopwords.words("english")

nltk.download('wordnet')

Used for: Lemmatization

WordNetLemmatizer().lemmatize()

nltk.download('omw-1.4')

Used for: Lemmatization support

WordNetLemmatizer().lemmatize()

nltk.download('averaged_perceptron_tagger_eng')

Used for: POS Tagging

pos_tag() · nltk.pos_tag()

nltk.download('maxent_ne_chunker_tab')

Used for: Named Entity Recognition

ne_chunk()

nltk.download('words')

Used for: NER support

ne_chunk()

NLP LinguisticAnalysis Guide

Introduction & Overview

1.1 What is NLTK?

1.2 The Four Levels of NLP Analysis

1.3 Full NLP Processing Pipeline

Lexical Analysis

Required Libraries & Setup

Lexical Analysis Pipeline

2.1 Tokenization

2.2 Stop Words Removal

2.3 Stemming

2.4 Lemmatization

Morphological Analysis

Syntactic Analysis

4.1 Part-of-Speech (POS) Tagging

4.2 Chunking / Shallow Parsing

4.3 Constituency Parsing

4.4 Dependency Parsing

Semantic Analysis

5.1 NER Entity Tags

NER Pipeline

Summary & Cheat Sheet

Full NLP Processing Pipeline

Quick Reference — NLTK Downloads

NLP Linguistic
Analysis Guide