Part 4: Semantics & Word Embeddings

Last update: June 2026. All opinions are my own.

NLP from Scratch · Part 4/10

"You shall know a word by the company it keeps." — J. R. Firth, 1957

In Part 3 we put structure back into bag of words. We could tell that "dog" and "boy" play different grammatical roles. But we still couldn't tell that "bank" in "I deposited money at the bank" is a different concept from "bank" in "we sat by the bank of the river".

That's semantics — the third level of the NLP ladder from Part 1. And this is the one where things get genuinely magical: the moment Word2Vec showed up in 2013, vectors learned to do algebra on meaning. king − man + woman ≈ queen is the headline. The math underneath is, as usual, mostly counting.

Semantics in one sentence

Semantics is understanding the combined meaning of words.

The key insight: we shouldn't be talking about words. We should be talking about senses.

A word sense is one distinct meaning of a word. "Bank" has two senses — the financial institution and the river edge. They share the same spelling, the same pronunciation, but they're different concepts. Once we accept that we're working with senses (not words), every interesting question in semantics becomes well-formed.

That gives us our central question: given a word in context, how do we pick the right sense — and how do we represent senses in a way a computer can compute with?

We're going to answer that twice. First the knowledge-based way — humans hand-build a lexical database (WordNet) that lists every sense and every relationship. Then the distributional way — count which words appear near which other words, and let the geometry of context do the work. The distributional approach is what eventually gave us Word2Vec.

The six semantic relationships

Before we talk about how to compute with senses, we need vocabulary for how senses relate to each other. There are six relationships that show up everywhere.

1. Homonymy

Same form (spelling or pronunciation), unrelated meanings.

• bank₁ = financial institution
• bank₂ = sloping land at the side of a river

Two sub-types:

• Homographs: same spelling. bank / bank.
• Homophones: same pronunciation, different spelling. write / right, piece / peace.

These trip up tokenizers and downstream classifiers alike. "I sat by the bank" is a different document depending on which sense you pick.

2. Polysemy (or metonymy)

Same form, related meanings — linked by a systematic relationship.

Not just two random meanings — meanings that follow a pattern.

• IE University — the organisation vs the building. (org → location)
• Jane Austen — the author vs her works. ("I'm reading Jane Austen.")
• plum — the tree vs the fruit.

The systematic part matters: once you know the organisation ↔ building polysemy pattern, you can predict it for Harvard, Google, the Pentagon, etc. Languages are full of these patterns.

3. Synonymy

Different form, same (or near-same) meaning.

• big / large
• automobile / car
• water / H₂O

The textbook definition: two lexemes are synonyms iff they can substitute for each other in all contexts and carry the same propositional meaning.

There are no perfect synonyms. Politeness, slang, register, genre — these all leak. You can say "my old man" but not "my old father" in most contexts, even though both refer to the same person. Don't trust a thesaurus to give you drop-in replacements.

4. Antonymy

Senses that are opposites with respect to one feature, very similar otherwise.

The trap with antonyms is that they look dissimilar, but they're actually highly similar in every dimension except one.

• dark / light, hot / cold, short / long — gradable opposites
• rise / fall, up / down, in / out — directional / reversive
• alive / dead — binary opposition

hot and cold both apply to weather, both modify coffee or day or bath, both come up in the same kinds of sentences. The vectors of antonyms tend to land close together (which causes real trouble for distributional methods — we'll see).

5. Hyponymy & hypernymy

The is-a hierarchy. One sense is a subclass of another.

• red is a hyponym of colour.
• colour is the hypernym (or superordinate — "hyper is super") of red.
• red, blue, green are co-hyponyms of colour.

The hierarchy can go deep: crimson → red → colour → property → attribute → abstraction → entity. This is what makes WordNet possible — and what later powers a whole family of similarity metrics.

6. Meronymy (and holonymy)

The part-of relationship.

• wheel is a meronym of car.
• car is the holonym of wheel.

Other classic examples: finger / hand, hour / day, chapter / book. Different from hyponymy: a wheel is part of a car; it's not a kind of car.

WordNet distinguishes a few sub-types — member meronym (faculty / professor), part meronym (table / leg), substance meronym (water / oxygen). For our purposes, the key idea is just: parts and wholes are linked by a different relation than is-a.

WordNet — the knowledge-based approach

OK, we have six relationships. How do we put them into practice?

The first idea — and the one that dominated NLP for two decades — was: build a database of every word, every sense, and every relationship. Hand-curated by lexicographers.

The most famous example is WordNet 3.0 (Princeton, 1985–ongoing). It's a hierarchically organised lexical database — a thesaurus plus aspects of a dictionary, with all six relationships encoded.

A synset = a set of synonymous senses sharing a meaning. So the set bass, bass voice, basso is one synset (meaning ≈ "the lowest adult male singing voice"); sea bass is another (meaning ≈ "a type of fish").

WordNet stores relationships at the sense level, not the word level. So when you see something like breakfast¹ → meal¹, that means "the first sense of breakfast is a hyponym of the first sense of meal". Sense numbers matter.

WordNet relationships

The relationships you find in WordNet, all linked at the sense level:

The hierarchy of senses

The really useful thing WordNet gives you is the hypernym chain — for any sense, walk upwards through is-a relations until you hit the root (entity).

For bass (sense 3, the singer): bass → singer → musician → performer → entertainer → person → organism → living thing → whole → object → physical entity → entity.

That chain is what makes thesaurus-based similarity possible. And it's the place WordNet's limitations start to bite.

Why WordNet is hard to use in practice

WordNet looks like a goldmine. In practice it's awkward:

1. You need word sense disambiguation first. If a sentence has "bass", you have to know which sense before you can query the database. Picking the sense automatically is itself a hard NLP task.
2. Building and maintaining one is slow. WordNet 3.0 was decades of lexicographer labour. Most languages don't have one. EuroWordNet exists, GlobalWordNet collects others, but coverage is patchy.
3. It doesn't update. New slang, new compound nouns, new domain jargon, language drift over time — WordNet won't have covid, yeet, rizz, or any 2020s vocabulary the day they emerge. Manual updates take years.
4. Verbs and adjectives are worse than nouns. The hyponymy structure is much weaker for non-nouns. "Beautiful" doesn't fit cleanly into an is-a tree.

So WordNet works — but only as a starting point. We need a way to learn relationships from data without hand-annotation. That's where word similarity and distributional semantics come in.

Similarity — the looser metric

Synonymy is binary: two words either are synonyms or they aren't. But we usually want something gentler: how similar are two words on a continuous scale?

• big and large are very similar (near-synonyms).
• big and small are somewhat similar (same dimension, opposite direction).
• big and pineapple are not similar at all.

Formally: similarity is properly a relation between senses, not words. The word bank is not similar to the word slope — but bank₂ (sloping land) is similar to slope¹. Still, in practice we usually compute over both words and senses.

The reason similarity is more useful than synonymy: similarity rolls multiple relationships into one number. big and small are similar because they're both adjectives that modify the same nouns, even though they're antonyms. red and blue are similar because they're both colours, even though they're not synonyms. That's exactly the kind of mixed signal a downstream task usually wants.

Two families of similarity algorithm

There are two ways to compute word similarity, and they correspond to the two answers we promised at the start of the post.

1. Thesaurus-based: use the WordNet hierarchy. Are two words "nearby" in the hypernym tree?

2. Distributional: use a corpus. Do two words appear in similar contexts?

Thesaurus-based requires WordNet. Distributional requires only a big text corpus — which is why it scales, and why it ended up winning.

Thesaurus-based similarity

If you have a hierarchy, you can define a distance on it. Several flavours.

Path-based similarity

Two senses are similar if they're near each other in the hierarchy.

Walk up from one sense, walk down to the other, count edges.

Problem with this metric: it assumes every edge is the same weight. But intuitively, coin → coinage → currency feels like a tighter relationship than natural-object → inanimate-object → entity. Edges near the root are much broader than edges near the leaves.

We want a metric that down-weights edges high in the hierarchy (because those nodes are abstract — entity covers everything) and up-weights edges low in the hierarchy (specific concepts share more concrete meaning).

Information content similarity

The fix: define similarity using probability.

For every concept c, let P(c) be the probability that any randomly chosen word belongs to a sense subsumed by c. The root (entity) covers everything, so P(entity) ≈ 1. Specific concepts (coast, hill) have tiny probabilities.

Define:

IC(c) = −log P(c)

That's the information content of a concept — same idea as in information theory. Common ancestors carry low information; specific concepts carry high information.

Now, given two senses c₁ and c₂, find their Lowest Common Subsumer — LCS(c₁, c₂) — the most specific ancestor they share. Define:

sim_resnik(c₁, c₂) = −log P(LCS(c₁, c₂))

That's Resnik similarity. Two senses are similar if their lowest common ancestor is very specific (rare) — because that means they share a lot of meaning before diverging.

Lin similarity

Resnik only uses the LCS. Dekang Lin (1998) extended it: scale by how much the two concepts contain in total.

sim_Lin(c₁, c₂) = 2·log P(LCS(c₁, c₂)) / (log P(c₁) + log P(c₂))

The intuition: similarity is the ratio of information shared to information described. If two concepts share most of their content, they're very similar. If they share only an abstract ancestor, they're not.

Worked example: sim_Lin(hill, coast) ≈ 0.59. They share geological-formation as LCS — moderately specific, so moderately similar.

Path, Resnik, and Lin are the three you should know — but they're members of a wider family. Wu & Palmer factors in the depth of the lowest common subsumer. Leacock & Chodorow uses path length normalised by taxonomy depth. Jiang & Conrath combines information content with path distance. They all answer the same question — how close are these two concepts in the hierarchy? — they just disagree on what "close" should mean.

The problem with thesaurus-based methods

OK, we have a family of metrics that turn WordNet into a similarity function. Why don't we just use this everywhere?

Because WordNet has fundamental problems:

• Coverage. Most languages don't have a WordNet. The ones that do are smaller and less curated than English's.
• Drift. Language changes faster than lexicographers update databases. Twitter, slang, covid, yeet — WordNet is always behind.
• Open class. Verbs and adjectives have weaker hyponymy structure than nouns. WordNet works much better for things than for actions or qualities.
• Manual scaling. Adding new domain vocabulary (medical, legal, finance) means hand-building hierarchies. Doesn't scale.
• Doesn't fit modern problems. If your task is to find similar tweets, the language of Twitter isn't reflected in WordNet. You'd have to extend it — which is hugely expensive.

It helps to see this against the older idea of a thesaurus. A thesaurus groups words by topic — big → large, huge, enormous. Useful, but flat. WordNet keeps the senses apart and organises them into a network — big (size) is not the same node as big (importance). That's the difference between grouping and linking meanings.

So researchers started asking the obvious question: can we learn similarity from text directly, without a hand-built thesaurus?

Distributional semantics — the big idea

The intuition is one of the oldest in linguistics:

"A and B have almost identical environments → they're synonyms." — Zellig Harris, 1954

"You shall know a word by the company it keeps." — J. R. Firth, 1957

If two words appear in the same contexts, they probably mean similar things. You don't need to know what tesgüino means — if I tell you:

A bottle of tesgüino is on the table.
Everybody likes tesgüino.
Tesgüino makes you drunk.
We make tesgüino out of corn.

…you've already worked out it's some alcoholic beverage. From context alone. That's distributional semantics: define a word's meaning by the words it co-occurs with.

This is the foundation for everything that follows in this post — and most of modern NLP, including transformers.

From distributional hypothesis to vectors

If meaning lives in context, then we can represent a word as a vector of counts: count how often the word appears with every other word, and that vector is the word's representation.

Two words with similar count vectors → similar contexts → similar meanings.

These are called vector-space models of meaning. They offer:

• Higher recall than WordNet — if a word appears in your corpus, it has a representation. No coverage gaps.
• Lower precision — vectors blur synonyms, antonyms, and topical neighbours into one similarity score.
• Automatic — no human curation. Scrape a corpus, learn vectors.

The term-document matrix — and why it's not enough

You've already met this in Part 2. Each row is a term, each column is a document, each cell is a count (or TF-IDF weight).

Each row is a word vector (counts across documents). Each column is a document vector (counts of words).

Two documents are similar if their column vectors are similar. Two words are similar if their row vectors are similar.

This is a real representation of meaning — but it has two problems:

1. A document is too coarse a context. Two words can appear in the same book without being related. A Shakespeare play has 200 pages — "fool" on page 5 and "clown" on page 200 doesn't mean they're related; the same is true for "battle" and "weather".
2. It's sparse. Most cells are zero. Most words don't appear in most documents.

The fix to (1): shrink the context window.

The term-context matrix — narrower is sharper

Instead of counting words by document, count words by local context — a small window of ±k words. A typical window is 20 words, or one sentence.

Now each row vector is a much sharper representation of the word's meaning — it's defined by the actual neighbours of the word, not just the documents it appears in.

Two words are similar if their context vectors are similar.

One nagging problem: raw counts

Raw counts have the same problem as bag of words: very common words dominate. The pair (the, computer) will have a high count just because the is everywhere — not because the is semantically related to computer.

We need to weight the counts to favour surprising co-occurrence: pairs that appear together more than chance would predict. We did this in Part 2 with TF-IDF for the term-document matrix. For the term-context matrix, the standard answer is PMI.

PMI and PPMI — weighting context counts

Pointwise Mutual Information measures how much more often two events co-occur than they would if they were independent:

PMI(w₁, w₂) = log₂ ( P(w₁, w₂) / (P(w₁) · P(w₂)) )

If w₁ and w₂ are independent, the joint probability equals the product of the marginals and PMI = 0. If they co-occur more than chance, PMI is positive. If less than chance, PMI is negative.

One practical wrinkle: negative PMI values are unreliable (they say "these two words appear less together than chance" — but with sparse data, that's mostly noise). The standard fix is Positive PMI: replace negatives with 0.

PPMI(w₁, w₂) = max(PMI(w₁, w₂), 0)

PPMI has a small bias: it favours infrequent events too much. Various smoothing tricks (add-2 smoothing, weighting) help. The takeaway: PPMI on a term-context matrix already gives you meaningfully good word vectors.

Using syntactic context instead of word-window context

A neat twist (Lin, 1998): instead of using a ±k window, use dependency relations as context. The contexts for the word cell become things like subj-of-absorb, obj-of-attack, nmod-bacteria. Much sharper signal.

This works because grammatical relations encode which words are actually related, not just which words are nearby. The downside: you need a dependency parser first (which has its own error rate — remember Part 3).

Word2Vec — the 2013 revolution

By 2013 we had decent word vectors via PPMI on context matrices. They worked. But they were sparse, big, and hard to scale.

Then Mikolov et al. at Google published Word2Vec. It changed everything.

The trick

Word2Vec doesn't compute a similarity metric. It trains a tiny neural network to predict context — and then throws away the network and keeps the hidden layer.

The task is simple:

Given a word, predict the words around it. (Skip-gram)

Given the surrounding words, predict the centre word. (CBOW)

Take Wikipedia. Slide a window of ±5 across every sentence. For each centre word, generate training samples (centre, context). Train a shallow neural net to do the prediction task.

The architecture (in the original paper):

• Input: one-hot encoded vocabulary, 10,000 positions.
• Hidden layer: 300 neurons. Linear. No activation.
• Output: softmax over the same 10,000-word vocabulary.

You train this network to predict context words. After training, the hidden layer weights are the word embeddings. The output layer is discarded.

Each word is now a dense 300-dimensional vector. Much smaller than the sparse PPMI matrix (10,000 dimensions). And — this is the magic — each dimension encodes some semantic property the network learned was useful for the prediction task.

Skip-gram vs CBOW

Two flavours of the same idea, swap the direction of prediction:

In practice both produce embeddings of similar quality. Skip-gram is the more common default.

The training samples

Slide a window of size 2 over the sentence "The quick brown fox jumps over the lazy dog." Centre = brown. Context window: [the, quick, fox, jumps]. Training samples:

(brown, the)
(brown, quick)
(brown, fox)
(brown, jumps)

Repeat for every position in every sentence in your corpus. Billions of samples. Train the network. Throw away the output layer. Keep the hidden layer. Done.

The king − man + woman = queen moment

This is the result that made everyone's jaw hit the floor.

After training Word2Vec on Wikipedia, the researchers tried subtracting and adding word vectors. They expected nothing in particular. They got something extraordinary:

• vec(king) − vec(man) + vec(woman) ≈ vec(queen)
• vec(Paris) − vec(France) + vec(Italy) ≈ vec(Rome)
• vec(walking) − vec(swimming) + vec(swam) ≈ vec(walked)

The vectors had learned analogies. Gender, country-capital, verb tense — the network discovered them as directions in vector space, even though it was never told to.

Why does this work? The skip-gram task forces the network to compress 10,000 words into 300 dimensions in a way that preserves which words appear near which. The most efficient compression turns out to be one that captures semantic regularities as linear structure.

Two practical consequences:

1. You can swap one part of meaning for another with arithmetic. That's wild.
2. You can solve analogies as nearest-neighbour search. Given "man : king :: woman : ?", compute vec(king) − vec(man) + vec(woman) and find the nearest word in vector space.

The bigger lesson: a representation that is good at one task (predicting context) can encode information useful for many other tasks — text classification, NER, retrieval. This is the philosophy that drives transformers and large language models. Word2Vec was the first time NLP people saw it work.

Advantages and limitations

Advantages of Word2Vec over PPMI / WordNet:

• Dense. 300 dimensions instead of 10,000+ sparse columns.
• Encodes semantics. Each dimension picks up some meaningful direction.
• Big lift on downstream tasks. Classification, retrieval, NER — all improved when you swapped TF-IDF for Word2Vec features.
• Automatic. No hand-curated database. Just a corpus.

Limitations of Word2Vec:

• No polysemy. The word bank gets one vector — a smear of "financial" and "river" senses. There's no way to disambiguate at inference time.
• No word order. "I had my car cleaned" and "I had cleaned my car" would produce the same bag-of-vectors. Order is gone again.
• Not domain-specific. A Word2Vec trained on Wikipedia is bad at medical jargon. You'd have to retrain on PubMed.
• No context-dependence. The vector for bank is the same regardless of the sentence it's in. (This is what BERT/transformers fix — but that's Part 10.)

Anything2vec

Once people saw what Word2Vec could do, the same recipe got applied to everything else:

• Doc2Vec — embeddings for whole documents.
• Med2Vec — embeddings for medical codes.
• Author2Vec — embeddings for authors based on their content.
• Citation2Vec — embeddings for papers based on their citation graph.
• Many, many more.

The pattern is always: define a "context" (whatever lives near what), train a network to predict it, keep the hidden layer.

What's next

We've turned senses into vectors. We can compute similarity, do analogy arithmetic, and feed dense embeddings into downstream classifiers. But each word still gets one vector — no polysemy, no context. And we've still got no model of sequence: order is gone again.

The next session is the bridge to modern NLP: Language Modeling. Given a sequence of words, what comes next? Once we can answer that, we get contextual embeddings, generative text, and (eventually) the entire transformer stack.

The single most important idea

Meaning is geometry. Words live in a vector space, and the relationships between them are directions you can compute on. WordNet hand-drew the map; Word2Vec learned it from text alone — and discovered, by accident, that analogies are linear.

Sources

• Firth (1957). A synopsis of linguistic theory.
• Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space. arXiv 1301.3781.
• Miller (1995). WordNet — A Lexical Database for English. Communications of the ACM.
• Lin (1998). An Information-Theoretic Definition of Similarity. ICML.
• Jurafsky and Martin. Speech and Language Processing, 3rd ed., chapter 6.