Post 4 — Feature Engineering Is the Key

Last update: June 2026. All opinions are my own.

ML Foundations · Post 4/10

Raw data → useful features

This is the practical implication of the curse of dimensionality. If most of the work in ML is making sure your features carry signal and not noise, then the most leveraged skill in ML is feature engineering.

A well-engineered feature can take a problem from impossible to trivial. The classic example: predicting house prices.

• Raw column: address — essentially useless to a model.
• Engineered: distance_to_city_centre, crime_rate, school_district_rating — suddenly the model has signal it can actually use.

Same underlying data; completely different model performance. The transformation is what made the difference.

The ML process is iterative

Feature engineering doesn't happen once at the start. It loops:

1. Understand the problem. Talk to whoever knows the domain. What are they actually trying to predict, and why does it matter?
2. Prepare / engineer features. Clean, transform, derive. Build the columns the model will actually see.
3. Train the model. Pick a representation, fit it.
4. Evaluate. Look at where the model fails. Those failures point back to step 2.

You don't move linearly through the steps. You spend most of your time bouncing between 2 and 4, asking what would the model need to know to get this case right?

GIGO: garbage in, garbage out

Even the best algorithm can't learn from features that don't carry the signal. Conversely, a mediocre algorithm with great features will outperform a fancy algorithm with raw, messy features almost every time.

That's why this post comes before any specific algorithm in the foundations: nothing downstream matters if the input is wrong.

---

Next up — Post 5: More Data, Better Data, and Domain Expertise.