Introduction to Feature Selection

05 Aug 2026

Ever since I started making summaries as a child, I wondered how to pick the most important information out of a textbook. When I first began, I used to underline almost every feature of the text.

As I grew older, I noticed that was not a good strategy. Even though I did not exactly struggle on the exams, I was not properly taking advantage of my study time. So I underlined less and started writing abstracts. Still, I did not always catch what would actually appear on the exam. Sometimes a little detail on a page that I considered unimportant mattered a lot to the teacher. Selecting the most important information from a book turned out to be an art—and not as easy as it seemed.

In Data Science we do something very similar when we try to keep the most useful features of a dataset. The table may have dozens or hundreds of columns, as our book had pages, but that does not mean we need all of them to predict well—i.e., to pass the exam in the analogy.

There are many different ways to perform Feature Selection

Here we will look at two complementary views: what objective the selector optimizes, and how the selector relates to the model that eventually predicts.

Different objectives

Depending on what we are trying to achieve, people usually distinguish:

  1. Idealized: Find the smallest subset that still describes the target. The shortest abstract that still gets the best mark.
  2. Target Feature Count / metric-driven: Find a subset that optimizes a chosen metric—Akaike Information Criterion, conditional entropy, mean squared error, and so on—without obsessing over how short the abstract is. Best result first; length is secondary.
  3. Approximate the original information: Prefer a subset that does not invent structure beyond what the full dataset already contains. An abstract that stays faithful to the book while cutting filler.
  4. Ranked selectors: Score every feature and cut with an ad-hoc threshold—either a fixed number of features (static) or a statistical relevance cutoff (automatic). Keep the chapters most likely to appear on the exam; drop the rest.

How selectors relate to the predictive model

Filter algorithms

Filters judge feature subsets from the data’s own statistics, without training the final predictor. That makes them fast and model-agnostic. The downside is familiar: ignoring the predictor can leave you with a subset that is theoretically neat and practically suboptimal.

This is closest to how we studied in practice. We looked at the textbook—how often the teacher stressed a passage, how central a definition seemed—and assigned importance from those inherent cues. We still did not know what would land on the exam.

Common filters include:

  1. mRMR: Maximize relevance to the target while minimizing redundancy among selected features, often via mutual information, an F-statistic, or both. Variants such as MID (Mutual Information Difference) and FCQ (F-test correlation quotient) show up often.
  2. ReliefF: Updates feature weights from nearest neighbors of the same and different classes, typically with Manhattan distance (any distance works in principle). Relatives include Relieved-F and MultiSurf, the latter adding a dead-band around the mean pairwise distance so borderline neighbors do not dominate the update.

Wrapper algorithms

Wrappers train a predictive model on candidate subsets and score them on a hold-out or against the full feature set. In the study analogy: write several abstracts and grade them on past exams. You search directly for the merit figure you care about; you also pay for it in compute.

Typical search patterns:

  1. Forward selection: Greedily add features. Useful when there are many candidate variables; it tends to overestimate how much variance each new feature “explains.”
  2. Backward selection: Greedily remove features from the full set. Often preferred under collinearity; it can struggle when features greatly outnumber rows.

Embedded algorithms

Embedded methods select features while the model trains—writing the abstract during the exam, not before. Because selection and fitting share work, they are usually cheaper than full wrappers (think Lasso-style penalties, tree importances built during boosting, and similar tricks).

Final remarks

We have sketched why feature selection matters and the main families people reach for day to day. There is rarely a single correct subset waiting to be discovered. Our job is closer to asking a better question, then chasing a good enough answer without pretending we have proven global optimality. We may be stuck in a local suboptimum (mathematical pun intended). The only reliable way forward is still to try, measure, and revise.