10 Probability Concepts for Machine Learning Explained Simply

--> 10 Probability Concepts for Machine Learning Explained Simply - KDnuggets

-->

Join Newsletter

Introduction to Probability Concepts

For a long time I treated probability as the vegetables of machine learning. The boring stuff you choke down before you get to the good part. Later on, I realized that probability is not just a prerequisite for machine learning but it makes much of machine learning work. But nobody tells you up front that a model is almost never sure of anything. Is this image a cat? Probably. Is this transaction fraud, or did someone just buy a lot of socks at 2am? Hard to say. A language model has no clue what word you're about to type next, so it doesn't pretend to. It spreads its confidence across a bunch of options and hands you the most likely one. Once that clicked for me, a lot of the rest stopped feeling like memorization. You don't need a stats PhD for any of this. You need maybe ten ideas. Here they are, the way I wish someone had explained them to me:

1. Random Variables

Let's start with one of the most fundamental ideas in probability.

You don't know if an email is spam before it shows up. You don't know if a visitor will buy anything before they arrive. You don't know what a model will spit out before you run it. Any value that depends on an outcome you haven't seen yet is a random variable, and in machine learning that covers basically everything: the features, the labels, the errors, the outputs.

By convention, random variables are usually written using uppercase letters such as X and Y, while specific observed values are written using lowercase letters such as x and y.

So for a spam classifier you'd write the label as:

\[

Y =

\begin{cases}

1 & \text{if the email is spam} \\

0 & \text{if it's not}

\end{cases}

\]

Y is binary. Before you read the email it could be either. The second it gets labeled, the mystery's gone and you just have a number.

In supervised learning, the usual cast is X for the input features and Y for the target, and the model is chasing this:

\[

P(Y \mid X)

\]

Given what I can see, how likely is each label?

Feed it the words, the sender, the suspicious links, and it might come back with:

\[

P(Y = 1 \mid X = x) = 0.92

\]

Translation: it's 92% sure this email is spam. Not certain. Just pretty confident.

2. Probability Distributions

Okay, so a variable can take different values. The follow-up question writes itself: which values, and how often? That whole map is a probability distribution.

The bookkeeping rule is simple. For discrete stuff, the probabilities have to add up to one, no more, no less:

\[

\sum_x P(X=x) = 1

\]

For continuous stuff it's the area under the curve that equals one:

\[

\int_{-\infty}^{\infty} p(x)\,dx = 1

\]

Different data wants different distributions. A yes/no thing like spam is a Bernoulli:

\[

Y \sim \text{Bernoulli}(p)

\]

where \(p\) is the probability of a 1. For example, if

\[

P(Y = 1) = 0.3

\]

then

\[

P(Y = 0) = 0.7

\]

and you're done.

Continuous values such as prediction errors, temperatures, heights, or sensor readings are often approximated using a Gaussian distribution:

\[

X \sim \mathcal{N}(\mu, \sigma^2)

\]

where:

\(\mu\) is the mean

\(\sigma^2\) is the variance

Probability distributions are important because machine-learning models are often trying to learn one. A regression model tries to estimate likely values for a continuous target, while a classifier tries to estimate a probability distribution over possible classes:

\[

p_\theta(y \mid x)

\]

Here, \(\theta\) represents the parameters learned by the model during training.

3. Expectation, Variance, and Standard Deviation

Suppose you repeat an experiment many times and record the results.

What value would you expect to see on average?

That average is the expectation, also called the expected value or the mean.

For a discrete variable:

\[

\mathbb{E}[X] = \sum_x xP(X=x)

\]

For a continuous variable:

\[

\mathbb{E}[X] = \int x p(x)\,dx

\]

Where this shows up is average performance. For example, say you have a model that predicts house prices. Its prediction error will vary from one house to another. The expected error tells us the average error we would expect across many predictions.

But, an average can lie to you. Two models, both averaging $10,000 of error. One model gives values near $10,000 almost every time. The other is off by $1,000 on one house and $50,000 on the next. Same average. Completely different behavior, and you'd want to know that before deploying either one.

That's what variance is for. It measures the spread:

\[

\mathrm{Var}(X) = \mathbb{E}\left[(X - \mu)^2\right], \quad \mu = \mathbb{E}[X]

\]

Take the square root and you get the standard deviation:

\[

\sigma = \sqrt{\mathrm{Var}(X)}

\]

I almost always reach for standard deviation over variance because it uses the same units as the original data.

4. Conditional Probability

A model basically never asks a question without context.

A model does not simply ask:

What is the probability that an email is spam?

Instead, it asks:

What is the probability that an email is spam given the information I can see?

This little "given" is conditional probability: the chance of one thing once you already know another.

\[

P(A \mid B) = \frac{P(A \cap B)}{P(B)}

\]

Most classifiers are trying to estimate:

\[

P(Y \mid X)

\]

which means:

What is the probability of a label given the observed features?

For example:

\[

P(\text{Spam} \mid \text{Email contains "free"})

\]

represents the probability that an email is spam given that it contains the word "free."

Suppose that among all emails containing the word "free," 80% turn out to be spam.

Then:

\[

P(\text{Spam} \mid \text{contains "free"})=0.8

\]

Notice how the condition changes the probability.

Maybe only 20% of all emails are spam overall. But once we observe a useful clue, such as the word "free," the probability increases significantly. This is how a model makes predictions: it observes features and continuously updates its estimate of each possible outcome.

5. Bayes' Theorem

Conditional probability naturally leads to one of the most famous formulas in statistics: Bayes' theorem.

Bayes' theorem has a reputation for being intimidating, and I think that's mostly a marketing problem. All it really does is tell you how to change your mind when new evidence shows up.

\[

P(A \mid B)=\frac{P(B \mid A)P(A)}{P(B)}

\]

It has four important quantities:

\(P(A)\): your initial belief (the prior)

\(P(B \mid A)\): how likely the evidence is if A is true

\(P(B)\): how common the evidence is in general

\(P(A \mid B)\): your updated belief after seeing the evidence

Let's return to the spam example. Let A be "the email is spam" and B be "it contains the word free" then Bayes' theorem becomes:

\[

P(\text{Spam} \mid \text{"free"})=

\frac{

P(\text{"free"} \mid \text{Spam})P(\text{Spam})

}{

P(\text{"free"})

}

\]

Suppose that spam is fairly rare, but when it does land, it practically screams "free" at you. So the moment that word appears, your suspicion should shoot up. This is what this formula actually does. This way of thinking is all over the place: Naive Bayes classifiers, Bayesian neural nets, the diagnostic systems that weigh symptoms, anything that has to mix what it already knew with what it just learned.

6. Joint, Marginal, and Conditional Distributions

Up to now it's been one variable at a time. But in machine learning, we often care about how multiple variables relate to each other.

When building that spam detector, you might track two things at once: does the email contain a link, and is it spam. Those aren't strangers. The joint distribution is the probability of both happening together, written as

\[

P(X, Y)

\]

If \(X\) is "has a link" and \(Y\) is "is spam," then

\[

P(X=\text{link},\, Y=\text{spam})

\]

is the probability that both events occur at the same time.

Sometimes, though, you only care about one of them and want the other to vanish. That's a marginal distribution, and you get it by adding up across everything you're ignoring:

\[

P(X) = \sum_y P(X, y)

\]

You sum over every possible value of \(Y\) until only \(X\) is left in the room.

And the conditional distribution is the one you've already seen above, dressed up in the joint:

\[

P(Y \mid X) = \frac{P(X, Y)}{P(X)}

\]

These three are basically the same family. A model often learns the joint story of, say, images and labels, then uses it to estimate the conditional probability: given this image, which label?

Now the fun one. Independence. Two variables are independent when knowing one tells you precisely nothing about the other:

\[

P(X, Y) = P(X)\,P(Y)

\]

When that's true, they ignore each other completely.

However, in the real world, true independence is rare. But pretending it's true anyway can make a model dramatically simpler, and a classic example is Naive Bayes. It assumes that features are conditionally independent once the class label is known:

\[

P(x_1, x_2, \dots, x_d \mid y)

=

\prod_{j=1}^{d} P(x_j \mid y)

\]

This is, to be blunt, a lie. Words in a sentence are deeply entangled. And yet Naive Bayes refuses to fail, especially on text. It's one of those cases where the wrong assumption somehow still gets you to the right place, which used to drive me a little crazy until I made peace with it.

7. Likelihood and Maximum Likelihood Estimation

When training a machine-learning model, we are trying to ask a simple question:

How well do the model's parameters explain the data we observed?

The number that answers it is the likelihood.

A likelihood measures how probable the observed data is under a particular set of model parameters.

Suppose a model with parameters \(\theta\) assigns probabilities to outcomes:

\[

p_\theta(y_i \mid x_i)

\]

For a dataset containing \(n\) independent examples, the likelihood is:

\[

\mathcal{L}(\theta)

=

\prod_{i=1}^{n}

p_\theta(y_i \mid x_i)

\]

You can think of this as multiplying together the probabilities that the model assigns to all observed training examples. A model that consistently assigns high probability to the correct outcomes will have a high likelihood.

This leads to a training strategy called Maximum Likelihood Estimation (MLE).

The idea is simple: choose the parameter values that make the observed data as likely as possible.

Mathematically:

\[

\hat{\theta}_{\text{MLE}}

=

\arg\max_\theta

\mathcal{L}(\theta)

\]

In practice, multiplying many probabilities together can produce extremely small numbers that are difficult for computers to work with.

To avoid this problem, machine-learning systems usually maximize the log-likelihood instead:

\[

\log \mathcal{L}(\theta)

=

\sum_{i=1}^{n}

\log p_\theta(y_i \mid x_i)

\]

The mathematics becomes easier, and the optimization process becomes more stable.

And if you've ever trained a classifier with binary cross-entropy, surprise, you were doing this the whole time. Maximizing log-likelihood and minimizing cross-entropy are the same act wearing different hats. The intuition behind all of this is simple. A decent model should give high probability to the things that actually happened. If your spam filter keeps seeing spam and keeps shrugging "looks fine to me," it's broken, and the parameters need to be adjusted.

8. Sampling, the Law of Large Numbers, and the Central Limit Theorem

Nobody gets the whole population. You get a chunk of it, and that chunk is your sample. A company might log half a billion clicks and you'll train on three million of them, because that's what fits and that's what's labeled.

The sample mean is:

\[

\bar{X}

=

\frac{1}{n}

\sum_{i=1}^{n}X_i

\]

Naturally, this raises an important question:

Can I actually trust a number I computed from a slice instead of the whole pie?

The Law of Large Numbers says, mostly, yes. Grow the sample a

[truncated for AI cost control]