OLS & Multiple Regression — Bayes Thinking Lab

Example

N = 30 Students X₁ = Study time (h/week) X₂ = Sleep hours (h/night) Y = Exam score (0–100 pts)

The Least-Squares Principle

Why does OLS minimize squared deviations — and what does this mean?

Study time (h/week) → Exam score

b₀

—

b₁

—

R²

—

RSS

—

RSS comparison (smaller = better)

Your line

—

OLS (min)

—

Model:
Y = — + — · X₁

Tutorial — The Least-Squares Principle

What you see
30 simulated students: Study time X₁ (h/week) → Exam score Y (points). The orange line is your estimation — draggable via two round anchors at the left and right edges. Enable ● Residuals: green bars = positive residuals (point lies above the line), red bars = negative residuals (point lies below). A residual eᵢ = yᵢ − ŷᵢ is the deviation of the observed from the predicted value.

What to do
Drag the anchors until your RSS (orange bar, absolute value) is as small as possible. Then click ◎ Show OLS — the blue OLS line appears. The second bar shows its RSS. Comparison: How close did you get to the minimum?

What is RSS? (Residual Sum of Squares)
RSS = Σeᵢ² = Σ(yᵢ − ŷᵢ)². OLS finds analytically the exact b₀ and b₁ for which RSS is globally minimal — uniquely and provably. No trial and error: b₁ = Cov(X,Y)/Var(X), b₀ = ȳ − b₁·x̄.

Why squares, not absolute values?
(1) For every OLS line Σeᵢ = 0 — positive and negative residuals trivially cancel out. (2) Squaring penalizes large residuals disproportionately: a residual of 10 costs 100, not 10. (3) The square provides a differentiable function → closed, unique solution.

Gauss-Markov: Under the 5 OLS assumptions (→ Module ③) OLS is the BLUE — Best Linear Unbiased Estimator. No other linear unbiased estimator has smaller variance.

What is a Residual? eᵢ = yᵢ − yᵢ is the deviation of the observed value from the predicted. Why square? First, positive and negative residuals are treated equally (|+3| = |−3|). Second, large residuals are penalized disproportionately: a residual of 10 counts 100, not 10. OLS finds the unique b₀, b₁ for which: Σeᵢ = 0 and Σeᵢ² is minimal — this solution is unique.

I — linear

II — quadratic

III — outlier

IV — leverage point

All four datasets have (nearly) identical statistics: b₁ ≈ 0.50, b₀ ≈ 3.0, r ≈ .816, R² ≈ .667. Yet scatterplot and residual plot show fundamentally different patterns. Conclusion: Coefficients and R² alone are not enough — residual diagnostics are mandatory.

Understanding b₀, b₁ and β

Slope, intercept and standardized coefficient — visually and formally

Regression line with coefficient visualization

b₀

—

b₁

—

b₁: Per +1 h study time the expected score increases by — points.
β = r = — — bivariate: β = r always.
b₀: Expected score at 0 h study time (extrapolated, often not meaningful in context).

Tutorial — Understanding b₀, b₁ and β

What you see
The OLS line through all 30 data points. Enable △ Slope for the slope triangle and ✛ Mean for the centroid (x̄, ȳ) in the plot.

b₁ = — — Slope (unstandardized)
Per +1 h study time, the expected score increases by — points. Unit: points/hour. Interpretation: descriptive, not a causal effect. b₁ is scale-dependent.
The slope triangle in the plot shows concretely: +3 h study time → +— points (= 3 · b₁).

β = — — standardized coefficient
Per +1 SD in X₁, Y increases by β SD. Bivariate: β = r (Pearson correlation) always. β allows comparison of predictors on different scales within one study.
Important: β does not directly indicate importance — with correlated predictors β can deviate strongly from the partial effect (→ Module ④).

b₀ = — — Intercept
Expected score at X₁ = 0 h study time. Since 0 h lies outside the observed range, this is an extrapolation — usually not meaningfully interpretable in context.

Centroid (x̄ = — h | ȳ = — pts)
The OLS line always passes through (x̄, ȳ) — mathematically necessary, since b₀ = ȳ − b₁·x̄.

Calculation formulas:
b₁ = Σ(xᵢ − x)(yᵢ − y) / Σ(xᵢ − x)² = Cov(X,Y) / Var(X)
b₀ = y − b₁ · x · β = b₁ · (SD_X / SD_Y) = r (bivariate)

          b₁ (unstandardized) is reported when
          the unit is substantively meaningful: "+1 h study time → +— points".


          β (standardized) allows comparison of
          predictors on different scales within one study.
          Caution: β varies with SD(X) and SD(Y) — comparisons between studies are
          not valid.


          In bivariate OLS: β = r = — always.
          In the multiple model (Module ④) β ≠ r.
        

Model Fit & Variance Decomposition

SST = SSR + RSS, R², adjusted R² and the assumptions of the OLS model

Total variance (TSS) — Deviations from mean ȳ

            TSS = SSR + RSS
          

SSR

RSS

            SSR = —
            SSE = —
          

R²

—

adj. R²

—

f²

—

r (Pearson)

—

R² = SSR/TSS = —% of total variance in Y is explained by X₁.
f² = R²/(1−R²) = SSR/RSS: small ≥ .02 · medium ≥ .15 · large ≥ .35

Tutorial — Variance Decomposition & Model Fit

What you see
TSS mode: ȳ as a blue dashed line; blue bars show each point's deviation from ȳ. SSR + RSS mode: Shows the OLS line's residuals — green bars = positive residuals (point above line), red bars = negative residuals (point below line). The bar shows the SSR : RSS ratio of the variance decomposition.

TSS = — — Total Sum of Squares
TSS = Σ(yᵢ − ȳ)². Total variance in Y — does not depend on the model.

SSR = — (—% of TSS) — Sum of Squares Regression (Model)
SSR = Σ(ŷᵢ − ȳ)². How far are the predicted values ŷᵢ from the overall mean ȳ? This measures how much variance the model explains. OLS maximizes SSR.

RSS = — (—% of TSS) — Residual Sum of Squares
RSS = Σeᵢ² — unexplained variance. TSS = SSR + RSS is mathematically exact.

R² and f² (Cohen's effect size)
R² = SSR/TSS: proportion of explained variance. Note: R² always increases with each additional predictor! Adjusted R² corrects with a penalty term per predictor.
f² = R²/(1−R²) = — → — effect. Cohen (1988): .02 small · .15 medium · .35 large.

①

              Linearity:
              The relationship between X and Y is linear. Violation: residual-vs-fitted plot
              shows a systematic pattern. Remedy: transformation or polynomial terms.
            

②

              Independence:
              Observations are independent (no autocorrelation, no clustering).
              Violated in longitudinal or nested data → mixed models required.
            

③

              Homoscedasticity:
              Variance of residuals is constant across all X values — no fan shape.
              Violation: heteroscedasticity-robust standard errors (HC3) or WLS.
            

④

              Normality of residuals:
              Only required for inference — not for OLS estimation itself
              (Gauss-Markov theorem!). At n ≥ 30 the central limit theorem applies.
            

⑤

              No perfect multicollinearity
              (multiple regression): If two predictors are perfectly correlated,
              (XᵀX) is not invertible.
              High collinearity inflates standard errors → VIF > 10 critical.
              See Module ④.
            

Multiple Regression & Added Variable Plot

What does "controlling for X₂" mean? Understanding partial slopes visually.

Step ① — Bivariate Model: Y ~ X₁

Step

Correlation X₁↔X₂

Coefficient comparison

Model	b₁	β₁	R²
Bivariate	—	—	—
Partial	—	—	—

AVP-Slope

—

Δb₁

—

Step ① shows the bivariate regression of Y on X₁ — as in Module ①. The bivariate b₁ still contains the influence of X₂ when X₁ and X₂ are correlated.

Tutorial — Multiple Regression & AVP

From line to plane
Bivariate (Y ~ X₁) the solution is a line in 2D space. With two predictors (Y ~ X₁ + X₂) it becomes a regression surface (plane) in 3D space. b₁ is the slope in the X₁ direction, b₂ in the X₂ direction — each holding the other predictor fixed.

Why multiple regression?
X₁ (study time) and X₂ (sleep) correlate with r₁₂ = —. The bivariate model does not measure the pure X₁ effect — b₁ also contains the influence of X₂. The multiple model holds X₂ statistically constant.

Reading the coefficients (table left)
Bivariate b₁: slope from Y ~ X₁ alone — contains confounding by X₂.
Partial b₁: slope from Y ~ X₁ + X₂ — adjusted for X₂.
Δb₁: the difference quantifies the confounding. At r₁₂ = 0, Δb₁ = 0.

Step ① — Bivariate starting point
What you see: The bivariate OLS line Y ~ X₁ (study time → exam score) — identical to Module ①.
Tip: Note the bivariate b₁ (table left), then change r₁₂ and compare.

Confounding comparison
bivariate b₁ = — · partial b₁ = — · Δb₁ = —
The larger r₁₂, the more the bivariate deviates from the partial b₁.

AVP Principle: An Added Variable Plot (partial regression plot) makes the partial effect of X₁ on Y visible — adjusted for X₂. To do this, X₂ is partialled out of both Y (e(Y|X₂)) and X₁ (e(X₁|X₂)). The slope of the regression line through the residual vectors equals exactly the partial coefficient b₁ from the multiple model.

Learning Cards — OLS & Multiple Regression

① OLS Criterion

OLS minimizes the sum of squared residuals: min Σ(yᵢ − ŷᵢ)². Squaring serves two purposes: signs are neutralized, and large residuals are penalized disproportionately more than small ones. The solution is unique and always yields Σeᵢ = 0.

② Coefficient b₁

Interpretation: "Per +1 unit X, the expected value of Y increases by b₁ units — all other predictors held constant (ceteris paribus)." In the bivariate case: b₁ = Σ(xᵢ−x̄)(yᵢ−ȳ) / Σ(xᵢ−x̄)². The numerator is the covariance, the denominator the variance of X.

③ Intercept b₀

b₀ is the expected Y value when all predictors equal zero. This is often not meaningful in context (e.g., 0 study hours, 0 sleep). b₀ is needed for prediction but should usually not be interpreted substantively. Always: ȳ = b₀ + b₁·x̄.

④ R² and adj. R²

R² = SSR/SST ∈ [0, 1] — proportion of explained variance. R² increases with every added predictor, even useless ones. Adjusted R² corrects for this: adj.R² = 1 − (1−R²)·(n−1)/(n−k−1). adj.R² decreases when a new predictor explains less than expected by chance.

⑤ Partial Slope

b₁ in the multiple model is not the same as in the bivariate model — it is the slope in the Added Variable Plot (AVP): regression of e(Y|X₂) on e(X₁|X₂). This value equals the partial slope: effect of X₁ on Y, after the shared portion of X₂ has been removed from both.

⑥ Standardization β

β = b · (SD_X / SD_Y) — the standardized regression coefficient. β indicates how many standard deviations Y increases when X increases by one SD. Allows comparison of predictors with different scales, but only within one sample. Between studies, β values are not directly comparable due to different SDs.

What is OLS?

Coefficient interpretation

R² and model fit

Effect size f²

When does OLS fail?

Related Tools in Bayes Thinking Lab