Fuel Economy Prediction
Comparative analysis of Lasso regression and Multivariate Adaptive Regression Splines in a multicollinear dataset where MARS's piecewise non-parametric approach substantially outperforms L1-penalized linear regression.
- Vehicles · 7 Predictors · Target: MPG
- 7.35
- Best MSE — MARS (deg=2, nk=20)
- MARS MSE Advantage Over Lasso
Built WithR · glmnet · earth · 10-Fold Cross-Validation
The Problem
Predicting vehicle fuel economy from engine and design characteristics is complicated by strong multicollinearity between predictors — and the question of whether variable interactions meaningfully influence predictions demands a method that can both handle collinearity and test interactions explicitly
Vehicle fuel economy (measured in miles per gallon) is determined by a cluster of interrelated design parameters: cylinder count, engine displacement, horsepower, weight, acceleration, model year, and country of origin.
75), creating a multicollinearity structure that inflates coefficient variance in standard least-squares regression and can produce misleading variable importance estimates.
Beyond multicollinearity, vehicle performance parameters interact: a heavy vehicle with high horsepower behaves differently from a light vehicle with the same horsepower, and the fuel economy penalty for weight likely depends on model year as manufacturing technology improves.
Standard linear regression — and Lasso, which extends it with L1 regularization — cannot capture these interaction effects. The analytic challenge is two-fold: selecting the most informative predictors from a correlated feature set, and determining whether variable interactions are present and materially influence fuel economy predictions.
The Solution
A direct benchmark of Lasso and MARS — both with 10-fold cross-validated parameter selection — comparing variable selection, interaction detection, and predictive MSE to identify which approach better characterizes the drivers of vehicle fuel economy
391 while retaining all 7 predictors. MARS is fitted using the earth package, with cross-validation over degree (1, 2, 3) and nk (10 to 70) — testing no interactions, pairwise interactions, and three-way interactions. 350 using 6 of 7 predictors with pairwise interaction terms.
Both methods are compared on MSE, number of retained variables, variable importance rankings, and ability to detect and quantify interaction effects. The analysis also includes degree=1 MARS (no interactions) as an intermediate comparison point — allowing the contribution of interactions to be isolated from the algorithmic difference between the two methods.
Key Outcome
MARS substantially outperforms Lasso — MSE 7.35 versus 11.39, a 35% advantage — with weight and model_year emerging as the dominant predictors across all configurations; variable interactions exist within the dataset but are weak, as the MSE improvement from degree=1 to degree=2 MARS is modest, suggesting the performance gap between MARS and Lasso is driven primarily by MARS's non-parametric piecewise flexibility rather than interaction modeling alone.
Technical Deep Dive
Methodology & Analysis
Stage 1 — Data Preparation & Multicollinearity Diagnosis
Stage 2 — Lasso Regression · glmnet
Stage 3 — MARS · earth · 21 Configurations (degree × nk)
Stage 4 — Comparative Evaluation & Variable Importance
Stage by Stage
- Stage 1
Data Preparation & Multicollinearity Diagnosis
The Auto MPG dataset contains 398 vehicle records; 6 instances with missing horsepower values are removed, leaving 392 complete observations. The 7 predictors span engine characteristics (cylinders, displacement, horsepower), physical attributes (weight, acceleration), and contextual variables (model year, origin). The name variable is excluded as non-predictive.
75 — a cluster of engine power proxies that will distort standard least-squares coefficients. This collinearity structure motivates the use of regularization (Lasso) and a non-parametric alternative (MARS) that handles correlated predictors through its forward-backward selection procedure rather than global coefficient shrinkage alone.
- Stage 2
Lasso Regression
0. 391 — the L1 penalty is small enough that no coefficient reaches zero. The coefficient path plot shows that as t decreases (equivalently λ increases), model_year and origin are the last to depart from zero — identifying them as the most robustly important predictors. 824.
Lasso cannot model variable interactions by design, making its variable importance assessment limited to individual coefficient magnitudes rather than interaction-adjusted rankings.
- Stage 3
MARS — Forward Selection & Backward Deletion
MARS builds a model of piecewise linear hinge functions via forward selection — iteratively adding basis function pairs that most reduce RSS — then backward deletion removes redundant terms using GCV to avoid overfitting. Cross-validation over degree (1, 2, 3) and nk (10 to 70) selects degree=2 and nk=20 as optimal.
At degree=2, MARS fits interaction terms including weight×model_year, horsepower×model_year, acc×model_year, and acc×origin — capturing how the fuel economy effect of weight, horsepower, and acceleration changes with vehicle era and market origin. Degree=3 performs worse (MSE deteriorates with nk), confirming that three-way interactions overfit this dataset.
Disp is unused across degree=2 configurations despite being a primary power driver, displaced by its highly correlated counterparts.
- Stage 4
Variable Importance & Comparative Evaluation
MARS provides quantitative variable importance via GCV and RSS criteria across all basis function subsets. 0 (unused). The interaction-adjusted ranking shifts acc ahead of horsepower compared to the no-interaction model — confirming that acceleration interacts with model_year and origin in ways that amplify its contribution.
Lasso produces coefficient magnitudes but no ranked importance under multicollinearity, making MARS the superior method for understanding which predictors drive fuel economy and in what order.
Key Methodological Choices
MARS over degree=3 — pairwise interactions optimal, three-way interactions overfit
Cross-validation over all three degree values makes the interaction order a data-driven decision rather than a modeling assumption. 696 at nk=10 — already worse than degree=2 — and deteriorates further as nk increases, indicating that three-way interactions are not supported by the data and produce overfitting when too many basis function terms are added.
350) with the stable plateau behavior that indicates a well-regularized model. The result confirms that fuel economy is shaped by pairwise relationships — how weight interacts with model_year, how horsepower changes meaning across eras — rather than higher-order joint effects.
Lasso at λ=0.001 retains all variables — minimum penalty needed to handle multicollinearity
001 is small enough that no coefficient reaches zero — the L1 penalty shrinks coefficients slightly but does not eliminate any predictor. This reveals that even under the collinearity present in the dataset, all 7 variables contribute marginal predictive information when taken together.
The practical implication is that Lasso's variable selection benefit — zeroing out redundant predictors — does not activate at the MSE-optimal regularization level for this dataset. 5.
MARS performance gap over Lasso driven by flexibility, not interactions alone
466 — modest. 575, much larger. This arithmetic isolates the source of MARS's advantage: most of the 35% MSE improvement over Lasso comes from MARS's piecewise non-parametric flexibility — its ability to fit different linear slopes in different regions of the predictor space — rather than from interaction modeling.
Variable interactions contribute additional improvement but are not the primary driver of the performance difference, confirming that fuel economy has non-linear relationships with individual predictors that Lasso's global linear coefficients cannot represent.
Results & Metrics
What the analysis reveals
- 7.35
- Best MSE — MARS
- At degree=2, nk=20 — pairwise interactions with 6 of 7 predictors retained
- 11.39
- Lasso MSE at λ=0.001
- 35% higher than MARS — all 7 predictors retained, no variable selection at optimal λ
- weight
- Dominant Predictor
- GCV score 100.0 in MARS — most important predictor across every configuration tested
MARS outperforms Lasso by 35% — piecewise flexibility beats L1 regularization on this dataset
MARS at degree=2 and nk=20 achieves MSE of 7.350 versus Lasso's 11.391 — a gap that persists even when MARS is restricted to no interactions (degree=1, MSE 7.816). The performance difference is not primarily explained by MARS's ability to model interactions: MARS without interactions (MSE 7.816) still substantially outperforms Lasso (MSE 11.391). The advantage is structural — MARS's piecewise hinge functions adapt the slope of each predictor's relationship with fuel economy across different value ranges, capturing non-linearities that Lasso's single global coefficient per predictor cannot represent regardless of how well λ is tuned.
Weight and model_year are the two dominant predictors across all methods and configurations
In every MARS configuration — degree=1 and degree=2, nk=10 and nk=20 — weight ranks first (GCV 100.0) and model_year ranks second (GCV 51–52). The consistency is striking: even as other predictors enter and leave the model depending on interactions allowed, and even as the MARS formula changes substantially across configurations, the top two rankings are invariant. This is interpretable: vehicle weight directly determines the power required to maintain speed (physics), and model_year captures the compound effect of technological improvements in engine efficiency, aerodynamics, and fuel systems over time.
Variable interactions exist but are weak — MSE improvement from degree=1 to degree=2 is modest
The optimal degree=2 MARS model (MSE 7.350) outperforms the optimal degree=1 model (MSE 7.816) by 0.466 units — confirming that interactions between weight, horsepower, acc, and model_year are present and detectable. However, the gap is small relative to the total MSE reduction achieved by switching from Lasso to MARS. The interaction terms in Eq. 10 are statistically non-trivial — horsepower×model_year and acc×origin capture meaningful cross-predictor effects — but their incremental contribution to predictive accuracy is modest, meaning the non-linear marginal effects of individual predictors matter more than their joint interactions for this dataset.
Disp is consistently excluded by MARS despite being a primary engine power metric
Engine displacement (disp) is unused in both the degree=1 and degree=2 optimal MARS models — receiving GCV importance score of 0.0. This is not because displacement is physically unimportant: larger displacement engines consume more fuel. Rather, displacement is so highly correlated with cylinders, horsepower, and weight (correlations above 0.75) that once those variables are included, displacement adds no independent predictive signal. MARS's forward-backward selection effectively identifies this redundancy and eliminates displacement in favor of its less-correlated surrogates. Lasso, by contrast, retains disp at the optimal λ=0.001 due to its global coefficient shrinkage mechanism — which does not enforce the hard exclusion that MARS's selection procedure achieves.
MARS provides ranked variable importance; Lasso cannot under multicollinearity
MARS's GCV and RSS-based variable importance measures provide a quantitative, ranked ordering of predictor contributions — weight (100.0), model_year (52.2), acc (24.3), origin (15.7), cylinders (11.8), horsepower (8.9) — that is stable across the degree=2 configuration. Lasso's coefficient magnitudes, while interpretable in isolation, are distorted by multicollinearity: when cylinders, disp, horsepower, and weight share variance, Lasso's coefficients reflect arbitrary splits of the shared explanatory power rather than independent contributions. For practitioners seeking to understand the engineering drivers of fuel economy rather than just prediction accuracy, MARS's interaction-adjusted importance rankings are the more reliable and actionable output.
Tools
Tech Stack
| Technology | Purpose |
|---|---|
| R | Statistical modeling environment and primary implementation language |
| glmnet (R package) | Lasso regression with L1 penalty — 10-fold CV over 12 λ values for optimal regularization |
| earth (R package) | MARS implementation — forward selection of hinge basis functions, backward deletion via GCV, CV over degree & nk |
| 10-Fold Cross-Validation | Parameter selection for both methods — λ for Lasso; degree (1, 2, 3) and nk (10–70) for MARS |
| GCV (Generalized Cross-Validation) | MARS internal criterion for backward deletion — penalizes model complexity during term pruning |