---
license: mit
tags:
  - tabular
  - regression
  - catboost
  - xgboost
  - cann
  - ft-transformer
  - tabm
  - localglmnet
  - drn
  - ensemble
  - house-prices
  - ames-housing
  - kaggle
library_name: catboost
pipeline_tag: tabular-regression
datasets:
  - t22000t/house-prices-tabular
metrics:
  - gini
  - mae
  - rmse
model-index:
  - name: house-prices-tabular-models
    results:
      - task:
          type: tabular-regression
          name: Tabular Regression
        dataset:
          type: t22000t/house-prices-tabular
          name: House Prices - Tabular
        metrics:
          - type: gini
            value: 0.2049
            name: Test Gini (XGBoost, best)
          - type: mae
            value: 17204
            name: Test MAE (XGBoost, USD)
---

# House Prices - Tabular Models (8 architectures + interpretability)

Pre-trained models for the
[t22000t/house-prices-tabular](https://huggingface.co/datasets/t22000t/house-prices-tabular)
dataset, covering all **eight** architectures from the
[tabular-data-modelling-pipeline](https://github.com/timothy22000/tabular_data_modelling_pipeline):
two gradient-boosted machines (CatBoost, XGBoost) and six deep-learning
models (CANN, CANN-GBM, FT-Transformer, TabM, LocalGLMnet, DRN).

> **v3 release** - adds the full interpretability stack: SHAP
> TreeExplainer for the GBMs, Captum Integrated Gradients for the DL
> models, per-layer attention for FT-Transformer, residual analysis
> for CANN / CANN-GBM, per-row coefficients for LocalGLMnet, and full
> distributional outputs (mean, variance, quantiles, VaR) for DRN.
> See [`INTERPRETABILITY.md`](./INTERPRETABILITY.md) and
> [`dashboard_dl_interpretability.html`](./dashboard_dl_interpretability.html).

## Results

All 8 architectures trained with default hyperparameters (no Optuna
tuning), 3-seed ensembles for the DL models, gamma family + log link.

| Rank | Model | Test Gini | Test MAE (USD) | Test RMSE | A/E ratio | n params | Training time |
|---:|---|---:|---:|---:|---:|---:|---:|
| 1 | **XGBoost** | **0.2049** | **17,204** | 29,716 | 0.999 | 462 trees | 0.3 s |
| 2 | CatBoost | 0.1996 | 29,223 | 42,467 | 1.161 | 499 trees | 2.1 s |
| 3 | LocalGLMnet | 0.1991 | 23,420 | 41,406 | 0.988 | 22,620 | 4.5 s |
| 4 | DRN | 0.1962 | 27,928 | 50,086 | 0.981 | 53,010 | 4.8 s |
| 5 | CANN | 0.1941 | 24,906 | 40,935 | 1.024 | 52,815 | 8.2 s |
| 6 | CANN-GBM | 0.1940 | 32,932 | 48,574 | 1.193 | 52,815 | 4.2 s |
| 7 | FT-Transformer | 0.0368 | 187,771 | 203,270 | 3,337.26 | 483,267 | 171.7 s |
| 8 | TabM | 0.0332 | 187,802 | 203,299 | 7,436.72 | 410,364 | 126.8 s |
| - | Stacked ensemble (NNLS) | 0.2049 | 17,204 | 29,716 | 0.999 | (9 weights) | - |

- **Test set:** 304 rows (20% of 1,460)
- **Target:** `SalePrice` (USD)
- **Loss:** Gamma deviance via `reg:gamma` (XGBoost) / `Tweedie:variance_power=1.99` (CatBoost) / explicit gamma NLL (DL)
- **Cap:** 99.5th percentile (= $555,355; 6 rows winsorised)
- **Random seed:** 42

## Why FT-Transformer and TabM underfit

With only ~1,150 training rows, transformer-class models have nothing
like enough data to learn meaningful attention patterns. They converge
to near-constant predictions (MAE ≈ mean target). This is the **expected
behaviour** for these architectures on small tabular datasets - the
literature consistently shows they need 10k+ rows to outperform GBMs.

For a comparison where these architectures actually compete, see the
companion drop on Bike Sharing (17k rows): **[t22000t/bike-sharing-tabular-models](https://huggingface.co/t22000t/bike-sharing-tabular-models)**.

## Interpretability (new in v3)

Multiple methods applied side by side so findings can be triangulated:

| Method | Applies to | What it measures |
|---|---|---|
| **SHAP TreeExplainer** | CatBoost, XGBoost | Per-row Shapley contribution of each feature to the model's log-prediction. |
| **Native importance** (CatBoost / XGBoost) | CatBoost, XGBoost | Loss reduction (CatBoost) / gain (XGBoost). |
| **Captum Integrated Gradients** | All 6 DL architectures | Gradient-based attribution for continuous features, averaged over 50 integration steps. |
| **FT-Transformer attention** | FT-Transformer | Per-layer multi-head self-attention weights, averaged across heads. |
| **CANN / CANN-GBM residual analysis** | CANN, CANN-GBM | Distribution of the neural network's correction to the GBM/GLM base. Here: mean=0.005, std=0.107 -> the NN added almost no correction beyond what the base already captured. |
| **LocalGLMnet coefficients** | LocalGLMnet | Per-row linear coefficients - exposes *how* the effective regression formula varies across the input space. |
| **DRN distributional output** | DRN | Full predicted gamma distribution per row. Here: mean shape ≈ 1.04, CoV ≈ 0.98, VaR95 ≈ $1.18M, VaR99 ≈ $2.5M. The wide upper tail is consistent with House Prices' heavy right skew. |

### Cross-method consensus (the high-confidence finding)

Both CatBoost and XGBoost ranked the following four features in their
**top-5 most important** for predicting `SalePrice`. Cross-method
agreement is a strong signal - independent methods identifying the
same drivers means the finding is unlikely to be a method-specific
artefact:

| Feature | CatBoost | XGBoost | Interpretation |
|---|---:|---:|---|
| `TotalSF` (derived) | 0.096 | **0.138** | Total square footage (basement + 1st + 2nd floor). Dominant predictor across both models. |
| `OverallQual` | 0.043 | 0.103 | Material and finish quality 1-10. Strongly monotonic with price. |
| `GrLivArea` | 0.043 | 0.032 | Above-grade living area sq ft. |
| `YearRemodAdd` | 0.038 | 0.025 | Year of most recent remodel - matters more than `YearBuilt`. |

Note the derived feature **`TotalSF`** (defined in
[`configs/example_house_prices.py`](https://github.com/timothy22000/tabular_data_modelling_pipeline/blob/master/configs/example_house_prices.py)
as `TotalBsmtSF + 1stFlrSF + 2ndFlrSF`) is more predictive than any
of its components individually - the pipeline's `derived_features`
mechanism paid off here.

Full breakdown - per-method top-10 tables, LocalGLMnet coefficient
distributions, sign-stability analysis - is in
[`INTERPRETABILITY.md`](./INTERPRETABILITY.md). For the interactive
plots see
[`dashboard_dl_interpretability.html`](./dashboard_dl_interpretability.html).

## Files

| File | What it is |
|---|---|
| `catboost.cbm` | Trained CatBoost (Tweedie:variance_power=1.99) |
| `xgboost.json` | Trained XGBoost Booster (reg:gamma) |
| `cann_member{0,1,2}.pt` | CANN 3-seed ensemble |
| `cann_gbm_member{0,1,2}.pt` | CANN-GBM 3-seed ensemble |
| `ft_transformer_member{0,1,2}.pt` | FT-Transformer 3-seed ensemble |
| `tabm_member{0,1,2}.pt` | TabM 3-seed ensemble |
| `localglmnet_member{0,1,2}.pt` | LocalGLMnet 3-seed ensemble |
| `drn_member{0,1,2}.pt` | DRN 3-seed ensemble |
| `evaluation_summary.csv` | Per-model train/test metrics |
| `ensemble_weights.json` | NNLS weights over the 8 base predictions |
| `dashboard_dl_models.html` | Performance dashboard (Lorenz, calibration, A/P scatter) |
| **`dashboard_dl_interpretability.html`** | **Interpretability dashboard** (SHAP, IG, attention, residuals) |
| **`feature_importance.csv`** | **Consolidated importances** across CatBoost + XGBoost |
| **`localglmnet_coefficients.csv`** | **LocalGLMnet per-row coefficients** (304 rows × 16 continuous features) |
| **`drn_distributional_outputs.csv`** | **DRN per-row distributional moments** (mean, variance, VaR95, VaR99) |
| **`INTERPRETABILITY.md`** | **Human-readable interpretability summary report** |
| `figures/fig_dl_*.png` | Standalone publication figures (incl. attention heatmap) |
| `model_summary.json` | Structured run record |

## Loading and inference

### CatBoost (single best non-XGB baseline)

```python
from huggingface_hub import hf_hub_download
from catboost import CatBoostRegressor
import pandas as pd

path = hf_hub_download("t22000t/house-prices-tabular-models", "catboost.cbm")
model = CatBoostRegressor()
model.load_model(path)

df = pd.read_csv("hf://datasets/t22000t/house-prices-tabular/train.csv")
features = [  # see configs/example_house_prices.py for the full list
    "LotArea", "YearBuilt", "YearRemodAdd", "TotalBsmtSF", "1stFlrSF",
    "2ndFlrSF", "GrLivArea", "FullBath", "BedroomAbvGr", "TotRmsAbvGrd",
    "GarageCars", "GarageArea", "OverallQual", "OverallCond",
    "MSZoning", "Street", "LotShape", "Neighborhood", "BldgType",
    "HouseStyle", "RoofStyle", "ExterQual", "Foundation", "Heating",
    "CentralAir", "KitchenQual", "SaleType", "SaleCondition",
]
preds = model.predict(df[features])
```

### XGBoost (best overall on this dataset)

```python
from huggingface_hub import hf_hub_download
import xgboost as xgb

path = hf_hub_download("t22000t/house-prices-tabular-models", "xgboost.json")
booster = xgb.Booster()
booster.load_model(path)
```

XGBoost predictions need the exact feature order used at training time.
Clone the [pipeline repo](https://github.com/timothy22000/tabular_data_modelling_pipeline)
to reproduce the preprocessing path.

### Deep-learning models (CANN, CANN-GBM, FT-T, TabM, LocalGLMnet, DRN)

Each architecture ships as a 3-seed ensemble of PyTorch state-dicts.
Reconstructing them requires the matching architecture class from
the pipeline package. Easiest path:

```bash
git clone https://github.com/timothy22000/tabular_data_modelling_pipeline
cd tabular_data_modelling_pipeline
pip install -e ".[all]"
```

Then load each ensemble member's state dict into the right module
(see `modelling/models/cann.py` etc).

## Training configuration

| Setting | Value |
|---|---|
| Pipeline | [tabular-data-modelling-pipeline](https://github.com/timothy22000/tabular_data_modelling_pipeline) v0.1.0 |
| Architectures | All 8 (catboost, xgboost, cann, cann_gbm, ft_transformer, tabm, localglmnet, drn) |
| Hyperparameters | Defaults - **no Optuna tuning** |
| DL ensemble size | 3 seeds per architecture |
| Family / link | Gamma / log |
| XGBoost objective | `reg:gamma` |
| CatBoost loss | `Tweedie:variance_power=1.99` (CatBoost has no native Gamma loss) |
| Train/test split | Random 80/20, seed 42 |
| Cap percentile | 99.5 |
| Hardware | Apple M-series, MPS device for DL |
| Total wall-clock | ~5.4 min |

To reproduce:
```bash
git clone https://github.com/timothy22000/tabular_data_modelling_pipeline
cd tabular_data_modelling_pipeline
pip install -e ".[all]"
python scripts/download_data.py --dataset house_prices

OMP_NUM_THREADS=1 python train.py \
    --config configs/example_house_prices.py \
    --input data/house_prices.csv \
    --skip-tuning --skip-interpretability \
    --architectures catboost xgboost cann cann_gbm ft_transformer tabm localglmnet drn
```

(`OMP_NUM_THREADS=1` is only needed on macOS arm64; Linux is unaffected.)

## Limitations

- **No Optuna tuning.** Defaults only. With tuning, expect ~0.02-0.04 Gini lift across the board.
- **FT-Transformer & TabM underfit.** Dataset is too small (~1.2k training rows) - see the companion Bike Sharing drop for a fair comparison.
- **No interpretability artefacts.** Captum attributions and partial-dependence plots were skipped to keep wall-clock manageable. The pipeline supports them - drop `--skip-interpretability` to compute on re-run.
- **Random split, not stratified.** SalePrice is right-tailed; a quantile-stratified split would give a more representative test set.
- **Trained on Kaggle's train.csv only** (test.csv is unlabelled). Not directly comparable to the Kaggle leaderboard.

## Intended use

- **Baseline for tabular DL research** on small-N regression problems.
- **Teaching** end-to-end tabular pricing pipelines (8 architectures with one CLI).
- **Sanity check** for reimplementations of any of the 8 architectures on gamma-family data.

## Citation

```bibtex
@software{tabular_data_modelling_pipeline,
  author = {Mun, Timothy},
  title  = {tabular-data-modelling-pipeline},
  url    = {https://github.com/timothy22000/tabular_data_modelling_pipeline},
  year   = {2026}
}

@article{decock2011ames,
  author  = {De Cock, Dean},
  title   = {Ames, Iowa: Alternative to the Boston Housing Data},
  journal = {Journal of Statistics Education},
  volume  = {19},
  number  = {3},
  year    = {2011}
}
```

Please also cite the individual architecture papers - see the
[main repo README](https://github.com/timothy22000/tabular_data_modelling_pipeline#the-eight-architectures)
for the full reference list.

## License

MIT for the model code and pipeline. Underlying dataset under Kaggle
competition terms (free use with attribution).

## Related

- 📂 [Dataset: t22000t/house-prices-tabular](https://huggingface.co/datasets/t22000t/house-prices-tabular)
- 🤖 [Companion: t22000t/bike-sharing-tabular-models](https://huggingface.co/t22000t/bike-sharing-tabular-models) - Poisson family, 17k rows
- 📦 [Pipeline: tabular-data-modelling-pipeline](https://github.com/timothy22000/tabular_data_modelling_pipeline)