Tabular foundation models must beat gradient-boosted local baselines

Tabular foundation models must beat gradient-boosted local baselines before I give them a production role. A model trained across many tasks brings a useful hypothesis about my data, not evidence that it improves my decision. I want the comparison against a credible XGBoost, LightGBM, or CatBoost pipeline on the actual prediction problem. A public benchmark earns a candidate an experiment; it does not earn it the deployment.

The baseline matters because gradient boosting is not a placeholder for teams that have not discovered transformers. The tabular research spans regularization and column subsampling in XGBoost, histogram-based learning in LightGBM, and ordered boosting and categorical statistics in CatBoost. These are different ways of making structured prediction practical. I would choose the baseline around the table’s characteristics rather than deliberately select an inconvenient tree implementation and call its defeat progress.

I also distinguish the neural candidates instead of treating them as one replacement wave. A feature-tokenizing transformer learns over feature representations. A prior-fitted network such as TabPFN transfers inference learned from synthetic tasks. An LLM-based predictor can consume a serialized row, while CAAFE uses an LLM to generate feature-engineering code. Those interventions change different parts of the pipeline. If generated features improve a boosted model, that is evidence for better feature construction, not proof that trees should disappear.

My first requirement is a shared information boundary. Every candidate must predict the same target using information available at the same decision point. Model-specific preprocessing is legitimate; giving one candidate access to labels or future observations is not. I fit learned preprocessing within the training partition and keep target-derived categorical statistics away from held-out labels. Before comparing architectures, I want to know whether the apparent winner learned the task or benefited from a leak.

The split is part of that boundary. I use stratification where class representation is the concern and grouped validation where repeated entities would otherwise cross partitions. If I select hyperparameters using cross-validation, I need an evaluation boundary that selection has not consumed; nested validation is one way to establish it. The same discipline applies to imputation, feature selection, and imbalance handling. A carefully named model cannot rescue a comparison whose training procedure has already seen the answer.

I set the selection rule before examining the results. The metric should reflect the decision, especially when classes are imbalanced, and I inspect important segments rather than letting an aggregate score settle the matter. I declare tuning budgets and record preprocessing effort, prediction latency, and resource requirements alongside predictive quality. An untuned tree against an extensively selected neural pipeline is not a useful purchasing argument. Neither is insisting on identical preprocessing when the architectures have different input requirements.

For row-as-text approaches, I add representation checks. Serialization, token budgets, and column ordering are explicit concerns in the tabular research. I want to know whether a harmless reordering changes predictions and whether a wide record loses relevant fields at the input boundary. These are testable properties of the proposed system, not reasons to reject language models by category. The comparison should include the adapter that turns the table into model input, because production will depend on it.

I make one precise concession: for small classification datasets in the regime where TabPFN has demonstrated strong results, I would test it immediately rather than require a lengthy tree-tuning exercise first. The evidence for in-context inference makes that a rational experiment order. It does not establish superiority on a different table or excuse dropping the local control.

After selection, I keep the baseline reproducible and retain the data split, feature logic, model version, and acceptance rule. Distribution monitoring tells me when the original comparison deserves another run; new labels let me check whether its conclusion still holds. The outcome may be a foundation model, a boosted ensemble, or an LLM-assisted feature pipeline. My commitment is to a repeatable local advantage, not to preserving either generation of architecture.