NeurIPS 2026 · Education Track Website
Tabular Foundation Models

An Interactive Guide to Tabular Foundation Models

See how pretrained transformers use labeled rows as context and predict targets for new rows in a single forward pass, without task-specific gradient updates.

Start reading Skip to the live model
scroll ↓

01 · The problem

Why tabular prediction is difficult

Tabular data underpins applied machine learning in spreadsheets, logs, transactions, and clinical records. Its columns can mix numbers, categories, dates, and missing values, while each dataset can introduce a new schema and prediction task.

Gradient-boosted trees such as XGBoost and LightGBM remain strong task-specific baselines. They fit new parameters for each dataset, while tabular foundation models try to reuse a prediction procedure learned during pretraining.

Online shopping sessions under stress

Five records from the UCI Online Shoppers dataset ↗. The final session is the query, so its Revenue label is hidden.

last row = query · ? = hidden Revenue target · select a column to inspect it

common approach

Fit a tree ensemble for this table

Gradient boosting builds an additive score function one tree at a time. XGBoost uses first- and second-order derivatives of a regularized objective to choose splits and leaf weights. [1] [2]

training
Labeled table 5 rows · (xᵢ, yᵢ)
Loss derivatives gradient gᵢ · Hessian hᵢ
Fit + add one tree Fₜ = Fₜ₋₁ + η fₜ
prediction
New row x
Sum fitted-tree scores F(x) = F₀ + Σ η fₜ(x)
Transform score sigmoid · softmax · identity
primary sources
  1. Friedman, Gradient Boosting Machine (2001)
  2. Chen and Guestrin, XGBoost (KDD 2016)

02 · Adaptation

Fit a model or provide context

A tree ensemble learns a new set of trees from each labeled table. Later predictions need the fitted ensemble and the new feature rows.

In-context TFMs such as TabPFN and TabICL keep their pretrained network weights fixed. They process labeled context rows and query rows together during prediction. [TFM book] [TabPFN]

The difference is where task adaptation happens: fitted parameters or inference context.

Five training rows and one row to predict
training data 5 labeled rows
prediction input 1 query row
A
gradient-boosted trees

Fit a model for the table

Gradient-boosted tree adaptation Five labeled rows first fit a tree ensemble. A copy of that fitted ensemble is then used to predict one new row without fitting again. TRAIN · 5 LABELED ROWS training data fit FITTED TREE ENSEMBLE copy fitted model PREDICT · 1 NEW ROW ? query row appears here SAME FITTED ENSEMBLE prediction
first fit on 5 rows; then apply the fitted model to 1 row
B
pretrained TFM

Provide examples as context

Tabular foundation model adaptation Five labeled context rows and one query row enter a fixed pretrained model together. Only the query row receives a prediction. PREDICT · ALL 6 ROWS ENTER TOGETHER ? 5 LABELED CONTEXT ROWS 1 query row is part of the table one pass PRETRAINED θ* · FIXED context and query interact inside query prediction 5 context rows + 1 query row → prediction
no new model is fitted; the 5 rows remain as context
If the model is not trained on this table, where did its predictive behavior come from?

03 · Theory

Prior-Data Fitted Network (PFN)

A prior-data fitted network, or PFN, learns a prediction rule across sampled supervised tasks. Each episode reveals context labels, hides query labels, and uses query loss to improve one shared predictor.

In the PFN formulation, minimizing expected query loss makes the network approximate the posterior-predictive distribution induced by the sampled task prior. [PFN]

the 30-second picture

PFN learns how to learn from a table

Instead of fitting a new model to one dataset, pretraining repeats the same four-step lesson across many temporary datasets.

  1. 01
    Sample a taskA fresh table comes from the prior.
  2. 02
    Make an episodeShow context labels; hide the query label.
  3. 03
    Predict the queryOne shared network reads the examples.
  4. 04
    Learn from the errorUpdate θ, discard the temporary table.

Repeat across tasks. The datasets disappear; the prediction strategy remains in θ.

At inference: freeze θ and use a new table only as context.

now, one step at a time

Follow one purchase-prediction episode

The same illustrative shopping task runs through every step below.

01
sample

Start with a fresh supervised task

The prior defines what kinds of tables and relationships the PFN practices on.

formula D∼p(D)
D
one temporary labeled table
p(D)
the task-generating prior
mini visual · choose another episode

Each click changes the episode below; the shared parameters θ stay the same.

02
split

Turn the task into context and a query

The network sees worked examples and one question whose answer is hidden.

formula D=Dc∪Dq Dc = {(xi, yi)}  ·  Dq = {(xq, ?)}
mini visual · selected episode D¹
C10 C21 C30 C41 C51 Q?
featuresvisible Revenuehidden Revenue
03
predict

Use the context to predict the hidden answer

The same network parameters serve every task; the context changes what prediction they produce.

formula qθ(yq|xq,Dc) fixed θ + changing context → task-specific prediction
mini visual · predicted probabilities Revenue for the query session
037%
163%
04
learn

Reveal the answer and update the shared predictor

This happens during PFN pretraining, then repeats with another temporary task.

formula yq = 1; NLL = −log qθ(yq = 1 | xq, Dc) = −log 0.63 = 0.46 θ←θ−η∇θℒ
mini visual · what survives?
reveal 1→update θ→discard D¹
θ′ continues to the next task
interactive · synthetic task generator

Watch a synthetic table come into existence

First sample one data-generating mechanism. Then reuse that same mechanism with fresh random noise to create many related rows.

Synthetic does not mean filling every cell with unrelated random numbers. One sampled SCM creates a coherent relationship; only the noise changes from row to row.
01
sample once per task

Choose a causal graph and its functions

See the sampled equations
z = f(ε₁, ε₂) x₁ = g₁(z, εₓ₁) x₂ = g₂(z, ε₂, εₓ₂) y = 1[h(z, x₂, εᵧ) > 0]
02
sample again for every row

Draw noise, then run the equations forward

fresh noise
ε₁–
ε₂–
εᵧ–
hidden cause
z–
saved row
X₁–
X₂–
y–
03
the resulting synthetic task

The generator creates X and y; the episode masks query y later

No rows yet. Generate one row to see its complete causal trace.

rowX₁X₂y
The sampled mechanism is ready. No observations have been drawn yet.
Generator output: complete rows (X, y) PFN input: context y visible · query y masked masked query y still supplies the training loss
put all four steps together

Minimize expected query loss across the task prior

With enough capacity and representative tasks, this trains qθ to approximate the prior's posterior-predictive distribution.

θ*= arg minθ ED∼p(D) [−log qθ(yq| xq,Dc)]

Important boundary: the approximation is relative to the chosen task prior, not a universal Bayesian predictor for every possible table.

Now connect the PFN idea to one concrete model architecture.

04 · Architecture

Try it.Train it.Run inference.

PFN describes a training formulation, not one required architecture. This tutorial makes it concrete with nanoTabICL, a compact implementation of the TabICLv2 graph. TabICLv2 applies the PFN objective to synthetic tasks; nanoTabICL exposes its core inference graph in a compact educational implementation. [PFN] [TabICLv2] [nanoTabICL]

part 1 · try it out

Compare four predictors on the same task

All four answer the same question: given labeled rows and one query, what class should the query get? Logistic regression and the tree fit task-specific parameters; the frozen TFM instead adapts through inference context without updating its weights.

kNN

Nearby labeled rows vote. Spatial distance decides which rows count as evidence.

Frozen TFM

The same context passes through fixed weights. Learned attention decides how each row is read.

Compare them side by side below: the labeled points and query stay fixed while each model produces its own decision field.

four-model comparison

Same data and query across four predictors

Linear, tree, kNN, and TFM use one shared dataset and query. Drag any ★ to update the complete 2×2 comparison.

side-by-side 2D

All four canvases use the same 2D context and query.

LinearLogistic boundary
TreeDepth-3 decision tree
kNNNearest-neighbor field
Tabular Foundation ModelTFM field
Loading Tabular Foundation Model
28 MB checkpoint 0%
Starting automatically… First load may take several seconds. After downloading, the browser prepares 8 views.

TFM note: its boundary is a single-view preview for speed; its displayed probability is the full 8-view ensemble. This 20-row context is an out-of-regime illustration: TabICLv2 was pretrained on 300 to 48K rows, and the official authors have not tested generalization below 300 rows. See the official FAQ ↗

part 2 · training

How does the model learn this behavior across synthetic tasks?

nanoTabICL implementation · TabICLv2 training graph

Follow one toy task through nanoTabICL’s training graph

The graph and tensor dimensions are faithful; the small task and output values are illustrative. Real TabICLv2 pretraining samples graph-SCM tasks in batches and does not run in this browser.

Step 01 explainerStandardize first, then shift and wrap features

nanoTabICL first standardizes every feature using statistics from the context rows only. It then follows the TabICLv2 paper convention: every position collects offsets 0, +1, and +3, wrapping past the final column.

context standardization x′ij = (xij − μj,context) / (σj,context + ε)
zero-based position j group(j) = [x′1+(j mod d), x′1+((j+1) mod d), x′1+((j+3) mod d)]
x₁[x′₁, x′₂, x′₄]
x₂[x′₂, x′₃, x′₁]
x₃[x′₃, x′₄, x′₂]
x₄[x′₄, x′₁, x′₃]

The shared Linear(3→128) keeps four output positions while giving each feature repeated local context. Context cells then receive y_embed_in(yᵢ); query cells receive no label embedding. The frozen browser checkpoint preserves TabICL 2.0.2’s cyclically shifted +1,+2,+4 ordering, while this training explainer follows the paper and current nanoTabICL source.

step 1 · prepare and embed

Standardize from context, then create repeated feature groups

Use context mean and standard deviation, group offsets 0, +1, and +3 with wrap-around, project each group to 128D, then add the first label embedding to context cells only.

10×4 → standardize → 10×4×3 → 10×4×128 + y_embed_in(context)
PFN paper ↗ TabICLv2 paper ↗ nanoTabICL repository ↗
part 3 · inference

How does the frozen graph use context to answer a query?

Inference repeats the same preprocessing and 3-column / 3-row / 12-ICL-block forward graph, but stops after prediction. Context labels are embedded; the query label is unavailable; pretrained weights remain frozen. There is no loss, backpropagation, or task-specific optimizer step in this diagram. [Qu et al., 2026]

same toy table as training 9 context rows + 1 query row 4 features 2 active classes weights frozen
same graph · inference mode · weights frozen

Trace the same toy table without an update step

This is a shape-faithful schematic. Select a stage to inspect what is reused from training and what changes at inference.

stage 1 · cells and cyclic grouping

Standardize, group each feature position, then embed it

Use context statistics, then gather j, j+1, and j+3 modulo d under the paper/current nanoTabICL convention. For d=4, x₁ maps to [x′₁, x′₂, x′₄]. Context cells receive y_embed_in; the query does not.

10×4 + separate context y → z-score → 10×4×3 → 10×4×128
The model graph is clear. Next, inspect a real forward pass.

05 · Live model explorer

Trace a real Iris prediction

The embedded explorer runs mapped upstream TabICLv2 classifier weights locally. Select a query, follow the context rows through the model, and inspect real selected-view attention and output probabilities.

Scope: its 12-row Iris context is deliberately small for tracing and is outside TabICLv2's documented 300 to 48K-row pretraining regime. Treat the output as an out-of-regime illustration, not evidence of expected model quality.

Implementation note: The main playground runs the complete eight-view browser classifier. This embedded inspector uses the selected-view nanoTabICL-shaped bridge adapted to the released TabICLv2 checkpoint so its intermediate tensors remain traceable. It is not the standalone nanoTabICL model.

The explorer scales its original 1880px visualization to this frame. Open full screen ↗

06 · References

References

Primary papers, implementations, checkpoints, datasets, and evaluation suites cited throughout this tutorial.

models

Tabular foundation models

evaluation

Evaluation benchmarks

TALENT

Broad toolbox and benchmark for classical and deep tabular methods.

TabZilla

A suite of 36 difficult datasets for comparing tabular algorithms.

supporting work

Methods, data, and materials

Code and checkpoints can use different licenses. Check upstream terms before reuse.

Authors: Jiarui Xu, Tianming Zhou, Tommy Tai, Joshua Pan, Alexander Statnikov, Nitesh Kumar