TAP Computation & Data Initiative Computational Workshop

Joshua Speagle, University of Toronto

When

Next event: 11 a.m. – 12:30 p.m., Dec. 15, 2026

Where

Steward Observatory, Room 550

Joshua Speagle, University of Toronto - Visit Dates: December 14-16, 2026

Two Sessions: Dec 15 & 16

Title: Exploring Tabular Foundation Models in Astronomy

Abstract / overview
Tabular foundation models (TFMs) such as TabPFN take a table of training data as input and return predictions for new rows in a single forward pass, with no training on the user's side. On many small and medium datasets, they now match or outperform carefully tuned methods. This hands-on workshop will cover:

1. What are tabular foundation models, and how do they compare with familiar tools?
2. How do they turn a training set into a prediction?
3. Why do they work as well as they do?

To answer the last two questions, participants will also learn how to run simple experiments on the model, including comparisons to Gaussian processes and "standard" machine-learning methods. Examples will be drawn from astronomy, including galaxy redshifts. No experience training neural networks will be needed.

Learning objectives:
At the end of the workshop, participants will be able to:
1. Describe what a foundation model is.
2. Discuss how a tabular foundation model differs from a custom model trained on your own data.
3. Apply a tabular foundation model to a problem and compare its predictions with standard machine-learning models.
4. Explain how the model produces a prediction from a training set given as input (in-context learning).
5. Test how its predictions respond to changes in its context and inputs.
5. Explain the hypothesis that these models approximate Bayesian posterior predictive distributions under a pretraining prior.
6. Design and run a controlled comparison against models with known assumptions to test why the model works on a given problem.

Suggested pre-reading (optional):
Müller et al. (2022), "Transformers Can Do Bayesian Inference," ICLR
Hollmann et al. (2025), "Accurate predictions on small data with a tabular foundation model," Nature

Resources Required:
Participants should have a laptop prepared, and either have access to Google Collab or a local version of Python with Jupyter and PyTorch and/or JAX installed (likely all done via this package: https://github.com/biprateep/lazy-tfm).

To pre-download some of the models, registered participants will receive more detailed instructions. No large setup required except confirming access to a live notebook environment and the ability to compile PyTorch/JAX against GPUs.