Hosted by the DSI Foundations of Data Science Center; Department of Statistics, Arts and Sciences; and Columbia Engineering
Speaker: Danqi Chen, Associate Professor of Computer Science, Co-Leader of Princeton NLP Group, Associate Director of Princeton Language and Intelligence, Princeton University
Registration for all CUID holders is preferred. If you do not have an active CUID, registration is required and is due at 12:00 PM the day prior to the seminar. Unfortunately, we cannot guarantee entrance to Columbia’s Morningside campus if you register following 12:00 PM the day prior to the seminar. Thank you for understanding!
REGISTER
From Needle-in-a-Haystack to Long-Horizon Agents: A Retrospective on Long-Context Language Models
Abstract: Language models' context sizes have rapidly increased from thousands to millions of tokens, reshaping how we build and use these models. In this talk, I will trace this evolution along three dimensions: (1) how we think about training long-context language models from data (and architecture) perspectives, (2) how our evaluation and applications have shifted — from synthetic retrieval tests to test-time scaling and long-horizon agents, and (3) how we should rethink inference and scaffolding to make better use of long context, beyond naively filling the context window. I will draw on recent work from our group on long-context model training, evaluation, and effective context management for long-horizon agentic tasks.