Why Exactly LLMs are Non-Deterministic and Can We Tame This Probabilistic Beast?
she/her
Rimma is a Senior Data Scientist at VGW with over nine years of experience in the field. Having built her career on the foundations of traditional machine learning, she has spent recent years diving deep into the architecture and utility of Large Language Models. A curious consumer of scientific papers, Rimma isn't content with just using LLMs as "black boxes." She is driven to look past the API layer and understand the hardware-level mechanics—such as floating-point precision—that dictate how these models behave in production. Rimma is a seasoned speaker within the Australian tech circuit, having previously presented his insights at conferences including YOW! Data, CDAO, and DDD.
You’ve set your temperature to zero, fixed your seed, and double-checked your parameters and yet your LLM still gives a different answer every now and then. In traditional programming, this would be a bug. In the world of Large Language Models, we’re often told it’s just “the nature of the beast”, we are told ""to grow up from our deterministic ideas of traditional software and simply accept the chaos"". When trying to build reliable agentic systems, these ""random"" hallucinations or slight shifts in logic can be the difference between a production-ready tool and a prototype that works ... until it doesn't. I initially chalked up LLM inconsistency to ""GPUs are messy"" concept. That changed when I encountered a research paper by Thinking Machines titled ""Defeating Nondeterminism in LLM Inference."" Reading it sparked a rabbit hole of experiments, debates with colleagues, and deep-dives into the hardware-level mechanics of AI. In this session, I will move beyond the theory and use animations to explain the exact source of non-determinism—specifically how floating-point non-associativity in matrix multiplication and batching dynamics create bit-level differences that compound into different answers. What we will cover: - The Matmul Micro-Grain: who knew matrix multiplication could be so riveting? - Local vs. Cloud: Comparing non-determinism across local LLMs and major cloud providers. - The ""Thinking Machines"" Solution: An investigation into the paper’s proposed fixes, including my own experiments with VLLM and batch invariance. Join me to find out if we can finally make our LLMs predictably creative and smart, giving the exact same good answer to the exact same question every single time.
