Savannah Thais | Measurement for Effective and Ethical AI

As AI systems increasingly shape critical aspects of society, the need for robust, transparent methods to measure their behavior has never been more urgent. Leveraging my background in particle physics, I have focused my research on designing rigorous, domain informed approaches to measuring AI performance and impact.

In the first half of this talk I will introduce a few of my recent research projects spanning benchmark development for agentic AI systems, novel policy impact analysis frameworks, and technical literacy development. In the second half of the talk I will introduce our exciting new Human Rights Benchmark project which aims to assess language model performance through the lens of internationally recognized human rights. I will outline our methodology, including prompt development and model evaluation strategies, and discuss how this work contributes to more accountable and socially aligned AI systems.

Dr. Savannah Thais is an Associate Research Scientist in the Columbia Data Science Institute with a focus on machine learning (ML). She is interested in complex system modeling and in understanding what types of information is measurable or modelable and what impacts designing and performing measurements have on systems and societies. This work is informed by her background in high energy particle physics and incorporates traditional scientific experiment design components such as uncertainty quantification, experimental blinding, and decorrelation/de-biasing methods. Her recent work has focused on geometric deep learning, methods to incorporate physics-based inductive biases into ML models, regulation of emerging technology, social determinants of health, and community education.

Dr. Thais is the founder and Research Director of Community Insight and Impact, an non-profit organization focused on data-driven community needs assessments for vulnerable populations and effective resource allocation. She is passionate about the impacts of science and technology on society and is a strong advocate for improving access to scientific education and literacy, community centered technology development, and equitable data practices. She was the ML Knowledge Convener for the CMS Experiment from 2020-2022, currently serves on the Executive Board of Women in Machine Learning and the Executive Committee of the APS Group on Data Science, and is a Founding Editor of the Springer AI Ethics journal.

Dr. Thais received her PhD in Physics from Yale University in 2019 and was a postdoctoral researcher at Princeton University from 2019-2022.