Incredible Yet Limited Large Language Models in the Wild

Date: Nov 8, 2:05pm – 3:15pm

Abstract: Large language models (LLMs) demonstrate remarkable capabilities in handling a wide range of tasks, from generating human-like text to answering complex questions. Despite achieving remarkable performance on existing benchmarks, the effectiveness and limitations of LLMs in realistic scenarios remainlargely underexplored. In this talk, I will introduce our work on evaluating LLMs across diverse domains, including psychology, arithmetic, and medicine. I will highlight several intriguing findings regarding LLM capabilities: 1) Like humans, LLMs fall into common cognitive traps in decision-making, exhibiting representativeness heuristic biases; 2) While LLMs can solve advanced math problems, they struggle with basic arithmetic
due to their reliance on symbolic learning; 3) LLMs can achieve passing scores on medical board exams but fail to answer challenging medical questions or provide meaningful explanations. Finally, I will discuss future directions for enhancing the explainability and performance of LLMs in real-world applications.

Biographical Sketch: Hanjie Chen is an Assistant Professor in the Department of Computer Science at Rice University, affiliated with the Ken Kennedy Institute. Prior to joining Rice, she was a Postdoctoral Fellow in the Center for Language and Speech Processing at Johns Hopkins University from 2023 to 2024. She completed her Ph.D. in Computer Science at the University of Virginia in May 2023. Her research interests lie in Natural Language Processing and Interpretable Machine Learning, with a focus on explaining and evaluating neural language models and enhancing their impact on real-world applications such as medicine and sports. Her work has been published in leading NLP and AI venues, including ACL, EMNLP, NAACL, and AAAI. She has served as an Area Chair for EMNLP 2024, COLING 2025, NAACL 2025, and a Senior Area Chair for ACL 2025. She has been honored with the Outstanding Doctoral Student Award, John A. Stankovic Graduate Research Award, Carlos and Esther Farrar Fellowship, Graduate Teaching Awards at UVA, and the Best Poster Award at the ACM Capital R= egion Celebration of Women in Computing.

Location and Zoom link: 307 Love, or https://fsu.zoom.us/j/7153751215