# Evaluation of large language models within GenAI in qualitative research **Author:** Supriya D. Mehta, Souvik Paul, Enid Awiti, Sophie Young, Garazi Zulaika, Fredrick O. Otieno, Penelope A. Phillips-Howard, Linda Mason and Runa Bhaumik **Published:** 7 October 2025 **Length:** 33 mins (8,596 words) ## In Summary Published in _Scientific Reports_, this study compares GPT-4o's analysis of focus group transcripts with analysis conducted by experienced qualitative researchers. While AI identified broadly relevant themes, it struggled to distinguish their significance and select supporting quotations. Errors included altered, truncated and misleading quotations, alongside limited cultural and contextual understanding. The researchers conclude that large language models can assist with identifying themes, keywords and potential biases, but cannot yet replace the interpretive rigour and judgement of experienced researchers. **Source:** [https://www.nature.com/articles/s41598-025-18969-w](https://www.nature.com/articles/s41598-025-18969-w) ## What might this mean for us? As Higher Education explores AI-supported documentation and analysis, how can it preserve the context and meaning of participants' contributions? Electric Dreams' dialogic capture pilots provide opportunities to test AI-assisted thematic analysis against collective human interpretation. Could careful verification of quotations, transparent analytical processes and participant review make these tools useful without surrendering academic judgement? ## Links in this reading - [Original research article, Scientific Reports](https://doi.org/10.1038/s41598-025-18969-w) - [Thematic analysis in psychology, Braun and Clarke](https://doi.org/10.1191/1478088706qp063oa) - [Google Colaboratory](https://colab.research.google.com/) - [OpenAI API documentation](https://platform.openai.com/docs/overview)