ChatGPT for Automated Qualitative Research: Content Analysis

Simple item page
aut.relation.journalJournal of Medical Internet Research
aut.relation.startpagee59050
aut.relation.volume26
dc.contributor.authorBijker, R
dc.contributor.authorMerkouris, SS
dc.contributor.authorDowling, NA
dc.contributor.authorRodda, SN
dc.date.accessioned2024-08-09T03:08:50Z
dc.date.available2024-08-09T03:08:50Z
dc.date.issued2024-07-25
dc.description.abstractBackground: Data analysis approaches such as qualitative content analysis are notoriously time and labor intensive because of the time to detect, assess, and code a large amount of data. Tools such as ChatGPT may have tremendous potential in automating at least some of the analysis. Objective: The aim of this study was to explore the utility of ChatGPT in conducting qualitative content analysis through the analysis of forum posts from people sharing their experiences on reducing their sugar consumption. Methods: Inductive and deductive content analysis were performed on 537 forum posts to detect mechanisms of behavior change. Thorough prompt engineering provided appropriate instructions for ChatGPT to execute data analysis tasks. Data identification involved extracting change mechanisms from a subset of forum posts. The precision of the extracted data was assessed through comparison with human coding. On the basis of the identified change mechanisms, coding schemes were developed with ChatGPT using data-driven (inductive) and theory-driven (deductive) content analysis approaches. The deductive approach was informed by the Theoretical Domains Framework using both an unconstrained coding scheme and a structured coding matrix. In total, 10 coding schemes were created from a subset of data and then applied to the full data set in 10 new conversations, resulting in 100 conversations each for inductive and unconstrained deductive analysis. A total of 10 further conversations coded the full data set into the structured coding matrix. Intercoder agreement was evaluated across and within coding schemes. ChatGPT output was also evaluated by the researchers to assess whether it reflected prompt instructions. Results: The precision of detecting change mechanisms in the data subset ranged from 66% to 88%. Overall κ scores for intercoder agreement ranged from 0.72 to 0.82 across inductive coding schemes and from 0.58 to 0.73 across unconstrained coding schemes and structured coding matrix. Coding into the best-performing coding scheme resulted in category-specific κ scores ranging from 0.67 to 0.95 for the inductive approach and from 0.13 to 0.87 for the deductive approaches. ChatGPT largely followed prompt instructions in producing a description of each coding scheme, although the wording for the inductively developed coding schemes was lengthier than specified. Conclusions: ChatGPT appears fairly reliable in assisting with qualitative analysis. ChatGPT performed better in developing an inductive coding scheme that emerged from the data than adapting an existing framework into an unconstrained coding scheme or coding directly into a structured matrix. The potential for ChatGPT to act as a second coder also appears promising, with almost perfect agreement in at least 1 coding scheme. The findings suggest that ChatGPT could prove useful as a tool to assist in each phase of qualitative content analysis, but multiple iterations are required to determine the reliability of each stage of analysis.
dc.identifier.citationJournal of Medical Internet Research, ISSN: 1438-8871 (Print); 1438-8871 (Online), JMIR Publications Inc., 26, e59050-. doi: 10.2196/59050
dc.identifier.doi10.2196/59050
dc.identifier.issn1438-8871
dc.identifier.issn1438-8871
dc.identifier.urihttp://hdl.handle.net/10292/17866
dc.languageeng
dc.publisherJMIR Publications Inc.
dc.relation.urihttps://www.jmir.org/2024/1/e59050/authors
dc.rights©Rimke Bijker, Stephanie S Merkouris, Nicki A Dowling, Simone N Rodda. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 25.07.2024. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research (ISSN 1438-8871), is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
dc.rights.accessrightsOpenAccess
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectChatGPT
dc.subjectTheoretical Domains Framework
dc.subjectnatural language processing
dc.subjectqualitative content analysis
dc.subject4203 Health Services and Systems
dc.subject42 Health Sciences
dc.subject08 Information and Computing Sciences
dc.subject11 Medical and Health Sciences
dc.subject17 Psychology and Cognitive Sciences
dc.subjectMedical Informatics
dc.subject4203 Health services and systems
dc.subject.meshQualitative Research
dc.subject.meshHumans
dc.subject.meshQualitative Research
dc.subject.meshHumans
dc.titleChatGPT for Automated Qualitative Research: Content Analysis
dc.typeJournal Article
pubs.elements-id564776
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
ChatGPT for Automated Qualitative Research.pdf
Size:
409.77 KB
Format:
Adobe Portable Document Format
Description:
Journal article
Download
Collections
School of Clinical Sciences - Te Kura Mātai Haumanu