Evaluation of ChatGPT Response Quality for Drug Information Queries: A Study of Medications Selected from National List of Essential Medicines

Abstract

Objective: To evaluate the quality of responses generated by ChatGPT regarding medications listed in the National List of Essential Medicines (NLEM), focusing on accuracy, appropriateness, completeness, consistency, readability, and safety, and to analyze factors associated with meeting the quality threshold. Methods: This cross-sectional analytical study evaluated 960 ChatGPT responses concerning indications, dosages, administration, and adverse drug reactions of both modern and herbal medicines in the NLEM. Questions included both open-ended and closed-ended formats, administered across four time points: days 0, 7, 14, and 28. On day 0, three consecutive queries were performed, whereas subsequent days involved a single query. Two expert pharmacists independently assessed the responses using a 5-point Likert scale across six dimensions. Inter-rater reliability was analyzed using weighted Cohen’s kappa and intraclass correlation coefficient (ICC), while factors associated with passing the quality threshold were analyzed using generalized estimating equations (GEE). Results: Responses generated by ChatGPT achieved mean scores exceeding 4.0 out of 5 across all evaluation dimensions. The highest mean score was observed in accuracy (4.60 ± 0.67), followed by safety (4.43 ± 0.69) and appropriateness (4.40 ± 0.69). More than 92% of responses in all dimensions met the quality threshold (score ≥ 4). Inter-rater agreement was low across all domains. GEE analysis revealed no statistically significant associations between the likelihood of meeting the quality threshold and question category, drug class, question type, or assessment time point. Conclusion: ChatGPT demonstrates significant potential in providing information on NLEM medications, with high-quality responses across all evaluated dimensions. This study found no significant variations in response quality over the assessment period. However, ChatGPT may still encounter limitations regarding completeness and consistency in certain contexts. Therefore, it should not replace clinical decision-making or professional counseling by healthcare providers. Further studies in more diverse and complex clinical contexts are warranted to support the appropriate and safe application of artificial intelligence for medication-related inquiries.