Analysing Credibility of Information on Thai Herbs Generated by the ChatGPT from Pharmacists' Perspectives
Article Sidebar
Main Article Content
Abstract
Objective: To evaluate the appropriateness of the ChatGPT's responses in the Thai language when questioned about Thai herbs, both in general information and the use for specific purposes. Method: The study selected five Thai herbs and asked the ChatGPT to provide information about them. The inquiry started by asking one general question on indications, precautions and instruction for use of each herb, with one answer in Thai requested. Subsequently, the following questions regarding their specific applications were posed, including 1) Can turmeric treat gastrointestinal symptoms? 2) I got COVID; can I take only Far Tha Lai Jone (Andrographis paniculata) to treat the disease; is it safe during pregnancy? 3) Could Krachine (Boesenbergia rotunda) prevent COVID infection? 4) Would it be harmful to take senna every day? 5) Is kanja oil safe to use? The answers to these specific questions were generated three times in Thai using the regenerate function of the ChatGPT. Two pharmacists with expertise in pharmacognosy independently assessed the appropriateness of the answers recommended by the ChatGPT in five aspects including indication, efficacy, safety, adherence, and cost. Additionally, the raters also assessed two more aspects, i.e., readability and practicality for use in real-life situations. Experts employed a 5-point Likert scale for the assessment. Results: The total of 220 evaluations ((5 general questions x 2 raters) + (5 specific questions x 3 generated responses x 7 aspects x 2 raters)) demonstrated that all aspects of evaluation only scored around 3 (maximum 3.80 with minimum 3.07 from the full score of 5). The results suggest that the responses given by the ChatGPT were rather ambiguous. Moreover, when considering a score of at least 4.0 as an acceptable level in the assessment, only readability (with 8 answers scoring more than 4 from the total of 15 answers) showed the highest proportion of acceptability, while the rest exhibited a substantially low proportion of answers scoring more than 4. None of answers met the stringent criterion of a minimum score at 4 across all aspects of assessment. Conclusions: At the time of research conduction, the ChatGPT demonstrates substantial limitations in responding to inquiries on Thai herbs when using Thai language and Thai herbal names in questioning and generating answers.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
ผลการวิจัยและความคิดเห็นที่ปรากฏในบทความถือเป็นความคิดเห็นและอยู่ในความรับผิดชอบของผู้นิพนธ์ มิใช่ความเห็นหรือความรับผิดชอบของกองบรรณาธิการ หรือคณะเภสัชศาสตร์ มหาวิทยาลัยสงขลานครินทร์ ทั้งนี้ไม่รวมความผิดพลาดอันเกิดจากการพิมพ์ บทความที่ได้รับการเผยแพร่โดยวารสารเภสัชกรรมไทยถือเป็นสิทธิ์ของวารสารฯ
References
Satyapan N, Patarakitvanit S, Temboonkiet S, Vudhironarit T, Tankanitlert J. Herbal medicine: affecting factors and prevalence of use among Thai population in Bangkok. J Med Assoc Thai 2010; 93 Suppl 6: S139-44.
Kanjanahattakij N, Kwankhao P, Vathesatogkit P, Thongmung N, Gleebbua Y, Sritara P, Kitiyakara C. Herbal or traditional medicine consumption in a Thai worker population: pattern of use and therapeutic control in chronic diseases. BMC Complement Altern Med 2019; 19: 258. doi: 10.1186/s12906-019-2652-z.
Onchonga D. A Google Trends study on the interest in self-medication during the 2019 novel coronavirus (COVID-19) disease pandemic. Saudi Pharm J 2020; 28: 903-4.
Bianch, T. Market share of leading desktop search engines worldwide from January 2015 to July 2023 [online]. 2023 [cited Oct 18, 2023]. Available from: www.statista.com/statistics/216573/worldwide-market-share-of-search-engines/.
Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educ 2023; 9: e45312. doi: 10.2196/45312.
Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language models are few-shot learners. Adv Neural Inf Process Syst 2020; 33: 1877-1901.
Seghier ML. ChatGPT: not all languages are equal. Nature 2023; 615: 216.
Boonrit N, Ruanglertboon W. Assessment of the appropriateness of responses in Thai from ChatGPT on the questions for recommendations of drug uses in common illnesses. Thai Journal of Pharmacy Practice 2023; 15: 1135-48.
Fang C, Wu Y, Fu W, Ling J, Wang Y, Liu X, et al. How does ChatGPT-4 preform on non-English national medical licensing examination? An evaluation in Chinese language. PLOS Digit Health. 2023; 2: e0000397. doi: 10.1371/journal.pdig.0000 397.
Hopkins AM, Logan JM, Kichenadasse G, Sorich MJ. Artificial intelligence chatbots will revolutionize how cancer patients access information: ChatGPT represents a paradigm-shift. JNCI Cancer Spectr 2023; ; 7: pkad010. doi: 10.1093/jncics/pkad010.
Hepler CD, Strand LM. Opportunities and responsibi lities in pharmaceutical care. Am J Hosp Pharm 1990; 47: 533-43.
Kanjanasirirat P, Suksatu A, Manopwisedjaroen S, Munyoo B, Tuchinda P, Jearawuttanakul K, et al. High-content screening of Thai medicinal plants reveals Boesenbergia rotunda extract and its compo- nent Panduratin A as anti-SARS-CoV-2 agents. Sci Rep 2020; 10: 19963. doi: 10.1038/s41598-020-77003-3.
Short SE, Mollborn S. Social determinants and health behaviors: conceptual frames and empirical advances. Curr Opin Psychol 2015; 5: 78-84. doi: 10.1016/j.copsyc.2015.05.002.
Kao YS, Chuang WK, Yang J. Use of ChatGPT on Taiwan's examination for medical doctors [online]. 2023 [cited Oct 10, 2023]. Available from: 10.1007/s10439-023-03308-9.
Singnoi U. A reflection of Thai culture in Thai plant names. Manusya 2011; 14: 79-97.
Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutier rez L, Tan TF, Ting DSW. Large language models in medicine. Nat Med 2023; 29: 1930-40. doi: 10.1038/ s41591-023-02448-8.
Hastie T, Tibshirani R, Friedman J. The elements of statistical learning: Data mining, inference, and prediction. New York: Springer; 2009.
Kaewdech A, Nawalerspanya S, Assawasuwannakit S, Chamroonkul N, Jandee S, Sripongpun P. The use of Andrographis paniculata and its effects on liver biochemistry of patients with gastrointestinal problems in Thailand during the COVID-19 pandemic: a cross sectional study. Sci Rep 2022; 12:18213. doi: 10.1038/s41598-022-23189-7.
Intharuksa A, Arunotayanun W, Yooin W, Sirisa-Ard P. A comprehensive review of Andrographis paniculata (Burm. f.) Nees and its constituents as potential lead compounds for COVID-19 drug discovery. Molecules 2022; 27: 4479. doi: 10.3390/mo lecules27144479.
Menz BD, Modi ND, Sorich MJ, Hopkins AM. Health disinformation use case highlighting the urgent need for artificial intelligence vigilance: Weapons of mass disinformation. JAMA Intern Med 2024; 184: 92-6. doi: 10.1001/jamainternmed.2023.5947.
Anon. Will ChatGPT transform healthcare? Nat Med 2023; 29: 505-6. doi: 10.1038/s41591-023-02289-5.
