Assessment of the Appropriateness of Responses in Thai from ChatGPT on the Questions for Recommendations of Drug Uses in Common Illnesses

Abstract

Objective: To evaluate the appropriateness of responses in the Thai language provided by ChatGPT when inquiring about drug uses in common illnesses. Method: This study posed 5 questions on drug uses to ChatGPT in the Thai language, which included: [Q1] experiencing knee pain, tried piroxicam without improvement, can I consider an alternative?; [Q2] frequent coughing with sore throat and fever, can I use amoxy?; [Q3] I tripped, have a scratch and bleeding, should I take an antibiotic?; [Q4] suffering from food poisoning and diarrhea, is an antibiotic necessary?; and [Q5] with a history of allergy to ibuprofen, is it safe to take diclofenac? For each question, the answer will be generated 3 times in Thai using regenerate function of ChatGPT. Two pharmacists independently rated the answers on appropriateness of drugs or treatments recommended by ChatGPT in 5 aspects including indication, efficacy, safety, adherence and cost. Two additional aspects including readability and the applicability in real-life situations were also assessed. Results: The total of 210 evaluations (5 questions x 3 generated answers x 7 aspects of evaluation x 2 raters) showed  that the recommendation obtained from ChatGPT had the highest scores in terms of adherence and applicability with average scores of 4.40 and 4.23, respectively (out of 5). In the aspects of readability, safety, indication, and cost, the scores were slightly lower, with ratings of 3.80, 3.80, 3.80, and 3.73, respectively. The information on efficacy yielded the lowest score at 3.37. Although ChatGPT may have errors in the use of Thai words or sentences, its overall responses were readable and relevant to the questions being asked. However, generating just one ChatGPT answer may not cover all the issues related to the questions. If ChatGPT responses were regarded as appropriate when all assessments on indication, efficacy, safety, adherence and cost were scored at least 4.0 (except for [Q5] where assessment of efficacy considered irrelevant), 3 out of 15 ChatGPT responses were appropriate. However, if the responses from ChatGPT were considered appropriate when the assessment in the above 5 aspects as well as the assessment of readability and applicability (totally of 7 aspects) scored at least 4.0 in all aspects, only 1 out of 15 ChatGPT responses was considered appropriate. Conclusions: At present, the use of ChatGPT has limited benefit for self-care with medications for uncomplicated diseases or conditions among Thai. Public are encouraged to seek further advice from physicians or pharmacists before taking any medication recommended by ChatGPT.