This study aims to evaluate and compare the accuracy and completeness of responses generated by three AI models—ChatGPT, Gemini, and DeepSeek—when prompted with patient-oriented questions regarding female urinary tract infections. The findings will be measured against evidence-based clinical guidelines and publications.

A cross-sectional design was employed. Researchers developed five standardized, patient-focused questions on UTI management based on recent evidence and authoritative guidelines. Each question was individually submitted to ChatGPT, Gemini, and DeepSeek in a private browser session. Two medical professionals independently evaluated each AI-generated response for accuracy (1–3 scale: Correct, Partially Correct, or Incorrect) and completeness (1–2 scale: Incomplete or Complete). Both raters compared the AI response with the AUA guidelines. Inter-rater agreement was used to assess the consistency of ratings between evaluators.

Regarding completeness, all three AI models received the highest score of 2 (Complete) from both raters in all five categorical questions. Rater 1 administered a score of 3 (Correct) for all three AI models as it pertains to accuracy in all five responses. Rater 2 provided DeepSeek with a score of 3 (Correct) in accuracy for all five responses. Rater 2 provided ChatGPT and Gemini with a score of 2 (Partially Correct) for their responses in the prevention category.

Inter-rater agreement was high across all models. Overall agreement for accuracy was 86.7%, while completeness ratings had 100% agreement. DeepSeek demonstrated the highest consistency, with 100% agreement between evaluators on both accuracy and completeness. ChatGPT and Gemini each showed 80% agreement for accuracy but maintained full agreement for completeness.

All three AI models produced generally accurate and complete responses to UTI-related patient questions. High inter-rater agreement, especially for completeness, suggests strong reliability of the content. However, small variations in accuracy ratings highlight the importance of consistent evaluation frameworks. DeepSeek demonstrated the highest overall consistency, indicating potential for reliable patient education support.