Hypothesis / aims of study
Based on international guidelines(1), expert decision-making schemes, and experience in previously managed cases collected, it is possible to develop a computational prediction model to support decisions for correct and structured surgical management of female urinary incontinence.
Study design, materials and methods
A tool was developed that employs AI and machine learning algorithms to assit decision-making regarding surgical treatment for female urinary incontinence. Synthetic clinical data are generated using the Monte Carlo Method to build a computational model to classify patients into various treatment categories(2).
For the design of the model, the following significant facts were considered:
1. Define Management Rules: We reviewed national and international guidelines for diagnosis and management of female urinary incontinence to establish decision-making rules for the proposed computational model. We also conducted expert interviews to incorporate local variations of management and surgical decision-making.
2. Define Inputs: Risk factors for surgical complications were found from literature and expert inputs. A database with records from 2014 to 2019 was accessed and anonymized to protect patient privacy. This data was analyzed, and fifty-three clinical variables considered essential for decision-making were selected.
3. Define Outputs: Only the type of surgery recommended will be the output variable, with eight options decided by expert consensus. Retropubic midurethral ling, transobturator midurethral sling, single incision sling, Burch colposuspension, bulking agents, autologous pubovaginal sling, adjustable retropubic midurethral sling, artificial urinary sphincter. (3)
4. Simulation of Patients in Seed Data Sample: A descriptive analysis of the chosen variables was performed, and a seed of 123 patients was generated using Excel. The experts assigned surgical treatments to each case, forming the basis for the subsequent expansion of the data.
5. Assignment of Treatments to Patients in the Seed Sample: An application (Fig 1) was developed to facilitate expert input on surgical recommendations for each patient (Fig 2). Using the Borda method, surgical management was prioritized based on expert consensus.
6. Expansion Data: Three thousand patients were synthetically generated using Python based on the seed data. The data were partitioned into training and validation sets for model development and testing.
7. Assigning Treatments to Patients in the Validation Sample: Experts reassigned surgical treatments to a synthetically generated sample of one hundred patients for validation purposes.
8. Descriptive and Exploratory Data Analysis: Further analysis of the extended dataset was carried, providing insights into patient characteristics and trends.
9. Computational Model Search: Several machine learning algorithms, including SVM, Random Forest, Naïve Bayes, KNN, Gradient Boosting, and Logistic Regression, were explored using Python. Model performance was evaluated to determine the most effective approach. (Fig 3)
10. Embedding the model in a tool: The developed model was integrated into a clinical application (Fig 1), with plans for continuous development and improvement
11. Evaluation of model performance: A final dashboard will display model performance metrics, iteratively updated with new patient data to ensure continued effectiveness.
The study received ethical approval from the Institutional Research Ethics Committee.
Results
The application of the Monte Carlo method facilitated the simulation of a seed sample of patients in an Excel database, where expert clinicians assigned surgical treatments. This process was streamlined with the use of the AppSheet tool, which allowed experts to easily assign clinical treatments. The inner workings of the tool also incorporated a a logistic regression (LR)-based model search.
Different machine learning models were developed with Python, using various approaches. These models showed an accuracy ranging from 37% to 100%. In particular, the LR model implemented in the application demonstrated an accuracy of 80%. However, it is crucial to note that the 100% accuracy achieved by some models can be attributed to possible biases in model development or failures in the simulation process. These issues must be addressed before deploying the models with real clinical data.
The initial design phase of the decision support system based on machine learning approaches is nearing completion. The goal is to integrate external models with the application model to obtain an optimized solution. However, it is important to emphasize that the system has not yet been launched for clinical use, as it needs to be refined and validated to ensure its effectiveness and reliability in real-world clinical settings.
Interpretation of results
The successful simulation of a seed sample of patients and the development of machine learning models mark significant progress towards the design of a decision support system for the surgical treatment of female urinary incontinence. However, several key points merit discussion to contextualize the results and outline future directions:
1. Variability of model accuracy: the observed variability in the accuracy of different machine learning models underscores the complexity of predicting optimal surgical treatments for female urinary incontinence. While some models achieved high accuracy rates, others fell short, indicating the need for further refinement and optimization. Investigating the underlying factors contributing to variability and addressing potential biases or shortcomings in model development is essential to improve predictive accuracy.
2. Use of the AppSheet Tool: The AppSheet tool streamlined the process of clinical treatments assignment by expert clinicians. While the tool proved useful in facilitating treatment assignment, its integration with the machine learning model deserves careful consideration. Ensuring seamless interoperability between the tool and the model is crucial to effectively deploying the decision support system in clinical practice;.
3. Importance of External Validation: Although machine learning models showed promising results in simulated scenarios, the absence of external validation using real-world clinical data is a notable limitation. Before the system can be deployed in clinical settings, rigorous external validation studies are needed to assess its performance, generalizability, and reliability in diverse patient populations. In addition, ongoing monitoring and validation are essential to ensure the continued accuracy and relevance of the system over time
4. Potential Clinical Impact: Although the current phase is limited to design and development, the final deployment of a robust decision support system can have a significant impact on clinical practice. By providing evidence-based recommendations tailored to individual patient characteristics, the system can help clinicians make more informed decisions, optimize treatment outcomes, and improve the quality of care for patients. Moreover, the integration of such technology can improve workflow efficiency and contribute to the advancement of personalized medicine in urogynecology.
Concluding message
The development of a computational model that uses simulation, expert input, and prior clinical data to support clinical decision-making holds great promise for improving the care of patients undergoing surgical procedures for female urinary incontinence. This innovative approach is expected to optimize process efficiency, effectiveness, and overall experiences for both patients and medical professionals, while helping to advance research and understanding of the clinical decision problem. However, further research and validation efforts are essential to ensure the translation of this model into a clinically viable tool. Through interdisciplinary collaboration and addressing identified limitations, the goal of improving clinical decision support in the surgical management of female urinary incontinence through innovative computational approaches can be achieved.