Assessing the Quality of AI-Generated Clinical Notes

AI can streamline clinical documentation, but is the output up to par? We partnered with the University of Waterloo to find out if AI-generated notes met the high standards of KidsAbility clinicians.

Problem

Clinical documentation, such as writing clinical notes, generates a significant amount of administrative work. This process is time-consuming and takes away valuable time that could be spent on direct client care. AI tools have the potential to assist with this documentation-heavy process. However, AI documentation tools are new and novel, and it is unclear what impact, if any, they have on documentation quality. Such uncertainty is problematic in regulated health services, where professional licences are connected in part to documentation practices. Uncertainty also challenges a given clinician’s ability to trust the output of AI.

Solution

As part of KidsAbility’s exploration of AI in pediatric rehabilitation we partnered with the University of Waterloo to investigate the impact of AI on the quality of clinical documentation. The principal investigator on the project was Dr. Maura R. Grossman, a Professor in the School of Computer Science at UW, Adjunct Professor at Osgoode Hall Law School, and an affiliate faculty member of the Vector Institute. The work was supported by Dr. Dan Brown and Solomon Amenyo, also from the School of Computer Science.
As part of the study, a group of KidsAbility clinicians participated in a blind evaluation comparing clinical documentation written solely by clinicians to documents that were written either exclusively by AI or with AI in the loop. The results showed that AI notes had higher quality on average than notes written why clinicians alone (without AI involved). Further, the AI-supported notes were far more consistent than clinician-only notes. For complete details on the study, see this report.
 

Benefits

In other research at KidsAbility, we have found AI can reduce the time it takes to produce clinical documentation by 25-50%. Such time savings have important implications for access to care, but important uncertainties exist about the reliability and accuracy of such technology. This research contributes to our understanding of the risks and benefits of using AI in regulated health professions. It clarifies that AI tools are unlikely to produce clinical documentation of lesser quality than a therapist would produce on their own, and should improve the level of trust therapists place in AI tools. The improvement in consistency of documentation is a benefit that can be added to the productivity boost they can provide.  

About KidsAbility

KidsAbility™ empowers children and youth with special needs to realize their full potential. 

We support over 17,000 children and youth to reach their communication, social, physical and behavioural goals each year. Using a blended model of virtual and in-person services, at KidsAbility, we help children, youth and families to achieve great things at home, at school and in their community. 

Demand for our services continues to exceed the resources available. The Rocket Discovery Centre is a symbol of our commitment to children and families that no child in our community will be left behind. Donations support life-changing therapy, innovative programs and the latest technology. Your support helps a child benefit sooner to say their first words, take their first steps, hold a pencil or spoon, or even learn how to make a new friend and participate fully in our community. Together, we can achieve this.

Rocket News

Get the Latest Mission Updates.

Scroll to Top

Thank you for signing up!

You are ready for blast off and have been subscribed to our KidsAbility Rocket Newsletter. Stay tuned for updates and news releases.