“Human in the Loop?”

These days, the application of AI in GXP lives rent free in my mind, occupying waaay too much space to be living for free.  For context – I am referring in this blog to the AIs being rolled out based on reinforcement learning models such as Claude, CoPilot, OpenAI, etc.  I’m sure this same rent-free situation is true for many quality professionals and regulators around the world, who are tasked with protecting participants and patients.  There must be a safe pathway forward into this exciting future, but our current trajectory is dangerous.  Nearly all pharma companies (both small and large) permit nearly unfettered access across their organizations to the most powerful decision-influencing tool we humans have ever created.  We find ourselves at a crossroads, and some have already chosen a path.

Why is this “dangerous”, as I claim above?  It is because these tools cannot be “validated” according to traditional ways of thinking.  Historically, pathways for computer system validation have been built on static deterministic models (both CSV and CSA): ‘if you press this button this happens’, or ‘if you enter the incorrect password 3x you will be locked out of the system’…  No computer system validation strategy is ever without risk or fully deterministic, as malfunction/error is still a possibility, although the probabilities are low if the right vendor is selected.  We periodically monitor performance of the validated system via monitoring just in case, as the stakes are high. 

This conclusion regarding ‘no system is free of risk’ applies to nearly everything in the universe and science – hence the use of the term “theory” vs. “truth” – a core pillar of the scientific method developed centuries ago that permits continuous advancement of our civilization.  There are too many variables that exist in the universe for us mere mortals to even consider treating scientific theories as truths.  This is the human ego at its worst, and to do such would be reckless and irresponsible, considering the infinite variables continuously at play.  We must accept and familiarize ourselves with the reality of risk in the spirit of progress and providing a service to humanity based on overwhelming probabilities that the system will work as intended with appropriate validation and risk-based strategies.  This is the best we can do with what we have!

Now back to AI… our traditional way of validation strategies no longer apply, as we cannot make a reliable conclusion regarding probabilities (as we do with CSV).  This whole “human in the loop” thing isn’t going to work on a large scale.  We would never sign-off on a half-hearted CSV and then just write and SOP for human oversight of the computer system for error detection! 

Hence the introduction of the term “credibility” in FDA guidance, as AI behaves more like a human when it comes to outcomes: mistakes, unexplained behavior and hallucination cases are evident everywhere (unless you live in a cave, perhaps?).  I draw a direct comparison to a human here for a reason, as we have been establishing the credibility of humans in GXP for generations – are they qualified to perform their assigned tasks?  We are familiar with this GXP requirement and (sometimes) manage risks appropriately.  This is the best we can do with what we have (human nature).  Demonstrating credibility of an AI is going to be more difficult than we seem to think – and data scientists already know this.  Human behavior is notoriously difficult to control – and as a result the regulators now task us with evaluating human-machine interactions in the context of “vulnerability” (see EU draft GMP Chapter 4).  We need to consider vulnerabilities with the introduction of AI as well, as controlling outcomes when outputs are drawn from unstructured data sets will be as difficult to control as a human performing a keyword search.  Not impossible, but difficult.  Companies who have objectively evaluated risks for non-deterministic systems such as a human or AI in terms of vulnerabilities (vs probabilities) are making medicines I personally would always choose vs. any competitor letting the ego take over and consider probabilities. 

We cannot let the excitement of reduced GXP timelines distract us from our core responsibility: protection of the participants and patients, as they have no voice.  I propose the following quick and simple checklist to consider when evaluating the potential for AI in GXP use cases:

• What is the criticality of the use case?

  • What might go wrong if our procedural control “human in the loop” fails to detect hallucination or errors?

    • Patient/Participant Safety Risks?

    • Business Risks?

• Have we gathered the necessary evidence to demonstrate that all relevant risks can be managed

  • Use caution when concluding procedural controls (e.g., “human in the loop”) are sufficient, as there are too many variables to consider