Ousmane, the lung-CT example is the one that really drives this home. The AI had all the data and still missed it, and what caught it was the radiologist knowing the patient. The problem is that a system performing well day after day is exactly the one that gets less scrutiny, not more. Your second condition, real authority to override, seems like the hardest one to actually deliver on. The way institutions are set up right now, overriding triggers paperwork, liability questions, justification to administrators. That's not authority, that's a deterrent. Do you think that's something design can solve, or does the institutional culture have to change first?
Thank you for these details. The radiologist case is an eye opener. The second condition requires institutions to remove all the frictions that make it impossible for clinicians to do their work properly. If all the barriers of justification are there for the clinicians because they override a suggestion by the system, it is no longer a real authority and the clinician is more of an observer. This is why a good governance system will enable the design of the system to give proper authority to Clinicians. I know that is not the case today in most places, hence it is important to bring attention to it.
I completely agree. A good tool should function the way most of us use AI, the expert directs it, pushes back on it, overrides it when something feels wrong. The moment the institution makes it easier to go along than to push back, the relationship is now inverted. The tool is now in charge. I would personally have a very hard time working like that, and I think most people who actually know their field would too.
Ousmane, the lung-CT example is the one that really drives this home. The AI had all the data and still missed it, and what caught it was the radiologist knowing the patient. The problem is that a system performing well day after day is exactly the one that gets less scrutiny, not more. Your second condition, real authority to override, seems like the hardest one to actually deliver on. The way institutions are set up right now, overriding triggers paperwork, liability questions, justification to administrators. That's not authority, that's a deterrent. Do you think that's something design can solve, or does the institutional culture have to change first?
Thank you for these details. The radiologist case is an eye opener. The second condition requires institutions to remove all the frictions that make it impossible for clinicians to do their work properly. If all the barriers of justification are there for the clinicians because they override a suggestion by the system, it is no longer a real authority and the clinician is more of an observer. This is why a good governance system will enable the design of the system to give proper authority to Clinicians. I know that is not the case today in most places, hence it is important to bring attention to it.
I completely agree. A good tool should function the way most of us use AI, the expert directs it, pushes back on it, overrides it when something feels wrong. The moment the institution makes it easier to go along than to push back, the relationship is now inverted. The tool is now in charge. I would personally have a very hard time working like that, and I think most people who actually know their field would too.
The tool should be in service of human beings, not the other way around. I am 💯 with you on that.