The Deep View: Conversations

# Swell AI Transcript: FERRUM POD_1.mp3

SPEAKER_02:
AI. Healthcare. Match made in heaven? Maybe. Jury's still out. The integration of AI and healthcare has been talked about a lot by literally everyone, but it's a prominent point of focus for the major developers, including OpenAI and Microsoft and Google, kind of based around this question of what can these systems enable in terms of enhancing or improving medical care. Now often they are talked about in a very science fiction-y magical way when you hear folks talking about this notion that AI will cure diseases, that it will cure cancer. What we do have is a series of powerful tools. And we have different applications of automation and algorithms that employed specifically with care and forethought in specific environments from diagnostics to drug creation, to precision medicine, to automating menial administrative tasks, hopefully returning time to the doctors. And the reality is getting that trust, gaining that trust, has been very, very difficult. And so that's what we're talking about today. My guest is Peilu Tran. He is a co-founder and CEO of Ferrum Health. Now, Ferrum's whole pitch is what Peilu describes as the last mile of adoption. So, you know, you have these developers that are kind of pushing the state of the art of what their systems can do. And then you have healthcare systems, which are operating largely on outdated IT infrastructure, and that are largely very cautious about adopting these new technologies. If you are new here, welcome. If you're not, welcome back. This is The Deep View Conversations. Pelu, thanks so much for joining me today. Likewise, thanks for having me. I want to start with your why, I guess, right? So I know you had a kind of personal reason at the foundation of the start of the company you run, which is Ferrum Health. And I wanted to start there. What inspired you to start this company, to take this approach?

SPEAKER_01:
Yeah, it really is very personal. And I think health care is personal for so many of us. But I had been in digital health previously building productivity solutions for doctors. So that story was actually inspired by an experience I had in health care as well as a medical student at Stanford. But over probably six years, I had built up a digital health company into several thousand employees and a path to going public. And in that course, I had actually had a chance to work with a lot of digital health companies that were doing really exciting things in AI. And this was back in 2016, 17, 18, when AI was actually going through yet another boom. But it was the machine vision boom that was allowing us to figure out whether something was a cat or a dog really, really, really efficiently. And at that time, was a mentor at StartX at Stanford and a number of other accelerators here in the Bay Area, helping companies figure out their commercial go-to-market. But then I actually had to take some time off to take care of a family member who passed away from a missed lung cancer. And this was my uncle. He actually raised me for several years growing up. And unfortunately, he was getting screenings, was healthy, doing everything he needed to but his cancer was missed by his doctors for over five years across three different imaging studies and so by the time his cancer was caught it was already metastatic it was too late to perform surgery and he passed away about around six months after his diagnosis and that experience for us as his family members really taught us that it doesn't really matter what technology is out there but if doctors aren't able to use it, then patient care unfortunately can't improve. And that was the case for him. Because there were at the time, over a dozen FDA cleared solutions on the market to detect lung cancer that were performing better than clinicians were, in many cases, but were really just struggling to be adopted. And so Pura was, on the one hand, the family member of a patient who really had a easily preventable death. And on the other hand, working closely with founders that were literally building the solutions that could have saved his life. And so that was what made us realize the genesis of Ferrum, which is that last mile for technology adoption in healthcare is incredibly broken. And until we fix that, until we make it easy for doctors and patients to have access to the AI tools that they want to and need to use, patient lives are going to suffer. And that was really why we started this.

SPEAKER_02:
I feel like everyone has a somewhat similar story. Even I, you know, even outside of oncology itself, the idea of just things that are missed. I can think of a lot of people in my life that that applies to, and it's a real challenge. But you're talking about adoption problems, and it's a real challenge on both ends of the spectrum. It's a challenge to be sitting here and saying, if you had adopted this, you might have saved lives. And then for the clinicians, it's a challenge as well of, you know, clinicians are generally slow to adopt new technology. And earning clinician trust is really difficult. And so I wonder what you think about that, that spectrum, where, you know, as much as we want the adoption for things that'll work, Why you find clinicians, like, why is it that that last mile that you're talking about is the hardest push?

SPEAKER_01:
Yeah, I think a huge part of it does have to do with trust, like you said, but I actually think that the problems in healthcare are far more. systemic than they are individual dependent. And, you know, the day of a doctor hanging up their shingle and opening up a private practice in a small town, like those are gone, right? We're well past the private equity M&A, we're well past consolidation. So the average doctor is a cog in a wheel. And that's separately, and we could spend an entire podcast just discussing that. But I think that the conservatism of doctors is no longer the reason why AI is not being adopted. There's some aspect of it, I mean, and we can dig into that, but I think the real reason why AI is not being adopted has a lot to do with the existing systems that are in place. That means the sort of legacy infrastructure, the fact that they're still using pages and fax machines and data centers and the things that we all know and love to make fun of, but healthcare IT is almost definitionally decades and decades old. sitting in data centers because of the sensitivity of patient records, and it's spread across hundreds of different disparate systems. And I think that is kind of the first and greatest problem is the sort of challenge of integrating these sorts of hyper modern GPU based cloud service leveraging AI point solutions that are being offered via these two grad students and their dog in a garage and 100K of AWS credits. And then trying to sell that and trying to deploy that into these sort of massive legacy monolithic systems that health currently operates on. And then that is kind of in a different lens, a lot to do with security as well, where patient data privacy, and the security of patient data kind of runs directly headlong into the data hungriness of AI. So I think it's, it's about legacy systems, it's about the overriding need to protect patient data, And it's about the fact that healthcare is probably the single most targeted industry for cybersecurity, ransomware attacks that's out there. I mean, virtually every hospital has been targeted by these sorts of acts. And so I think in that IT environment, it is really obvious why it's hard to adopt these tools. And unfortunately, I don't think that that's a problem that hospitals or AI vendors are going to solve.

SPEAKER_02:
How does that problem get addressed then? I mean, if you're talking about beyond the individual clinician conservatism, right? And into, it's just really outdated technology dispersed across so many different disparate systems. I guess you would imagine that would have to start with like an IT consolidation and revamp, but that hasn't happened. Why do you think that is?

SPEAKER_01:
Yeah. I mean, honestly, I almost feel like it's a little bit of a different approach to the entire concept of adopting technology that needs to occur. And we've been in this mindset that we need to get hospitals comfortable with public cloud, we need to get them comfortable with sending their patient data across the interwebs, because that's how everyone else does things. And I think if you went to a financial services company, or you went to the Department of Defense, or you went to any number of other hypersecure industries, and you said, well, you know what, like, too bad, DoD, you just got to send your military records to these AI vendors. Like, no, like, they're like, no, you're going to deploy them the way we want you to deploy them. I think that's actually the solution. And the solution is saying, and recognizing, hey, AI companies are the ones that are modern. They're the ones that have the ability to do things like containerize and deploy at the edge and encrypt. And so they should be the ones that actually provide their solutions in the way that can be deployed within the environments that customers want. And that usually means fully containerized at the edge within the hospital data center. And if you actually take that view, then you realize, oh, we can actually solve all these problems. So we can deploy AI securely for these hospitals. We can address this problem of point solution legacy integrations. You can just stand up an integration with a legacy system once, connect it into a kind of AI control plane that is governing and managing and hosting these models. And you can run these models the way hospitals want to, behind their firewall, in their data center, in their private cloud, without putting their patient at risk. And so that was actually where Ferrum came in and we said, listen, we want doctors to be able to deploy any AI application from anywhere in the world as easily as your eye downloads an app on your iPhone. And I think that was the, kind of initial goal we had, and we realized that it solved a lot of problems for hospitals and for AI vendors once you did that.

SPEAKER_02:
And since you brought it up, right, we're a few minutes in, but we haven't talked about Ferrum yet, but I'd like to. And I guess you kind of alluded to it, but what is, I guess, your big pitch as Ferrum? What are you accomplishing for hospitals as it seems like a kind of intermediary between the tech and the adopters?

SPEAKER_01:
Yeah, so we're the last mile for healthcare AI. And so our entire mission is to eliminate all friction that exists between AI developers and the hospitals that they serve. And we do that by providing a governance platform that takes care of the IT, clinical and administrative deployment of these AI tools for everything from providing a secure compute environment, and secure integrations into existing systems to standardizing, normalizing, and connecting AI models that need to be running on patient data to the systems that they need to run on. Then ultimately, we mentioned this a bit earlier, is making sure that these solutions are safe, effective, and performant for hospitals. I think there's a huge need for partnership around model performance, model bias and drift, and ultimately model security that we take on a lot of the burden for both hospitals and for AI developers.

SPEAKER_02:
You've mentioned a bunch of things. We've talked a lot about a bunch of these things. Uh, it seems that forefront on your mind is security because in order for these things to do what we want them to do, they have to train on patient data, which is raises red flags and understandably. Um, and, and we'll, we'll get back to that, but first I wanted to kind of focus on, you were talking about performance and. you know, how do you validate the systems that you work for? And what does that validation process look like? So that when you are working with a hospital, you can say, Hey, here's what this thing does. And we can guarantee you that it'll do what it says it does.

SPEAKER_01:
Yeah, it's, it's, it's honestly a much harder question than you would think. In the rest of healthcare, the answer is, well, they run a clinical trial, FDA takes a look at it, they approve it, and we can generally believe that it works. And that's because medical devices and drugs, if I run a trial and it works on a patient, it's not going to stop working a year or two later. The problem with AI is that it will, because there's drift in data, there's incredible differences in kind of patient demographic and population. And so FDA is kind of a, it's a milestone for AI developers, but it does not give the sort of security to hospitals and clinicians that FDA clearance does elsewhere. And that as a result puts all the burden on hospitals. There was a really, really eye-opening metric that came out of Hopkins a few years ago. where they looked at the real world performance of AI models after they had been FDA cleared. And they saw that 81% had a decrease in performance in the real world compared to their FDA clearance. And so we're not talking about the occasional misstep, we're talking about the entire industry having a real issue with trust. So the burden falls to doctors. And that means that at least in today's standard, you are asking hospitals to spend millions of dollars on data science, data labeling, clinician committees, spending hundreds and hundreds of hours of doctor time on just looking at these models, comparing it to historical records and seeing if they are performant, not just in general, but across subsections of your population, right? Different minorities, different types of like equipment you might have, different demographics, socioeconomically, et cetera. And that has actually meant that the average hospital doesn't really do any sort of validation or monitoring despite the underperformance of these AI tools. They just kind of turn them on, see if the doctors are going to really complain. And if they don't, they'll leave it on and ask them a year after at renewal, hey, are you still using this tool? Half the time, they're not. And even when they are, usually it's not performing really the way they want to. That was a long way to answer how it's being done now and how it shouldn't be done. I think the key to all of this is realizing that the limiting factor is labeling. And it's how do you generate labels automatically on patient records that will allow you to automatically both validate initially the model's performance, so you're not putting things into the wild that are going to harm patients or underperform. but then also in an ongoing manner, how do you automatically label patient records so that you can actually track the model's performance as time goes on? And so that was the key for us, was actually realizing that it's turtles all the way down. The only way to automatically label patient records in order to make sure AI works is actually with AI. And so we developed an AI system that generates a ground truth for any sort of diagnostic or decision support AI tool that's being deployed, And as a result of the automatic surrogate ground truth, we're able to automatically report on performance of these tools, make sure they're safe, and do so without requiring any meaningful additional time from doctors. And that's actually how we solved the problem, was by saying, to govern AI, you need AI. And when we did it, we saw that you could actually have really, really granular, down to the patient level, insights and visibility into model performance, and that was giving clinicians the trust they needed to actually keep proceeding with using the tool.

SPEAKER_02:
Well, and I guess, how do you ensure the ongoing performance of the model that you're using to assess the models?

SPEAKER_01:
Would you believe me if I said the answer is more AI? But the answer is more AI. At some point, right, there is a human audit in place to regularly revisit and to check a subsection of your concordant findings. So the studies that agree with the doctors and the AI agrees, you have a human audit on some small percentage of that. But what we discovered was a lot of these models, they're language-driven at this time. It's off of radiologist reports, off of free text and other sort of unstructured data in the report. We've actually solved that. I think large language models, natural language processing, that is very much a solved problem at this point. I think we all can see the difference between that and the performance of those tools today versus even machine vision. We're not answering essay quotes as our CAPTCHAs these days. We're still trying to tell what a blurry little motorcycle is on the image. So I think that that's a representation of the fact that Some is better than others, the AI that used to govern these sorts of decision support and diagnostic tools is much more precise, since you can get an extremely accurate picture of what the doctors ultimately said downstream. And that can be used to actually govern these diagnostic AI systems that are actually far more critical and far more sensitive.

SPEAKER_02:
Yeah, I mean, the point about language is interesting, especially in the context of medical stuff. And this is something I wanted to ask you too, which is, you work with a lot of systems, but what kind of systems, what kind of algorithms are you working with? Is it predominantly large language models that have been tuned to do certain things? Because there's a lot of other, like when it comes to healthcare, there's a lot of just you know, not LLMs, but like small language models or just older machine learning models that are trained to help do, you know, identification of X, Y, Z thing, basic pattern matching. I wonder what the, I wonder what kind of things that you're working with and how that impacts how you go about validating.

SPEAKER_01:
So, uh, given our company's genesis, you can probably guess that we have a real passion for the diagnostic and the decision support are things that actually are going to drive improved care. And. When you look at the numbers, you look at over 1,000 FDA clearances for AI tools that are out there, 80 plus percent are actually some sort of imaging tool. Again, going back to the machine vision being this really constant presence within healthcare. And that doesn't necessarily mean radiology. It's cardiology, orthopedics, emergency medicine. There are a lot of different areas that imaging ends up being used in healthcare. The vast majority of tools out there are actually imaging tools. And so that's actually where, if I were to look at the market, it's where the vast majority of AI adoption today is. And then you have this world of agentic AI and LLMs and foundation models, and you have this kind of current generation of AI tools that is all the hype and all the rage. but actually it's disconnected from what most people are actually using day to day. And so I think that's an area where we're agnostic, right? Anything that touches a GPU is better hosted on Ferrum, better hosted in the hospital's firewall, better managed through a single control plane and integration. Like it doesn't matter what AI you're deploying, whether it's an agent or whether it's a vision model or, you know, like literally like a linear regression tool, but, um, Uh, what matters is the fact that you're giving hospitals a way to manage, host, deploy, and, and govern those, those tools. Um, most of them do end up being imaging, um, with a rapidly, rapidly growing part of our business being these LLM and agent models and kind of unstructured texts and data that, that I feel like is, is just taking over the world. I'm sure you use chat GPT every day. And so do I.

SPEAKER_02:
I test ChachiPT often. I don't actually use it too much. Really? Yeah, really. And I think that this has been studied a little bit. This is a total tangent. I guess it depends what you use it for. My day job, what I do is I'm a writer. And so I enjoy writing. So I'm not going to have it write for me. For a number of reasons, but at the core of it is that this is what I am enjoying doing, and I'm not just going to sit here and edit a model. There are ways that models can be used in my line of work. They tend to add more time, which is interesting because, and I mean, we've kind of talked around this, but you're talking about performance and stuff and like medical models and stuff, but even among language models, their big performant ability is also their weakness, which is that Being able to output coherent text is not a necessary quality of understanding that text. And they tend to get things wrong. This has been called hallucination. A lot of people hate that term. I'm in that camp. I don't like the term hallucination. It's too anthropomorphic. I was reading a paper the other day, and they kind of called it Frankfurtian bullshit. Which is like a technical sort of term that is like these systems can't distinguish between truth in the tokens they're outputting versus not truth, so they'll just output whatever that's on us to go through. I think one big use case that people have talked about for journalists is like, use it to process research papers. You don't have time to read a research paper. I'm like, okay. But if you know how to read a research paper, it's really not that hard. You start with the abstract, go to the methodology, hit up the conclusions, see what's going on there, and then you just do targeted. Like, I'm not going to read it like a book. But if I had a language model process it, even if I could trust that everything it outputted to summarize was accurate, which I can't automatically because I have to go to the first source and that automatically adds a second source. It's like asking my brother to summarize something for me. He might get it right. He might get it wrong, but I don't know what it's not telling me. And that's my big, like, there might be one little line in a paper that, you know, I might find myself and the model would choose not to put into it. And that just adds a layer of, you know, for me to have to go back through and essentially read the paper to validate whatever the output was. I'm just going to read the paper. And so, yeah.

SPEAKER_01:
I, that is exactly what you see. And it's going to be one of those really, really hard challenges to solve. And it contributes to the whole trust issue. I mean, it's one thing to kind of have it reading a research paper as background context is another thing for it to be actually determining whether a patient has had sepsis, for example. So that and if you looked at the, you know, your research papers are incredibly well structured, they've got well-written texts are reviewed by peer. That is a very different world than if you imagine the real-time data that's flowing into the guts of a health record that you're asking an AI tool to go in and figure out if a patient's decompensating or if they're a fall risk for readmission. All these questions we're asking them, it's no surprise these AI tools underperform. I think the more kind of severe the use cases are, the more critical they are, like the more you need really, really strong safeguards and governance in place.

SPEAKER_02:
Yeah, well, and that's the key challenge, right? You want a safety net that doesn't impede any current safety nets, right? Because it might get stuff wrong. And so how do you go about, and I think you mentioned this earlier, but it was a good point, that adoption is, it goes beyond the hospitals themselves, and it goes beyond the developers themselves. And I think that's absolutely true, that this is a big, question, the reliability of these models, and different models have different types of reliability, different models, the degradation of reliability that you're talking about, which if I'm a doctor, that's terrifying to hear. How do you integrate this in a way that it enhances your diagnostic capabilities without reducing the safety net? Because hospitals, and I think this is a fine line, and I wonder what you think about this. In some aspects, AI as a technology is kind of about efficiency. Like that's kind of the whole goal. You know, you can call it what you want. Again, AI is one of those terms. I don't know if it's the greatest term. It's the term we've got. Call it automation. It's automating, you know, in medical imaging, reading these things, whatever it is. It's about efficiency. In a hospital, I don't know that we necessarily want efficiency. We want to make sure that we fill the gaps. Like we want the hospitals to operate efficiently, but hospitals should also have tons of redundancies in place to make sure that no one slips through.

SPEAKER_01:
I regret to inform you that that ain't the way our healthcare system works. We are in a fee-for-service environment. We are in a fee-for-service environment that rewards efficiency above all else. And we're in an environment that has fewer and fewer doctors that are getting reimbursed less and less, that's more and more private equity owned, that has a rapid exponential growth in amount of medical data and kind of detail to it, and this aging population. Just like take those factors and combine them all and I genuinely believe that if you were to look at the Standard of care that we receive as patients today like you or I when we go into a hospital system No matter how good it is I truly believe that there is a hidden cost and pain to the fact that you have doctors incredibly distracted, you have so much more data. There's been a shifting of the responsibility in healthcare to the patient. And it reminds me a lot of the way that, you know, plastics companies invented recycling, or, you know, we talked about carbon footprint as a way to deflect responsibility for healthcare or for for for a problem that the industry created onto the consumer. Like I think in healthcare, You know what, like if you want to proceed in your care journey, if you want to get to a specialist, if you want to get a lab test, you want like it's somehow it's become on us to do all this stuff. And I think that that used to be different. Right. You used to have providers that would actually care for you and take care of your care journey. And that's all been sent to us. And that's the first of many, many, many things that I feel like has degraded with the amount of additional work that we've piled onto the healthcare system and the drive towards productivity. So I don't think it's a problem that we're unfortunately going to solve as an American healthcare system anytime soon. And so I think you have to basically look at productivity in the eye of, well, these people are already drowning. They're already up to their eyeballs. They're already having to churn through 20, 30 patients a day and, you know, spend seconds on a given image to be able to figure out if there's cancer, pneumonia, whatever else it may be. They're already kind of at their like razor thin limit. And all you can do is hope and pray that if we give them tools that can let them do a better job and let them do it faster, that you know, maybe, maybe we'll be able to give them time to do it better, instead of just piling even more on top of it. Like, I think, you know, I've been in the industry long enough that I know that maybe it'll be some of both. But the very least, we're helping an already underutilized resource that is not operating at the top of its license that is spending so much of his time doing menial labor, reading low value scans, or making low value decisions. At least we're allowing them to operate at the top of the license. We're removing a lot of the menial work from their day. And I like the hope that yeah, letting the humans do the things that humans do best rather than the really crappy stuff that machines do pretty well, I really hope that that'll at least allow us to lead to some systemic improvement.

SPEAKER_02:
The only other point on the productivity side, right? I'll push on it one more time, and then we'll talk about something else. But the system that you're talking about right now, and this is something that I've heard from everyone who is in the health tech, health tech AI space, is exactly what you said. We don't have enough doctors. We don't have enough nurses. The doctors and nurses are overworked already. The electronic health record is not helping. We have an aging population. We're going to have a massive doctor shortage in the next couple of years that we do not have capacity to deal with. And I guess, and it's interesting, but just as a thing to think about, right? The, the world that you're talking about right now, where they have a second to review an image, they're seeing 30 patients a day, specifically in hospital ecosystems. There's just, they're incentivized for as much as you can do possibly. Efficiency tools, productivity tools, AI tools to read these things. I think there is a world to be super pessimistic, right? Where that gets worse, where they're even more incentivized to see not 20 patients a day, but 50, because now you can. And so I think the ideal is we would like them to be able to spend more time with the patient because, you know, the model can, whatever, read that X-ray for you. Uh, and then you can just verify real quick and then you could talk to the patient for 90 seconds instead of 22. Um, I, but it, it seems like there's a risk that kind of the way the electronic health record was intended, whatever to make things better, but it ended up making things worse on just the side of overwhelming them with that kind of administrative work, which I don't think they saw coming at the time. It seems like there's a risk that this might happen to here. And is that something that comes down to the hospitals? And you mentioned the PE systems, all this consolidation to be like, you know what? No, we're not going to do it like this. Is there any other mechanism in place to make sure that that doesn't happen?

SPEAKER_01:
Uh, I, I, you'll probably tell that I somewhat share your pessimism on the space. Uh, the, the way that I choose to view it, uh, to be willing to get up in the morning and go to work is, I think if you talk to the average doctor or the average nurse about what percent of their day is spent on actually high value tasks, I think you'll be pretty depressed because the overwhelming majority, I mean, maybe it's 60, maybe it's 50, maybe it's more likely 70, 80, 90% of the day is spent on just stuff that doesn't really need to be them doing. guess what, there is actually this intersection of where AI ends up being better at and able to do stuff like documentation, like looking at really simple studies for really simple findings. And so, in my mind, the way I think of the system is, okay, we have a lack of executive decision-making capability that leads to problems, just as much as we have a lack of clinician time. And so, if we can give healthcare back all the time it's currently spending on administration. If we can eliminate and automate large swaths of the 10 headcount that every hospital has to support one doctor, you're still driving costs down in the system. You're still enabling limited resources to spend more time in decision making. Maybe they'll get pressured to not do so, but you're still creating capacity. You're still at least allowing the people that are out there every day working to spend our time working on stuff that matters. And I think the AI is actually the tool that comes in and strips out cost from the system in a way that we haven't been able to. I mean, just take every single problem that's been solved in healthcare historically, like care coordination, or care gap closure, or patient communication, and the answer has been hire another body, and that's been the case for all of healthcare's history. And so I feel like I choose to believe that creating capacity and allowing clinicians to operate at the top of their license will lead to a better healthcare system. And I choose to believe that there'll be some compromise between the fiduciary powers that be that currently drive decisions and hopefully us as patients and doctors getting more ability to do the right thing. And hopefully, we all can be part of that.

SPEAKER_02:
Well put. Yeah, I'm with you there. Hopefully, a lot of this stuff is hopefully. And it kind of brings me to another point. I think we've established, and if you're a regular listener of the show, you know this, everything in AI is so complicated. Like there's no, uh, black and white, good, bad, yes, no answer to anything. Like it's, it's nuanced. A lot of it depends on deployment, how you think about it. I wonder how you think about, you know, where you are, you're working with hospitals, with clinicians, you're, you're that last mile that you've talked about, right? How do we take something that someone made and help you adopt it and make sure that you can trust that you can adopt it safely? I wonder how you think about communicating what these tools do, can do, and can't do, and shouldn't do to the clinicians who are adopting them, because I think a big problem we're seeing in the industry beyond healthcare, but prominent in healthcare too is that, you know, developers might be the AI experts, and they're trying to get adoption in industries where they're not domain experts. And then those other domain experts are not AI experts. And so how do you make sure that a doctor of medicine not a doctor of computer science, understands like very well how a tool should be used and how it shouldn't. I mean, especially in the context of all the hype, how do you think about that?

SPEAKER_01:
If only you knew, if only you knew, but man, the number, so we've got over 70 third-party AI models on our platform. And the number of times that I have seen an ROC curve presented in a meeting with a clinical committee with the expectation that some chair of radiology in some part of the country is going to be able to understand the impact it has on patient care. It's kind of become the norm. And so I think the company is doing a terrible job at it. I think the clinicians themselves are not being given anything different. And so they're just expecting, well, it must be something I don't know. And doctors are gluttons for punishment. They wouldn't have entered medicine otherwise. And so they're now here trying to understand what a confusion matrix is or trying to figure out what hallucination is as a side gig to delivering patient care for 10 hours a day. So the problem is exactly the way you're describing. I think that a few comments on that. The first one is so much AI that's available on the market. is super, super simple. It's, is there breast cancer? Is there a stroke? Is there a fracture? Is there lung cancer, like in the case of my uncle? And for those tools, the binary classifiers, there is some question, right? Like, oh, is that lung nodule actually cancer? Did this need to be intervened on? Are we over-diagnosing? There are questions to be asked. But a lot of this stuff, it's, Simple. And at least for the simple ones, we shouldn't let this view of AI being scary or being risky get in the way. I think where it does end up being problematic is when you actually start to talk about, OK, so what do clinicians actually need to look at to judge performance, if not an ROC curve or confusion matrix of some model's performance in the vacuum? And it's actually, you need to connect it to downstream data points. You need to connect it to the ultimate diagnosis of cancer based on the pathology report, you need to connect it to the actual outcomes, the staging, the interventions that were provided, the ultimate survival. Those are actually the endpoints that these AI models need to be connected to. And that's something that we've realized that hospitals don't do ever. Literally, you will have one academic center, maybe publish one paper 15 years after an AI tool has been launched, And I think AI has been in health care since the early 2000s. It's been around for years. And you see these longitudinal studies published on AI's actual ultimate impact on the population literally decades later. So the question there is, OK, well, how do you get clinicians the actual metrics that they need, even in the beginning? Like, hey, does this tool work in your patients? Does it work consistently across your patients in different demographics? Does it keep working the same over time? Those are the questions that we realized had to be answered that weren't being answered by really anybody. So I think it's a simpler problem for so many of these AI tools in healthcare. It doesn't mean they're simple tools, but it means the thing they're finding is relatively straightforward and simple. The end downstream data points you're comparing it to are complex from a healthcare viewpoint. But they're simple from a data science and engineering viewpoint. And so it's really just trying to pull in all these disparate threads and create a consistent view of these models' performance by connecting all them and by being able to run and have visibility into the iTools outputs and then Also by automating that analysis, so you're not adding it more headcount to healthcare in order for them to do the right thing.

SPEAKER_02:
But to your point, yeah, a lot of these things are, especially where I know you said a focus is on the diagnostic side of things. These things are a little less complicated. It is yes or no. And in that region, right, I guess if a model said yes, then it kind of activates a chain of you know, then it will definitely get to a human reviewer who is now looking specifically to validate that there is cancer on that node. Um, and, and so I, I guess it kind of activates the thing like the, the healthcare system faster.

SPEAKER_01:
That's scary too, right? Because now you've got, you've got AI that if the AI misses it, right. You said you need safety nets that will make other safety nets worse. So, um, that's, that's truly one of the problems that, that, that exists in the industry. that is not solved. And I just want to take a moment to, I guess, even even in the world of generative AI and AI agents, and this next wave of extremely unstructured, and kind of wild out there use cases for AI. Guess what, like like it or not, if you're gonna affect clinical decision making, if you're gonna actually change patient care, Like, you've got to lock that stuff down. And so the reason these, none of these tools are being really used in actual clinical decision-making yet is because we do have safeguards in place. We do have ways that we need to require that they are observable, predictable, you know, a little bit more definitive and less probabilistic in their outputs if we're going to deploy them on patient care. And so, thankfully, safeguards do exist in place. I mean, to some degree, it is slowing adoption of these tools. But when tools actually do enter the market and they are being used for clinical care, even if they're running on unstructured data, there is an expectation, thankfully. that those tools have some amount of predictability, visibility, and you can use that as a basis off which you build your downstream validation and monitoring systems.

SPEAKER_02:
I was going to ask you if you think there are systems that have not been adopted and that should be. And I think the point you just made is really interesting in the context of the gap between where the cutting edge of the developers might be versus where actual use cases might be. And you see this in enterprise adoption as well. You know, wild agents is not necessarily something that's going to earn the trust of doctors. You need elements of determinism so that you know it'll verifiably work the same and consistently. You don't want an unpredictable diagnostic tool. You want it to be predictable in that it'll do what it says it's going to do. It'll perform the same no matter the person's gender or race or these other things, right? Algorithmic bias is an issue. I think that really, you know, it's another one of those interesting things where you have these different pools of experts that are sitting in different places and so much of the push in generative AI and agentic AI is Harder autonomy, more autonomy, more autonomous decision-making, less structure, less guardrails, right? But you need a little bit more of that stuff in order for these things to be used in critical environments.

SPEAKER_01:
Oh man, where to begin with this one? I think that... it's a little bit of a baby in the bathwater situation, which is, I think there are plenty of things that these AI agents like can and should be doing, you know, like, if a patient's never going to go in and read a doctor's note anyways, I mean, maybe, maybe you and I would, but there's value to AI, you know, giving it a recap or give a summary, like things like that, right? There's, there's, I think there's value to AI being used instead of human bodies in this kind of absurd, nuclear arms race between payers and providers. Like, you know, just honestly, instead of just hiring another 300,000 people at United Healthcare to go to bat with against another 500,000 people at the health systems, just just have AI go back and forth and find like United's probably gonna be able to afford a better AI than hospitals will. But at least we're not all spending our healthcare dollars on on bodies that are whose sole job is just to go back and forth and argue about, you know, patient care and data. So I think there is so much of healthcare that is just administrative, has nothing to do with patient care, where I just cannot wait for AI to just replace all of it. But then when it comes to clinical care, like you said, even, you know, many aspects of patient communication, like how do you know when an AI model really should have escalated it but didn't? There is a blind spot to AI being overconfident. You learn in medicine that overconfidence is literally the worst thing that you can be as a young, unframed student or trainee. And yet that is the problem that AI has most painfully right now. So I fully agree. We believe that the way you address it, at least partially, is, you know, guess what? Like if the AI messes up at some point downstream, It's not like that diabetes went away, or that stroke disappeared, or that patient isn't going to fall. So at some point, there are very clear downstream metrics in health care. Health care is an atoms-based business. It's going to result in outcomes. And so that AI models underperformance at some point will result in a real-world impact, just not in the way you want it to. And so I think we think about this as an ROC curve is just the beginning. But if you allow these AI models to be consistently viewed against downstream data points, whether it's the signed report, whether it's downstream diagnoses, whether it's downstream coding data or billing data, like, guess what? If the AI misses a stroke, it misses a lot of strokes. I'm going to see that in your data. The question is just, is anyone looking? Do you have the capability to ask that question? Do you have the bandwidth? Do you have the even time, or do you even care to? have seen a lot of hospitals do, and a lot of the reason why many people aren't adopting AI is because they don't trust it. They don't know that it's going to work. And they know that if it doesn't, they're going to kill people. And so I think giving them the ability, maybe it's a little too late. Maybe ideally, we find that the AI tool works before it misses 50 heart attacks. But if we're going to deploy anyways, and if these tools are already being used, at least we can try to build a safety net for it the way that I just described.

SPEAKER_02:
You just mentioned a point that we've been making this entire time, which is that hospitals don't really trust it, largely, and so they're adopting very slowly, very cautiously. I don't know that that caution is a bad idea, but at a certain point, you know, if developers don't meet them where they are, then there's nothing to adopt. Like if, if that doesn't kind of evolve. And then, so the, the validation system that you're talking about, which is connected to downstream data. And so we can, we can tell you whether or not this is making mistakes. we can tell you whether or not this thing should be trusted, you know, in real time, right? Obviously it would be great if this could be completely validated so that you don't miss the heart attacks, but you know, you're going to miss the heart attacks beforehand. Um, and I guess that's just one of those things where it's kind of imperfect. Um, and perfection is a, so it comes down to acceptable error rates, really like nothing is going to be perfect, right? People, People always throw out this thing where it's like, you know, you're harping too much on the fact that AI makes mistakes. Humans make mistakes too. I don't really love that, but also because people have different expectations. If we're adopting a machine, the idea is that it wouldn't make the mistakes that a human would make. And if it's going to make different kind of mistakes, then we have a much harder job of dealing with it.

SPEAKER_01:
I can tell you now, machines make different mistakes.

SPEAKER_02:
Exactly. And that makes it hard. I mean, you talked about the young doctor, the young, overconfident doctor. The mistakes that the young, overconfident doctor would make are somewhat predictable in nature. You know the areas that he's going to go in are like a new student who's just out of medical school, just doing their residency. A patient's crashing and they kind of freeze up or something, right? Like there are human failure points that, you know, since we've been doing this for a while, we kind of, we, we, we know the machines fail in different ways. And the thing about generated AI, especially, um, Is that they fail in unpredictable ways, which makes it really like to, to the point of everything that we've been talking about, it makes it really hard when you, when you talk to these hospitals. And when they sit down and when you talk to them about methods of validation and stuff, and when you talk to them, I imagine about, you know, why they've been reluctant to adopt. I, what can you tell me about where the hospitals are at that you speak with when it comes to how they think about adopting this stuff and how they are doing it now?

SPEAKER_01:
Um, we work with some of the leading health systems in the country, some of the most progressive, some of the most innovative, I mean, they wouldn't be working with us if they weren't. And. they're still just trying to get their employees to stop sending patient data to ChatGBT. They're just trying to get these Fathom note-takers and note-meeting things to get out of their sensitive meetings because its data policies are pretty rough. So if you want to have a generative AI, they're not even close to even asking these questions about it. But the doctors are turning them on anyways. They're turning these tools on anyways. They're already so far behind in governance, and they're already so far behind in even understanding what it means to know that these tools are safe. But yeah, we're nowhere even close. I feel like the actual system view I have of this is the silver lining to everything that we just discussed. Unlike other aspects of health care technology, whether it's a drug or device where an iteration on it takes a decade, AI is advancing every single week, every single month, every single quarter, even just in the LLM world, like a year ago to now. And so on the timeframe that health care operates in, I think the bigger problem is that hospitals are making buying decisions. And the problem is that the buying decisions are going to be for another decade. And if they are, you're gonna have hospitals stuck with an LLM from this year, or an agent from this year, in 10 years when, you know, AI's already, you know, you'll have our AI overlords, you know, governing our economy, and hospitals will still be using ChachiBT 4.0. And like, that's kind of what we do now, right, with fax machines and pages, I joke about it, but. That's kind of the issue, I think, is when hospitals adopt tools, the amount of friction it takes to turn a tool on, they hold on to them for five, seven, 10 years. And with AI, that just cannot happen. So I think if you give them not just the ability to validate and ensure these tools, I think that's just the beginning, because guess what? You'll validate a tool, you'll say it works, you'll turn it on, you'll say it's relatively safe, and then next month, there's gonna be a tool that is even better, that does even more, that's even safer, even more powerful. And you're gonna be stuck with the tool you just bought for the three-year or five-year term that you bought it on. So I actually think the problem systemically is making sure hospitals have the ability to be flexible and to be fungible with the models they're using, allowing them, again, to turn applications on and off, like you or I turn on or off on our phones. And that, given the pace of AI innovation, combined with the ability to validate and ensure safety and understand what is better. I think that's actually the secret sauce, right? It's not just making sure a tool's safe and then just letting them off the race. They say, we're gonna let you know a tool's safe. We're gonna let you know the update that the vendor launched is safe. We're gonna let you know that the competitors that they're going up against are safe. And we're gonna let you hot swap, replace, update your AI ecosystem and like all of the sort of different systems that are connected within your enterprise And that kind of constant drumbeat of innovation is going to be what fixes these problems, rather than a conservatism and a holding back and a waiting until the self-driving car never kills anybody. Because that, I think, is a fool's errand for healthcare, and millions of patients will die and be harmed while we wait.

SPEAKER_02:
It's an interesting point. The flexibility that you're talking about, that's gonna be core. The long terms.

SPEAKER_01:
Yeah.

SPEAKER_02:
It comes with safety.

SPEAKER_01:
Flexibility plus the validation, plus the monitoring, plus knowing, it's knowing what's better. It's knowing that this tool that just entered the market is actually better and how much better it is and what that means for our patients. I feel like so many companies and kind of IT departments and healthcare, they just focus on the flexibility. They're just like, we need to be able to innovate faster. And so many companies are like, let's disrupt healthcare.

SPEAKER_00:
And it's like, well you can't, you don't want to disrupt it unless you know that the disruption's positive. And as I mentioned before, we're not really measuring the disruption. So let's get that in place before we do too much.

SPEAKER_02:
Yeah, let's not disrupt healthcare. Let's enhance healthcare. It's all terminology. It's how these things are sold. It's flexibility that's tied with some sort of ecosystem that enables safe flexibility. But the long terms of buying XYZ product. Yeah. Especially given the fact that these things degrade over time, like even advancements aside, you know, if advancement stops today, but you have to be able to understand that, you know, if you're buying a five-year term on a technology and on a given system, and it might degrade, uh, in, you know, 20% of the way into that term, maybe there's a different competitor you switch to, like maybe there's something else going on. And clearly there's the ecosystem isn't primed for that kind of movement.

SPEAKER_01:
One of my favorite things to just hear is, oh my God, like if I were a trainee, I would not become a radiologist or I would not become a pathologist. Like, oh my God, AI is gonna replace doctors. And what I say is, listen, do you know how many doctors there are in the US? There are a bit over 900, right under a million doctors in the US. So, you know, you've got that. Do you know how many healthcare workers there are? 22 million. And so you've got less than, you know, like, a tiny fraction, a half a percent of all of the humans that are in healthcare are doctors. And many are nurses, and there are many other kind of clinical workers that need to be there. But AI is going to start from the bottom. It's going to start from the lowest skilled tasks. And I have no doubt that someday AI is going to replace doctors. but they're going to be the last people it replaces. And so I would love to be in the system for the rest of my life where it has replaced everyone but care workers. I would love for that to be the case. And at some point, it might replace my physical therapist. It might start replacing care workers in our lifetime. And that's great. But the more it does, actually, I think the better off all of us are. But I think doctors are going to be last. And I think I don't know that's going to happen anytime, anytime soon. And same with nurses and same with a lot of these people that kind of technology savants have gone headlong to try to confront.

SPEAKER_02:
Yeah, I, I, the whole replacement front is an interesting thing. I, I don't, I don't expect to see any like medical replacement anytime soon. Um, unless it's the insurance people that you were talking about that the kind of insurance back and forth, maybe, maybe, but all these other ones, we just have such a shortage and to replace them would require so much. that, you know, in some ways the ideal situation is if we just didn't have a, if we didn't have a shortage, if we were able to just snap our fingers and make, you know, millions more nurses and doctors appear, we can't do that. And so this is going to be a kind of necessary tool to help them, but it, but it's the, It's interesting, and we'll leave off on this point, but the different ways to look at these things. We're talking about artificial intelligence, and there is a tremendous amount of hype from the tech side of what it can do, and a lot of the perspective from the tech side to all these other domains, including healthcare, is, as you mentioned, disruption, replacement. They always shoot for the top thing they could possibly shoot for, but there's a different way of thinking about it, which is just the same way that a hydraulic press can squash something way flatter than I could ever get it, it's not gonna change the fact that sometimes I'm gonna squash something with my hands, but that enhances my strength. and here I think looking at it as more of we're developing additional tools Like the computer or, you know, the fax machine today is outdated, but at the time it was great. Like that, that was, that was such an enhancement on the speed of communication. Right. And looking at it from that perspective of, you know, if we're not coming at it from a, from a place of attempting replacement, we're just coming at a place of, we got to come up with ways to make your job better, to make you better at your, like whatever it is. Then it's a much different conversation. And I think that takes a lot of the fear out of it too, because there might be among nurses, like there might be a perspective of, you know, I'm cautious to adopt for all the reasons that you and I've talked about, but also I don't want to adopt the thing that's going to take my job away. And, and, and I think talking about it a different way is probably going to be beneficial because you know, that, that stage of replacement, right. Where you got like a medical droid kind of from star Wars, right. A lot would need to happen to get us to that point. And even then, I think people would still want human caretakers if they could access them.

SPEAKER_01:
Yeah, I totally agree. I think we, at the end of the day, we're going to need so much help over the coming decade and two in health care. And I kind of see this defunding slash collapse of our social safety nets and Medicare and social security, like there is there are a lot of unsustainable financial sequelae of the current trajectory we're on in health care. And so that's something that I feel like we just are going to one way or another be forced to reckon with. I'm hoping that what we're doing at Ferrum is a way to bend that cost curve and try to introduce at least a few more years of good health and safety within the healthcare system. But yeah, we all need to be pushing in the same direction here. And I think there's a lot of opportunity and I do think AI could not come at a better time to help fix some of these problems we have as a society. But I think it's got a lot, it's got a lot of missteps and it's got a lot of scary landmines in its way.

SPEAKER_02:
Yep. Your, uh, your job is not easy. I said this before and I'll say it again. It's not easy. Um, but you know, yeah, we'll be interesting to see if we can go that last mile. Totally agree and hope.

SPEAKER_01:
I mean, I, I know we can, I know that there's the right way to do it and I'm hoping that, uh, you know, the rest of the industry recognizes that sooner, sooner rather than later.

SPEAKER_02:
Hopefully. Cool. Well, Peilu, this has been so much fun. Thank you. Likewise. Thanks for having me.