What can software engineers learn from post-incident reviews that physicians do in the emergency room? In our ninth episode, Christina, member of the Blameless strategy team, guest-hosts the podcast to interview both Kurt Andersen and Al'ai Alvarez, MD (@alvarezzzy). 

Dr. Alvarez is an assistant clinical professor of Emergency Medicine at Stanford. Clinically, he’s an emergency physician. He has led residents as an associate residency program director, and recently became their department’s Director of Well-Being. His work focuses on physician well-being and humanizing physicians in the context of high performance teams. 

The trio talk about fostering blameless cultures in their domain, dealing with self blame, and what leaders can do to make learning from bad outcomes safe and fruitful. Read the organized transcript below. 

New to our podcast?

Resilience in Action is a podcast about all things resilience, from SRE to software engineering, to how it affects our personal lives and more. This podcast is hosted by Kurt Andersen. Kurt is a practitioner and an active thought leader in the SRE community. He speaks at major DevOps & SRE conferences and publishes his work through O'Reilly in quintessential SRE books such as Seeking ARE, What is SRE?, and 97 Things Every SRE Should Know.‍

Speaker Introductions

Kurt Andesen (00:29): Hello, I'm Kurt Andersen from Blameless. And welcome back to Resilience in Action. Today I have a guest host, Christina Tan, who works with me at Blameless. And Christina, can you introduce yourself and then lead right into the podcast?

Christina Tan (00:43): Sure. Thanks, Kurt. Hi, everyone. I'm Christina. I'm on the strategy team of Blameless. And I'm really excited to introduce you to my lovely friend Al'ai here. So, the context for today is that we've wanted to have this conversation for a long time, because there's so much parallel in learnings and lessons between incident management or disaster response to some extent in medicine and also in software and finance that Al'ai, coming from medicine, I think, could teach us a lot about how we respond to incidents. So without further ado, Al'ai, please introduce yourself.

Al'ai Alvarez (01:23): Hi, everybody. Thanks, Christina. My name is Al'ai Alvarez. I'm one of the associate  residency program directors at Stanford Emergency Medicine. Clinically, I'm an emergency physician at Stanford. And also, most of my work is on physician well-being. And along with that is quality and process improvement, our work on the first. I also interface with Stanford WellMD as the co-chair of the Physician Wellness Forum.
Why is Blameless Important to Your Domain? 

Christina Tan (01:50): Thank you, Al'ai. And I'm sure the audience already knows a lot about Kurt. So Al'ai, the company name Blameless, when we first met, seemed to pique your interest. I remember seeing your eyes light up. Can you tell me more about why that's interesting to you?

Al'ai Alvarez (02:05): Yeah. Thanks for that question, Christina. And I do remember that. As you know, in our work in medicine, medical errors do happen. In fact, medical harm happens, and calling it medical error creates a stigma of blaming, in the blaming culture in medicine. In our departments, we change the name of our peer review to case review to really strip away that notion of naming, blaming, and shaming somebody who may be part of an adverse event, whether intended or not.

Al'ai Alvarez (02:42): So oftentimes, there's bad outcomes that happen, just simply because of the natural course of the disease, then there's bad outcomes that happen because of a misdiagnosis or complications of a procedure, all the same, or even just simply taking care of really sick patients. We call it vicarious trauma or the second trauma syndrome. We take this to heart. And this impacts the way that we see things, the way that we interact with each other.

Al'ai Alvarez (03:12): And that idea of psychological safety, when we're chatting about the case, the name Blameless was very interesting for me because I think focusing on the events and what happened without attributing blame to someone has been very helpful. And as you know, in our work with compassion, also taking that blame away from yourself, so practicing that self-compassion, allows somebody to really take on the processing of the events, the whys and the how’s as opposed to who did what and being stuck in that front.

Christina Tan (03:53): Thank you, Al'ai. Kurt, do you mind speaking to why Blameless is significant in the software domain?

Kurt Andersen (03:59): Sure. It's very much the same thing that when there is something that goes wrong, understanding the wider context of the system that allowed the wrong action, or perhaps fostered an incorrect mental model on behalf of the participants with the system, it's more productive to look at the system as a whole instead of pointing fingers or taking a heads shall roll perspective, which is where the blame takes you. When you bring blame into the game, people shut down, and their creativity disappears. And these are all not conducive to improving either the system or the way that people operate within and on behalf of the system.

In emergency medicine, we change the name of our peer review to case review to strip away the notion of naming, blaming, and shaming somebody who may be part of an adverse event, whether intended or not. 

Opposite of Blame: Recognition

Al'ai Alvarez (04:49): And I think similar to the opposite end of blaming is recognition. So, I know that we like recognition by names as well. But oftentimes, you recognize just one person as opposed to understanding the process that led to a successful event. And so even stripping away just nominating one person, and just looking at the team, creates an environment where people are more likely to continue the process that led to successful events as opposed to just focusing on just one person.

Kurt Andersen (05:19): Yeah. I was going to ask you, Al'ai, when you talk about your case review process, do you do case reviews in situations where things work very well or a near miss, or do you only do them in the case of an adverse outcome?

Al'ai Alvarez (05:35): Yeah. Those are two different things when you said near miss and also very, very well. Very, very well is like we call it the save of the month and so we celebrate saves. But in the past year, what we're realizing, the saves really recognize only one person. And so since January, what we started with the residents is having this amazing and awesome event. And so, we have a lot of time in our journal club to have that. So whoever got nominated for the save of the month, then they get to talk for 15 minutes about the individual, like how did this particular event happen like as a success?

Al'ai Alvarez (06:12): And then they have to comment on the individual, the team, and also the environment, what were the team aspects that led to the save, what were the environmental aspects that led to the success? And so then they get to talk about it. The whole class, the whole residency program, so 60 residents, there have been faculty asked them questions like, "Oh, how did you do this? What happened?" And really, again, creates this sense of fostering the positive conversation instead of focusing on the negative ones. 

Al'ai Alvarez (06:40): The near misses are different. We do that as part of our case review. And in those scenarios, again, we focus on the system. So what level in the system prevented the mistake to happen, the error to happen, and also, what did the team do to prevent that from happening? And so I think, focuses on a lot of systems area as opposed to, again, focusing on one person who may have screwed up or not even have screwed up but there was just the unlucky person that took care of that patient and now is having to address many of these explanations of what happened.

Christina Tan (07:14): I love that you have positive celebrations. Kurt, is there something like that in software?

Kurt Andersen (07:20): Wow, nothing that comes to mind. I love the save of the month but I also like the concept of expanding that beyond the particular individual, because it's been shown that like Employee of the Month programs do have some counterproductive aspects, because of the competition that engenders. But to be able to expand the focus to the team and the context, systems that supported that accomplishment is a great way of countering that aspect.

Al'ai Alvarez (07:54): And I also want to follow up with that, that I think it's one thing to do to save months of amazing and awesome events in a public forum. So, people talk about it, people witness what's happening, I think what's more important is actually the debrief immediately after the event. And so, for me, one of the things that I really encourage, whether it's a good outcome or bad outcome when we take care of patients, is to have a debrief, to just have that shared mental model as you were explaining because it's not always a shared mental model.

I really encourage, whether it's a good outcome or bad outcome when we take care of patients, is to have a debrief immediately after the event, to just establish a shared mental model.

When Bad Outcomes Happen

Al'ai Alvarez (08:22): The leader has a mental model and then the rest of the team had their idea of what's happening, and simply debriefing that and sharing like, okay, what worked, what didn't work. I'm using Blameless here. I think in the idea of Blameless, we use it in a form of at least when I introduce the topic of debrief, we talk about it, again, individual, the team, and the system. If I can change 1% to better that outcome, what is that 1%? As opposed to what could I have done differently, which I think is a lot of pressure for somebody, especially when you're talking about losing life or having significantly devastating events, and so that it takes away that moral distress of answering that question. Just 1%, what is that 1% that I could have done better to make this better for the next iteration?

Christina Tan (09:10): That sounds like something we can learn from. Kurt.

Kurt Andersen (09:14): It is. And actually, it puts me in mind of some of the specifics that Steven Spear talks about in his book The High-Velocity Edge where he thought the functions of the check-in and registration at a hospital in Pennsylvania, I believe it was, where they were constantly overwhelmed and patients were sitting for hours waiting to get into the emergency room. And they took it in a very piece by piece approach that I think it was one of the head nurse managers, or I don't recall the exact title level, said, "Okay, what can we do to take this process that takes 15 minutes and make it happen in 10?" And then they just took cumulative, they gained more time by making each of the processes more efficient to the point where they eventually had nobody had to wait any significant amount of time.

Christina Tan (10:13): Yeah. And speaking to the point of moral distress, sometimes an incident could cost a company millions of dollars. And the engineer in question most definitely has feelings that I think are difficult to resolve. Al'ai, what do you do to help resolve that feeling, because there's the actual resolution of the case review and then there's what the person feels, like you said, the secondary trauma?

Al'ai Alvarez (10:40): Yeah. I think there's a few things that we can do as leaders. For me, one of the things that I find effective is modeling that vulnerability. And so, my residents know that I share a lot during shifts like I totally screwed this up. So, there's a lot of maybe self-deprecation, but really just sharing with them that I will normalize that this is not easy. So in those debriefs, when I talk about the self, team, and then the environment, the self is usually me sharing like I was really nervous about this decision, or I don't know if you noticed this, but like, I made this mistake.

Al'ai Alvarez (11:13): So I kept on asking for this particular task to happen. And the third time I said it, I realized I was saying it out loud without pointing to one particular person. And so nobody knew who I was talking to. In a simple acknowledgement like that, you can see people are like, "Oh, my gosh, the attending just accepted this." And then it's actually pretty cool, whenever I do this, people start sharing, like, oh, I did this and I want to do... So again, it's not blaming themselves but just taking accountability for the 1% that they could have done better.

Al'ai Alvarez (11:44): And it was not a big deal. All I had to do was next time, that 1% is I need to tell somebody directly or look at them in person and say, "Hey, I need this to happen," as opposed to just saying it out loud like I'm thinking. So vulnerability, I think, is definitely a key thing there. And it's not easy, which means that there's a lot of practice of self-compassion to be comfortable with that. And as a leader, I think that's very, very important. 

Al'ai Alvarez (12:10): Second, I think you have to do multiple touch points. You can't just do this out in a big forum. So, when I do the debriefs, I had to tell you, if I know there was a bad outcome and I know that there's something that I personally as a faculty could have told the resident like, "Hey, you should have done it this way," I pull them aside ahead of time and say, "Hey, just so you know, I will bring this up. What are your thoughts?" Or at least think about this so that you know you're not going to be sideswiped in front of everybody to have to respond to this question, so that they feel empowered. And I also level them up at the very beginning, like, "Hey, Alpha Chief, Christina has done a great job. I want her to lead this discussion." I will just start with my mistake because I did this and then people have that sense of openness when they talk about the case.

If I can change 1% to better that outcome, what is that 1%? 

Dealing with Self Blame

Christina Tan (12:55): I love that. Students who are residents ever encounter a situation where they feel so guilty that they're not sure how to handle that emotion and come to you?

Al'ai Alvarez (13:04): Of course, 100%. Yeah, it's actually very common. And so that's when I normalize things for them. I would say like I can't believe I did this, and just so you know, it's normal to feel this. Or even the opposite end, when you've really done an incredible resuscitation but it's not the best outcome, I've pulled my resident aside and say, like, "Hey, I just want to pause and take this moment to let you know this is what it feels like to save a life." Because oftentimes, they're so focused on blaming things and focusing on the screw-ups that we don't recognize when we're doing amazing things, and we don't remember that feeling.

Al'ai Alvarez (13:44): And so, I take moments like, "Hey, this is how it feels like to do this procedure. This is so cool that you did this." Now when you pause on that and then you see that smile, and also when they're feeling down, I just want you to know, this is what it feels like to feel like you didn't do a good job in that case. And let me explain my thought process and why I think you actually did a good job. So I'm not trying to take it away from them. I think it's important to feel the negative aspect, but it's also important to put it in context that they're not alone in this experience, that everybody else has felt this way, and then there's ways to improve if they're willing to, if they're actually open to exploring that.

If a resident has done an incredible resuscitation but it’s not the best outcome, I’ve pulled them aside and said, “Hey, I just want to pause and take this moment to let you know this is what it feels like to save a life.”

Christina Tan (14:23): Wow, that's beautiful. I feel like a company with that type of culture would really help employees empower them to grow and really be at their best. Because one of the hardest things when we make mistakes is we feel alone and it's like there's something wrong with me as a person. Kurt, how have you been in this kind of guilt that's internalized within the team and how have you seen this be addressed?

Kurt Andersen (14:51): It certainly happens in IT when there's outages. I'm sure that there are a number of people at Fastly today who are going through this and struggling with this themselves, because they had a rather significant outage of only an hour. I mean, just for context, this was only an hour earlier today. And it ended up on the front page of the New York Times who was affected, and the front page of writers and worldwide was smeared all over the place saying, "Oh, the internet has a huge outage." Well, yes, but putting it in context, it wasn't the end of the world. People probably did not die as a result of this outage. They may not have been able to get their news fixed in the morning, but there are other things in life.

Kurt Andersen (15:44): And allowing people to talk about those feelings and at the same time being able to look at the overall system, and I don't know any details, no details have been released yet, I'm sure that they will be writing up a retrospective and publishing that in due time. But letting people process both the emotional aspects and then also looking at the system level changes that led to incidents is important to do on both sides, because you've got to recognize the people and what they're dealing with, and the systems that led them in that direction or misled them in that direction, potentially.

Christina Tan (16:26): Something that I found really inspiring in software is this idea that failures are inevitable, to be human is to err, and with complex systems they inevitably fail. And I think also in the context of teamwork, things inevitably go wrong. It's more about how we respond to them, rather than... I don't think it's always about prevention or perfection but there's a great component of just being able to be resilient against issues that are out of our expectations.

Al'ai Alvarez (16:58): I think it's more than just being resilient that's outside of our expectations. I think it's also being able to sit with that discomfort, being able to process a lot of these negative things. I don't like the notion of stripping away those negative feelings and just trying to be positive, because that creates an internal conflict, I am allowed to feel crappy when I fail at something. And at the same time, I think what's more important is to realize that that is not the end-all-be-all, and that also I'm not alone in feeling that I have failed at this particular project.

Al'ai Alvarez (17:33): And so, for me, I think having somebody to reflect back and normalize that, yeah, this is bad, this is like what we're feeling is bad. And then another way is to put it in perspective, as Kurt was saying, in 5 years, in 10 years, in 15 years, will this really matter? And so for me, then, that takes away a lot of this pressure, unrealistic pressure, that we put on ourselves that is like, "Oh, I didn't do this," and therefore X happened. Maybe like in 5 years, 10 years, 15 years, it will still matter. But most of the errors that happened do not have significant consequences. And at least it allows you to then think, "Okay, what can I do next," as opposed to "Oh my gosh, I'm going to lose my job," or X, Y, Z.

It’s more than just being resilient. I think it’s also being able to sit with that discomfort, and process a lot of these negative things… At the same time, realize that it’s not the end-all-be-all and you’re not alone in feeling the sense of failure.

Christina Tan (18:19): Yeah, thank you. I think that's so powerful. I think the fear of job loss is very real in IT, because sometimes not all leaders understand that technology inevitably fails. So maybe Kurt can speak to this. I've known Fortune 500 companies where one executive will stand in the middle of a meeting room and point to another executive and say, "This outage happened because of this team." And they want to hold someone accountable. And you mentioned this, too. I love that you talked about accountability in terms of facing forward and doing that 1% difference, whereas I think there's a propensity for humans to find the bad apple and say this person did it and if we remove the person, we'll remove the mistakes.

Kurt Andersen (19:09): By dehumanizing other people, you can pretend that you're not equally susceptible to be the person in the hot seat. And so, there is some degree of blame shifting that goes on if you're going to be dealing with blame. But it's even better to go beyond that. And as Al'ai was pointing out, be able to feel the feelings when something goes wrong and that it is a bad thing, but at the same time be able to look at it and say it's not the end of the world, and how do we get better. And being able to do both is amazing. I think that it comes across as very compassionate to me and it's a unique perspective, a strength that I hear in what you're saying, Al'ai.

Al'ai Alvarez (19:53): Yeah. And I mean, in my field, I mean, somebody will die often, right?

Kurt Andersen (20:00): Right.

Al'ai Alvarez (20:03): And I'm not minimizing your work, but I'm just putting in a perspective of when errors do happen in my fields, people usually end up having long-term complications or they die. And so, how can you tell somebody to say that that's okay? And I think that's hard. But I think part of this is to realize that there's only so much that we can do in our role in this disease process versus the role of the entire system. How can one smart person who's trained for four years of college, four years of medical school, four years of residency training miss a heart attack?

Al'ai Alvarez (20:43): It's inconceivable. And yet, something happens that somebody will miss. And it actually happens a lot in this country. And it's because of interruptions, it's because maybe you are thinking about something else and then it was the wrong context, or whatever else presentation that led to that. That's why we look at it from a systems perspective and really incorporate that whenever you're looking at every single case, that's why it's a case review, as opposed to a peer review.

Al'ai Alvarez (21:08): That person that missed that heart attack is already blaming themselves. They don't need to be blamed by anybody else. They're going to go home, they're going to beat themselves up because they're like, "I can't believe I missed this." And we ruminate all the time. But I think until we get that compassionate level of, "Huh, I wonder why I missed this," that curiosity of, "I know what heart attacks look like on a piece of paper. I looked at that piece of paper and it didn't click. I wonder why that happened."

Al'ai Alvarez (21:35): And then when you realize there's all these systems, then those are the things that we can really address. Because in 5 years, 10 years, 15 years, another doctor is going to miss that heart attack again and not that person. That one person that missed it, I bet you, they're going to look at that EKG, they're going to stare at that and make sure that they're not going to miss that again. But everybody else may not learn from that experience. And so, that system review is really, really important.

Kurt Andersen (22:00): I think that's one of the things that is highlighted in the Checklist Manifesto about the importance of having... Even for well trained, extremely expert people, is having this procedural standard that people can follow.

Christina Tan (22:17): Yeah. I remember our compassion teacher often says, Robert, that compassion is holding opposites. It's okay to both feel really bad about what has happened but to also feel like you've done your best.

Al'ai Alvarez (22:35): Yeah.

What Makes Debriefs Successful

Kurt Andersen (22:36): Al'ai, do you have a specific framework or procedural template for how you do your debriefs? You've touched on doing them a number of times, but do you have a way of how you go about it?

Al'ai Alvarez (22:49): So I don't have anything formal, but that's a great idea. I think that's something that I'll take away from this, formalizing debriefs. But what I usually do is, number one, I gather everybody. I give them an expectation, "Hey, in 10 minutes," because I know that people as soon as we leave one trauma bay or one resuscitation, everybody does something else and they're so busy.

Al'ai Alvarez (23:11): So if we have time, I do it immediately. That way it's fresh and everybody is contained. But if we don't, I tell them a time point like, "Hey, in 10 minutes, let's debrief." And so it creates a space like we are allowing ourselves to meet. So that's number one. Number two, any critical things, again, like I have to step back and figure out is this a one person thing or is this everybody else can learn? And if it's a one person thing, then I would just pull them aside later on.

Al'ai Alvarez (23:36): If it's something that I know that people will be perseverating with, or it's a decision that many people will not agree with, I will quickly find time to just pull aside that person and give them a heads up, "Hey, we're going to talk about this. This is what I saw. I wonder what your thoughts are." That way they can respond and they have a few minutes to just collect their thoughts and not feel defensive. The last thing I wanted to debrief is to put somebody in a defensive mode.

Al'ai Alvarez (23:59): Preface the debriefs once we're all in the same space together and I say, "Hey, I know we've done multiple debriefs. I just want to remind us, because not everybody has been in my debrief, and here's what I want us to focus on. I want to structure it so that if you're going to say something, focus on the individual, yourself only, and then the team and then the environment. So there's no pointing finger." So I explicitly say that, what can you do as a person? Because if you know that Kurt did this and you want to say that, it's probably not helpful to delay in front of everybody, because it'll just embarrass Kurt. And then he'll just shut down and then there's no conversation.

Al'ai Alvarez (24:37): And then that last part that I said, and if there's 1% that you can do better, what would that be? And I'm not asking for a grand change, just 1%, what would be the one thing that you would change for me? And that's it, really, and then the rest I facilitate, "Oh, I like that." Oh, I miss something that's very, very important. And then I start, I start with something, a mistake that I made. There's always a mistake that I make. Everybody has some sort of mistake. And so I own up something that I can normalize to everybody, and then it creates that vulnerable psychological safe space for them to feel like, "Okay, my boss can own up to their mistake. Maybe I can as well."

Kurt Andersen (25:16): So a quick question. You say you get everyone together. Does that mean like the doctors, the residents, the nurses? Who is everyone?

Al'ai Alvarez (25:26): The whole team, nurses, the techs, the rest of our therapists, my residents, if I have a medical student the medical student. I really tried to get as many teams there. So, the recent one that came to memory is a pretty bad case. And the surgeons were there. The head surgeon was just upset, not happy with how things were. And so I pulled him aside and like, "I know you just came on to this and I know that you have a lot of opinions on this. Let's talk this out now because I don't want this to be like a butting heads in front of everybody. I'll share with you my perspective of why we went the way that we did and I want to hear your opinion." And so we did that.

Al'ai Alvarez (26:04): And I actually didn't get a chance to pull my senior ahead of time and give them a heads up that he's going to be leading this. And so I stood next to him during the entire debrief to create that, like I support him, because again, it didn't go so well. And I'm sharing this because I want to highlight something really, really positive. A senior surgeon started talking like, "I screwed up. I should have done this. I should have done that." I was like, I never would have expected a trauma surgeon to say that in front of nurses, other emergency positions, the techs, and acknowledge that.

Al'ai Alvarez (26:40): And for me, that was a signal that was like, "Huh, something that we said here, this space right now is safe enough for that surgeon, a head surgeon, to acknowledge their mistake." And that was pretty cool. Because then, you bet you, some leader like that starts saying something about their mistake, everybody else starts sharing like, "Oh, and I could have done this, next time I will do this." And again, it's not focused on the screw-up, it's more of like, "Next time I'm going to do this."

Everybody has some sort of mistake. And so I own up something that I can normalize to everybody, and then it creates that vulnerable psychological safe space for them to feel like, "Okay, my boss can own up to their mistake. Maybe I can as well."

Creating a Safe Space for Learning

Christina Tan (27:06): Kurt, as reliability engineer, do you see this leading with vulnerability? And how does that happen? What does it look like?

Kurt Andersen (27:14): I do. I mean, some of it comes from the idea of doing the retrospective or having a more neutral facilitator to lead the retrospective. Also, one of the most important things that has come out in practice is keeping people from making counterfactual statements. So, I should have seen this. I should have done that. I should not have done this, because then nothing would have happened or whatever, the system wouldn't have broken. Those are all counterfactual, but they're not grounded in reality.

Kurt Andersen (27:50): And so, keeping people factual, and the first person I saw, it appeared to me that it was this system and so I went over and investigated this, and it turned out not to be that system or something. It's a much more constructive way of understanding the incident and how people responded to it from their particular perspectives at the time given the information that was available to them. And I think, really, having come to the retrospectives from the point of view that everyone was doing the best that they could, given the knowledge and the system understanding they had at the time, really does make a big difference.

Kurt Andersen (28:34): And be careful of the language that you use. Courtney Eckhart had an awesome talk a couple of years ago that she gave, where she pointed out that just using the word “why” tends to trigger a judgmental stance, and it asks for, in some ways, an agent that was responsible, therefore, subject to blame. And so avoiding why questions and talking about what or how is a more constructive way to elicit information.

Just using the word “why” tends to trigger a judgmental stance. It asks for an agent that was responsible, therefore, subject to blame. Avoiding why questions and talking about what or how is a more constructive way to elicit information. 

Al'ai Alvarez (29:04): I love that a lot. And having trained in the east coast in the Bronx, I've had to learn that the hard way moving to California, because I would always ask before, like that's how I was trained, like why did you do this? Why this? So now it's more of, "Huh, I noticed this. Can you walk me through your thought process?" or "I saw this. What was your thought process?" And you're right, it's a little hokey but it does make sense. But you said something, Kurt, that really resonated, and I love that, which is I think aligning the goal from the very beginning, I think, that's something that I'll take away from this conversation.

Al'ai Alvarez (29:38): When we do debriefs to say, "Hey, I just want to remind everybody that everybody here today showed up with a goal of doing X, Y, Z, which is the goal of saving this person's life," I think nobody will argue with that. Because, nobody shows up to work and says, "I'm going to be a not nice person today," or "I'm not going to be X, Y, Z," we all have good intentions. And I think reminding people that can be really helpful, especially in debriefing tough scenarios.

Christina Tan (30:08): Yeah, absolutely. Kurt, do you mind sharing with Al'ai the retrospective structure that site reliability engineers often use?

Kurt Andersen (30:15): I'm trying to think about what you have in mind. But one of the practices that is emerging, I'll say more recently within the last few years, has to do with this idea of doing individual interviews with key participants beforehand, and by this neutral facilitator, ideally neutral, otherwise somebody who can at least have a degree of objectivity. Getting that information from them individually, because as you point out, if something went wrong, it can be easier to talk about it one on one than to feel like you're being put on the spot in a group setting.

Kurt Andersen (30:54): And then assembling that into a group discussion that is more focused on the team and system dynamics, after you get the individual points of view, seems to be the most constructive way to move forward. The other aspect that has shown up recently is that separating the action items, we've got to do this to fix the problem or prevent it from happening in the future.

Kurt Andersen (31:23): Separating that action item stage from the group meeting itself tends to result in not only better quality action items but it also results in better learning about the whole system by people who are involved in the group meeting. Because then they're not so focused on what are the things we can do to Band Aid over the problems, which often ends up being the case when you want, okay, we've got to come up with three items or five items or whatever, to fulfill some checklist, check the box, we're going to fix things because we did these three things. Those often end up in a ticketing system somewhere and never get action. But learning out of the process is much more enduring.

Separating that action item stage from the group meeting itself tends to result in not only better quality action items but it also results in better learning about the whole system by people who are involved in the group meeting.

Feedback Loop for Learning 

Christina Tan (32:13): And is there a feedback loop today both in medicine and in site reliability engineering so that essentially takes the learning and feeds it back into the system or the people?

Kurt Andersen (32:24): There should be. In systems that are working well, yes, there's a learning feedback loop, I'll say.

Al'ai Alvarez (32:29): Yeah, for us, we track. So for each case that we look into, we then categorize it as this is like a human factors thing, this is like a systems thing, and then from each system that we have like different categories of where, and so we track that. So over the course of the year, you know how many are systems related and then which ones are feedback that you're giving to the clinicians. And then you also track how many times it's happening, because it may be an isolated thing and then you realize, five months later, a similar event happened.

Al'ai Alvarez (33:01): And so then we can study it further and it's like, "Huh, this seems to be happening. Is there a new thing that happened? Is there a new workflow that we changed that's leading to this error to happen?" And then I think the one thing that Kurt mentioned as well that reminded me of what we do with our residents is so when I get a letter from my medical director, who's also the chair of quality of our case review committee, I still get palpitations.

Al'ai Alvarez (33:25): So I normalize everybody else, I still get home, like whenever I get those letters, there's a case that you're involved in. So I told my medical director a few years ago, "l still get triggered, even now like at this very moment, when I get that." And so, what I asked her is like, "Will you let me know ahead of time if you're going to reach out to my residents?" And as somebody who's done a lot of work on wellness, a lot of things on mindfulness and practicing self-compassion, I still feel those feelings. I can only imagine if you're not equipped with handling that emotion.

Al'ai Alvarez (34:02): And so then what I do is I translate that into simple terms. I reach out to them, so I give them that formal email instead of coming from the medical director. It's still big coming from their associate program director, like to say there's a case that's being reviewed. And then simultaneously, I text them, "Hey, heads up, there's a case. There's a formal email that you're going to get from me. No big deal. Actually, this is it. Let's talk if you want to talk now. You can call me or you can even set up a time to meet." But again, it gives them different options to talk about it as opposed to, in my experience in the past, when I get this email, I look at it. It's a secure email, which means I have to log in somewhere.

Al'ai Alvarez (34:40): And if I'm on my phone, I have the entire night just ruined because I can't look at what happened, what did I do, and I don't even know what the complaint is, but there has to be something bad. And then I look at it, it's actually like, "Congratulations, you did something awesome." And so now, when I send those emails, I'm mindful of the impact of that. And also, afterwards, I asked for a debrief like, "Hey, how did I do? Did that work? Do you want me to text you next time?"

Al'ai Alvarez (35:05): In the past, I've texted people and then they get panicked that I get a text from an APD, like what works for you? So, you have to individualize it, which is hard. For me, I have 60 residents. But I think just the act of trying, just me normalizing like, "Hey, I know how it feels when I get these. I tried this to make it less scary for you. Did that work? What would work better?" Again, it gives some more agency on how to receive the news, and also it gives me ideas of like what else can I do in the future?

Agency and Compassion

Kurt Andersen (35:35): You've talked a number of times about agency, empowering people. What's the connection between that, if any, and compassion?

Al'ai Alvarez (35:45): Yeah. That's a tough question, I think. Because compassion for me is just being able to either be kind to others or be kind to yourself. I think agencies, sometimes, you feel like you have no choice in our work, like you're stuck in this, "I just have to do this and I'm doing this." Which for me, if you're not empowered to learn, if you're not empowered to act on something, then you're always going to rely on your seniors, your attending, your boss, and I'm not creating an independent physician.

Al'ai Alvarez (36:21): And my goal in training is to make them independent. The compassion there, I think, comes with "I am letting you know that you will have many mistakes, just like me." And that's where compassion comes in. Just like me, when I make mistakes, I need to be able to be kind to myself in order to actually look at that email in order to process what's happening as opposed to start blaming myself, the shoulda coulda woulda of the difficulty of that scenario.

Al'ai Alvarez (36:49): I don't know if that answered your question. But for me, they're related in the sense that you have to put in the "just like me" perspective and also being able to be kind to yourself in order to not just do the job, but actually own that job, be accountable for what you're doing and be accountable on how you're going to change to be a better person in the next iteration of this, because there's going to be many other versions of this. May not be the same case but there's going to be another scenario where you're going to screw up, you're going to make a mistake, and how you're going to take care of yourself is going to mean a lot after that.

Kurt Andersen (37:21): It sounds to me like at least in a scenario where you're dealing with yourself and you're trying to be compassionate to yourself, that it is a way of taking yourself from a victim status to having the capability of making an improvement. And then likewise, honoring that same concept within the people that you work with.

Al'ai Alvarez (37:41): A hundred percent. There's this thing called the dreaded triangle. I don't know if you're familiar. So there's the victim, persecutor, and savior.

Kurt Andersen (37:51): Oh, yes.

Al'ai Alvarez (37:52): And so you want to avoid that. Well, again, I can easily see that picture. I sent you an email, you screwed up, and then you're going to feel the victim. And then I'm the rescuer, I'm going to be like, "Oh, that's too bad. You're not that bad. You're so good." This doesn't help anybody. It makes that person feel a little better at that moment but they're not going to look into why they screwed up for this example. If I rescue them and I'm going to be that person that's like, "Oh, it's actually not that bad," they don't learn from that. What I'm saying is actually to tell them, yes, bad things did happen. What can we do for empowerment, what can you do to be better?

Kurt Andersen (38:32): Yeah, it makes sense.

I am letting you know that you will have many mistakes, just like me.

Measures of Success

Christina Tan (38:34): So, it's so interesting because we're here in an article on blame right now and thinking about who is to blame. I think, oftentimes, a lot of the blame comes from inside. Many emotions are things that we don't have control over. It's just a natural human response to things that go wrong. And so, I think objectively at times that the blame comes from leadership, but at the same time, oftentimes, the blame can come from the self, too.

Christina Tan (39:03): And there's so much in self-compassion such as recognizing humanity that everyone makes mistakes can help move the person from panicking or guilt to having the curiosity to wonder what can be done differently moving forward. And, I mean, speaking from the leadership perspective, transitioning to that, I want to just hear what metrics are people being measured on for both site reliability engineers and residents? And how does that relate to when things go wrong? Are there metrics? What are the expectations, say, for reliability or expectations for procedure success rate?

Al'ai Alvarez (39:50): I find it fascinating that you went directly to metrics when you're measuring this. And I think one of the reasons why we struggle so much with self-compassion is because in our culture, we're basing everything on evaluations, we're basing everything on metrics. There's always this criteria that we have to make and not making that is a failure. So even with our choice of what success looks like, does that mean that if it doesn't look like success, it's not successful? And so, maybe that's a little bit more philosophical.

Al'ai Alvarez (40:27): But for me, the metrics for success, in this case, from a perception or from the perspective of medical errors, is when people can volunteer their case and talk about it. If they're willing to share with me that, "Hey, I screwed up," for me, those are one of the coolest experiences when I work. A resident comes up to me, "Hey, I screwed up. Can we talk about it?" Because oftentimes, people are so self-critical and they're not going to be able to share those because they're embarrassed, they're ashamed.

Al'ai Alvarez (41:00): And for me, a good way of a healthy environment is people sharing many of these things that they could have done better, not to deprecate themselves but to actually push the entire institution forward to, yes, we're doing great, here are better ways, here's 1% that I can take home with me that I will do the next time to make this organization better. Those few steps that Kurt was saying earlier about the waiting room process, I think that is a good metric to measure, which is hard because oftentimes, it's actually the opposite, somebody declares that they screwed up.

Al'ai Alvarez (41:39): Amongst people who are very comfortable with talking about this from a case review perspective, then you learn more. But those others because now that you mentioned it, there's tracking, now does that mean that this person is going to be put on a list of the screw-ups? And then what are the ramifications about their jobs in the future when bad things do happen consecutively and you need to make a decision about letting somebody go? Do those things that they volunteered count as well?

Al'ai Alvarez (42:07): And I think that's the danger of this. And so, creating a safe space truly requires you to be open about anything. And the pilots have figured this out. They're brutal about talking about areas that are not doing so well. And yet, they're all super protected. It's not like they're doctors and they're going to get sued and they have to deal with all of these ramifications and credentialing and the money and the litigation behind it. We don't have that in medicine. And so, really, right now, it's a lot of these grassroots projects that we're doing. Individually, each of the different departments across the country and somewhere somehow I think we need to go past that.

For me, the metrics for success from the perspective of medical errors, is when people can volunteer their case and talk about it.

Leadership Traits that Forge Blameless 

Christina Tan (42:48): Yeah. And Kurt has told me about these metrics, even in SRE that essentially measures learning. So, how many people actually read the case reviews or retrospectives? How long do they spend on those? And that's a much better measurement of learning. So, if you wave a magic wand, we can kind of propel leadership to respond to these incidents differently. What would you wish for leaders to do differently for both of you?

Al'ai Alvarez (43:19): I think the humanizing aspect of our work requires us to see ourselves as fallible as well. So as leaders, it's important for us to be vulnerable, it's important for us to recognize that we will make mistakes and be able to model that to people. I think, only then can we feel comfortable about seeing other people's vulnerability. Because right now, if it's only the junior people sharing a lot of these mistakes, then they're not going to be feeling secure enough that there's not going to be retribution.

Al'ai Alvarez (43:52): But if their leaders are also sharing, well, like, "I screwed up, I did this," then you're more likely to say, "Well, if my boss is screwing up and can stand up to everybody and own up the areas that they can improve, then maybe I can as well." And that's for me, I think, is very important, especially in a medical education world, where you have to signal that this is not going to be like you wake up, you become an emergency physician. You wake up and you show up every single day and you get to make a ton of mistakes, and eventually you're going to get here and even when you're here, you're still going to make a lot of mistakes. And that's the lifelong learning that you're going to have to develop.

Kurt Andersen (44:28): I think, for me, the epitome of what would be amazing is leaders who are willing to stand up and take responsibility for mistakes, and at the same time give credit for all the amazing awesome stuff to their teams, being able to take the responsibility for mistakes and share the credit to their teams for the good stuff that goes on engenders psychological safety, engenders creativity, builds teams. And I think that would be my magic wand.

Al'ai Alvarez (45:04): Yeah.

Christina Tan (45:05): Well, thank you so much Kurt and Al'ai. Really appreciate your time today and it was a really wonderful conversation. And thank you to the audience for listening. We'll see you next time.

‍

‍

‍