Videos

In this episode, Liz Fong-Jones, Developer Advocate at Honeycomb.io, and Jean Clermont, SRE at Flatiron, join Matt Davis and Kurt Andersen from the Blameless team to search the dark and scary corners of technical debt. What's "dark debt"? Do you have haunted graveyards? And what's the difference between debt and toil? Register to watch the full conversation!

In this episode, Alyson van Hardenberg, Engineering Manager at Honeycomb.io, and Varun Pal, Staff SRE at Procore, talk to Matt Davis and Jake Englund from the Blameless team. Find out how their engineering orgs pick an incident commander. What type of technical knowledge should this person have about an incident? And does it take a certain type of personality to excel as incident commander? Register to watch the full conversation!

Our first episode covers a juicy topic to kick us off — What’s difficult about on-call? We invited Yvonne Lam, Engineering Manager at Kong, and Charles Cary, CTO at Shoreline.io, to chat with Kurt Andersen and Matt Davis from the Blameless team. Watch the full conversation, honest and unscripted, where the four discuss their personal experiences and learnings about on-call duty.

Communication can make or break the effectiveness of a team responding to an incident. On-call needs to know real-time status updates plus all relevant tasks. Support, customer success and functional owners across the business need assurance that it’s being handled. Blameless’ new powerful CommsFlow™, core to the platform, is designed to operate timely, context-rich and automated communications, from start to completion. The lift becomes much lighter, everyone is updated, and most importantly, engineers have time to focus.

Having good reliability means that incidents are nothing special, merely variations of our regular work. Such a perfect dream of Site Reliability Excellence means that there are clear paths to expertise and common grounding between teams happens frequently. To make this vision a reality, we run an open session at Blameless that builds on musical traditions of improvisation. Inspired by western jazz and Indonesian percussion orchestras, our weekly session seeks to build group intuition through a discipline of iterative collaboration. In this talk I introduce our approach to continuous learning, Practice of Practice Gamelan. What we've created is an opportunity for coming together in a collaborative way without the anxiety of performing under pressure. We share mental models through the telling of stories, playing of games, and riffing of ideas. Different areas of our socio-technical system are explored as seen through different eyes. By learning about how our coworkers view the system we operate together, we continuously build new connections through our newly shared perspectives. We not only learn how our teammates strive towards Reliability Excellence in their daily work, we also reduce unknowns about the system itself, giving us more flexibility to adapt around inevitable ambiguity. Come see how we want incidents to be just another time to get together and jam about some fascinating part of the system that has suddenly revealed itself as a wrong note we can learn by.

On call doesn’t have to suck. Right? Some vendors even have this as a tagline. Truth is some on-calls do suck (a little) especially if it’s a Sev0. Ever been in a chaotic on-call environment? Unclear on who’s doing what and how to run the playbook. Is there a playbook? This session shares approaches that simply work and more importantly how teams learn and improve over time. From living through inevitable incidents and reviewing retrospectives, everyone up-levels and learns. Happier DevOps teams, happier customers. Presented by Paul Chu, Head of Customer Success at Blameless.

A retrospective (or post-mortem) is a critical part of Incident Response. If you don’t document, share, and learn, everyone misses out on a critical learning opportunity. Retrospectives are designed to help teams collectively learn, iterate, and improve. It involves taking the time to story-tell exactly how an incident took place and discussing how to do better next time. The key is to shine new light on processes, tools, and systems so that you know how to fine-tune.

In SRE, a core message is that failure is inevitable. No matter how much you prepare, there will always be incidents you can't foresee. This doesn't mean preparation is useless, though. This talk will focus on one extremely valuable type of preparedness: having backups and restoration processes for the worst disasters. When your system experiences a total outage, an effective option is often to switch to a backup system before trying to solve the issue itself. This will restore service as fast as possible. However, just making backup systems isn't enough. This talk reveals complacency and blind spots when it comes to backup systems. Many organizations feel comforted by having created backups, but aren't actually prepared to use them. There will be practical advice given on how to improve backup systems for organizations of all sizes. The talk will cover looking at backup systems from the perspectives making them more reliable, more robust, and more resilient - based on the definitions given by Dr. David D. Woods. In order to make the advice inclusive, there won't be much technical detail. Instead, the focus will be on mindsets and strategies. Black swan events are highly impactful incidents that are so unlikely or unimaginable that effort isn’t made to prepare for them. You'll learn how to conduct thought experiments of "meteor strikes" and other worst-case scenarios, such as ransomware, to feel ready for other problems you can't yet imagine. You'll also see how backup systems can still be useful for such disasters. This is how a resilient backup system is created - one that can still handle what falls outside your expectations.

Alerts go off. Acknowledge it’s real! Start an Incident. Blameless and PagerDuty work brilliantly together. Blameless has a bi-directional PagerDuty integration to access on-call assignments and escalation policies from within the Blameless product to find and allocate the right engineers on-call. Bring your questions and energy. Excited to share new Blameless updates!