Navigate Incident Management Like a Pro: MyFitnessPal's Sr. Director of Engineering Shares Insider Strategies with Lee Atchison
How much time are engineering teams spending on incidents?
Are you trying to set your engineering team free to do their best work? Read our new case study to learn how Blameless can help you do that.

Changelog

Subscribe to stay up to date
Black X   - Blameless Images
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Recent Posts
Empower Secure Collaboration with Incident-Level Permission Management
April 18, 2024
While anyone at your organization should be able to start Blameless incidents, not everyone should have access to all current and past incidents (nor retrospectives), particularly when those incidents are handling sensitive and confidential information related to security issues. With Blameless you can already restrict the creation of incidents and access to incidents (and retrospectives) on a per incident type basis. Now, you can also allow Blameless users only to access the incidents (and their corresponding retrospectives) that they create, while preventing access to all other incidents of the same incident type. Additionally, at any time during or after the incident, if deemed necessary for security or confidentiality reasons, users who created the incident or joined the incident, can have their permissions to access the incident easily revoked. Meanwhile, users joining incidents without any pre-assigned Blameless identity roles, can be granted more permissions (Reader, Writer) to access and interact with the Blameless incident (and retrospective).

While anyone at your organization should be able to start Blameless incidents, not everyone should have access to all current and past incidents (nor retrospectives), particularly when those incidents are handling sensitive and confidential information related to security issues. With Blameless you can already restrict the creation of incidents and access to incidents (and retrospectives) on a per incident type basis. Now, you can also allow Blameless users only to access the incidents (and their corresponding retrospectives) that they create, while preventing access to all other incidents of the same incident type. Additionally, at any time during or after the incident, if deemed necessary for security or confidentiality reasons, users who created the incident or joined the incident, can have their permissions to access the incident easily revoked. Meanwhile, users joining incidents without any pre-assigned Blameless identity roles, can be granted more permissions (Reader, Writer) to access and interact with the Blameless incident (and retrospective).

Assign the new IncidentCreator role to users

Restricting incident creators to create and access only incidents they created in Blameless is pretty easy. Under Identity Management in Blameless, a Blameless administrator with the IdentityAdmin permission, just needs to assign the new IncidentRole to users via a group, which is automatically created by default per incident type, along with the IncidentReader, IncidentWriter and IncidentAdmin roles (see incident roles).

Users who have been granted the IncidentCreator role for a specific incident type are automatically invited to the incident they created and are granted permissions to execute a limited set of commands corresponding to the IncidentReader role (see incident roles), but only for incidents they created. Additionally, they can only access the incidents and retrospectives they have created via the Blameless incident. Other incidents (and retrospectives) will not be accessible to users who have only been granted the IncidentCreator role.

Additionally, users who have been granted the IncidentCreator role gain Reader permissions to the incident they created, which provides access to the same Blameless commands that the IncidentReader has access to today (see incident roles).

Note that you should make sure that users with the IncidentCreator role, are also not assigned one of the other default incident roles such as the IncidentReader, IncidentWriter or IncidentAdmin for a selected incident type, otherwise it would override the restricted permissions set by the IncidentCreator role, still granting them access to all incidents of the same type.

Updating permissions during the incident

Conditions may change after incidents have started, so Blameless provides the flexibility to grant more Blameless permissions to users as they join incidents or to revoke permissions to access the Blameless incident (and retrospective) if the incident has to be handled and accessed only by a reduced number of authorized users.

You can view and change user permissions (grant/revoke) at the incident level only (local roles) without leaving the incident channel in Slack using the new /blameless set permission command or from the incident detailed page in the Blameless web UI. For security reasons, note that only users with the IncidentWriter role for this incident type have permission to change the local user permissions. 

View permissions

You can view the user permissions (local and global roles) for any users joining the incident, including the creator of the incident and anyone who joins the incident.

  • Users joining an incident may have permissions set either at the incident-level only (local) or at the global level per incident type (or all incident types)
  • Note that if a user has a role set at both the global and local level, the permissions at the global level prevail.

Changing permissions

You can change permissions for users with local roles only. Permissions inherited at the global level (per incident type) can only be updated under Identity Management in Blameless.

Call to action

To benefit from this new role-based access control (RBAC) enhancement (permission per incident), contact your Blameless administrator responsible for managing user access in Blameless under the Identity Management section, to start granting users with the permission to access only the incident (and retrospective) they create. Check the instructions on how to configure and change permission per incident here: https://docs.blameless.com/hc/en-us/articles/22773768650007-Permissions-per-incident 

Integrate Blameless with your Backstage service catalog
April 4, 2024
By integrating Blameless with your Backstage catalog, your incident response teams can now more accurately tie incidents to the exact affected parts of your production environment (e.g. services, components, APIs, resources, etc), providing critical context to drive the investigation more rapidly in the right direction and fast access to a wealth of information stored in your Backstage catalog. Then post-incidents, using Reliability Insights in Blameless, each team responsible for those services or components can draw some new learnings by more effectively identifying the most vulnerable parts of their production environment based on the severity and number of incidents, among many other dimensions.

By integrating Blameless with your Backstage catalog, your incident response teams can now more accurately tie incidents to the exact affected parts of your production environment (e.g. services, components, APIs, resources, etc), providing critical context to drive the investigation more rapidly in the right direction and fast access to a wealth of information stored in your Backstage catalog. Then post-incidents, using Reliability Insights in Blameless, each team responsible for those services or components can draw some new learnings by more effectively identifying the most vulnerable parts of their production environment based on the severity and number of incidents, among many other dimensions.

Mapping Backstage resources to incidents

Blameless provides the flexibility to specify one or more entities as defined in Backstage, immediately at the start of an incident from the Blameless web UI or when using the Blameless app directly from within Slack or Microsoft Teams.

Blameless purposefully follows Backstage's naming convention for entities, which uniquely identifies them and makes it easier to search for the kind of entity you are looking for and the namespace the entity belongs to. This is particularly useful when the same entity name is used across multiple namespaces: <kind>:<namespace>/<name>

As the list of Backstage entities could be very long, you can quickly find the entities you want to map to your new incident with a type-ahead search approach in Slack and Microsoft Teams. For additional convenience, if you start incidents from the Blameless web UI, the list of Backstage services can be pre-filtered with the “Backstage entity type” dropdown.

Slack Incident Creation (Left), Microsoft Teams Incident Creation (Middle), Blameless Web UI (Right)

Impacted services linked to Backstage

After you start the incident, all selected Backstage services (entities) are displayed in key areas of the Blameless application, readily available to incident responders during incidents and post incident resolution. Each impacted service is hyperlinked to its description page on your Backstage web server, providing incident responders with direct access to a wealth of information about the affected services (owners, dependencies, release dates, software repositories, etc).

Slack Incident Channel Backstage Services (Left), Blameless Web UI Backstage Services (Right)

Tracking Blameless incidents from within Backstage

As part of the integration with your Backstage server, you must install the Blameless plugin for Backstage on your Backstage server (see installation instructions) which provides the following capabilities:

  • How frequently you want to synchronize Blameless with your Backstage catalog
  • List of Blameless incidents

Without leaving Backstage, you can explore the history of incidents that have affected your Backstage services (entities), helping draw attention from your engineering team towards the vulnerability level of different parts of their production environment.

Reporting on incident statistics related to Backstage services 

Ultimately, Blameless Reliability Insights provides all the key insights your leadership needs to assess the overall reliability and vulnerability of your production environment, projecting incident statistics (e.g. mean time to resolution) across multiple dimensions (e.g. severity, incident type, Backstage service and entity kind, incident tags, timeframes, etc) leveraging a large selection of chart types (pie, bars, lines, etc) to produce custom and powerful dashboards.

Getting Started

Integrating Blameless with your Backstage server is very easy. Contact your Blameless administrator and Backstage administrator and have them follow those installation instructions: https://docs.blameless.com/hc/en-us/articles/22151438605719-Backstage

Enhanced Blameless App for Microsoft Teams
March 21, 2024
Blameless is excited to share how we’ve transformed the user experience of incident responders using the Blameless app in Microsoft Teams. By enabling the use of group chats to run incidents, we’ve streamlined the incident management workflow in Microsoft Teams. This also allows responders to reserve the thread in their team channel for the core investigation. Users can still start incidents from any channel in Microsoft Teams and access all the resources of Blameless wherever they choose to do their work. After two years on the Microsoft AppSource store, Blameless remains the leading choice for incident management for organizations relying on Microsoft Teams. These changes have all been built in collaboration with Microsoft Teams users, and we are excited to announce their availability in the form of a more intuitive Blameless experience.

Blameless is excited to share how we’ve transformed the user experience of incident responders using the Blameless app in Microsoft Teams. By enabling the use of group chats to run incidents, we’ve streamlined the incident management workflow in Microsoft Teams. This also allows responders to reserve the thread in their team channel for the core investigation. Users can still start incidents from any channel in Microsoft Teams and access all the resources of Blameless wherever they choose to do their work. After two years on the Microsoft AppSource store, Blameless remains the leading choice for incident management for organizations relying on Microsoft Teams. These changes have all been built in collaboration with Microsoft Teams users, and we are excited to announce their availability in the form of a more intuitive Blameless experience.

With Blameless, Microsoft Teams users benefit from a broad collection of customizable automation, from automatically recruiting on-call engineers to the incident channel to automatically updating internal stakeholders on incident progress. Moreover, with Blameless, your reliability journey does not stop when the incident is resolved. All incidents run from within your Microsoft Teams workspace, producing a wealth of knowledge that you can leverage to produce higher-quality retrospectives and analyze, painting your overall Reliability map directly in Blameless.

Whether you have been using the Blameless app for Microsoft Teams over the past two years or you are new to Blameless, you will immediately notice the following new capabilities but also the power of the capabilities we kept intact in this updated version of the app:

🚀 What’s new

  • Run incidents in dedicated group chats instead of team channels
  • Start incidents from team channels (and group chats) where the Blameless app is installed
  • Customize the incident group chat name with incident title, severity, status and more
  • Faster and easier access to most commonly used Blameless actions, through the @blameless suggestions and action buttons in the welcome message.
  • More verbose responses when triggering alerts via PagerDuty and OpsGenie 
  • Centralized tag management in the web UI
  • Easier incident type configuration in the web UI
Action buttons in welcome message
Suggested Blameless commands

🛠️ What’s still there

The @blameless command to display and execute selected Blameless commands

  • Invite users, Assign roles, Manage tags, Trigger Alerts, and more
  • The powerful incident Overview tab
    • Quickly check and update the incident title, description, status, severity and type. It also provides visibility into the incident timeline, your team (automatically matching users in your incident group chat)
    • Access to a few key resources (Jira and/or ServiceNow incident tickets and link to the dedicated video conference such as Microsoft Teams, Zoom, etc)
  • Auto-inviting users paged via PagerDuty or OpsGenie
  • Single sign-on with Blameless instance and your Microsoft Active Directory authentication service

For more information about the Blameless app in Microsoft Teams:

Call to action

To benefit from those enhanced capabilities, existing customers who have previously been using the Blameless app in Microsoft Teams, contact your Blameless administrator to re-authenticate your Blameless instance with your Microsoft Teams workspace using the same credentials you previously used.

If you are integrating Blameless with Microsoft Teams for the first time, just follow these instructions provided here.

Enhancing your Experience: Latest Product Updates
March 7, 2024
The team here is hard at work expanding Blameless, adding new, powerful features and integrations to help transform how engineering teams approach incident management. But every now and then, we take a step back from all that innovation to focus on the little details. Here, we’re dotting the i’s and crossing the t’s to make sure you have a fantastic experience using Blameless every day. So here’s an update coming out of our recent bug smash where we tackled the backlog of feature enhancements you’ve been asking for. Usability improvements, UX tweaks, feature upgrades, we’ve got it all here for you. Enjoy!

The team here is hard at work expanding Blameless, adding new, powerful features and integrations to help transform how engineering teams approach incident management. But every now and then, we take a step back from all that innovation to focus on the little details. Here, we’re dotting the i’s and crossing the t’s to make sure you have a fantastic experience using Blameless every day. So here’s an update coming out of our recent bug smash where we tackled the backlog of feature enhancements you’ve been asking for. Usability improvements, UX tweaks, feature upgrades, we’ve got it all here for you. Enjoy!

🛠️Improved Task Creation in Blameless Create Task Modal in the UI

In our latest update, we've streamlined this task-creation process by removing the option to select a status when creating a new task. This helps users focus on task creation by eliminating the redundancy of having to select a “status” for the task when one can already be inferred by Blameless. This optimizes workflow efficiency and ensures that tasks are consistently assigned with the appropriate status, ultimately fostering improved team collaboration and overall productivity. 

🔒 Enhanced Reliability Insights: DashboardReader Permissions

We’ve made some interesting enhancements to Reliability Insights that make it easier for your teammates to access the custom dashboards you create. We’ve given members of your team with the “DashboardReader” role read access to custom dashboards. This allows DashboardReader role holders to seamlessly access and glean insights from custom dashboards, empowering them to make informed decisions and contribute meaningfully to their teams' objectives. This change has the power to amplify the utility and relevance of all the dashboards you create while ensuring that every user on your team has the tools they need to thrive in their role. 

🔍Improved Timeline Display: Seamless Time Zone Synchronization!

We've addressed a common issue affecting timeline clarity and efficiency. With our latest timeline enhancement, all events are accurately displayed according to the chosen user's time zone. This eliminates any ambiguity around the timeline of critical events and makes collaborating during the post-incident retrospective process even easier.

🚀 Exciting Update: Enhanced Incident Type Auto Invite Functionality!

We're thrilled to unveil a powerful enhancement to our Incident Type Auto Invite Users feature. Previously, users encountered difficulty distinguishing between Slack users with identical display names when selecting auto-invites for incident channels. We've improved this process by adding an additional field for user distinction. Now, you can effortlessly identify and invite the right team members, ensuring swift and effective incident response every time. This enhancement not only streamlines collaboration but also reinforces our commitment to providing intuitive solutions tailored to your needs. 

🌟 New Feature Alert: Enhanced Retrospective Reminders!

We've made some big updates to our Retrospective tool by enhancing the Reminders feature. We’ve given users the power to exclude weekends and US Holidays from their retrospective reminders, ensuring that teams only get notified when they are working. This is a really important step forward in helping teams avoid engineering burnout by eliminating unwanted notifications to your team on their days off. 

Track PagerDuty actions in Blameless incident timeline
February 22, 2024
Have you wondered how long it takes your team to complete critical tasks like acknowledging alerts or escalating an incident to the next level in PagerDuty? Making the team more efficient at completing these types of tasks can have a big impact on their overall effectiveness. These types of data points are great to examine during the retrospective process, but only if you capture them in the first place. Blameless now captures all those PagerDuty actions in your incident timelines as “critical events”. Each one clearly displayed, easily accessible, and ready to be discussed at your next retrospective meeting.

Have you wondered how long it takes your team to complete critical tasks like acknowledging alerts or escalating an incident to the next level in PagerDuty? Making the team more efficient at completing these types of tasks can have a big impact on their overall effectiveness. These types of data points are great to examine during the retrospective process, but only if you capture them in the first place. Blameless now captures all those PagerDuty actions in your incident timelines as “critical events”. Each one clearly displayed, easily accessible, and ready to be discussed at your next retrospective meeting.

Notifications, acknowledgments, transitions to other teammates, manual or triggered escalations, priority changes, manual resolutions, all of these event types can now become part of your analysis of each incident. These new dimensions can help you get right to the heart of your team's standard incident response performance.

PagerDuty events in the Incident timeline

Out of the box, Blameless automatically captures the following critical PagerDuty actions in the incident event timelines:

  • Triggered PagerDuty alerts
  • Acknowledged PagerDuty incidents (related
  • Unacknowledged PagerDuty incidents
  • Escalated incidents
  • Changed incident priorities
  • Changed incident status (resolved)

Each record captured in the Blameless incident timeline includes a link back to the related PagerDuty alert or incidents, including the name of the responder who triggered the action in PagerDuty, so that you can easily correlate who did what when.

Then, when doing retrospectives, the incident timeline is readily available to users collaborating on crafting their retrospective and blamelessly assessing what could have been done differently to resolve incidents faster.

Reporting PagerDuty event statistics using Reliability Insights

With all these new PagerDuty events captured in Blameless, team leaders can produce powerful dashboards and reports to extract critical insights and trends using reliability insights. These can be shared with key stakeholders to identify more opportunities for improving the performance of the incident response team.

For example, reporting on the time from event triggering to acknowledgment in PagerDuty offers valuable insights into operational efficiency, but also SLA compliance, team performance, root cause analysis, real-time visibility, and continuous improvement, ultimately contributing to more effective incident management and better organizational resilience.

As you build new tiles (tables or graphs) in your dashboards in Reliability Insights, you can view and filter such PagerDuty events matching the following event types by filtering those starting with the word PAGER:

  • PAGER_TRIGGERED
  • PAGER_INCIDENT_RESPONDER_ADDED
  • PAGER_INCIDENT_PRIORITY_CHANGED
  • PAGER_INCIDENT_ACKNOWLEDGED
  • PAGER_INCIDENT_RESOLVED
  • PAGER_INCIDENT_ESCALATED

Call to action

To benefit from those new enhancements, immediately contact your Blameless administrator to disable and re-enable the integration of your Blameless instance with your PagerDuty account.

For brand new integrations with PagerDuty, Blameless will automatically place the new Blameless webhook in PagerDuty for Blameless to automatically capture those PagerDuty actions in the incident timeline.

Incident Assistant: Automatically generating incident summaries that are easy, instant, and up-to-date.
February 6, 2024
Every incident responder has the same problem when they first join an incident. They need to get up to speed “yesterday” so they can figure out how to help the team move forward. Trying to read through, and glean the context they need, from a chaotic Slack thread is not an efficient use of time and energy. Relying on your incident commander to onboard each participant to the incident is just as slow and unreliable. Don’t worry, Blameless Incident Assistant is here to solve this problem for you. Incident Assistant generates a comprehensive incident summary for every member of your team. Instantly available on-demand at any time during or even after the incident has been resolved.

Every incident responder has the same problem when they first join an incident. They need to get up to speed “yesterday” so they can figure out how to help the team move forward. Trying to read through, and glean the context they need, from a chaotic Slack thread is not an efficient use of time and energy. Relying on your incident commander to onboard each participant to the incident is just as slow and unreliable. Don’t worry, the Blameless Incident Assistant™ is here to solve this problem for you. Incident Assistant generates a comprehensive incident summary for every member of your team. Instantly available on-demand at any time during or even after the incident has been resolved.

Whether you are part of the incident from the very beginning, or you’re pulled in later to help out, the Blameless Incident Assistant™ Will hand you every piece of critical information you need to contribute from the moment you enter the incident channel. The incident summary is comprehensive and focused on what you need most: A short and long description of the issue, its actual impact, the list of impacted capabilities, and how the investigation has progressed so far to mitigate the issue.

Powered by OpenAI, the Blameless Instant Assistant™ is available now in Preview mode, ready to assist anyone involved directly or indirectly with the incident when they need it most!

Blameless Incident Assistant™ does more than just get your responders up to speed, it can keep your team on the same page from declaration to resolution. Your team can easily prompt Blameless Incident Assistant™ at any point throughout the process to provide a succinct, comprehensive assessment of what is known so far about the incident. 

Invoke the incident summary at any time

It just takes a few messages in Slack at the beginning of the incident to generate a succinct and useful incident summary. Anyone who joins the incident channel in Slack has multiple ways to access the incident summary whenever they need it.

First, whenever incident responders join an incident, they are provided the option to view the latest AI-generated incident summary with a View Summary button presented in the main incident welcome message at the beginning of the incident and also in the personalized welcome message presented to users each time they check-in to the incident channel in Slack.

Option to view the latest AI-generated incident summary with a View Summary button presented in the main incident welcome message at the beginning of the incident
Personalized welcome message presented to users each time they check-in to the incident channel in Slack

Additionally, during the incident, users can use the following easy-to-remember Blameless command to display the incident summary at any time during the incident.

/blameless view summary

Note that users must have the IncidentReader or IncidentWriter roles per incident type to be able to execute this Blameless command. 

Understanding your incident summaries

Because the incident summary is continuously being updated with the latest messages captured in the incident channel, you can access the summary in a pop-up window that also be minimized or closed at any time using either the Blameless command or clicking on the View Summary button.

As you can see, the incident summary contains the following succinct but powerful, easy to read in a blink of an eye overview of the incident and what the team has done so far, etc:

  • Issue: one sentence description of the issue
  • Detailed: short summary of what is known so far about the issue,
  • Impact: how this issue is impacting your customers and/or users, 
  • Impacted capabilities: list of capabilities affected by the incident
  • Mitigation: what has been accomplished so far

Integrate with your own OpenAI account

Out of the box, by default, the integration with OpenAI is disabled in Blameless, giving you the option to enable it, when you are ready, using your own OpenAI account as followed:

After you log in to your OpenAI account, go to https://platform.openai.com/api-keys generate and copy/save a new OpenAI API key

Then log in to your Blameless instance, and go to the OpenAI integration settings (Settings > Integrations > Generative AI) to enter your new key and enable the integration.

Use GPT-4 for your ChatGPT Model while using Incident Assistant

Call to action

Ask your Blameless administrator to integrate your Blameless instance with your OpenAI account or use Blameless OpenAI account as described above. If you are using Blameless with Microsoft Teams, expect to have access to the Incident Assistant in the very near future.

Better ways to recruit team members to Blameless incidents using PagerDuty
January 29, 2024
Finding the right subject matter experts to join an investigation is something you wish happened yesterday. Failing that, you’ll probably settle for “As soon as humanely possible”. Fast recruitment of the right experts when an incident kicks off can be a difficult task, but it can all be the difference between quick resolution and a problem spiraling out of control.

Finding the right subject matter experts to join an investigation is something you wish happened yesterday. Failing that, you’ll probably settle for “As soon as humanely possible”. Fast recruitment of the right experts when an incident kicks off can be a difficult task, but it can all be the difference between quick resolution and a problem spiraling out of control. 

That moment when an incident starts is really stressful, so we’ve made finding the right people to help you as easy as possible. We’ve expanded our integration with PagerDuty and given you new ways to bring experts into your incident sooner, faster, and at any time during an incident. 

We know recruiting styles can vary greatly from one organization to the next so we’ve made sure we can support them all. Whether you’re using PagerDuty as a true service catalog, as a series of escalation policies, or just as a way to page the same person for different incident types, we’ve got you covered. 

With Blameless, incident responders can trigger PagerDuty alerts multiple times, at any point during the incident. NOW, those responders can select one or more PagerDuty escalation policies or PagerDuty responders explicitly, directly from their incident channels in Slack. This is perfect for scenarios where incident responders may not initially know which specific services or teams are affected, but they know which people might be able to help. This gives your team significantly more flexibility to choose the best way to recruit experts throughout the duration of an incident.

Do more with the same Slack command

Incident responders don’t need to learn anything new to take advantage of those new capabilities. Simply continue to use the existing Blameless command to trigger PagerDuty alerts at any time during the incident from within their Slack incident channels: /blameless trigger alert

Note that Blameless users working in Slack, need to be granted the IncidentWriter Blameless role for each specific incident type to be able to execute this command. For more information on Blameless RBAC functionality go here).

More ways to escalate with an improved user experience

When this command is executed, Blameless presents incident responders with a new Slack pop-up window where they can decide the best way for them to recruit the right responders, either by selecting a service, one or more escalation policies or one or more responders.

However, if escalating by directly selecting an escalation policy and/or a responder is not the way you want your incident responders to operate, you can decide to hide one or both new options to enforce your preferred method to recruit specialists to your Blameless incidents. Simply have your Blameless administrator, enable or disable those options in the PagerDuty integration configuration settings:

Already built-in automation

If you are new to Blameless, you should also know that Blameless automatically invites all Pagerduty users to your Slack incident channel whether they have been notified by selecting a service, an escalation policy or directly by their username. However, to be invited to a Slack channel, the notified PagerDuty user has to have a user identity in Slack matching the same email address.  

Additionally, if you have a simpler escalation model for incident response operations, while keeping your service catalog, team members and escalation policies up to date in PagerDuty, Blameless can make your life easier. By simply mapping Pagerduty services to Blameless incident types, you are telling Blameless to automatically recruit incident responders which are currently responsible for the services when starting an incident, without having to manually trigger an alert from your Slack incident channel. And, if you still need to recruit additional specialists to your incident, you can then manually trigger alerts using the above Blameless command at any time. 

Call to action

Ask your Blameless administrator to enable one or both options in Blameless under the PagerDuty integration setting page at your earlier convenience and let us know how this is helping you recruit team members faster and more easily.

Customize the Comms Assistant™ to your communication style using your own OpenAI™ account
January 11, 2024
Blameless Comms Assistant™, powered by OpenAI™, makes sending effective incident updates to various audiences, easier than ever. Comms Assistant can now be integrated with your own OpenAI account. So, if you have policies governing the use of Generative AI products and have your own OpenAI’s ChatGPT Enterprise subscription, you can leverage your OpenAI account to keep your incident data private. Blameless lets you integrate with either the ChatGPT engine version 3.5 or 4.0, so you can choose which generation of model works the best for your team.

Blameless Comms Assistant™, powered by OpenAI™, makes sending effective incident updates to various audiences, easier than ever. Comms Assistant can now be integrated with your own OpenAI account. So, if you have policies governing the use of Generative AI products and have your own OpenAI’s ChatGPT Enterprise subscription, you can leverage your OpenAI account to keep your incident data private. Blameless lets you integrate with either the ChatGPT engine version 3.5 or 4.0, so you can choose which generation of model works the best for your team.

Frequently crafting incident updates that share the perfect amount of information, and strike the right tone can be incredibly time-consuming. Keeping all your stakeholders assured and the right slack channels updated requires more forethought and manual customization than you can afford in a crisis. So, in addition to its powerful default prompts, Blameless now gives you full control over the Generative AI prompts in Comms Assistant.  This allows you to build a fully tailored communication strategy by audience for each incident update. Then let the system take over so you can get back to putting out the fire.

Integrate with your own OpenAI account

Out of the box, by default, the integration with OpenAI is disabled in Blameless, giving you the option to enable it, when you are ready, using your own OpenAI account as followed:

  1. After you log in to your OpenAI account, go to https://platform.openai.com/api-keys generate and copy/save a new OpenAI API key
  2. Then log in to your Blameless instance, and go to the OpenAI integration settings (Settings > Integrations > Generative AI) to enter your new key and enable the integration.

Customize the Comms Assistant

New Comms Assistant Settings

The Comms Assistant comes with its own setting page which can be found in the Settings section of your Blameless instance under the new Communication section (go to Settings > Communication > Comms Assistant).

Customize the audience list and design your own OpenAI prompts

Out of the box, Comms Assistant, gives you three default audiences with prompts carefully engineered by Blameless to effectively tailor your incident updates to either an Executive, Sales, or Customer Support team.

However, if you want more control over how incident updates should be generated, Blameless puts Generative AI at your fingertips. With the option to add new, additional audiences, each with their own custom prompt. You can experiment with these custom prompts directly in Blameless before ever engaging Comms Assistant in Slack.

Feature summary
  1. Activate/deactivate audiences - hides/shows the audience in the `send comms` modal in Slack
  2. Includes 3 default audiences out of the box, with default prompt engineered by Blameless, with the flexibility to edit the audience display name as well as the list of default Slack channels to be added to the `send comms` modal in Slack.
  3. Add, test, and activate new audiences with custom prompts, engineered by you, with the option to customize the list of default Slack channels, by duplicating any existing audiences with its current prompt (including the default audiences).

Test your prompt directly in the Comms Assistant settings

Using the Comms Assistant in Slack

Once you have configured the Comms Assistant to your custom needs and enabled the integration with your OpenAI account, incident responders can now more easily and more rapidly than before craft very effective incident updates directly from Slack, using the following Blameless send comms command at any point in time during the incident: /blameless send comms

Sending reminders using CommsFlow

Additionally, do not forget to create reminders in CommsFlow to remind your communication leaders (or any role name you choose) to send those ad-hoc incident updates at specific transitions of the incident of your choosing.

Getting Started!

Export Retrospectives to Google Docs with more flexibility and security
December 8, 2023
Blameless retrospectives can be automatically exported to Google Docs, and with this new update, can NOW be stored flexibly in any location across your folder system. This makes it easy to provide quick access to retrospective documentation for a specific team or your entire organization. This lets you quickly disseminate learning while leveraging Google access permissions to control who can engage your retrospective docs.

Incident response and retrospective is a “multi-team” effort! For engineering organizations to embrace the principles of incident learning at scale, certain conditions must be met:

  • Teams must be able to work in tandem without impeding each other's progress.
  • Resources and learning created along the way must be reasonably organized or structured in order to be valuable.
  • Every team member must have the capability to access and share this new wealth of knowledge, built through the production of high-quality retrospectives.

Sounds like a lot, but we’ve made it simple.

Blameless retrospectives can be automatically exported to Google Docs, and with this new update, can NOW be stored flexibly in any location across your folder system. This makes it easy to provide quick access to retrospective documentation for a specific team or your entire organization. This lets you quickly disseminate learning while leveraging Google access permissions to control who can engage your retrospective docs. 

Additionally, the integration between Blameless and your Google Drive is now more secure. With the help of a Google user or service account, you have complete control over granting Blameless access to your Google Drive. 

Use Cases

Think about how you organize your Blameless Incident Types today. Maybe you organize Incident Types by Internal Teams, Products you support, or Use Cases, such as Customer Incidents, Security Incidents, and Planned Outages. When you start a Blameless incident, the Incident Type helps you categorize incidents and controls who has permission to read, write, or access the incident at all. 

With the new Google Docs Export Folders enhancement, you can predefine where a Blameless retrospective will be stored within your Google Drive. This lets you control permissions internally regarding who can see your post-incident documents. Security Incidents might have different users in Google Drive that can view your Retrospectives than Customer Incidents, for example. The moral of the story is that Blameless’ Google Docs Export capabilities are flexible to fit your needs!

Key Enhancements

  • Integrate Blameless with a Google Drive using a dedicated Google service user account
  • Export all retrospectives as Google Docs to a default folder
  • Configure multiple folders where you want to route and store your Retrospective as Google Docs based on selected incident types
  • New Export settings page under Retrospective settings to enable the automated export, and configure your default folder and custom folders.
  • Using Google Drive native folder picker modal

Call to Action

Export Enterprise Reliability reports to keep your teammates in the loop
December 1, 2023
To ensure the right Reliability dashboards make it to intended audiences for ad-hoc or end-of-month reporting, the owner of any given dashboard can export it directly as a PDF file by clicking on the 3 dots at the top right of the dashboard.

High-impact IT Operations groups are always on the lookout for opportunities to improve their product and process. Blameless Reliability Insights supports these teams by delivering key insights about infrastructure and team performance. Visualizing trends in data drawn from across your monitoring, observability, on-call management, ticketing, and incident management tooling. But in order for insights like these to effectively guide your engineering investments, they also need to be easily accessible and digestible for your business and technical stakeholders. Reliability Insights dashboards can now be easily exported as a PDF to share with colleagues via Slack, Microsoft Teams, email, or your own corporate portals like Confluence. This takes the pressure off your stakeholders to locate dashboards in Blameless and makes it easy for IT Operations leaders to keep their stakeholders up to speed.

Exporting dashboards on demand

Dashboards in Reliability Insights are highly customizable reports that can be used to examine incident and retrospective data against things like your team structure or service architecture. Each dashboard contains charts (tiles) and tables that can be customized to specific audiences like an individual engineering team, your executives, customer support representatives, etc.

To ensure the right Reliability dashboards make it to their intended audience for ad-hoc or end-of-month reporting, the owner of any given dashboard can export it directly as a PDF file by clicking on the 3 dots at the top right of the dashboard.

The generated PDF is a multi-page, well-formatted document, automatically downloaded to the folder your web browser uses to store any downloaded files. You can now choose to attach this PDF file to an email, Slack message, etc or also post it on one of your internal web portals. 

Go try it yourself now, and see how your dashboard charts and tables are automatically and properly organized in the rendered PDF document, in the order they appear in your dashboard (left to right and top to bottom), with up to two charts per row, and with tables displayed after the charts, allowing all table rows to be printed. All pages are numbered, with the name of the dashboard as the title of the document and the date the document was generated, at the top of the first page.

Getting Started Video

Capturing follow up actions in Blameless is easier than ever!
Anything you learn from past incidents and through retrospectives eventually results in some form of action that your organization wants to take to prevent similar incidents from recurring, whether it is about improving processes, software stacks, or professional skills, just to name a few. To make it easier to capture and assign such action items via the Blameless web UI, we have streamlined the process of creating follow up actions and linking them to the appropriate external ticketing system of your choice.

Anything you learn from past incidents and through retrospectives eventually results in some form of action that your organization wants to take to prevent similar incidents from recurring, whether it is about improving processes, software stacks, or professional skills, just to name a few. To make it easier to capture and assign such action items via the Blameless web UI, we have streamlined the process of creating follow up actions and linking them to the appropriate external ticketing system of your choice.

When creating a follow up action via the Blameless web UI, you now have the following options, which can be used concurrently for as many follow up actions as you need to create per incident:

  • Created in Blameless, with the option to link the follow up action to an existing ticket from any external third party ticketing systems (e.g. Github issue, Linear, Shortcut, etc)
  • Created in Blameless and integrated with Jira in the following 2 ways:
  • Linking it to an existing Jira ticket, automatically importing the Jira ticket summary as the title of the follow up action.
  • Linked to a Jira ticket created by Blameless, mapping data from the Blameless incident to required Jira ticket fields, which is customizable per incident type.

To benefit from the option to integrate your follow up actions with Jira, you first need to integrate your Blameless instance with your Jira account, otherwise this option will remain hidden to not confuse users.

Follow up actions are now created using the exact same workflow from a single “Add Action” button, whether from the Follow Up Actions tab under the incident detailed page or from the corresponding retrospective detailed page under the Overview tab. Both provide access to the same list of follow up actions, whether linked to an external ticket or to a Jira ticket or without linking it.

Starting now, we are inviting you to create your next follow up action via the Blameless web UI, and contact your Blameless administrator to enable the integration of your Blameless instance with Jira if not already integrated. 

Next, stay tuned for more great news about which next ticketing system Blameless is going to integrate follow up actions with very soon!

Getting Started Video

Generate Swift and Powerful Incident Updates using Comms Assistant™
October 31, 2023
Communicating effectively with a broad set of stakeholders is a key characteristic of good Incident response. Everyone who has a stake in recovering from a high severity issue wants to be reasonably informed of status or progress towards resolution. However, crafting and delivering regular updates to satisfy these folks takes time and focus. Two things that incident responders often can’t afford in the midst of a high severity incident. To empower communication leaders and incident responders in providing fast, frequent and meaningful incident updates to various audiences (executive, sales, support), Blameless is introducing the first generative AI-powered Automated Incident Communications With Comms Assistant™, available now in preview mode.

Communicating effectively with a broad set of stakeholders is a key characteristic of good Incident response. Everyone who has a stake in recovering from a high severity issue wants to be reasonably informed of status or progress towards resolution. However, crafting and delivering regular updates to satisfy these folks takes time and focus. Two things that incident responders often can’t afford in the midst of a high severity incident.

To empower communication leaders and incident responders in providing fast, frequent and meaningful incident updates to various audiences (executive, sales, support), Blameless is introducing the first generative AI-powered Automated Incident Communications With Comms Assistant™, available now in preview mode.

From the moment an incident starts, Comms Assistant™ makes it simple to quickly tailor updates for your respective audiences:

  • Executive leadership looking for progress updates, business impact or tactical response plans.
  • Sales or customer success interested in resolution status and looking for messaging for their customers.
  • Support teams seeking technical details.
  • Communications teams assisting with market notification.
  • Customers looking for updates on resolution or when they can resume usage.

Each can receive their own specially tailored, and fully automated updates, thanks to the combination of Comms Assistant™ and CommsFlow™.

To access Comms Assistant™ through Slack, incident responders and communication leaders can leverage Blameless’ manual communication command. This is a fast and easy way to send incident updates to multiple users and audience on an ad-hoc basis via both email and Slack:

  1. First, you will provide a simple draft of your message. 
  2. Then, below the message field, select one of the three default audiences provided. 
  3. Comms Assistant™ will automatically generate a message tailored to the audience type you’ve selected, which you can further customize manually by editing it directly in the message field.

If you want to modify your original message and try again, the dropdown button gives you the option to restore your original message in the message field.

Before submitting your AI generated incident update, make sure to specify the Slack channels, users and/or emails you would like to send this higher quality message tailored to your targeted audience.

Reminder to send an update

Additionally, out of the box, don’t forget that Blameless provides a very convenient way to automatically send reminders, which can be used here to remind incident responders or even just the communication leader to submit incident updates at frequent intervals.  In fact, the higher the severity of the incident is, the more frequent you might be asked to send an incident update, potentially down to every 15 minutes, depending on your communication strategy. So, go now to set up your reminders in CommsFlow™!!

Getting Started

  • Contact your Customer Success Manager at Blameless to request access to a preview of this new feature.

Configure Blameless Settings via Terraform!
October 26, 2023
Whether you’re a Site Reliability Engineer (SRE) or a DevOps Engineer, Terraform is probably one of your favorite tools. Enterprise Reliability Management (ERM) best practices rely on incident response and retrospective workflows, that conform to Standards of Operations (SoPs) and change management control. Engineering organizations seeking a high level governance and control can now download and use the Blameless Terraform Provider to enforce critical configuration guardrails for incident workflow management in Blameless.

Whether you're a Site Reliability Engineer (SRE) or a DevOps Engineer, Terraform is probably one of your favorite tools. Enterprise Reliability Management (ERM) best practices rely on incident response and retrospective workflows, that conform to Standards of Operations (SoPs) and change management control. Engineering organizations seeking a high level governance and control can now download and use the Blameless Terraform Provider to enforce critical configuration guardrails for incident workflow management in Blameless.

Blameless Terraform Provider provides the expanded flexibility to programmatically configure and update the following Blameless resources:

  • Blameless organization settings
  • Blameless severities and roles
  • Blameless incident types, including Slack functionalities, task checklists and retrospectives.

Engineering organizations that use Blameless can now leverage the power of Terraform to reliably and securely track configuration changes (version control), track who applied the changes and when (auditing) and automate Blameless configuration changes along with their existing CI/CD workflows.

Managing incident types settings using Terraform

With the Blameless Terraform provider, managing incident types settings is easier than ever. While you can explicitly configure incident type settings on a per severity basis, you can also decide to configure the incident type settings once and apply the same settings to multiple severities. All thanks to the flexibility of the Terraform modular resource definition architecture.

   

Blameless Terraform provider opens the door for engineering operations leaders to do things like:

  • Create a new incident type from an existing template (check Blameless resource templates available on Github)
  • Update a retrospective analysis template across multiple incident types
  • Ensure that incident channels created in Slack are made private for security sensitive incident workflows
  • Temporarily deactivate or reactivate incident types, or deprecate an incident type entirely all by using Terraform.

How to get started

  1. Make sure you are up to date with the latest Terraform version
  2. Download the Blameless Terraform provider from the Terraform registry
  3. Download and customize the Blameless resource templates from our public repository: https://github.com/blamelesshq/terraform-provider-blameless/tree/main/modules

Initialize, plan, apply, and off you go!!

Take action as soon as you join the Incident in Slack
September 20, 2023
Starting incidents should not be stressful for anyone, especially for the incident creator and leaders (aka commanders). Key recurring actions must be taken right at the start, such as recruiting more incident responders, assigning roles to delegate tasks, paging on-call engineers, and quickly checking the typical set of tasks that you and your incident team is expected to complete as soon as possible.

Starting incidents shouldn’t be stressful, especially not for the incident’s creator and leaders (aka commanders). Recruiting more incident responders, assigning roles to delegate tasks, paging on-call engineers, and affirming the set of tasks that the incident team needs to complete are all key actions that need to get underway as quickly as possible.

We want these types of tasks to be obvious and intuitive. So, we’ve streamlined the Blameless welcome sequence in incident channels in Slack, to include default and dynamic buttons to trigger the following most commonly used Blameless actions at the start of an incident:

  • Assign roles
  • Invite Team Members
  • View Tasks
  • Trigger Alerts - only displayed when you have integrate Blameless with Pagerduty or Opsgenie
  • View incident summary - only displayed when the incident summary is excluded from the welcome sequence
  • More actions

Rather than relying solely on inputting /Blameless slash commands, you and your incident responders can rapidly initiate the incident investigation and get engaged as a cohesive and productive team, with just a simple mouse-click. 

Moreover, instead of displaying the task checklist at the start of the incident, it is now accessible on-demand using the new View Tasks button.

These buttons can be triggered throughout the life of the incident and the output of these Blameless commands will be displayed at the bottom of your channel like any other new direct messages.

Optionally exclude the incident summary

Additionally, if your team would rather not display the incident summary right at the top of the incident channel, you can now ask your Blameless administrator to exclude it from any future incident channels, which is customizable as a new setting under the Invite and Notify Users section on a per incident type basis:

Get started now!

Previously created incidents will not be affected by this change. Only new incidents created starting as of today will benefit from this improved and streamlined welcome sequence.

Collaborate on Retrospectives using Google Docs
September 28, 2023
Running high-quality retrospectives is a team effort. Every member of the team collaborates to analyze incident data and converge on the best plan of action to prevent similar incidents from reoccurring. Offering the best retrospective tool in the industry has been one of the pillars of our success at Blameless. Now, all the automated data collection and templated process standardization that Blameless Retrospectives provides can be seamlessly ported into Google Docs. Engineering organizations relying on Google Workspace, can now more efficiently produce, collaborate, and share high-impact Blameless retrospectives by automatically and securely exporting incident and retrospective data into Google Docs.

Running high-quality retrospectives is a team effort. Every member of the team collaborates to analyze incident data and converge on the best plan of action to prevent similar incidents from reoccurring.

Offering the best retrospective tool in the industry has been one of the pillars of our success at Blameless. Now, all the automated data collection and templated process standardization that Blameless Retrospectives provides can be seamlessly ported into Google Docs. Engineering organizations relying on Google Workspace, can now more efficiently produce, collaborate, and share high-impact Blameless retrospectives by automatically and securely exporting incident and retrospective data into Google Docs.

Automatic Data Export

Once incidents are resolved and the Google Drive integration is configured, Blameless automatically generates and exports incident and retrospective data into a newly created Google Doc. If, by policy, the incident type does not require a retrospective, no Google docs are created.

Effortless & Secure Access

A link to this Google Doc is automatically added to the retrospective details page within Blameless. Giving users quick and easy access and ensuring retrospective documentation doesn’t become a lost artifact in Google. Once in Google, users can add content and securely share access with a larger audience as needed, leveraging Google Docs' permission system (View, Comment, or Edit access modes).

Comprehensive Data Coverage & Easy to Navigate

The exported data comes pre-styled and properly converted to a Google Docs format. It includes incident and retrospective details like title, description, severity, incident type, timeline (including images captured from Slack or Microsoft Teams), incident tags, war room information (links to the Blameless incident, and corresponding Jira or ServiceNow tickets), the custom questionnaire, and the analysis template, which is customizable per incident type.

Refresh Data On-Demand

While the Google Doc is automatically created when the incident is resolved and fed with incident data collected up to that moment, users capturing more data post incident using the Blameless Retrospective questionnaire, tag management, and custom tab can update the linked Google Doc on-demand with the latest captured updates.

Secure Integration

Blameless only needs to know the specific Google Drive where you want Blameless to automatically create and store Google Docs. In this way, Blameless can operate within the folder system you assign without accessing other locations within drive that contain sensitive material.

Call to action

To get started creating and exporting retrospective data to Google Docs, ask your Blameless administrator to integrate your Blameless instance with a shared and secure Google Drive today!

New Dynamic Bookmarks in Slack
September 12, 2023
With the new Dynamic Bookmarks, incident responders are provided fast access to vital incident information at any time during the investigation.

New Feature Release

New Dynamic Bookmarks in Slack

At Blameless, one of our on-going missions is to maximize the productivity of your incident responders collaborating on incidents via Slack. 

As additional areas of expertise become necessary, and new team members join the investigation, these new players need to quickly assess the urgency and importance of the issue at stake.

Now with NEW Dynamic Bookmarks, automatically added to each new incident channel, incident responders are provided fast access to vital incident information at any time during the investigation. Dynamic Bookmarks are a welcome complement to the incident summary, displayed at the top of the incident channel and on-demand using the `/blameless show incident` command.

Blameless' dynamic bookmarks

From the very start of an incident, Blameless automatically adds the following key bookmarks to each incident channel in Slack:

  • Severity: one of your custom severity labels
  • Status: Investigating, Identified, Monitoring or Resolved
  • Required tasks: 1/5 - dynamically updated as required tasks are completed per incident status (not across all statuses)
  • Incident - Link to the incident detailed page on your Blameless account
  • Retrospective - Link to the retrospective detailed page on your Blameless account
  • Join Meeting - Link to the video conference

Required Tasks count is particularly powerful as it tracks the completion of steps necessary to progress the incident forward and can serve as a clear indicator of the team’s progress. This is an excellent reminder to task owners and incident leaders to complete the required tasks as soon as possible.

Immediate call to action

To benefit from this new feature enhancement in Slack, ask your Blameless administrator to re-authorize the Slack bot in Blameless at your earliest convenience.

Slack - Customizable start incident modal
May 1, 2023
Incident responders can now be optionally prompted to enter incident tags at the start of the incident creation process in Slack, effectively helping users capture and share context about the incident as soon as possible.

Feature Releases

Slack

Customizable start incident modal

Incident responders can now be optionally prompted to enter incident tags at the start of the incident creation process in Slack, effectively helping users capture and share context about the incident as soon as possible.

The list of tag categories (aka custom fields) displayed dynamically changes based on the incident type selected by users in the incident modal.

In the graphic below, the tag selection feature is available after starting a new incident by typing /blameless start incident in Slack.

Start_Incident.png

After the incident is started, incident responders can still update and add more tags to the incident directly from within the incident channel. This is done by entering /blameless add tags in Slack.

To leverage this new capability, ask your Blameless administrator to do the following:

In SettingsTag Management, configure selected tag categories to be visible as custom fields at the start of an incident in Slack. If no incident type has been selected, the tag category field is automatically visible to any selected incident type.

Incident Types

Enhanced incident type setting layout

The layout of the incident type setting page has been updated to provide a faster and more reliable user experience when creating new incident types. In addition, settings are now grouped together into expandable / collapsible sections and the order is aligned with an incident's workflow to facilitate ease of management.

The setting categories are as follows:

  • Manage Incident Details
  • Invite & Notify Team Members
  • Trigger Alerts
  • Set Affected Services
  • Create Ticketing
  • Assign Roles & Tasks
  • Create Retrospective Reports

A few additional enhancements have been made available to improve your experience with incident types:

  • To gain faster access to the configuration page for a selected incident type, you can directly click on the incident type from the list.
  • In case you have a long list of incident types, we make it easier to find your incident type by searching and sorting your incident types by name.
  • If you forgot to proactively click Save, you will be automatically asked if you want to save your changes before leaving any tab of the incident type configuration page.

Identity Management

Self-serve activate & deactivate users

As your team evolves, it is often necessary to activate and/or deactivate users ⏤ Blameless's new user status toggle feature now enables you to do this on demand.

Deactivating a user relinquishes their access to Blameless but does not delete the user profile, giving you the option to activate them again later.

To toggle user status:

  1. In the sidebar of the Blameless UI, select Identity Management, and then select the Users tab.
  2. Under the Status column heading, select the toggle switch to activate or deactivate a user as needed.
  3. In the confirmation dialog box, select Deactivate or Reactivate.
RBAC_Enable_disable_user.png

Bug Fixes

Slack

Type ahead search when adding tags to incidents in Slack is not working

Issue:

When adding tags via Slack, type ahead search (i.e., autosuggest) was no longer possible in Slack making it difficult to search for specific tags to be added to an incident.

Fix:

The type ahead search capability for tag categories in the Add Tags dialog box in Slack has now been restored for tag categories that have predefined tags, whether users select only one single tag or multiple tags.

Tasks would occasionally be duplicated when completed

Issue:

When completing a task in Slack, an identical message confirming the completion of the task would occasionally and randomly be displayed multiple times.

Fix:

This rare issue typically occurred when multiple tasks were rapidly being completed at nearly the same time. The processing of multiple tasks being completed nearly concurrently has been improved to prevent multiple appearances of the task confirmation message.

Jira

Jira incident ticket is not created when starting a new incident

Issue:

When the Reporter Jira system field was set as a required Jira field, the creation of incidents would fail. However, when the Reporter field is not required, Blameless correctly sets it to the default Jira user account used to integrate Blameless with Jira.

Fix:

If the Reporter Jira system field is set as a required Jira field, the creation of incidents now succeeds as Blameless does set its value automatically to the default Jira user account used to integrate Blameless with Jira, the same way it is set when the Reporter field is not required.

Role-Based Access Control
April 1, 2023
To make the assignment process more intuitive for new and existing instances, Blameless has introduced three new default incident groups. These groups reflect the most common types of incident response team roles, with each group possessing its own unique permission set.

Feature Releases

Role-Based Access Control (RBAC)

New default groups make access assignment more intuitive

To make the assignment process more intuitive for new and existing instances, Blameless has introduced three new default incident groups:

  • Observer: Users who participate in discussion and complete tasks (i.e., minimum viable permission set).
  • Responder: Users who frequently participate and/or drive incident resolution.
  • Lead: Users who are frequently incident commanders and have reliability management duties.

These groups reflect the most common types of incident response team roles, with each group possessing its own unique permission set.

Use Case

An enterprise wants to enable its teams to use Blameless, but has a Zero-Trust Policy. By specifying Observer as the default incident group, the organization ensures that the hundreds of users it is adding into Blameless are automatically routed and assigned the minimum viable set of permissions. If changes need to be made, the company can select users and modify their group placement at any time.

Additionally, here is a group-role matrix for each of the new groups:

New_Default_Groups.png

Slack

Unresolved Incidents now viewable from Blameless Slack App Home Tab

Incident responders using Blameless in Slack can now leverage the new Home tab enabled on their Blameless app to gain access to the following enhancements:

  • Brand new sorted view of your top unresolved active incidents.
  • Shortcut buttons designed to facilitate quick access to useful tools and features, such as the web app, commands used in Slack, and a getting started tutorial.
Unresolved_Incident.png

While on the Home tab, a list of your top unresolved incidents appears with a link to each incident's corresponding dedicated Slack channel. In addition, you can sort your top unresolved incidents by severity, creation date (oldest or newest first), status, or in alphabetical order by incident title.

These features enable users to quickly identify the most pressing incidents and then collaborate with colleagues to investigate and resolve.

We made sure that the list of unresolved incidents is refreshed from your Blameless account each time you change the sorting option or access the Home tab.

To benefit from these new enhancements, make sure to add the Blameless app to your app portfolio in Slack (Go to the bottom left and select + Add app). To navigate to the Home tab after the Blameless app has been added, browse to the Apps section and select blamelessbot.

Select_blamelessbot.png

Bug Fixes

Slack

Latency: Ticket and Zoom links not displaying as expected

Issue:

Occasionally, after starting an incident in Slack, the links to the Jira ticket and/or the Zoom video conferencing could take up to a few hours before being provided in the pinned incident summary at the top of the incident channel. Additionally, it would incorrectly indicate that the integration with Jira or Zoom was disabled despite being actually enabled and healthy.

Fix:

Now, after starting an incident in the Blameless app in Slack, Blameless will indicate that the links to resources on external ticketing or conference systems are still being created (e.g., Creating ticket...) and will update the links to those resources as soon as it receives confirmation from them (e.g., Jira, Zoom, Slack, etc.).

Severity levels with no label could be selected and prevent an incident from being created

Issue:

If one of the default severity levels had no label provided, this blank severity started to show up as an option in the severity dropdown when starting an incident in Slack, and when selected it would prevent the incident from starting.

Fix:

Severity levels with no labels no longer show as options in the severity dropdown when starting an incident in Slack.

Multi-line text captured in Slack are not rendered properly in the incident timeline

Issue:

In Slack, when users were adding long multi-line formatted text as a message in a Blameless incident channel in Slack and then capturing the message as an highlight to be saved in the incident timeline, the entire text would be rendered as a single line in the incident timeline in the Blameless web UI, ignoring the end of lines.

Fix:

Messages formatted with multiple lines captured as highlights in Slack are now correctly rendered in the incident timeline in the Blameless web UI.

Self-serve API Keys for authenticating services with Blameless
May 15, 2023
Looking to authenticate services and machines to Blameless while building out automation in your incident management response workflow? We’ve been listening.Previously, this feature was available by contacting one of our friendly CSMs. We are now happy to offer self-serve on demand API key generation from the convenience of the Blameless UI.

Feature Releases

Slack

Self-serve API Keys for authenticating services with Blameless

Looking to authenticate services and machines to Blameless while building out automation in your incident management response workflow? We’ve been listening.

Previously, this feature was available by contacting one of our friendly CSMs. We are now happy to offer self-serve on demand API key generation from the convenience of the Blameless UI.

Here's how to do it:

  1. In the sidebar of the Blameless UI, select Identity Management, and then select the Key Management tab.
RN_05-15_Navigate_to_Key_Management.png
  1. Select + Generate New Key.
RN_05-15_Select_Generate_New_Key.png
  1. In the Enter a Name for the Key field, enter a memorable name for the new API key and then select Create.
RN_05-15_Create_New_Key.png
  1. A new API key will be generated. Select Copy to copy the key to your clipboard, and then save the key in a secure location. When done, select Close.
  2. Note: This key will not be displayed again.
RN_05-15_Copy_Generated_Key.png
  1. Your new key appears in the key list table. You can have up to 10 active keys at a time.
RN_05-15_Key_List.png

After you have generated a Blameless API Key (long-lived), you can then exchange it for an API Token (short-lived) via a POST request to an API endpoint.

Background on Authentication

Blameless provides OAuth flows to authorize access to the Blameless API. The client credential’s OAuth flow is used for machine-to-machine application and allows authentication of the application without involving an end user. Authentication tokens are passed using an auth header and are used when interacting with the API. All requests to the Blameless APIs require authentication.​

Bug Fix

PagerDuty

The PagerDuty webhook fails to start an incident in Blameless when the PagerDuty user does not match a Blameless user

Issue:

In PagerDuty, configurable webhooks can be triggered upon the creation of an alert for a specific service, and then automatically create an incident in Blameless. If Blameless is unable to match the email address of either a) the creator of the PagerDuty alert or b) any on-call users defined in PagerDuty for the specific service, incidents are not started.

Fix:

Now, an incident will start when a PagerDuty webhook request is received, even if Blameless is unable to match the email address of a PagerDuty user (creator or on-call users) to a Blameless user. In this case, Blameless does not automatically set a Blameless user as the creator of the incident, but incidents are still started, as specifying the name of the creator of a Blameless incident is optional.

Role-Based Access Control
March 15, 2023
As a result of newly-introduced RBAC roles under IAM, Blameless has now extended access control from the platform into Slack. These enhancements empower teams to consistently align Blameless’s platform and Slack incidents with least privilege policies.

Feature Releases

Role-Based Access Control (RBAC)

Access control extended to Blameless for Slack

As a result of newly-introduced RBAC roles under IAM, Blameless has now extended access control from the platform into Slack. These enhancements empower teams to consistently align Blameless’s platform and Slack incidents with least privilege policies.

Key features include:

  • Users can assign new incident roles that map to controls in both the Blameless platform and Slack.
  • Three distinct roles are created automatically for each custom incident type, enabling users to effectively isolate access permissions by incident type (Incident Reader, Incident Writer, and Incident Admin).
  • The new Incident Reader role allows users to do more within incidents (e.g., such as completing tasks and follow-up actions).
  • Permissions are enforced via the Blameless web UI and Blameless APIs.
  • Role to Slack Command mapping:
  • Incident Reader:
  • View Your Tasks
  • View All Tasks
  • Complete Your Tasks
  • Show Tag(s)
  • Show On-Call
  • Show Escalation Policy
  • Help - Commands (show)
  • All Emoji usage in channel
  • Incident Writer:
  • All Reader items, plus:
  • Create Incident
  • Assign Incident Role
  • Resolve Incident
  • Set Incident Status
  • Set Incident Severity
  • Set Incident Title
  • Set Incident Description
  • Add Task
  • Assign Task
  • Create Follow Up Action
  • Create Swimlane
  • Add Tag(s)
  • Remove Tag
  • Trigger Alert to On-Call (PagerDuty)
  • Includes Invite to On-Call
  • Incident Admin:
  • All Reader and Writer items, plus:
  • Delete Incident

Slack

New customizable Slack incident channel naming option

Blameless has introduced additional flexibility to customize incident channel names in Slack via custom text (e.g., team name) and dynamic insertion of Blameless incident data into the channel name.

This option provides incident responders with more meaningful and descriptive names for incident channels in Slack. This enables responders to find urgent incidents more efficiently by aligning incidents with existing naming conventions.

Configurable as a new option for the incident channel naming scheme setting under each incident type, you can now specify the syntax of your incident channel names in Slack through a combination of custom text and variables. The variables represent common incident data and can be specified in any order.

They include the following:

  • Incident creation date
  • Incident title
  • Incident severity
  • Incident status
  • Incident ID

Additionally, as the incident progresses or if its title changes, Blameless automatically updates the incident channel name to reflect the latest state of the selected variables (e.g., status changing from investigating to monitoring).

Bug Fixes

Slack

Auto invite for multiple Slack users fails if any invitation to a user fails

Issue:

When auto-inviting multiple users into an incident channel in Slack (SettingsIncident TypesManageInvited Slack Groups or Users), the request could fail for all users if at least one invitee was not recognized as a user in your Slack workspace.

Fix:

Blameless detects and ignores auto-invited Slack users that do not exist (or no longer exist) in your Slack workspace. All other Slack users from the list of auto-invited Slack users that do exist in your Slack workspace will be invited to join the incident channel in Slack.

Incident Types

Unable to save updated Jira settings

Issue:

After updating Jira settings for an incident type (SettingsIncident TypesManageJIRA Settings), users were unable to save their updates. Additionally, it would delete all the tasks from the checklist configuration under the same incident type setting.

Fix:

Updates made to Jira settings for an incident type can now be properly saved and do not delete the tasks from the checklist.

Incidents

Events are duplicated when entered manually via the incident detail page

Issue:

When manually entering events to the incident timeline via the incident detail page in the Blameless web UI, the event would be duplicated in the incident timeline. Users could still manually delete one of the two events from the incident timeline.

Fix:

Events added manually via the incident detail page are no longer duplicated, whether captured as text or via image upload.

SCIM provisioning: Blameless-Okta integration
June 1, 2023
Does your organization manage user access across multiple SaaS services? If so, then more than likely you're using a third-party authorization provider like Okta to centrally manage your users. Good news! We are happy to announce that you can now manage Blameless user groups directly from Okta via our new SCIM integration capabilities.

Feature Releases

Identity management

SCIM provisioning: Blameless-Okta integration

Does your organization manage user access across multiple SaaS services? If so, then more than likely you're using a third-party authorization provider like Okta to centrally manage your users. Good news! We are happy to announce that you can now manage Blameless user groups directly from Okta via our new SCIM integration capabilities.

Key features include:

  • User activation
  • User deactivation
  • Group pushing
  • Group linking
  • User to group bi-directional updating (when groups are linked)

At a high level, the process works as follows:

  1. Create your groups in Okta.
  2. Assign your application for authorization to Blameless for each group.
  3. Assign multiple users to a group within the groups tab in directory.
  4. Select and push these groups to Blameless.
  5. Visit your Blameless instance to see your groups and the users you selected.
  6. Apply permissions to the group within Blameless that reflects the needs of your group.

Please see Okta to learn how to integrate Okta and Blameless. Additionally, to get up-to-speed on creating groups and adding users in Okta, please see User Management with Okta.

Bug Fixes

Slack

Starting an incident fails from a private channel without the Blameless app

Issue:

When attempting to start a Blameless incident from a private channel without the Blameless app added to it, the /blameless start incident command allowed users to open and complete the Start incident modal, but the creation of the incident would fail after submitting the request.

Fix:

Now, the following warning message is displayed in the private channel when users attempt to run any /blameless command when the Blameless app has not been added to it:

BL_RN1.png

Additionally, the following warning message is displayed in direct messages when users attempt to run any /blameless command when the Blameless bot user had not been invited to that direct message channel:

BL_RN2.png

Incident types

Once the auto assigned role to a creator is set to a role, it can no longer be reset to none

Issue:

In SettingsIncident Types, after selecting a new default incident role for an incident creator, users were no longer able to change it back to None.

Fix:

Users can now reset the default incident role for the creator of an incident to None from any previously-saved value.

Jira

The links to the Blameless incident and the incident channel in Slack no longer appear in the Jira ticket Description field

Issue:

When the integration with Jira was enabled in your Blameless account, the Description field in auto-created Jira tickets (per Blameless incident) stopped including both a link to the Blameless incident and the incident channel in Slack.

Fix:

If the integration with Jira is enabled in your Blameless account, both links to the Blameless incident and the incident channel in Slack now appear in the Description field for each auto-created Jira ticket (when a Blameless incident is created).

Enhanced Opsgenie Integration
January 15, 2023
Blameless is providing a more comprehensive integration with Opsgenie for alert management. This makes it easier to recruit any teams defined in Opsgenie into a Blameless incident.

Feature Release

ALERTING (OPSGENIE)

Enhanced Opsgenie Integration

Blameless is providing a more comprehensive integration with Opsgenie for alert management. This makes it easier to recruit any teams defined in Opsgenie into a Blameless incident:

  • Integrates with all services and teams defined in Opsgenie (service catalog and team directory)
  • The user can manually trigger Opsgenie alerts by selecting Opsgenie services or teams from within the Blameless incident channel in Slack (`/blameless trigger alert`) and Microsoft Teams (On-Call > Trigger Alert)
  • The user can automate Opsgenie alerts when starting incidents by preselecting services per incident type
  • The user can clear alert confirmation responses inside the incident channel (Slack/Teams) displaying the name of the notified on-call responders
  • Track all triggered alerts as an event in the incident timeline, with link to the alert, team or service in Opsgenie, providing clear context about what happened when, for the retrospective report and downstream analysis

SLACK

Self guided Tutorial

The Blameless Slack self guided tutorial allows new users the chance to "kick the tires".

These users include incident responders, site reliability engineers, those who are evaluating Blameless for the first time through our self-sign up trial program, and also new users within an organization already using Blameless.

The tutorial allows them to learn in a few guided steps how simple it is to start and run an incident using Blameless.

The Self-guided Slack tutorial gives the user instructions in how to:

  • Choose the Severity, based on a suggested level
  • Enter a title--something that makes sense in their environment
  • Provide a description (optional)

When you launch the `blamelessbot` app, you will see an invitation to try out the tutorial. Simply click on the "Start Tutorial" button to begin.

The tutorial creates a sample channel, then walks the user through a series of five steps to:

  • Invite team members
  • Assign roles to the team
  • Update the Incident status
  • Report the incident to the team and shareholders

And that's it in just a few minutes; you have learned how to start a Blameless incident in Slack, invite team members, set the incident status, and communicate details to the team and stakeholders. Once complete, the tutorial invites you to learn more capabilities Blameless has to offer.

No matching items.

Empower Secure Collaboration with Incident-Level Permission Management

April 18, 2024
While anyone at your organization should be able to start Blameless incidents, not everyone should have access to all current and past incidents (nor retrospectives), particularly when those incidents are handling sensitive and confidential information related to security issues. With Blameless you can already restrict the creation of incidents and access to incidents (and retrospectives) on a per incident type basis. Now, you can also allow Blameless users only to access the incidents (and their corresponding retrospectives) that they create, while preventing access to all other incidents of the same incident type. Additionally, at any time during or after the incident, if deemed necessary for security or confidentiality reasons, users who created the incident or joined the incident, can have their permissions to access the incident easily revoked. Meanwhile, users joining incidents without any pre-assigned Blameless identity roles, can be granted more permissions (Reader, Writer) to access and interact with the Blameless incident (and retrospective).
Read More

Integrate Blameless with your Backstage service catalog

April 4, 2024
By integrating Blameless with your Backstage catalog, your incident response teams can now more accurately tie incidents to the exact affected parts of your production environment (e.g. services, components, APIs, resources, etc), providing critical context to drive the investigation more rapidly in the right direction and fast access to a wealth of information stored in your Backstage catalog. Then post-incidents, using Reliability Insights in Blameless, each team responsible for those services or components can draw some new learnings by more effectively identifying the most vulnerable parts of their production environment based on the severity and number of incidents, among many other dimensions.
Read More

Enhanced Blameless App for Microsoft Teams

March 21, 2024
Blameless is excited to share how we’ve transformed the user experience of incident responders using the Blameless app in Microsoft Teams. By enabling the use of group chats to run incidents, we’ve streamlined the incident management workflow in Microsoft Teams. This also allows responders to reserve the thread in their team channel for the core investigation. Users can still start incidents from any channel in Microsoft Teams and access all the resources of Blameless wherever they choose to do their work. After two years on the Microsoft AppSource store, Blameless remains the leading choice for incident management for organizations relying on Microsoft Teams. These changes have all been built in collaboration with Microsoft Teams users, and we are excited to announce their availability in the form of a more intuitive Blameless experience.
Read More

Enhancing your Experience: Latest Product Updates

March 7, 2024
The team here is hard at work expanding Blameless, adding new, powerful features and integrations to help transform how engineering teams approach incident management. But every now and then, we take a step back from all that innovation to focus on the little details. Here, we’re dotting the i’s and crossing the t’s to make sure you have a fantastic experience using Blameless every day. So here’s an update coming out of our recent bug smash where we tackled the backlog of feature enhancements you’ve been asking for. Usability improvements, UX tweaks, feature upgrades, we’ve got it all here for you. Enjoy!
Read More

Track PagerDuty actions in Blameless incident timeline

February 22, 2024
Have you wondered how long it takes your team to complete critical tasks like acknowledging alerts or escalating an incident to the next level in PagerDuty? Making the team more efficient at completing these types of tasks can have a big impact on their overall effectiveness. These types of data points are great to examine during the retrospective process, but only if you capture them in the first place. Blameless now captures all those PagerDuty actions in your incident timelines as “critical events”. Each one clearly displayed, easily accessible, and ready to be discussed at your next retrospective meeting.
Read More

Incident Assistant: Automatically generating incident summaries that are easy, instant, and up-to-date.

February 6, 2024
Every incident responder has the same problem when they first join an incident. They need to get up to speed “yesterday” so they can figure out how to help the team move forward. Trying to read through, and glean the context they need, from a chaotic Slack thread is not an efficient use of time and energy. Relying on your incident commander to onboard each participant to the incident is just as slow and unreliable. Don’t worry, Blameless Incident Assistant is here to solve this problem for you. Incident Assistant generates a comprehensive incident summary for every member of your team. Instantly available on-demand at any time during or even after the incident has been resolved.
Read More

Better ways to recruit team members to Blameless incidents using PagerDuty

January 29, 2024
Finding the right subject matter experts to join an investigation is something you wish happened yesterday. Failing that, you’ll probably settle for “As soon as humanely possible”. Fast recruitment of the right experts when an incident kicks off can be a difficult task, but it can all be the difference between quick resolution and a problem spiraling out of control.
Read More

Customize the Comms Assistant™ to your communication style using your own OpenAI™ account

January 11, 2024
Blameless Comms Assistant™, powered by OpenAI™, makes sending effective incident updates to various audiences, easier than ever. Comms Assistant can now be integrated with your own OpenAI account. So, if you have policies governing the use of Generative AI products and have your own OpenAI’s ChatGPT Enterprise subscription, you can leverage your OpenAI account to keep your incident data private. Blameless lets you integrate with either the ChatGPT engine version 3.5 or 4.0, so you can choose which generation of model works the best for your team.
Read More

Export Retrospectives to Google Docs with more flexibility and security

December 8, 2023
Blameless retrospectives can be automatically exported to Google Docs, and with this new update, can NOW be stored flexibly in any location across your folder system. This makes it easy to provide quick access to retrospective documentation for a specific team or your entire organization. This lets you quickly disseminate learning while leveraging Google access permissions to control who can engage your retrospective docs.
Read More

Export Enterprise Reliability reports to keep your teammates in the loop

December 1, 2023
To ensure the right Reliability dashboards make it to intended audiences for ad-hoc or end-of-month reporting, the owner of any given dashboard can export it directly as a PDF file by clicking on the 3 dots at the top right of the dashboard.
Read More

Capturing follow up actions in Blameless is easier than ever!

Anything you learn from past incidents and through retrospectives eventually results in some form of action that your organization wants to take to prevent similar incidents from recurring, whether it is about improving processes, software stacks, or professional skills, just to name a few. To make it easier to capture and assign such action items via the Blameless web UI, we have streamlined the process of creating follow up actions and linking them to the appropriate external ticketing system of your choice.
Read More

Generate Swift and Powerful Incident Updates using Comms Assistant™

October 31, 2023
Communicating effectively with a broad set of stakeholders is a key characteristic of good Incident response. Everyone who has a stake in recovering from a high severity issue wants to be reasonably informed of status or progress towards resolution. However, crafting and delivering regular updates to satisfy these folks takes time and focus. Two things that incident responders often can’t afford in the midst of a high severity incident. To empower communication leaders and incident responders in providing fast, frequent and meaningful incident updates to various audiences (executive, sales, support), Blameless is introducing the first generative AI-powered Automated Incident Communications With Comms Assistant™, available now in preview mode.
Read More

Configure Blameless Settings via Terraform!

October 26, 2023
Whether you’re a Site Reliability Engineer (SRE) or a DevOps Engineer, Terraform is probably one of your favorite tools. Enterprise Reliability Management (ERM) best practices rely on incident response and retrospective workflows, that conform to Standards of Operations (SoPs) and change management control. Engineering organizations seeking a high level governance and control can now download and use the Blameless Terraform Provider to enforce critical configuration guardrails for incident workflow management in Blameless.
Read More

Take action as soon as you join the Incident in Slack

September 20, 2023
Starting incidents should not be stressful for anyone, especially for the incident creator and leaders (aka commanders). Key recurring actions must be taken right at the start, such as recruiting more incident responders, assigning roles to delegate tasks, paging on-call engineers, and quickly checking the typical set of tasks that you and your incident team is expected to complete as soon as possible.
Read More

Collaborate on Retrospectives using Google Docs

September 28, 2023
Running high-quality retrospectives is a team effort. Every member of the team collaborates to analyze incident data and converge on the best plan of action to prevent similar incidents from reoccurring. Offering the best retrospective tool in the industry has been one of the pillars of our success at Blameless. Now, all the automated data collection and templated process standardization that Blameless Retrospectives provides can be seamlessly ported into Google Docs. Engineering organizations relying on Google Workspace, can now more efficiently produce, collaborate, and share high-impact Blameless retrospectives by automatically and securely exporting incident and retrospective data into Google Docs.
Read More

New Dynamic Bookmarks in Slack

September 12, 2023
With the new Dynamic Bookmarks, incident responders are provided fast access to vital incident information at any time during the investigation.
Read More

Slack - Customizable start incident modal

May 1, 2023
Incident responders can now be optionally prompted to enter incident tags at the start of the incident creation process in Slack, effectively helping users capture and share context about the incident as soon as possible.
Read More

Role-Based Access Control

April 1, 2023
To make the assignment process more intuitive for new and existing instances, Blameless has introduced three new default incident groups. These groups reflect the most common types of incident response team roles, with each group possessing its own unique permission set.
Read More

Self-serve API Keys for authenticating services with Blameless

May 15, 2023
Looking to authenticate services and machines to Blameless while building out automation in your incident management response workflow? We’ve been listening.Previously, this feature was available by contacting one of our friendly CSMs. We are now happy to offer self-serve on demand API key generation from the convenience of the Blameless UI.
Read More

Role-Based Access Control

March 15, 2023
As a result of newly-introduced RBAC roles under IAM, Blameless has now extended access control from the platform into Slack. These enhancements empower teams to consistently align Blameless’s platform and Slack incidents with least privilege policies.
Read More

SCIM provisioning: Blameless-Okta integration

June 1, 2023
Does your organization manage user access across multiple SaaS services? If so, then more than likely you're using a third-party authorization provider like Okta to centrally manage your users. Good news! We are happy to announce that you can now manage Blameless user groups directly from Okta via our new SCIM integration capabilities.
Read More

Enhanced Opsgenie Integration

January 15, 2023
Blameless is providing a more comprehensive integration with Opsgenie for alert management. This makes it easier to recruit any teams defined in Opsgenie into a Blameless incident.
Read More