What is an Incident Timeline and How Do You Create One?

Incidents are unavoidable in software development and IT. As a Site Reliability Engineer (SRE), one of the tools you’ll use frequently is an incident timeline. The incident timeline provides a real-time report on any incident, including alerts, system updates, issue severity changes, manual chat entries, and more.

The incident timeline gives you access to the incident from start to finish and is a necessity for an SRE team because it helps you see what happened, share information between team members, and train new SREs on common issues and best practices.

What is an Incident Timeline?

An incident timeline is a report used during the postmortem (post-incident review) to help understand what happened, why it happened, any changes that occurred during the incident, and other details that will help SREs better prevent similar issues in the future.

The incident timeline is essentially a report that includes every step of the issue, such as anyone involved, and changes made to the system during the problem.

The following steps take place throughout the timeline:

1. The timeline begins at the initial detection of the problem. Maybe it was caught by user reports or an automated monitoring system. Either way, it can now be identified.

2. Incident identification usually occurs once a member of the SRE team realizes there is an incident and the response process begins.

3. Information is collected using error messages and system logs, among other useful resources.

4. SREs begin an investigation to see how the issue started. They may need to change code, examine code, implement debugging, or review system metrics.

5. A solution is created, and the issue is stopped and rectified if possible.

6. Updates are sent to other departments involved in the issue or solution, and shareholders or other important parties may be updated on the event.

7. A postmortem begins to review the above timeline and determine how to avoid the issue again in the future.

These steps may differ slightly from company to company.

Why You Need an Incident Timeline

An incident timeline is used anytime a problem occurs that could affect the system or software being developed/used. These incidents often directly impact the overall performance and reliability of the system.

SREs use the incident timeline to collaborate with other departments, create a detailed report of what happened, why, and how it was fixed, and analyze these details to streamline future incident responses. This creates learning opportunities and instills accountability within the SRE team.

Each incident report is more than the sum of its parts. Reviewing incident reports, you may find that a large problem can be more easily broken down into smaller issues. Maybe your system has been shutting down at random and you can’t figure out why. This is a big deal, stopping productivity, halting communication, and maybe impacting revenue. Incident reports help you get to the bottom of this by keeping thorough reports of everything that goes wrong behind the scenes of your system.

Being diligent in tracking incidents and being transparent in reports keeps you knowledgeable and prepared for future incidents.

How to Create an Incident Timeline

Like any timeline, an incident timeline shows the history of a system problem step by step as it occurs. The time is important because it helps SRE teams see:

• If there is a correlation between time and incident
• How long the incident went on before being noticed
• How long it took to stop or fix the problem once it was reported
• When the system began working normally again

An incident timeline is created by either manually or automatically capturing incident details as events occur. Each event is an action that directly impacted, or happened because of, the incident at hand. 

The events are combined into a single document to detail what, how, and when an event happened, as well as who was involved, and which platforms or applications it occurred on/with/through. All of this information is important to effectively communicate the issue to shareholders, management, and other teams, as well as to create a fix.

The timeline should include:

• Initial detection and response
• Incident escalation
• Incident duration
• Communication between SREs and other departments
• Incident resolution steps
• Post-incident review and documentation

There are many types of tools and software for creating incident timelines, including:

• GitHub
• IncidentIO
• OpsGenie
• Splunk

You can also count on Blameless for incident timelines. Use Blameless’ incident management software to resolve issues faster with automated task assignment, real-time data capture, and report assembly.

Use of Incident Timelines During Postmortem Process

Analyzing the incident report after the fact helps identify the root cause of the problem, which lets SREs go straight to the source and fix it before the issue repeats itself. During the postmortem, the incident timeline acts as a map key to decode:

• What went wrong?
• Where it happened?
• Why it happened?
• Who was involved?
• How to prevent it in the future?

Collecting all these details into one spot makes the information accessible to multiple departments, easy to break down and understand, and provides all the evidence of the issue in a single document.

In short, an incident report could save you hours of your team’s time spent pouring through systems, talking to different departments, and trying to share details individually, instead of all at once in the report.

Incident response is improved significantly by this timeline analysis. Seeing when things happen in real-time offers clues to intercept future issues more quickly.

Conclusion 

Incident timelines are an important part of the Site Reliability Engineer’s job. These reports help you maintain a more reliable site by providing clues on areas of weakness and potential fixes.

Use incident timeline tools to streamline documentation with standardized reporting. Many of these tools automate the reporting process in real-time and provide easy-to-follow templates. Simply input the details of each issue, and share it with teams across your platform.

Interested in learning more about the incident management process? Blameless can help. Book a demo or free trial today.