Kurt: Today, we're talking with Betsy Beyer, the writer behind such well-known books as the 2016 Site Reliability Engineering book, the follow-on SRE Workbook, and Building Secure and Reliable Systems. I'm sure she's also written about 10,000 other things that have not seen the light of day outside of Google. I wanted to start by asking you, Betsy, what's it like to write a bestselling, at least technical, book like the SRE book?

Betsy: So you beat me to the chase with naming all three books that I've worked on in the past few years. All of those were really group efforts. We had dozens of contributors for each of those, each of whom was super smart, generous, on the ball. Then each of the books also had a core team, a handful of people, and our job was to herd all the cats and impose some uniformity there. I think what's really interesting about this approach of a group effort in writing a book is that it's really a lot different from how most books and even most nonfiction books are written. So none of these books were a single person sitting down writing a Magnum Opus on their own. Instead, and I'm going to quote my friend and mentor Phil Beavers here, he likes to equate writing a book like this more to like building a cathedral, than making a sculpture.

Kurt: Can you explain that?

Betsy: I think instead of that one person really crafting this great work on their own, it involves a lot more collaboration, a lot of project management, a lot of coordination. There's also a pretty big task of uniting everything into a coherent whole. And that's a lot of what fell to the core team, those four people generally who are the editors of the book. So that team was responsible for having a vision of how we wanted to shape this book, what the outline was going to look like. From the beginning, shaping that big overall vision, and then more towards the end, making sure that every chapter fell in line with the tone and voice and style that we were looking for, for the book.

Kurt: What led to this shift, I'll say, it seemed in Google, that prior to the SRE book coming out, things were very relatively secretive. There were a few people who had left Google and sort of the concept of SRE was leaking out into the community, I'll say, but it wasn't widely recognized. There's still people who think that DevOps came before SRE came into being, which is wrong, because SRE was developed circa 2003 by Ben Treynor Sloss, and DevOps was not even a coined term until 2009 or 2010. But, the secrecy and sort of, this is our internal thing, seem to prevail until like 2015-ish or sometime in the teens. Do you have any background as to what made that change?

Betsy: I think over time, in the past 10 years or so, that there's just been a lot of traction and popularity with concepts like SRE and DevOps, because there are just a lot more engineers out there building large and more automated systems, and also people realizing that maintaining them with that traditional operations model just isn't sustainable. In the first place, there was an appetite for people to see how we were doing SRE at Google. On the other hand, Google realized that we think we have a pretty good way of doing things with SRE internally, but we also can stand to learn from other people in the industry how they're doing stuff. So I think opening that information to the whole community, it's good for everybody else in the community, and it's good for Google, too, because just because we invented SRE doesn't mean that we have the best answer to how to do everything. So I think we can all learn a lot from each other.

Kurt: How long does it take to assemble a book like the SRE book?

Betsy: The first book definitely took the longest. It was from conception to publication probably about three years. So it was Niall Murphy, he's one of the editors on the book. It was his idea originally. He kind of put it out there, and people were talking about it within SRE at Google. Then it really started to get traction once we got a program manager on board, [inaudible 00:05:36] PM extraordinaire really got that project off the ground and started getting all the organization in place to actually getting these dozens of people to write on it.

There was a lot of process that had to be defined for the first book. As you can imagine, when you have dozens of people working on something, that's just going to take a long time, A, to write stuff; B, to get it reviewed; C, to shop it around to publishers; D, to edit it. That was a pretty long road. I would say the next two books, we're able to take a lot of that process that we already had in place and repurpose it and improve upon it. I'd say each of those took probably more like a year-and-a-half from start to finish. Cutting down that time in half is pretty good, but it's still a long road, especially when you're looking at an almost 500-page book, it takes a while.

Kurt: All the books are pretty substantial. How's technical writing different from other types of writing?

Betsy: There's kind of a meme about technical writers out there, that a lot of them are working on the next great American novel. That is not me. I like to say I love technical writing because I love writing, I think I'm good at writing, but I'm never going to write a work of creative fiction. To me, what I find satisfaction in is taking ideas and content that's in a really nebulous [inaudible 00:07:27] form out there and figuring out the best way to present that and how to structure it and make it accessible and understandable to people. So I think at the end of the day, that is the goal of technical writing. The number one task there of course is figuring out your audience, because if you don't have your audience well-defined, then you're going to end up with a piece of writing that no one wants to read.

Kurt: Makes sense. How do you conceptualize the different audiences that might be consuming a piece of technical writing?

Betsy: I've worked on a lot of different kinds of writing, technical writing, in my career. So I've done a lot of internal stuff for Google, and now I focus mostly on externally facing stuff. Usually someone either comes to you with an idea of something they want to write or request or some kind of documentation or article, et cetera, that they want written. So really sitting down with that requester and saying, "Okay, what's the end goal here? Who do you imagine reading this?"

From there, I think the next useful step is coming out with an outline and then getting early feedback from people who might be consuming this to say, "Hey, this is what we think seems useful for the audience who is hopefully you. Do you find this useful? How could we approach this in a more useful way for you, if it's not?" Or "Are there any gaps here that we're not addressing?" Things like that. So really feedback is important at all steps along the way, and the earlier you can get it so you don't waste your time putting a lot of energy into something that no one's going to read, the better.

Kurt: If you were going to coach an engineer about writing, what would you advise besides know your audience?

Betsy: To me, I think the hardest step in writing is getting something, anything, down on the page to start with, because it's always easier to iterate from and edit whatever drafting piece of content you have than just getting it on the page. For that, just getting it on the page, don't let perfect be the enemy of good. Just brain-dump something and go from there. Not everyone finds this useful, but I always personally find it super useful to have an outline before I start writing, again, to go back to that concept of feedback all along the way. If you have an outline, you can shop it around and get feedback before you invest a lot of my energy in writing stuff. To me, that's just always useful exercise, because nobody likes to feel like they're wasting their time.

Kurt: What kind of feedback is helpful? How do you get people to give you feedback? I know you've probably got a well-developed network now, but in the early days, how did you get started with having a feedback group or network?

Betsy: I would say a big task for a technical writer is figuring out the best way to get input from your subject matter experts. The truth of that is, is that a lot of different people operate in different ways. Some subject matter experts, it might be useful for me to schedule a meeting with them and say, "Hey, I have this outline. I want to sit with you in a room. Let's go through the outline point by point so I can get your thoughts on it."

Other people hate meetings and will actually respond to an email, if you send them an email request. Some people want you to Ping them over Chat. Personally, that would drive me crazy if someone were submitting individual work requests via Chat, but hey, it works for some people. You got to cater your tactics to whatever's going to work for people. Also, developing the fine art of the nag, how often to remind people to get back to you, when to maybe give up if you're hitting a brick wall. Those are kind of things you figure out over time. Good news, this does translate to other areas of your life, learning how to nag people.

Kurt: What has surprised you about your career? When you started out, did you envision this as what you'd be doing?

Betsy: I did not. I started working at Google right after I graduated from college, because I graduated and I was like, I need a job. Hey, and I responded to kind of random posting on Craigslist. And I was working in AdWords at Google, which was an interesting learning experience, but not a good fit for me, as that role became more salesy, because I'm not a salesperson. A couple years into working at AdWords, I had my first good manager ever, who actually did some career planning with me and was like, "Hey, Betsy. You don't like advertising. Let's try to figure out something maybe you would like to do at Google and see if we can get you there." I thought, hey, I really like writing. I'm good at writing, I work at a tech company, so what about technical writing?

From there, I just found a lot of mentors at Google who were willing to basically teach me everything I know from scratch. All that being said, I would say the most... Maybe I'm not necessarily surprised by this at this point, but I'm consistently delighted by just the openness and generosity of people who are willing to share their time and their wisdom with you. And whether that's me asking someone for information on a technical topic or for career advice, I've gotten so much help along the way in the past 10, 15 years from people both inside and outside of Google who are just very giving of their time and their wisdom. I do try to keep that in mind and pay it forward when somebody else needs advice or help.

Kurt: A couple of things come to mind from that commentary. First, the idea that it took you a couple of years before you got what you considered a good manager. That seems sad, but besides, I'm so glad that it did happen. Was it the fact of being interested in your career and helping you to shift into an area that really resonated with you, that characterized that manager as a good manager, or one of your best managers? Were there other characteristics that people can take to heart if they're managing people, that they might want to think about improving as their skills?

Betsy: I guess that was the first manager who seemed more interested in how they could help me develop into a strong contributor, rather than what they could get out of me as a report. I think considering... It sounds really trite, but it's true, that if someone cares about what they're working on, they're going to be a lot more invested in it and put a lot more energy into it. So I think it is something that really pays off in the long run, if a manager is invested in their reports in that way and is really trying to help them find a good match for their skills and interests. But I've never been a manager, and it's hard to manage people, I'm sure. If you're managing 20 people, how do you do that for 20 people? I don't know.

Kurt: I have no experience doing so either, but at least a view from the trench's side can be helpful for people to think about, and I think you've provided some good nuggets there.

Betsy: Yeah. That was just really eye-opening that, that manager I had, Steven, he provided a model, that it made me realize for the first time out in your career, it's so important to have a good manager, and it's almost impossible to be happy in your job if you don't have good management. If I'm going to dole out any career advice here, it would be, when possible, if you're looking between different open positions, if you can select the one for which it seems that the manager's really going to be on your side, that's always the job you should go with.

Kurt: That seems to fit with what I've read in Gallup surveys in terms of the importance of a manager and engaged employees, and so on. You also touched on the concept of mentors and more or less informal support in your career. Can you talk a bit about that, and what is mentorship from your point of view? How have people helped to propel your career in that regard?

Betsy: Well, as I was talking about before, my first tech writing mentor, basically I came to this woman and was like, "Hey, I want to be a tech writer. I have no clue what to do." And she invested... It's funny. I actually live with her now because she's my best friend and roommate. Twelve years later we still have a great relationship, but at least career-wise, I would hope that she would say, too, that it paid off for her, too, because she trained me up. I became a good technical writer. And then in turn, I was able to be a strong producer for her team. Then we were documenting internal self-help stuff for Google. Once she did train me up, I was able to be a strong contributor to that and give back. After that, I've gone on to mentor a handful of people at Google myself and would like to think that I've spread out that mentorship love to an umbrella of people who followed after me. So I'm trying to remember what your original question was.

Kurt: How do you characterize, or how do you think about a mentor/mentee relationship? I know it's going to be a little bit different for each personal combination there, but how do you think about it?

Betsy: I think what's valuable coming from a mentor, is really kind of high level, long view guidance, is kind of the overarching most important thing. It is important to have somebody you can go to, to ask really annoying nitty-gritty questions to. But when I think about what stands out in my mind about people who have helped, it's people who have been in your field for a longer time than you, who are able to take a long view and, in that way, help you think more strategically about not just what you're doing today, but where you want to be headed in a year or a few years from now. Because you need a balance of those things. You need to be able to be successful in your day-to-day, but if you're never looking towards your next step or the next big thing, then it can get easy to stagnate, too.

Kurt: Some of the other questions that we had had talked about in the preparation for this podcast is, why three books? Trilogies are awesome, but are there more books on the horizon? Why more than just the first one, I guess?

Betsy: The first book was really kind of a runaway hit and a lot more successful and popular than we were even hoping for. So at that point we're like, okay, well, this was enough of a hit that it seems like a good idea to write another book. People are asking for this. The first SRE book was really about theories and principles of SRE, and the next book, the SRE workbook, that was more hands-on advice, work examples, in that volume. Then the third book that I worked on, which was Building Secure and Reliable Systems, we didn't build that as a third in the SRE series, per se. It was more of a companion book that looks at the intersection of security and reliability. There was also really a strong appetite for what people had to say about security, because to go back to your earlier question about all of a sudden SRE had been the secret sauce, and then we decided to talk about it, there's a very similar appetite to hear what we were doing for security.

Besides books, we have since started to focus on other types of content, too, which this is a convenient place for me to plug Google.com/sre, where we have all of the SRE-related content we've been working on for the past several years there. You can read all the books online for free.

Kurt: That's awesome.

Betsy: Yeah, and we also have articles. We have some longer format pieces. For example, there's a report on training SREs. It's been one of our most popular pieces. We have online trainings and audio interviews. So there's a lot of content on there if you, the audience wants to go and poke around, and it's all free.

Kurt: Thank you. Thank you for bringing that up. I am regularly aiming people to that site, because there is a tremendous amount of awesome content there.

Betsy: Good. Thanks.

Kurt: You started with books and now you mentioned you've got like articles, short form, long form articles. How are they received? Presumably, you've got tracking on all of this to see how popular things are. Are you finding the particular formats work significantly better than other formats? Do you know?

Betsy: Yeah, it's interesting. We found that articles do tend to perform pretty well. There are lots of pluses and minuses to longer format versus shorter format. And that's this push and pull we have between, oh, do we want to write another book right now? A book is great, because a book's got a certain gravitas and authority, but it's also a point in time snapshot that's not really evolving. So the shorter form contents, they're just a different way to consume content, to reach a wider audience.

We've also found that even shorter form like a blog post can be successful for other types of content. So we've been working on beefing up the SRE blog lately and our Twitter handle. I'm not a huge social media person, so my coworker [inaudible 00:22:58] is awesome at doing all of our Twitter posts. I got to give him a shout-out for that, because I cannot take credit. We've tried to cover all the bases with the content types there, because different people like different kinds of content. We've also wondered would it makes sense to do a podcast. Maybe I'll pick your brain about that later, Kurt.

Kurt: [inaudible 00:23:25] there. I think the most interesting thing I've found about podcasts are the variety of guests that participate in them. It's an interesting opportunity to have a conversation and hear from another point of view. So speaking of another point of view, what has your progression over time with the books and now the other articles and things, taught you about SRE itself? Because as a technical writer, you're coming in from an outside perspective to a certain degree, and now you're still inside of Google, but what have you learned about SRE as a profession?

Betsy: If we want to get kind of meta here, I think working on the books, especially, I think we learned a lot about writing books about SRE is very SRE-like in and of itself. As I mentioned right when we first started, knowledge about distributed systems is distributed. That's why each of these books had so many contributors. It was also really important to engage in capacity planning for writing these books. For example, the first book had something like 34 chapters, and we needed a tech writer assigned to each of those chapters. So we really had to work with the manager of the SRE tech-writing team to make sure that we had enough coverage for that load of work. We also found it was important to design for redundancy. SRE is all about N+1, N+2. Because we talked about that timeline for the books.

Over the course of a year, year-and-a-half, inevitably some subject matter experts are going to become unavailable or leave the company or whatnot. So we always wanted every topic to have a couple spare SREs that we could pull in.

Then, finally, I think we unsurprisingly saw that SREs have a lot of opinions.

Kurt: That's shocking.

Betsy: Yeah. And some topics in particular are really prone to shedding. So I think, gosh, one of the biggest headaches in the first book was the topic of configuration. We tried drafting that chapter like five different times, and we just couldn't think of anything. We couldn't come to a consensus of anything useful to say. Eventually, we were just like, okay, we're not doing a configuration chapter. Yeah, I would say those are all the SRE lessons we learned from the book.

Kurt: Design for redundancy, that's a good point, a good takeaway. I was curious as to whether you found... I remember reading Daniel Kahneman's Thinking, Fast and Slow book where he talked about the planning fallacy, where even he and a group of people were writing a textbook, and they knew about the planning fallacy, and they tried to compensate for it and figured, oh, well we still think that we can get it done in a year, or some, I can't remember the exact timeframe. Do you feel like your planning for the books ended up being fairly close? Did they take a lot longer than you expected? Or did you magically get them done a lot sooner than you thought along the way?

Betsy: It's funny, because I was in a meeting with somebody who had worked with me on a book, and somebody else in the meeting said, "Well, more time is always better, right? So let's just extend the timeline." [inaudible 00:27:05] I could see his face just cringing on the meeting, because I think one huge thing we learned is it's not always useful to give people more time on a deadline. People are going to fill the time that you give them. So we try to stick with pretty strict deadlines for chapter timelines, but I will say inevitably in a huge project like that, it always ends up taking a little bit longer than you would have hoped. You go into it knowing that, and then you hope it's just one month instead of four months extra.

Kurt: You talked about the idea of starting with a draft, a very drafty draft or an outline for content when you're working with somebody to develop a chapter or an article or something like that. How iterative is the process? Is it sort of throw it over the wall and hope that they get back to you by the deadline? Or is it more of a, hey let's do a daily standup. I'm taking all these memes from software development. But tell us a little bit about the process of working with the subject matter expert.

Betsy: For the second and third books, one process improvement we have that I think really helped was, we had chapter managers. So every chapter had its own little mini project management thing going on. We could really scale the efforts, so the overall program manager wasn't trying to manage every single chapter on their own. So really per chapter, that chapter manager tried to tailor their approach to what worked for that team. Some chapters might've had just one author. Then it might not make sense to have, say, a weekly meeting between the authors to see how it was going. Or some of them maybe had three authors who were kind of interweaving their narratives into each other. In that situation, it did make sense to have regular meetings for those people. Some people did a fine job of syncing every email and keeping [inaudible 00:29:32] that way. Whereas other people, seeing a face in a meeting every week helped keep them on task more.

We really tried to flex to whatever worked well for a particular chapter team. I think we're largely successful with that. I'd say kind of, in the most recent book, I would say most chapter teams probably defaulted to a somewhat regular check-in meeting. Because it's easier to cancel a meeting if you don't have anything to update on than to add one to people's calendars. That's kind of the de facto model that a chapter used, but, yeah, it's important to figure out what people's working styles are. And if somebody is not going to show up to a meeting, that they will respond to an email, just go that route.

Kurt: [inaudible 00:30:21] pragmatic and flexible, too. So that's good.

Betsy: [inaudible 00:30:26].

Kurt: That's the [inaudible 00:30:32] I don't know if that would be a good title for the podcast or not. You had suggested a question that people might have around how they get into book publishing or just generally publishing work. What thoughts do you have on that topic?

Betsy: My thoughts on that are, if you're interested in writing about tech, I think it maybe makes sense to start kind of small with a blog or some kind of similar format like that. That way you can start with somewhat bite-sized pieces of information, help build credibility and your audience, too. I think if you have success with that, what you do then depends on what you want to accomplish. If your intent is to share with as many people as possible, I think a blog or social media is really a good medium to do that. Whereas, a book's a good path to I want other people to take up my ideas. You can thump a book down on the table and say, "This is the answer, do it this way." I don't personally know a ton about self-publishing, but I know that's also a possible avenue if you don't want to go the whole industry publisher route, but, yeah, it seems like these days a blog or something like that is really a good way to get started.

Kurt: So maybe to wrap up our conversation, are there any particular resources that you'd recommend to people, whether they be conferences, other writers that might have blogs, or any articles on getting better as a writer? If you're an engineer, let's focus this, because I think mostly engineers are going to be listening more so than tech writers, sorry. For an engineer who wants to, recognizes that they want to get better at writing, what resources might you recommend? And we'll put links on the podcast's link.

Betsy: That is a very good question. I'm going to admit that I don't have an immediate answer for you. I have lots of internal resources at Google that I point people to. We have a tech writing 101 class that we give to engineers. I'll say that, generally, if you're interested in writing, one of the best things you can do is pay attention to people's writing who you like, and ask yourself, what about this piece makes it successful? But Kurt, let me look around and see if there's some resources that I would recommend for that. Because I don't want to just throw something out there that I haven't really thought about and lead people astray.

Kurt: Fair enough. We'll follow up with any resources that Betsy comes up with. We'll put them on the podcast links down below. I don't know if it's exactly your tech writing 101 course, but there is a Google course for tech writing that is published publicly that we have just aimed our own engineers at recently. So we will include a link to at least that one. Maybe there's multiple parts to it.

Betsy: Okay. Yeah. I'll check that out, too.

Kurt: Any parting thoughts, Betsy? And we can draw this conversation to a close.

Betsy: Well, I really enjoyed the podcast, Kurt. Now that we're thinking about maybe doing one, maybe, again, pick your brain, too, about podcasts.

Kurt: I'd be happy to chat with you anytime. Thanks again for being our guest on Resilience in Action. And we'll look forward to talking again in the future.

Betsy: Thanks, Kurt.

Kurt: Bye-bye.

Betsy: Bye-Bye.