Conversations about Software Engineering

Conversations about Software Engineering (CaSE) is an interview podcast for software developers and architects about Software Engineering and related topics. We release a new episode every three weeks.

Transcript

Sven Johann: Hello, and welcome to a new episode of The CaSE Podcast. Our today's guest is Birgitta Böckeler. She is a technical principal at ThoughtWorks, where she spends her time on software delivery teams with coding, architecting, coaching and consulting.

Sven Johann: At the O'Reilly Software Architecture Conference in Berlin in 2019 she gave a talk about cultivating architectural principles, which we will discuss today. Birgitta, welcome to the show.

Birgitta Böckeler: Hi, Sven. Thanks for having me.

Sven Johann: What are architecture principles?

Birgitta Böckeler: The best definition I have heard is by Eoin Woods, and I hope I'm pronouncing his name right... He says that an architecture principle is a declarative statement made with the intention of guiding architectural design decisions in order to achieve one or more qualities of a system. So if you break that down - it's a statement, and you're writing it down because you want to have guidance for your decision-making about the architecture and the design, and that guidance should help you achieve qualities of your system. So it's something like your architecture requirements, your cross-functional requirements, for example.

Sven Johann: What would be an example?

Birgitta Böckeler: An example could be loose coupling, or we want to encapsulate our legacy, or it could be an agreement on what size a microservice should be in your context, in your organization, or it could be--

Sven Johann: It's in your head...

Birgitta Böckeler: In your head, for example, yeah... And then what does that mean, "fits into your head"? There's different interpretations of that as well. Or something quite common is also guidance around how to use cloud. For example, what level of cloud vendor dependency is acceptable, or a principle like saying "AWS-first", which clearly gives guidance on what the cloud strategy of the organization is.

Sven Johann: So besides a catchy title, how does such a principle look like?

Birgitta Böckeler: Yes, a catchy title is definitely not enough, because if you think about it, something like loose coupling - you could say "Oh, that's just common sense." There's lots of principles out there in blog posts and just good practices... So just having the title would probably not be enough to justify saying "Okay, we need to write this down for our organization." So what's really important here is that you don't just have the title, but also have a description, and you write down why this principle is important for you, and what it means in practice on the teams.

Birgitta Böckeler: The TOGAF framework, which is like an open framework that describes how you could do enterprise architecture - it actually has a template for architecture principles that I find quite useful... And the template says "You have a title, a description", and then you write down the rationale, so the Why, which could for example be a description of your hypothesis, how this principle will help you achieve those architecture requirements you have... And you should also write down, according to this template, the implications. So again, those other things - what does this mean in practice? So if you don't add all those things, then it doesn't really help teams guide their decision-making, which is the ultimate goal of the principle.

Sven Johann: I think the link to TOGAF - we will put it in the show notes. But can we pick one principle and fill the template with a little flesh? So if we have AWS-first, what would be a brief description and rationale for that principle?

Birgitta Böckeler: It's of course hard to create something like this on the fly...

Sven Johann: Yes, exactly.

Birgitta Böckeler: ...because there could be a lot of words missing involved in this...

Sven Johann: I mean, of course... We're not expecting completeness, or something, but just give one or two simple sentences on that.

Birgitta Böckeler: I mean, something like AWS-first would show, as I said, the cloud strategy of the organization. In the rationale you would describe why you decided to choose AWS maybe over Google Cloud, or Azure (another one of those things, that I don't know how to pronounce). And the nice thing here is also that writing down this rationale really forces you to write down how you came to that decision... So it's kind of, in this case, a bit of a decision record as well.

Sven Johann: Okay, an architecture decision record.

Birgitta Böckeler: Yes, yes. And if you think about the qualities of the system that you want to achieve with this principle, it might be something like a faster time to market, because you want to reuse the services that come out of the box in AWS. And then in the implications, maybe it would say something like "This means that whenever you need a service, first consider if there's something suitable out there on AWS, and only if you find there's nothing suitable, then fall back on other things we have in the organization. And only if that's not suitable, then fall back on something totally different." Kind of give teams a set of questions to ask themselves to decide which service to use.

Sven Johann: Okay. So now we know how an architecture principle looks like, but why do I need them? Of course, we want to disagree with architectural decisions, or we want to enforce architectural qualities, but you might be able to do that otherwise... So why should I choose architecture principles for that?

Birgitta Böckeler: Most often I would say they're used as a tool of governance, and that is also I think where TOGAF places them. So as your IT organization grows, it will split into more and more teams, so that you're able to scale, and as the number of those teams grows, you also need more decentralization... Because you can't have a central group anymore that knows enough to make good decisions for everything. So you need those teams to be as autonomous as possible, because they are the ones close to the problem, and they need some empowerment to make their own decisions.

Birgitta Böckeler: So that is kind of a necessity as you scale, to have those autonomous teams, but of course, it comes with its own challenges, like risk of duplicate work, risk of needless variation, you need to coordinate all the dependencies between the teams, and so on... And especially for that needless variation, you might want some kind of governance, because full autonomy is basically a myth, right? There always needs to be something so that people are going in a similar direction, so organizations use principles to try and create some alignment between the teams... So governance is the biggest motivation behind them I think.

Birgitta Böckeler: But for me, I try to look at them more as a tool for change management, and to use them for learning between the teams, to kind of make it easier for them to do the right thing for the big picture, as opposed to their local problem.

Birgitta Böckeler: I mentioned before that a lot of these principles - loose coupling is the best example for that, I think - they are just good practices that are out there anyway, that you might have learned at university, or at a conference, or in a blog post. But if you choose a few of your own, then that can give you some focus to focus on the few areas that you want to improve in. So that's what I mean by change management.

Birgitta Böckeler: For me, architecture principles in an organization shouldn't be a comprehensive list, because then you'll just start writing down the whole internet again... But it should be a few principles that help you decide what you want to focus on and improve in that moment in time. And maybe if you successfully use them, then there's no need to highlight them anymore and you can scratch one of them.

Birgitta Böckeler: And another reason for me -- so focus is one that I think is a reason why it might make sense. Another reason for me is to get more consensus on what they mean. We had that example of "How big is a microservice before?" and there's lots of different opinions on this... And it might be valuable in an organization to get a common understanding of this. If you're going all the way to having super-small services, and then the respective support for that in the infrastructure as well, or if you maybe want a different size with that... Because the more abstract these principles are, and if you want them to be abstract because you still want the teams to be autonomous, so the more abstract they are, the more different interpretations there will be... So just the process of creating these principles can be kind of like a journey to get a common understanding and learn from each other. There will be people with different experiences in the company, and then you can come to a consensus on what these things mean for you.

Birgitta Böckeler: And finally, in the context of change management, principles in an organization can also help you get kind of an official blessing for some of the things you want to do. A good example for that I think is if you have something around experimentation or failing fast, which has been all the rage for a few years... But there are some organizations where their company culture up until that point might not have been like that, where maybe people still feel like they might be punished if they experiment and then it does work. So if you write down a principle around this, and then the leadership in the company or the management actually says "Yes, this is what we want you to do", then it gives people more freedom maybe to really do this.

Sven Johann: Currently, I'm at a customer... They had a very conservative delivery organization, and just by saying "You build it, you run it", basically every team really got the blessing to deploy and run their software on their own... And even if you make some mistakes -- obviously, in the beginning if you're very inexperienced with that, stuff happens... And everyone kind of - I say "kind of" because it was not perfect, but people felt relatively safe to do mistakes. So yes, that's true...

Birgitta Böckeler: Yes, and also the principle might be an opportunity then for the organization to also acknowledge what that means for teams, and what the support will be. When you have "You build it, you run it", you also should think a lot about what your on-call structure will be... And then that can be part of that. People also see now -- management didn't just say "Oh yeah, you build it, you run it. Go ahead", and kind of off-load everything, all the responsibility on them... But you can also show that you thought about the implications of that; again, that's part of that template... And for example, then write down what that means for on-call, and what will be in place to support that, and to compensate people, and so on.

Sven Johann: Possibly which problems will arise in the beginning, but will disappear after a couple of weeks...

Birgitta Böckeler: Yes, yes.

Sven Johann: ...so that people don't go too crazy if problems occur in the beginning. They just know this is kind of a normal thing, and it will be better in 2-3 weeks.

Birgitta Böckeler: Yes. And the implications can also contain things like what negative sides of this might come out... As we know, everything we do in architecture - there's a trade-off for everything, right? So we could also think about what smells might come up, that show us that this is not working, that this is not actually helping us achieve those system qualities that we want to achieve.

Sven Johann: Also, easy decisions are easy because there is no significant downsides to the decisions. But if you have to make a decision where each option has a significant downside; so if you choose one, of course, you have this downside, and then people start complaining about it, but it's good to have this decision record principle which says "Okay, we know this, but it's still the best solution."

Birgitta Böckeler: "It's still a trade-off that we accept", yeah. And I mentioned the word "hypothesis" before... For me, also a principle is kind of -- in the beginning, especially, it's a hypothesis for us, that we think that if we do this, we will achieve this system quality. With the loose coupling, usually that's a bet or a hypothesis that's very well supported by a lot of experience in the industry, that if you have loose coupling, then hopefully you will be able to move faster. But if you end up realizing that it's actually going too far, and you are actually not going faster, or the evolvability of your architecture doesn't increase, then maybe you haven't chosen the appropriate level of coupling, right?

Sven Johann: Okay, so for each principle I need some sort of measurement, if it pays off, so to speak...

Birgitta Böckeler: Yes, so if we go back to that definition that I quoted in the beginning, that a principle is made with the intention of guiding decisions in order to achieve qualities of a system, then it's always kind of a bet on "Ultimately, we want to achieve something else. We're not doing this for the sake of loose coupling, we're doing this to achieve evolvability, or maintainability", stuff like that. And I think that's always important to write that down, so we don't forget that we're not just doing this for the sake of following the canon, or cargo culting.

Sven Johann: Yes, yes.

Birgitta Böckeler: And again, if we have people with different experience levels in the company, if you have for example a lot of people who don't have that much experience yet, there's a much higher risk that they are prone to this cargo culting, because they've heard "This is how you do it, so I'll just build super-small microservices all over the place." Especially for those people, it's important if you have these more experienced owners of these principles who also try to share some of their experience and previous learnings from their careers with everybody, and remind them of what the ultimate goal of this is.

Sven Johann: Yes, and the fact in almost every organization that you have different experience levels.

Birgitta Böckeler: Yes.

Sven Johann: One thing you said also - the principle is a way of learning together... So there is someone who thinks about this principle, and then there are teams who might not really understand the principles. Maybe some do, others don't... If you have, let's say, five or ten teams, how do you distribute such a principle, and how do you make sure that it's properly understood? How is this distribution and feedback mechanism working?

Birgitta Böckeler: In terms of the distribution, again, the first step is that you don't just have the titles, but you have a bit more information available when people look it up. A recognizable visual is also sometimes helpful... One way to structure principles is this form of having three columns; in one column you write down the business goals of the organization, in the second one you write your principles, and in the third one you have a list of practices that actually bring the principles to life. That kind of neatly ties it all together, as in your business goals show you, again, some of the rationale, why you're doing this, and the practices bring it more to life, and the principles are in the middle.

Birgitta Böckeler: If you have these three columns, and maybe make a nice, neat table view out of it and design it a little bit, and then you put it up around the spaces in the organization... If you're collocated, of course; it doesn't work if everybody is remote, unfortunately. I haven't thought so much about how to do it there... But yes, a recognizable visual helps. And everybody knows "Ah, there's this thing." So ideally, it leads to - when people are taking decisions, that they're like "Oh, I saw this thing flying around; those three columns that are supposed to help you with this", and then you go and maybe look things up.

Birgitta Böckeler: If you have something like an engineering town hall maybe, you could also try and find ways to remind people of these things. For example, if you know about things happening in teams that can be related back to the principles, you could point out examples every now and then, like success stories, or also things where people tried something and it didn't work.

Birgitta Böckeler: And then, of course, if you have a role, something like a chief architect or a head of technology - maybe there's a role in the leadership of the organization that feels like they're the owner of these principles, it's also sometimes worth maybe going around the teams and having sessions with them about this, including explaining to them why you're having the principles in the first place.

Birgitta Böckeler: So all the reasons I've mentioned before, why that'd be useful - maybe in your case change management is your most important one... And then you explain that this is not a comprehensive list, but this is what you want to focus on as a group of teams at the moment, and then if people understand why they're there, and that they're not there to control them and to check -- you know, somebody's not going to come by and ask "So this principle - how have you implemented it? And if you haven't done this, why have you not done this?" If they know that that is not what they are for, but that they are for sharing knowledge and for helping them, then they will be much more open to considering how this will help them.

Sven Johann: Yes, that's a good point. It's a fine line between saying "Here's some guidance" and then at the same time making it happen that the guidance is used correctly, but teams are not feeling they're controlled, or something... Because then they immediately move into a defensive position.

Birgitta Böckeler: Yes, almost like doing some user research... Let's say you're the owner of the principles, and either there's (like I said) a single role or a single person that feels responsible for them, or maybe there's an architecture guild who as a group feels responsible for them, then the people on those autonomous teams - they are the users of the principles... So maybe every now and then you can do a bit of user research, like "Was this helpful for you?" or "How do you consume them? Do you even know that they're there?"

Sven Johann: Yes... We made both experiences - the good and the bad. When it comes to going to teams -- we are a very distributed team, so we visit teams and basically explain what we try to achieve... But also show that we are not perfect; that's what we think, based on our limited view... Because you know, if there are many teams, you are not in the details of everyone... "This is what we think we should do, and that's how you can do it. But what do you think about it? Do you think that works for you?" And if you have this communication style and this behavior, we could experience that people try to achieve something, and they want to work with us... And we have other examples where people show up and say "Here are the requirements. This is what you have to do. We make a status meeting every now and then, and then you have to prove that you made it right." Obviously, that didn't work.

Birgitta Böckeler: Yes, I think one of the key questions to ask is "Does this help you make decisions?" Again, if you go back to the definition - this should be a tool for guiding decision-making. So if teams are taking decisions and they're not looking at this, or they don't think it's helping them, then it's not yet helpful, right?

Sven Johann: Yes, the people who create those principles - they are kind of a service organization to the teams. I think this service organization - I learned it from someone from Google; if you want to do something at Google, you have to be this service organization and you have to prove that what you're doing is valuable to development teams, whatever you offer. And for architecture principles it's similar, I think.

Birgitta Böckeler: And then sometimes you have to acknowledge that one of them maybe is not useful, and you have to scratch it. And then for us as human beings it's very hard. We might have developed some relationship to this particular principle, because we've written all these sentences, and we thought this was a really good idea. But then we have to acknowledge to ourselves - "No, this doesn't help the team, so maybe it's not worth it", or "I have to change it."

Sven Johann: Yes, exactly. It's hard to let go... So I said that we are a service organization to teams, but who are we? So who creates those principles? I think you already mentioned a technical leader of an organization - is it a chief architect? Is it a group of people?

Birgitta Böckeler: So definitely I would say it shouldn't be one person by themselves, because for something this important that's probably never a good idea. You should always involve more people than just yourself. And yet, depending on the size of the organization, you might have a role that feels responsible for this, like the CTO, or a chief architect, or a head of technology... Or you might have a guild, where as a group you might create them.

Birgitta Böckeler: I think it's also at some point really valuable to include business or product people, because they are doing a lot of the prioritization in the backlogs of teams. Another very common problem - developers and teams complain that they don't get enough time for removing technical debt, or working on things that "just" improve the architecture, and might not have direct, obvious impact on what the users are seeing... So if you include product people in this, I think you get multiple things from that. One is that you really have to make sure that you can explain the value of this to them, which if you cannot do that, then maybe you haven't understood it enough yourself, or there's still something missing there... Because again, your ultimate goal is that the architecture serves the business, and not that it wins a beauty prize at a conference...

Birgitta Böckeler: And then if you have them on board, it also because easier for teams to talk to each other - developers and product-side - about priorities and why something is important that might be more on the technical side than a user-facing feature.

Sven Johann: I think that's really difficult. In one of our episodes we had Philippe Kruchten, one of the old, big people of software architecture, and he was also like "Yes, explaining the value of software architecture - that's always difficult." But you have to. The episode was about technical debt, and he said "In most organizations, the only way to get your technical debt paid back is create an item, put it in the backlog, and try to convince -- you have to be able to explain to a product owner, for example, why this is important. If we don't do that, then the following happens." The business needs to understand why they have to pay money to do that. I think for principles it's the same. If you can't explain it, then maybe you shouldn't do it. There are always exceptions, but in general I think it's -- there will be someone who's asking "Why are people stealing time for playing around with technology, and not implementing features?", so you really have to bring business people on board, I think.

Birgitta Böckeler: And I also think that as developers there are a lot of situations where we tend to lay around or optimize technology that maybe isn't the highest priority in that moment... And involving product people more and really being forced to discuss it with them and make them understand might actually make us realize "Oh, this is actually really not that important right now."

Birgitta Böckeler: A really great example of that I think is security. It's something that -- we're all, as non-security expert developers, kind of insecure about it as well, maybe... So there's so many things that could be secured, and then sometimes we just run at this one thing and make it really secure, and we just throw the word security out there and the product owner gets scared and says "Okay, do it." But if you do something like threat modeling not by yourself as developers, but you actually involve product people because that's an integral part of threat modeling, where you go through what are actually the potential attacks that could happen, and which of our assets could be compromised by it, which of our data stores", for example, and then you think about how probable is this, and what would actually be the damage to the business - and this is something that product people can usually evaluate a lot better than us as developers. And then involve them in that, and really evaluate all those probabilities, and so on - that helps us as a group to feel a lot less anxious about this, and really do better prioritization for the big picture, not just for that one security loophole that I happen to know a lot about, so I try to fix it.

Sven Johann: Or because I always wanted to do that...

Birgitta Böckeler: Yes...

Sven Johann: Yes, that's a good point... Especially security - it's a hole where you can throw in a lot of money by scaring people. We had the discussion on getting started with principles... It obviously takes time to distribute a principle, to explain it to all kinds of teams; if you have ten teams, you basically have a lot of work... So you cannot just start with ten principles at a time, because then everyone goes crazy, because the whole world is only about principles. So how do I prioritize principles? And what's a good size of principles to get started with?

Birgitta Böckeler: Yes, not only to get started with, but maybe for a longer time, as well. I almost wonder if ten principles - even if you start with less and then extend it to ten - might even be too much. Because as I said, my personal attitude towards principles - I see them as this change management tool, and for that, you don't want to have ten things that you focus on at the same time, right? It's usually less. So my gut tells me that maybe 3-5 are a good number to get started with. Then, of course, maybe at some point you feel like "Oh, this was useful. Let's extend it to more." But then I would maybe consider throwing some of the older ones out.

Birgitta Böckeler: So if you take that attitude towards principles, that it's for change management - like I said, others might think of it more as this governance thing, and then they might want more of these things, and they might want a more comprehensive list.

Birgitta Böckeler: So then how do you prioritize them? The first thing to have in mind would be what are the ultimate goals of my principles - again, the system qualities that I ultimately want to improve. And for that, ideally you know what your business or organization's goals are, and you know what your architecture requirements are. Or maybe a better term for architecture requirements would be cross-functional requirements.

Birgitta Böckeler: You, for example, have already thought about what in the area of security and compliance does your architecture need to fulfill; performance, or availability. So if you've already defined those things - and this exercise might actually force you to do that, because then you have to think about what your ultimate goals are - then can help you figure out what you need to focus on... Because you might realize the areas in which you are not fulfilling those requirements yet, and then you can come up with focus areas that will help you improve that.

Birgitta Böckeler: Another way to think about prioritization or to get started with prioritization is if you maybe start with kind of like a retrospective on how your software delivery is going at the moment. You think about what is going well, what is not going so well... So you come up with your strengths and your weaknesses. And then based on that, you can evaluate what is a higher priority to get rid of one of the weaknesses, for example... And what do you not have to focus on, because you're already really good at continuous delivery, for example.

Sven Johann: Yes, I think it hurts a little bit... So if your personal favorite is continuous delivery, and this works well and other areas don't work well, and then you have to say "Okay, we give up on getting even better at it, and move to something I'm not really interested in, but we really have to do it", that's also something of letting go...

Birgitta Böckeler: Yes... And it might not mean that you totally give up on getting better at something that you're already good at. It might just mean that we don't put as much focus on it. So if you have two ideas and one of them is a way to make one of your focus areas better and one of them is to do yet another tweak in the way you do continuous delivery, then maybe you spend time on the other thing.

Sven Johann: Yes, basically that's also how we do it. Different people are interested in or lead different cross-cutting concerns, and if let's say your cross-cutting concern actually works quite well already, and then take back and say "The time teams spend on architectural qualities should now be in another area, because 'my area' already works reasonably well, so we need to give other people the opportunity to improve their concerns."

Birgitta Böckeler: Yes. And one way to make that easier - let's take the example of performance. Let's say you have a public-facing website, and you have certain requirements for your performance of that website... And it might make people anxious if you say "Okay, this is not our main focus at the moment", because maybe it's good at the moment, but every day people are pushing you code, right? So anybody across like five autonomous teams who somehow can affect that website's performance might push something today that makes it worse, right? But then before you say "Okay, this is not our main focus right now", you should at least make sure that you have something in place that measures your performance regularly... So you can actually see if the changes all of those teams make to the architecture and to the code have a negative effect on it. And then when you know you have those measures in place and you actually get an alert that tells you that this is going down, then -- yes, maybe it's kind of institutionalized that way. This performance principle - you don't have to write it down separately, because you have all these things in place that make it part of your rituals, and it happens anyway.

Sven Johann: Yes, exactly. That's a good example, because that's exactly the example we have... For performance, for example, you could do all kinds of things, like automated performance tests in your continuous delivery pipeline, or nightly -- whatever... But maybe it's just enough to, let's say, have a service-level objective on performance, and have an alerting on it, and just a run book if an alert goes off, and that's it... And then people basically know what to do.

Sven Johann: Is there some sort of lifecycle of a principle? Can principles be retired? We already said that a principle can have a higher priority at the beginning, but once it works, we just lower the priority... But do we also retire principles, and say "We don't do this anymore"?

Birgitta Böckeler: That's an idea I have in my head, but I've actually never seen it happen... But that is something that I would imagine that you -- again, in this goal of principles for change management, at some point maybe you can say "Oh, this is successfully institutionalized now. We don't need it anymore." Or your architecture requirements might have changed. Your business might have changed, and you might not need it anymore. Or the details changed.

Birgitta Böckeler: As you know, I'm a consultant as well, and I usually stay with an organization 12-18 months, and I've never seen one retired. But that is a nice success criteria that I would have in my head, about a principle like "It wasn't success if you feel like you don't need it anymore."

Sven Johann: Yes, it became normal. Everyone is looking at you if you don't do it.

Birgitta Böckeler: Yes. And then throughout the life of a principle, I would say that if you have those details, like the rationale and the implications, I would also say you have to have an eye on maybe updating those implications with maybe more guidance, or more examples that happened over time, so that you have this little history or this organizational learning for each of them, and when new people come in, they can see what happened with this over time.

Sven Johann: Yes. Okay, now the last question from my side - is this for everyone? If we have multiple teams, a large organization -- I mean, it doesn't have to be large, but let's say minimum 20-30 developers; if I have 100 developers it's mandatory to have that. But do I also need principles if I just have a ten-person team which sits in one room?

Birgitta Böckeler: Good question. I think it could also help, especially for a benefit I talked about around consensus, and kind of getting on the same page about what is meant by a microservice, or what is meant by loose coupling. Of course, I don't know if you have to write down principles for that. You can maybe have those discussions in a different way... But if you have those discussions in some way, it's sometimes useful to write down what your conclusions were.

Birgitta Böckeler: Sometimes you can notice that this is missing if you feel like you're having the same conversation on the team over and over again. Every time you have to make certain decisions -- let's say you start a new service and you talk about what it's going to look like, and you feel like déjà vu all the time, like "Haven't we discussed this before?", then sometimes it helps with some of those, if you can bring it back to some basic principles, if it feels like you have to figure it out again and again. Then that might be a sign that it's worth writing some of those things down. If you call them principles or something else, that's a different story, but it might be a good label for it.

Sven Johann: Okay. Alright, so then, Birgitta, thank you for your time. I really enjoyed the conversation.

Birgitta Böckeler: Me too, thank you.

Sven Johann: Good bye.

Birgitta Böckeler: Bye.