Adam Tornhill on Software Design X-Rays

Transcript

Intro

Sven Johann: Hello, everyone. Welcome to the GOTO Book Club. I am Sven Johann. I work for INNOQ. I like improving systems and I like to talk about it, to teach it, to write about it. And today I have, as a guest, Adam Tornhill. Adam, would you like to introduce yourself?

Adam Tornhill: Thanks, Sven, and thanks for having me here. I'm Adam Tornhill. I'm a programmer and a psychologist. I’ve been working in the software industry for over 25 years, and I spent the past 5 years building a platform called CodeScene working with code analysis. I'm also the author of a couple of books and I like to think that the best-known books are "Your Code as a Crime Scene" and "Software Design X-rays."

Sven Johann: Actually, quite an interesting combination. How did it come that you studied computer science and psychology? It's a rare combination.

Adam Tornhill: It might be. I started to work in the software industry in 1997 and after a couple of years, I kind of noticed a pattern that succeeding with software at scale is incredibly hard. I saw one software disaster after another. At some point, I just wanted to try to understand why it is so hard to write good code. And I kind of decided to seek the answers within psychology because psychology has a lot to offer to us in our traditional technical field. You know, psychology is about the way we think, how we reason, how we solve problems and also how we collaborate with others.

Adam Tornhill: Originally, I just signed up for one semester but psychology is really, really fun. So, after that semester, I thought, "Okay, let's dive into one more topic and then one more topic." I ended up spending six years at the university and I took my degree by accident, more or less.

Adam Tornhill: Behavioral code analysis

Sven Johann: Oh, okay. I once did a Coursera course, an introduction to psychology. Basically, it was only the most famous psychology cases, and it was very fascinating. But I stopped after one semester. And I think this rare combination brought, to me at least, a totally new thinking about improving systems. Before I read your first book, "Your Code as a Crime Scene" I used tools like Sonar. I don't think that Sonar is a bad tool. I really like it. It helps you see problems with your code, but it makes it really hard to prioritize your work. If you run a Sonar analysis and it tells you you have 3 million findings and your technical debt is 2,000 years, it's just not helpful. I have to invest a lot of time to understand what's really important and also, even if I figure out what's important, the tool cannot really help me.

Sven Johann: That's how we worked in the past, but then your book came along accompanied by tooling and I think it made it really easy to prioritize. Maybe we should talk about the way of prioritization. You called the type of prioritization, if I say that correctly, a behavioral code analysis. Is that correct?

Adam Tornhill: Yes. That's correct, and it's pretty much about taking my psychology background and putting it on top of the technical work that I do. I agree with you, to me, the whole idea of static analysis is also a valuable practice. It's something that works very well as a low-level feedback loop when writing code. But static analysis, like you point out, has this property that it treats all code as equal because there is no such thing as a business priority when you just look at the source code and that leads to having thousands of findings.

Adam Tornhill: What I do instead is that I look at how we as a development organization interact with the code we are building. In practice, that means looking at things like version control data. You could then combine that with Jira data because then you can calculate the impact of each one of the findings. That gives you a window into all those quality issues and tells you that this is what's important and that this might be problematic long term but we can most likely live with it for now.

Adam Tornhill: So that's one of the things. The other thing that I noticed with static analysis is that once an organization grows beyond, I would say, just a handful of people, then organizational factors like coordination between different teams, team coupling, key personnel dependencies, knowledge distribution, all that kind of stuff tend to become maybe even more important than any technical properties of the code. And static analysis was simply not designed to help us with those aspects, right.

Sven Johann: Exactly.

Adam Tornhill: That was pretty much why I decided to call it behavioral code analysis. It's more about the behavior of the organization rather than the code.

Sven Johann: When I tried to prioritize, I also thought about our customers. You know, a lot of code is often used or executed because of customer use. Would that also be a part of an idea to prioritize by how customers are interacting with the system?

Adam Tornhill: I mean, it could definitely be an aspect. You know, software is interesting because it has power laws everywhere. You look at code and like 80 percent of your code is never touched. You look at your feature set and it's most likely the same thing. Most customers use a smaller subset of your features. So clearly there is a priority aspect there as well.

Sven Johann: Yes. For me it was easy to convince, for example, a product owner to improve code when I could say this is part of the 80 percent usage pattern or something like that.

Sven Johann: You wrote the book "Your Code as a Crime Scene" and now you published a new book, "Software Design X-rays". What was the motivation and the new learning, that you had to write a follow-up book?

Adam Tornhill: "Your Code as a Crime Scene" is a book I wrote in 2014, and what I did back then was to capture a number of techniques that I had used myself in my previous life as a software consultant. Techniques for prioritizing technical debt to communicate with non-technical stakeholders. It's stuff that I found useful myself. I published this book and it got quite a lot of attention. I realized that if I want these techniques to become mainstream, then there has to be some good tooling around them. So shortly after publishing that book, I founded CodeScene, the company to build those tools.

As part of those early years with CodeScene, I was fortunate to work with so many organizations across the globe on using different technologies of all kinds of scale. I learned so much from that experience. Basically, in "Software Design X-rays" I try to reflect on those learnings: how well do these techniques work in practice, and even more important, how do you act upon them? Because information is only good when actionable. So that was the main motivation.

Sven Johann: I liked a lot about the book, that it usually has recommendations on what to do with those findings, but we can dive into that a bit later.

Sven Johann: For me still, the most fascinating part of your work is the hotspots and X-rays. So, what is a hotspot?

Adam Tornhill: A hotspot is complicated code that we also have to work with often. It's a combination of technical factors like code quality issues and design problems with frequent development activity in that part of the code. That's a hotspot. And the whole hotspot concept came out of my experience of using static analysis to try to prioritize findings. We talked about it earlier, I simply lacked the priorities from the business side, and I also faced these challenges of communicating with non-technical stakeholders, because very often I try to explain that, "Okay. We have the business pressure for this feature but maybe we should take a step back and improve what's already there so that we can implement it safely and on time."

Adam Tornhill: I found that those conversations were extremely hard to have because software simply lacks visibility. There's no way I can take my software system, pick it up and show it to my non-technical stakeholders. At the same time I was wrestling with these problems, I was also in the middle of my psychology studies and I took a bunch of courses in forensic psychology. And that's where the hotspot concept actually came from.

Adam Tornhill: I was inspired by a technique called geographical offender profiling that is used in forensics. What you do is basically you look at the geographical distribution of different crime scenes in a city or in an area and then you mathematically calculate a probability surface that can kind of help you predict what's the home base of the offender so you know which area to patrol and supervise.

Adam Tornhill: When I learned about that, I think something just clicked. I thought, "Wow. What if we can do the same for software? What if we can take this large software system, project a probability surface onto it and say, hey, these are the parts of the code that are very likely to change in the future too and these are the most stable parts of the code?" Then we can suddenly get some real priorities on what needs to improve and where code quality is most important. That was a very lengthy answer to a simple question.

Sven Johann: You do that by combining static code analysis and mining software repositories. So, code that has a lot of commits and that's kind of problematic that we should work on. For me, it's so easy now to explain, let's say, to business stakeholders, why this is important because of technical debt. I know the metaphor technical debt lags a little bit with financial debt but still, you can say where we have an interest and we have a principal we have to pay back. And what is actually the interest rate? And you answered the question. The interest rate on code means I have to spend more time thinking about pretty bad code when I add a new feature because the code is not ideal for implementing the new feature. And of course, if I have code that is really bad and I have to change it quite often, the interest rate is very high. So, to me, it really sticks to my mind.

Sven Johann: What is your experience with your customers? I am only one single person who is doing it rarely but you do it all the time. Does that speak to, let's say, non-technical stakeholders all the time or rarely. How does it work for you?

Adam Tornhill: The short answer is that I have a good experience which is great. Otherwise, I wouldn't have food on the table and I'm happy about that. Because of these techniques I originally targeted engineering organizations and occasionally, as a bonus, non-technical stakeholders. My experience is that non-technical stakeholders like it because the hotspot analysis sends a positive message. It basically tells you that, "No, you don't have to fix all technical debt. You have to fix this debt. That's really critical. But you can safely live with this technical debt. You need to be aware of it, of course, but you don't need to pay it down now."

Adam Tornhill: It's a positive message for them because they are also wrestling with these long- and short-term priorities, like you have all these features that the business pushes for and at the same time, there is some awareness that we need to improve things. I like to think that's the main contribution.

Sven Johann: Exactly. There is always something to improve. If someone asks me when we have a software system to improve, there are a gazillion things to do. The question is what brings the most value to your work. You always have, of course, opportunity costs. And, you know, if I improve this, I cannot improve the other thing. I think the hotspot analysis makes our not-to-do lists longer and it shortens the to-do list.

Trends analysis: bad code is more or less bad from the beginning

Sven Johann: What you also have is a trends analysis, and I like trends. If you have a negative trend it tells you you have to improve something. It's also the other way around. How do you analyze those trends? How are you doing it with your tooling?

Adam Tornhill: To me, trends are what make all the data truly actionable. I very often meet organizations that might be struggling with something like a legacy code migration or they have an existing system that they're earning money on. They need to continue to be able to maintain it and you have this constant pressure for new features. So, I always tell them that to manage technical debt, step zero, the absolutely first thing to do is to avoid taking on more debt. Simply put a quality bar on what's already there and make sure it doesn't get worse no matter where you start from.

Adam Tornhill: I think that's again something that resonates with managers and technical leaders because I'm yet to meet anyone that tells me, "We want our code to get worse." You never hear that, right? It's actionable, that's the first thing. The other reason I like trends is because they carry so much more information than any absolute values. For example, in CodeScene, we have this concept of code health. It's a metric that goes from 10, healthy code, that's easy to understand and low risk, all the way down to one which is code with severe maintenance issues. And let me say that I will point you to a hotspot with a code health of five. Is that good or bad? Well, if that hotspot had a code health of eight the week before, then it's disastrous because it degrades rapidly. It's something you need to act upon now. But on the other hand, if that hotspot had a code health of two the week before, then it saw a dramatic improvement that needs to be celebrated. So, trends carry this information whereas absolute values don't.

Sven Johann: When I used CodeScene and I saw my first analysis, I was like, "Okay, now I see the red parts." We also have to say we have some screenshots on that one and you have those parts which are bad code change often. And that triggers me. All the red things, those are the ones which should be on my to-do list. But how do I see, let's say, parts which are good but they are getting worse and worse but they don't show up red on CodeScene? So how do I figure out negative trends which are not horrible yet? Do you have those? Can I look at something there in CodeScene?

Adam Tornhill: Yes, you can. If I remember correctly, I had a chapter discussing this in the "Software Design X-rays". I think it's the last chapter where I talk a little bit about how you can use the trends as an early warning system. I think this is really, really important. It's so important that we spend a lot of effort building it into CodeScene as well. We integrated things like total requests and build pipelines, so you can get an alert when things like that happen. I think it's vitally important because one thing I learned that actually surprised me was that, like so many others in the software industry, we have this idea that code starts out fine and then they would add the pressure of the business and the code kind of degrades over time. But that's not what's happening in general. What's happening, in general, is that if you find a module with quality problems, low code health, most likely those problems have been introduced very, very early on, often in the first version of that module. And once a module degrades in code, that tends to be a self-amplifying loop because the hurdle to refactoring is large. So instead, we kind of squeeze in more complexity and more complexity and it gets worse and worse.

Adam Tornhill: Using this kind of information you can actually set different quality gates and say for new code, this is the minimum level of quality or health that we accept. And then you can use it as an early warning system which allows you to prevent a ton of future problems.

Sven Johann: For me that was a surprising insight from the book that bad code is more or less bad from the beginning. But you also wrote that bad code usually stays bad. So, it's bad, you fix it but then it's coming back. Why do you think that's the case?

Adam Tornhill: I think there are several reasons. It's one of the things I noticed early on, shortly after writing "Your Code as a Crime Scene" when I started to work with this in organizations. I kind of figured out that, we found these really, really problematic hotspots. Then you start to look at the trends and you see that these hotspots have been problems for years. And at first, I couldn't explain that. So why hasn't it been refactored? There can be multiple reasons but one of the reasons that I think is very common is that in those really problematic hotspots, there are some code quality issues that tend to lead to organizational problems as well.

Adam Tornhill: If you look at a different behavioral code analysis, you look at the main contributors and you look at how fragmented the contribution activities to that file are. So you look at a purely human perspective, and you will most likely see that that code is not written by one person. In a large organization, it might be written by 50 or 60 contributors and that most likely means that no, you don't have 50 people that know that code. You have no one that actually knows it because everyone has their small piece of information and the original authors might long be gone. It might often be that we don't really understand the problem domain well enough in that code, we might not understand the solution well enough and that kind of, again, raises the hurdle of refactoring it. And in particular, we don't feel any kind of ownership for us as a team. So that makes us much less likely to invest the risk into refactoring it. That's my hypothesis.

Adam Tornhill: System mastery

Sven Johann: What does it mean for an organization if I have many people working on the code? You said, the problem is not well understood. What are you recommending companies to fix that bad code which is not going away? What can you do to make it happen that it doesn't come back?

Adam Tornhill: I like to think that the single most important thing you can do is to get situational awareness so that everyone in the organization, engineering, business, everyone knows where the problems are, how severe they are and what they actually mean to the business. Because once you have that awareness, the next steps tend to be not easy but doable at least. And then it depends on what kind of issues you have.

Adam Tornhill: Here’s another behavioral code analysis technique that looks at something called system mastery for example. System mastery is not a technical property. It's about how much of the current code is written by the current team. Because I often see this as well, that social and organizational factors like system mastery tend to influence how we perceive the complexity of code. We tend to overestimate the complexity of unfamiliar code. And that means you can hear complaints about, "Hey, this piece of code is overly complicated. It's a mess." Whereas in reality, it might simply be that the team never had a chance to get onboarded properly and understand the domain and the solution.

Adam Tornhill: If you have that situational awareness and know what your actual problem is, then you can also address it and put the right measure upon it.

Sven Johann: One interesting finding of yours for me was, when you analyzed all sorts of systems, you had systems that have code which is, let's say, rarely touched and then you have code which is often touched but only by a handful of people like one or two or three people. And then you have the code which is changed often by many people but also it's not immediately changed. It's changed from time to time and that puts another burden on it. If we have a mental model on the code because we work very often with it, we usually don't introduce parts. But if we rarely touch code, we are not 100% familiar with, then we introduce new problems. That was also something I found quite interesting.

Adam Tornhill: Yes, I definitely think that it's a very common problem. I see that you get lots of contributors on a particular piece of code and if that code is also a hotspot, then the situation isn't particularly good because what it basically means is that even if I write the piece of code myself today, then look at it three days later it looks completely different because five other people have been working on the code, so it's virtually impossible to maintain a stable mental model.

Adam Tornhill: And what's so fascinating about this is that very, very often it's that technical properties of the code possibly lead to organizational issues. So, what you will often see is that when you have that kind of code that attracts many different contributors, it's probably because it has good reasons to do so. It's typically a module that's very low on cohesion, meaning it has tons of different responsibilities. So consequently, different teams working on different features, all end up in the same part of the code because it ties everything together. It's a problem that you simply cannot solve with a reorganization because then you will introduce other bottlenecks.

Adam Tornhill: The proper solution there would actually be a technical refactoring or redesign which would improve the organizational fit. And this is one of the things I found fascinating myself.

Adam Tornhill: Software design X-ray

Sven Johann: Yes. Let’s move the topic to the X-ray. So, what is a software design X-ray?

Adam Tornhill: An X-ray is basically a hotspot analysis at the function level. The reason I started to develop the X-ray analysis was that after writing "Your Code as a Crime Scene", when I analyzed all the systems I noticed that the hotspots that I tend to find that are most problematic are typically really, really large files. It could be thousands of lines of code, occasionally, tens of thousands of lines of code. It can be extreme cases like that.

Adam Tornhill: If you take that large system and you narrow it down to a single hotspot and can say, "Hey, this is your biggest problem." But if I go to a client and say, "Rewrite this code. It's only 20,000 lines of code," they are not going to be happy. It's not actionable. An X-ray analysis takes that complex hotspot, parses it into separate functions and then it looks at the git log. Where do each committed over time? We sum that up and you get hotspots at a method level. And that's what I've been using successfully as a starting point to prioritize refactorings in large, complex processes and modules.

Adam Tornhill: The more metrics the better?

Sven Johann: Yes, if you tell someone that 20,000 lines of code class are a problem, they would say, "Well, we figured that out by ourselves." I think it's also used in many places. For example, Simon Brown has the C4 architectural model and there you also have this kind of zooming or Gernot Starke has arc42. There, we start from a very abstract view and then we dive always deeper. It's like Google Maps when you look at the map and then you can always go deeper and deeper. So, you're not lost in details. You can navigate quite nicely. And I think that's also quite a good idea.

Sven Johann: What I was wondering you use very simple metrics. When I look at a typical tool like Sonar, there are lots of metrics which tell me what is a problem. I think you only use lines of code of a class or a method or intention from lines. So, no dependency analysis and things like that. This is based on some research. Why do you focus on very few metrics and why is that still super powerful?

Adam Tornhill: I like to think we evolved that a lot over the past years but basically, the thinking process is that if you look at the research, you will see that a simple metric like the number of lines of code is a horrible metric. It's really, really bad. The thing is that most other metrics are just as bad. And lines of code is intuitive, it's simple to calculate, simple to reason about. So that's basically where I started out. We need a complexity dimension for hotspots. Let's use lines of code. And it surprised me how far that actually gets us. It takes us really, really far.

Adam Tornhill: One thing I learned after writing "Your Code as a Crime Scene" is also that lines of code works well, it correlates with most other metrics. But I also learned that at some point, you want to dig a little bit deeper and provide some actionable advice around what type of refactorings do you need to do and stuff like that. What I've been working on a lot over the past year is to add different static analysis techniques into a behavioral code analysis via CodeScene. What we tried to do is to look more at the design level like identifying brain methods, drive violations, you know, the stuff that really, really matters for maintainable code.

Adam Tornhill: So, these days I might still use the number of lines of code as an initial visualization, to get situational awareness but when I dig deeper, I typically use this more elaborate set of metrics. I hope that makes sense.

Sven Johann: Yes. So, when I look at the hotspots, I look at lines of code and let's say a cyclomatic complexity represented by intention. If that's the correct word. But clone detection, violating the DRY principle, I was not aware that this is also part of the hotspots.

Sven Johann: Change coupling

Sven Johann: Thank you. In the first episode, we talked about behavioral code analysis, hotspots, X-rays and now in the second episode, we want to start with change coupling and then move on to what change coupling actually means, not on the code level but also on the team level. What is change coupling?

Adam Tornhill: Change coupling is a very interesting software analysis technique. It's interesting because it's something you cannot do just based on the source code. As a developer, we typically refer to coupling as some kind of dependency. We have some piece of code over here and it uses some piece of code over there. They depend upon each other. Change coupling is different because it's more about logical dependencies. In a behavioral code analysis, you measure how the organization interacts with the code they are building. In change coupling, we pick up patterns and it could be anything from the git log to more advanced architecture analyzers, we pick up patterns in Jira later and figure out that each time I need to modify a piece over here. I have a predictive modification in the module over here. So, there is some kind of logical dependency between them. And using change coupling we can get a lot of interesting feedback on how well our architecture supports the way we work with it. So that's the short description.

Sven Johann: We all know the situation where we change, let's say, a piece of code and then we have to change five completely different unit tests because of some copy-paste coding. How does the change coupling relate to clone detection? Because clone detection is the thing I know if I change something and then I have all those subsequent changes.

Adam Tornhill: Yeah, so that's an area where I've done a lot of work. The original challenge I had was that this was way before CodeScene, way before I wrote the book. I was working as a software consultant with a team that was very heavy on test automation. The problem they had was that they got a lot of flaky tests. This was automation at the system level, at the application level. The problem was of course that, as a developer, you pushed some changes, you updated some tests and then you had three other tests failing because they also depended upon some logic.

Adam Tornhill: Now using change coupling, what we could do was we could, first of all, visualize the changes. So, we could figure out that you make this tweak in the application code and now we have to modify this whole cluster of files, right, so that we kind of highlight the problem. But to make it actionable, we wanted to drill deeper and figure out if the reason that we have this change coupling, is due to drive violations, violations of the don't repeat yourself principle. And adding copy-paste metrics on top of this proved to be really, really useful.

Adam Tornhill: To me, change coupling is an excellent way of benefiting from clone detection tools and copy-paste detection tools because the main problem with copy-paste is that there's simply so much of it. So, you look at the research and you see that there's somewhere between 5% to 20% of all code out there is duplicated to some extent. And not all of that application is bad per se. So, using change coupling, it gives us a window into those massive amounts of duplicated code and we can figure out that, "Okay. These are the software clones that are actually expensive to maintain, right, because we have these predictive modifications."

Adam Tornhill: The copy-paste dilemma

Sven Johann: The clone is not an exact clone, right? Usually, you copy something and then you paste it and then you change this tiny little bit. But of course, if one thing changes here, you have to change all of the rest there. Clone detection tools also find those half-baked copy, let's say, copy-paste-edit pieces of code, right?

Adam Tornhill: Yeah, that's right. I think that clone detection software is a pretty old technique. It's something that's been in industrial use for almost three decades. But again, I think that simply the sheer amount of duplicated code makes it very hard to act upon it because to me, I think one of the main problems with copy-paste is that we, as developers, myself included, are kind of conditioned to despise copy-paste. We have learned over and over again that it's a bad thing to do, right? The problem is that it's simply not true as a general statement because let's say that you have a piece of code here. You copy-paste it and now there are these two copies. They evolve in completely different directions. I could very well argue that copy-paste was a good starting point.

Adam Tornhill: Or maybe you duplicate some code and you never need to touch it again. It might not be ideal but is it a really a big problem. So, the problem starts when copy-paste kind of drives us, our behavior, in a negative direction. We have to make those predictive modifications. It's easy to miss updating one of those clones. And that's why I think this combination of change coupling and clone detection really, really adds value to our daily work.

Sven Johann: I agree that it's not always bad. I tend to do first-time copy-paste-edit but when I do it the second time, then it's time to say, "Okay, now I need an abstraction". Daniel Terhorst-North once said "DRY is the enemy of decoupled." So that's something I'm wrangling with but in terms of microservices for example or just different, completely independent modules. Whenever I change something far over here, I have to change something far over there even in a totally different repository. Then things become a bit problematic. I wonder how do I find out if I have a problem because maybe I only have this onetime thing as you said or it's a real problem. How can I find that out?

Adam Tornhill: That's a very interesting question because I think the incredible hype behind microservices over the past five to six years kind of flipped the whole architecture and design thing around a little bit because there are definitely scenarios where you want to violate drive principle because loose coupling is simply more important. It's a more important capability than minimizing the amount of duplication. At the same time, what you want to avoid is to have exactly the same behavior in two different places if that's a behavior that's been frequently modified. Because if you have that, it could very well indicate that maybe you missed a microservice to take on that shared responsibility. Maybe it's a separate business capability, right?

Adam Tornhill: There are two aspects to it. One is getting the data. How do we figure out change coupling across git repositories? The second challenge is how do we act upon it. And I wrote about this in the book. So, change coupling in simplest forms is based on finding patterns in the commits, files that change together over and over again as part of the same commit set. And that clearly doesn't work across repositories. What we do there is that you simply look at a ticket system instead. Things like Jira tickets, Azure DevOp tickets, whatever and see that if the commits reference the same ticket, then we know that they're kind of depending upon each other. And if it happened often enough, then that's a very strong signal. So that way you can detect change coupling down to the function level, even across git repositories.

Adam Tornhill: This is something that's very powerful and you can, of course, add clone detection on top of that as well and then see is this a drive violation that's desirable or is it an actual problem that we lack some level of encapsulation or abstraction.

Adam Tornhill: To me, that it's a hard engineering problem but the more challenging aspect is to make that decision, right. Do we live with this or do we ignore it? What I tend to recommend is I use a general heuristic that I call the heuristic of surprise. I simply visualize the change coupling and then I take everything I know about the architecture and the problem domain and any surprising coupling that I find is usually bad because a surprise is simply one of the most expensive things you can ever have in software architecture.

Sven Johann: Yeah. Rarely people are positively surprised.

Adam Tornhill: Yeah.

Sven Johann: It's always "You know, don't make your manager being surprised because usually, it's negative." But I mean, what would be an example for a surprise. Let’s move that question a bit to the back. You said you wanna show the coupling. So how do you show the coupling? I remember from when I looked at CodeScene and also when I remember the book I show two files, right? And then there is the amount of coupling of those files and how often they change together, so to speak. So, what's the thinking about this approach? You know, showing the files and the amount of coupling.

Adam Tornhill: The interesting thing with change coupling and a lot of those other behavioral code analysis techniques that we talked about in the previous episode like hotspots is that they scale at different levels. You can use change coupling at the function level. You can use it at the file level and you can use it at an architecture level. And that's where I usually start when I pick up a system. I look at the different architectural elements. They could be either different layers, different components or they could be microservices. And I simply figure out the change coupling between, let's say, the different services. So, I typically visualize it. You have all your services like a wheel and then you see the connections between those sections. And that's usually my starting point.

Adam Tornhill: I've been fortunate to work with so many organizations across the globe at all kinds of scale and I see several organizations are doing a really good job with it. But occasionally there is a surprise. And the surprise is very often that even though you put all these efforts into defining proper service boundaries, we know about bonded context, domain-driven design and all that stuff, but that is incredibly hard in practice because we basically have to be domain experts to do that. So, the surprise tends to be that, "Hey, we're actually starting to build our distributed monolith." It's very easy to highlight that with change coupling. When you change the service you have a dependency to five other services and then you look at the details. What kind of functions or couple do you often see that you implement a new capability here on your business capability and to pull that off, you need to query two different services and update the state in a third one. So that's a typical warning sign that I tend to find.

Sven Johann: As we said, maybe we have two code clones in two different services. And then we change it once and that's fine. It just happened once and that's probably not a problem. If we change it the second time, the third time, the tenth time, 100% of the time, we change those files together in different services. Is there any recommendation when I should start worrying about those temporal dependencies?

Adam Tornhill: I find it very hard to give general recommendations in this area because with hotspots it's usually quite easy. If you work on a part of a code a lot and that part of a code has a high degree of technical debt, then it's very easy to say that, "Okay, we need to fix this." But with change coupling, it's so much harder because it depends on the lifecycle of the code base. If it's something you just got started on, there's probably going to be a lot of experimentation and learning going on. So, I do expect a higher degree of change coupling and more complex change coupling.

Adam Tornhill: In a more stable system, once the basic architecture is set, I would be worried about any change coupling that's over 20%. That would worry me.

Sven Johann: Okay.

Sven Johann: Change coupling gets more expensive with distance

Adam Tornhill: I just wanted to add that the other heuristics I use is basically that change coupling gets more expensive with distance. And what I mean with distance is first of all architectural distance like are these two completely unrelated parts in theory but in practice tied together. Then that might very well be one of those surprises.

Adam Tornhill: The other one is dependencies that cross team boundaries. That very quickly becomes expensive because the teams end up in all these coordination meetings and they might have conflicting changes in their API and stuff like that.

Sven Johann: Yeah. I can sing a song about those kinds of problems. I think if you have different repositories and one team is responsible for that, okay. It's not nice. But it's still let's say manageable and it's also easy to fix because it's one team. Just a couple of people own different repositories and can think about reducing or getting rid of that coupling. But across teams that's going to be tricky. So, what are some interesting examples from your experience on that problem, you know?

Adam Tornhill: I have so many. I’m not even sure where I should start. I think one of the most common issues I've seen, and this used to be more common maybe a few years back, like, two, three years ago, was that several organizations that I worked with asked me, "How should we organize our development teams?" So, we started out with component-based teams. Then we noticed that we had very, very long lead times for new features because you had to do these handovers all the time between change coupling components. So, we changed to a feature team and now we kind of noticed that our whole quality aspect just went south because suddenly we find that we have 10 different teams all working the same parts of the code.

Adam Tornhill: It's a much harder answer because the organizational aspect always goes hand in hand with the software architecture. You really, really need to balance these two. And if you want to have component teams, then I think a much better approach is what the "Team Topologies" people would recommend, stream align the teams. That's the only way I found that actually scales. You have the teams separated based on business capabilities and those business capabilities are reflecting the architecture. I really haven't seen anything else that works at scale.

Adam Tornhill: Team structure visualised in the code

Sven Johann: Yeah, I was about to say. In the "Team Topologies" book they have this quote from, I believe, Ruth Malan. There is the software architecture but there is also the communication architecture basically on Conway's Law. And if you, as an architect, don't care about team setup and communication structure between teams, you give the architecture task to someone who cares like a non-technical manager. And then you end up in those problems you described.

Sven Johann: Is there a way how to use those insights on a change coupling based on different teams working on the same code base to, let's say, to refactor teams towards those stream aligned teams?

Adam Tornhill: Yes, there are. And I like to think this is one of the most important contributions by behavioral code analysis and "Software Design X-rays" and what we do with CodeScene as well of course. It brings visibility to the people’s side of code. I always talk about this like the grand tragedy of software design, that the organization that builds the code is invisible in the code itself.

Adam Tornhill: Using behavioral code analysis, you can actually visualize that. You can show which team works where in the source code. And you can overlay that with change coupling information so that you can easily show those bottlenecks that when we modify this service here, that team has to coordinate with four other teams. You can visualize that. And from there, it's mostly about domain expertise to figure it out. Quite often, it's not enough to just shift the teams around. Quite often, you have to do a more fundamental change and maybe those two different services or components are actually the same component, so you need to merge them together. And occasionally, you find that an organization simply lacks a team to take on a new responsibility.

Sven Johann: So it's the sames pattern maybe we discussed the last time where the hotspots actually create a not-to-do list. You have so many problems but with the hotspots, you get a focus on the problem. And here we would have the same. If you need to think about your team setup or your "Team Topologies", to use the word, you can look at those parts where many teams own a lot of the same code and also where are teams actually quite distinct from each other in the codebase. And I can use the problematic parts to identify where we need to create better boundaries, team boundaries.

Adam Tornhill: Yeah, and I think that's important because the moment you're able to visualize that, then you can start to have a meaningful conversation between engineering and the business. In my experience, even with organizations that have a fairly high degree of team autonomy, a team can usually not decide what they want to do. They can decide what they want to do differently but they cannot decide for the whole organization. We need some kind of buy-in and being able to show that data really, really helps in my experience.

Sven Johann: Yeah. If you just say you have a problem or you have a feeling, usually people don't react to it. And in your book, you mentioned that you can test your architecture. Let's say architecture in terms of dependencies on teams. If you implement a new feature, how many people do you need to bring into one room to find out how good or bad your real dependencies are, either your hard dependencies or your soft dependencies? If I want to implement a new feature and I have to call a big staff event, I obviously have a problem. What's your experience when you do those tests with your clients?

Adam Tornhill: My experience is that it's very easy to end up doing local optimizations. Organizations always know themselves that, "Hey, we have a lot of coordination meetings and we have a lot of sync meetings." And what typically happens is that many organizations tend to stuff additional ceremonies on top of that. We have a problem with sharing information for example. So, let's do additional information-sharing meetings. Let's do additional status meetings so that everyone involved is up to date and stuff like that. And in my book, that's basically just the way of covering up the symptoms, not the real root cause. Because you really do want to keep coordination and meetings to a minimum. And the only way of doing that is to make sure that each team can operate autonomously, so that they can have this fit between an organizational unit and the actual architecture. So that's where I come from.

Sven Johann: Yeah. I feel the pain already. I've been in meetings with around 60 coordinators and at least I was thinking, "This cannot possibly be true." How do you visualize those dependencies between teams? To me right now it feels like it's something you should use in each and every project to really reduce your team dependencies.

Adam Tornhill: Yeah, and I find it fascinating in particular if we talk about stuff like microservices where we're very heavy up on the measuring all kinds of things. We are measuring performance, we're measuring scalability and all that stuff. We have an alert system. But then we have these other architectural properties that are so important like loose coupling, independent deployability, autonomous teams and we are not really measuring that. So that's where I think behavioral code analysis can kinda fill that gap. And it can work as a monitoring system, as an alert system for architecture. And that's the way I've been using it over the past couple of years and I have a very good experience with that.

Sven Johann: Yeah. And even alert triggers. What are the typical steps to fix the problem?

Adam Tornhill: So, what actually surprised me the most because it was so unexpected for me was that a lot of those problematic dependencies between different teams or coordination bottlenecks, very often their fix turned out to be technical. It's something we got wrong either in the high-level design or in the architecture. We stuffed too many responsibilities into one module and now multiple teams have a reason to touch that module, right, or maybe we do have a very modular architecture but it's the wrong granularity on it. Maybe it's the wrong modeling concepts that we use. And very often it turns out that, as software architects, it tends to be very common that we identify technical building blocks and we kind of build our architecture around that and that's, like, more or less asking for heavy team coupling.

Sven Johann: Yeah. I like that. Little code changes to fix big problems. I am currently in a project with, let's say, 300 people. Without the data visualization on the coupling it's incredibly hard to have a meaningful conversation about what the problems are.

Adam Tornhill: Yeah. And I think that even minor improvements at that scale have a big payoff because imagine just the staff cost of 300 people. If we can save 1%, that can be quite some party.

Sven Johann: Yeah. Exactly. I mean, coordination meetings with 50 people every once in a while, why not analyze the code base instead?

Adam Tornhill: You could almost buy a private jet for that money you save, right?

Adam Tornhill: Outro

Sven Johann: Exactly. From my perspective, I could have many more questions but let's close it here. Thank you, Adam, for the conversation. I can really recommend not only reading your book but also check out the tool. Thank you.

Adam Tornhill: Thanks a lot for having me. A pleasure.