Transcript
Stefan Tilkov: Welcome, listeners, to a new episode of the CaSE Podcast, another conversation about software engineering. My guest today is Mike Sperber. Mike, welcome to the show!
Mike Sperber: Hello! Thank you for having me.
Stefan Tilkov: Mike, let's start off by you introducing yourself. Who are you and what do you do for a living?
Mike Sperber: I'm Mike Sperber, I'm heavily involved in functional programming, and have been so for most of my professional life. I started out as a researcher, and currently I am CEO of a software consultancy in Tübingen, Germany, where we do all kinds of project development and training using functional programming.
Stefan Tilkov: The topic of our podcast today is going to be “Does functional programming matter?” This is actually the result of an interesting conversation we had. We've known each other for a while, and we had a conversation over Twitter about whether or not the style of programming - functional, imperative, whatever - matters if you look at it from a larger distance, matters in the greater scheme of things. You definitely do believe it does, and I was sort of skeptical, even though I'm a fan of functional programming. So that's going to be the topic of our conversation. Maybe we should start by defining what it is we're talking about... So what is functional programming?
Mike Sperber: The term originally -- of course, it says "function" right there in the term, so originally the term was defined as programming with functions. I think this term was coined in the '90s to distinguish itself from object-oriented programming. There, functional programming meant you could have functions as objects. Most object-oriented languages have acquired functions as objects one way or another, so it's no longer of much use for distinguishing it.
Mike Sperber: I would say these days functional programming has two distinguishing characteristics. One of them is that we program with immutable data. That comes from the other definition of functions - you have functions in mathematics, where really you just have things that transform input into output, and you don't have the side effects and the manipulation of state that is typical of object-oriented programming. So that's one thing. Pure functions and immutable data.
Mike Sperber: The other aspect more has to do with the actual practice of functional programming - we really believe in high-level models. A good functional programming language will let you achieve abstraction at a level that is maybe one or two layers higher than an object-oriented language, and we believe that that is tremendously useful in modeling. So we use what's called higher-order abstractions that involve functions, and we also use techniques from mathematics and modeling, which is very different from the culture in object-oriented language, which is more rooted in looking at real things... If that makes any sense.
Stefan Tilkov: Okay, let's maybe elaborate on that a bit. What kind of languages are we talking about here? What are good examples of functional programming languages that you refer to here?
Mike Sperber: Good examples are Haskell, for example... F# is a popular language on the .NET framework. On the Java platform you have Clojure and Scala, there's Racket, there's Erlang, there's Elixir... These are all typical functional languages. OCaml I should mention...
Stefan Tilkov: So for the purpose of our discussion, does it matter which of these languages people use?
Mike Sperber: They all let you do things very differently from object-oriented languages, but they do differ significantly. First of all some of these languages are typed and some are untyped, and the typed functional languages typically have very fancy typed systems, if you will, that let you do a lot of the modeling using the types built into the programming language... So usually, the models that you build in typed languages differ from those in untyped languages, where you make active use of the fact that they're untyped. Erlang, Racket and Clojure, for example, are typical untyped languages, whereas Haskell and F# and OCaml are typed languages.
Mike Sperber: So that's one thing that matters... It really depends on your style of doing business. I'm not very religious about the whole static vs. dynamic types discussion, and we aren't, as an organization.
Mike Sperber: The other distinguishing characteristic is the platform that it runs on. For real-world project that we do, generally a large part of the decision on which of these languages we pick comes from the platform that the customer runs on their site. For the Java platform we would pick between Clojure and Scala. On the .NET platform F# is pretty much a given, and then the other platforms might be native code platforms where efficiency might be an important factor, or other aspects that have to do with platform, operating systems and so on.
Stefan Tilkov: Okay. If I look at this number of languages that we're talking about here, I completely agree that they vary very much... So is it a useful thing to talk about how modeling changes if you use a functional language versus an object-oriented language, or should it be more about whether you use a language that has a static type system, versus one that doesn't? Or is it a mixture?
Mike Sperber: I think there's distinctions in both places. Generally, when we do modeling in functional languages, there's more commonalities than differences. And the difference to typical object-oriented modeling is greater. Having said that, the difference between static and dynamically-types systems is still significant enough to be worth talking about. But definitely we always do things in a different from what we would do in object-oriented settings.
Stefan Tilkov: I think we're going to get into more detail about that, but I find it very refreshing that you're unreligious about this whole thing. So it's not going to be like a Haskell zealot kind of talk... Not that I know any Haskell zealots, of course... Not that they even exist, of course. But let's keep the discussion very down to earth; that's actually what intrigued me in the first place. My organization has been applying functional programming as well, but most of the time - at least that I know of; I may not know some of the things that some of my colleagues or co-workers are doing, but most of the things that I'm aware of, the choice of programming language is more of an implementation detail. It's not unimportant, definitely not, but it's not the most important decision. The most important decisions are other kinds of decisions, like "How do you separate the system into larger subsystems?" or "Where does it run? How does it manage its persistent data? How does it communicate with the outside world?", all those things.
Stefan Tilkov: What you very strongly seem to suggest is that it matters a lot in terms of actually building representation of the users or the domain expert's model. Can you elaborate a bit on that? How does that change?
Mike Sperber: The most visible change just comes with immutability. When you're doing object-oriented programming, you tend to see everything as objects that have encapsulated state... And independent objects, in particular, that send each other messages. So that's a very particular outlook on the world around us, and that's very different from functional programming, where we think more in terms of our perception of the entire landscape of what goes on around us. We tend to think in terms of consistent snapshots of our entire system, rather than individual objects that have state that changes independently.
Mike Sperber: That's one change that has a lot of consequences, and I find it hard to overstate those. Those have many architectural consequences on things like testing, on things like coupling... For example if you have state in object-oriented systems, they typically tend to imply a lot of coupling between objects that you don't really see in the signatures of your methods and your UML diagrams and what have you. So that's a very practical and down-to-earth consequence.
Mike Sperber: As far as domain modeling is concerned, for some reason we really tend to try to write things down in code, that at least in their structure and layout correspond to what a customer would tell us the structure of their domain objects is like... And I rarely see that in object-oriented systems. We tend to somehow always these evolving objects that change... Even though this idea that you would have a domain model reflected in your software that reflects the thinking of your customer or the domain expert is really an object-oriented idea. So in that sense, functional modeling is really just a hardcore implementation of this idea that we've always had in object-oriented modeling, if that makes any sense.
Stefan Tilkov: Let me start with the first half of that argument, the immutability thing. I think that the common wisdom - at least in the object-oriented languages that I'm familiar with; and I'm not a huge fan, by the way... I'm not like an object-oriented zealot; I actually like functional programming. But even within the object-oriented space I think immutability is now considered to be a very good thing. So if you can make something immutable, you should make it immutable. If something is of value, then it's of value, and the value doesn't change. It's something that refers to that value that might refer to a different value at another point in time, and then that gives you that value object pattern that's very dominant, very visible in the domain-driven design community, for example... But it has also been for a number of years, maybe a decade or more now, in many projects. Does that change anything?
Mike Sperber: That's much older than that though, actually... If you look at the design history of object-oriented programming per se -- there's a paper by Alan Kay on the history of... Alan Kay was the inventor of SmallTalk (or one of the inventors of SmallTalk) and coined the term "object-oriented programming." And if you look at a paper he wrote on the history, he said "Our goal with object-oriented programming was first to make a more flexible version of assignment (his word for things with state) and then eliminate it altogether." So we're really talking about the natural continuation of the original ideas of object-oriented programming... And that is much older than just the last 10-20 years. That's more like 40 years old, that idea.
Stefan Tilkov: I'm sure that's true. All I wanted to say is that specifically in the Java space, which I'm most familiar with--
Mike Sperber: Yes, somehow that dream got lost, right? We've got Java, right?
Stefan Tilkov: Yes, okay... So I love Java bashing as much as the next person, but I still think that -- you know, even in C++ or in Java you can program in very different styles, and the mainstream style has changed within the last years, and I think it's changed to words becoming more focused on immutability wherever possible... Because people in the object-oriented community definitely know or understand the value of immutability, I think. So they make everything immutable that they consider to be a good idea for. So there is some movement in that direction - I think probably everybody can agree on that.
Mike Sperber: Absolutely.
Stefan Tilkov: Now, the hard part is how do you deal with the cases where you wouldn't make something immutable in an object-oriented language, even if you could? In Java you can just have a constructor, no setters; it gets its value once it's constructed, that's it... But you can't do that with everything, because there are some things that -- or of course you can, but you wouldn't, if you're building an idiomatic program.
Mike Sperber: So where do you think you wouldn't...?
Stefan Tilkov: Well, I think we're going to talk more about this... Let's take domain-driven design and the terminology from there as an example. In domain-driven design a value object would follow the immutability pattern, but an entity or an aggregate wouldn't. So it would allow you to reflect with meaningful methods, not just simple setters - not like having data holders, or anemic data models; it would allow you to do meaningful things to that object that might or might not change its internal state, which serves none of your business if you just invoke that method. What's wrong about that?
Mike Sperber: I think what's wrong about that is really that idea that I described earlier - you have entities that are independent of one another. In domain-driven, which I have to admit, I don't understand that well, you try to have transaction boundaries in your system that manage dependent mutations in your objects, but it's always a headache. You have this entity, it has state, you call a method on it, that state changes... And in a concurrent world which we're in, people are going to look at that object and will hopefully see a consistent state, but you really have to take special care to make that happen.
Mike Sperber: Generally, if you look at mutable objects, there's all sorts of consistency issues that you deal with. As you pointed out, in object-oriented programming, it's long been considered a good idea to keep many things immutable. Functional programmers are just much more radical about that entire idea. So in order to represent what DDD considers an entity - especially if you're dealing with pure languages, or largely pure languages like Clojure or Haskell - you don't even have an option of putting a setter in there, or putting a method in there that will mutate your object. So really, we tend to think of this as "Well, we're modeling our perception of some real world or domain entity." The state of that entity changes, and that means we create a new representation rather than changing the old one.
Stefan Tilkov: Okay. So anything that you want to do, any sort of business logic that you write that does something meaningful will always return a new object or a new data or record (whatever it's called) as a result of that function, right?
Mike Sperber: Yes.
Stefan Tilkov: Okay. So I can see the benefit in that. I can see that you get rid of the mutability problems, or the problems connected to mutability. And maybe you should explain why that does not create a severe headache in terms of copying stuff all the time; why is that not a sever performance headache.
Mike Sperber: Yes, that's a good point. Very often when we have a data structure, even in object-oriented programming, that's internally represented by pointers. When we have a person, an object, and it contains an address object, or field of the type address, and the address itself has street number and so on, that typically does not mean that there's going to be a copy of the entire data structure inside the person object; instead, there will be a pointer. Right now I'm not talking about specific programming techniques in C++, where you maybe would copy that...
Mike Sperber: First of all, that means that if I create a new object from an old one to represent a new state, I typically only have to do -- well, on the implementation side you would call it a shallow copy. So I don't have to copy all the dependent data structures. And I really don't have to copy them because they're all immutable. I don't have to be afraid that somebody's gonna change the address of something underneath, so really it's safety using it in that way. So that means there is not all that much copying involved as you'd think.
Mike Sperber: Also, modern functional languages have highly-efficient (what's called) functional data structures - their implementations of things like lists, or maps, or sets are functional in a way that if you just create a map from a new one by changing one entry, it will share most of the storage between the old and the new version, and it will do so quite efficiently. Moreover, for quite a while now, especially thanks to Java, people have implemented highly efficient memory subsystems in the garbage collector, for example in Java... Deals with many new objects being created quickly quite well.
Mike Sperber: So this overhead from functional data structure -- I mean, there is some overhead, definitely, but that overhead tends to be smaller than we think... Even though we think we have a lot of experience doing that. So we don't see that that's being a factor.
Stefan Tilkov: I've seen these data structures - you call them functional data structures, and I know them as persistent data structures... Do you have any idea about that term - is that the same? It's the same thing, right?
Mike Sperber: Well, it depends on what era... The old algorithms community calls them persistent data structure, but then people think of databases... So the seminal work is just a book by Chris Okasaki and it's called Functional Data Structures; it lays out the land on this... So that's the term that we tend to use.
Stefan Tilkov: Okay. So let me try to play devil's advocate here again...
Mike Sperber: Please, please.
Stefan Tilkov: ...why all of this doesn't really matter that much. Okay, so immutability is very cool, and essentially just to get used to the idea, if I were a long-time object-oriented programmer, I would just have to get used to the idea that my functions maybe now don't belong to a class that creates instances of that object, depending on the language I use, so maybe put them somewhere else... Everything becomes a parameter, and I return whatever I want as the return value; I create something new as the return value, as opposed to mutating something existent. Now, in that regard, that seems to shift things a bit around, so it's a different place... And of course, it has a lot of effect because of the immutability aspect, because I get a lot fewer headaches in terms of concurrent access to stuff, or weird things happening while iterating over something, and then all of those things just continue to work... And that's pretty cool.
Stefan Tilkov: So how do the other aspects of object-oriented programming get replaced, or what replaces them? Let's say, for example, I use polymorphism to create a flexible representation of the outside world in my object-oriented program. What do I replace that with if I use a functional programming language?
Mike Sperber: I don't think you replace it with anything. You evolve it, and it just happens that you get different ideas. To give one classic example from functional programming, which is on the representational financial contracts... So in a former life I actually worked in finance on these things... So you have a complicated financial product, and essentially what it will do is it will generate payments over its lifetime according to complicated rules. In the old world, in the object-oriented world, you would represent this contract as an object, and that object would sort of mutate its state over time to track what payments have already been done, or what date it is, or something like that... You would call a method on it to evolve it over time.
Mike Sperber: So the actual semantics of the contract would be described by the implementation of a method that would mutate the internal state of a contract. So you would have an algorithm, if you will, implemented as part of the method, and you would have some state of the contract sitting in the class that represented the contracts. And that's all good and well...
Mike Sperber: In the late '90s, a couple of functional programmers got together with a trader - or the other way around - and thought about a different way of representing contracts. First of all, they didn't even have this idea of using an object with mutable state to represent the contract lifetime. What they thought about was "Well, what is a contract? Can we describe it in a high-level way that will make the structure and the semantics of the contract clear, that will create reusable components?"
Mike Sperber: So they didn't think about what happens to the state of that contract as the defining characteristic of the contract, they thought of "What is that contract?" It seems a little silly... So what they did is they developed a little library of building blocks for contracts; there's a couple of primitive components, like there's a fixed payment of 1 Euro, or something like that... And building up from that you have combinators; that's a defining characteristic of functional models, where you build larger contracts from smaller contracts. You start with a tiny contract that says 1 Euro, and then you have a combinator that says "Well, any contract I have, I can scale it by a certain factor.” So I could scale something by 10, and suddenly I have 10 Euros... I could combine it with another contract of (whatever) 10 Pounds, and reverse the direction of that.
Mike Sperber: So I get a toolbox of small building blocks and combinators that allow me to build smaller contracts from larger contracts... And that is a very powerful idea. It could be an object-oriented idea, but I've never seen the object-oriented community within the banks that I worked for come up with that. So that gives you a static view of the structure, and also the semantics of a contract... And that leaves open the question "Well, how do you deal with the evolving state?" You still have state that you model, so when you deal with that state... Well, we had that idea - you have an object, it has some opaque, internal, encapsulated state, and with this idea of contracts, they came up with a completely different idea that over the lifetime of a contract -- well, something happens; the contract generates a payment, and it leaves a residual contract as things happen... So the contract sort of over time gets smaller and smaller, until it expires and it goes down to zero. That's given us a very different view of how these things should be modeled.
Stefan Tilkov: I think maybe the analogy of the way -- I've seen it done a number of times... The first half of your explanation is actually that people sort of create an object-oriented representation of a model that represents the contract language. It seems to be that it's just one added level of indirection here... Because if I understood you correctly, what you would do in your version is you would essentially write down the contract in a pretty human-readable form, because you would have those building blocks, and the main expert would be able to see that that's sort of the point of the whole thing, right?
Mike Sperber: Yes, exactly.
Stefan Tilkov: And in an object-oriented language I think you would see an object-oriented structure being created, that represents sort of the same thing. You'd have some "add clause" or "add rule"... It would all be object-oriented programming, but it'd be used to create something that would then be interpreted to do the same thing... So it's at a different layer, but it's sort of the same idea of getting something into the hands of the domain expert.
Mike Sperber: Yes, so as I said, it's essentially an object-oriented idea, but this is not what I've seen in practice. In practice, people were all obsessed about the state of the contract, rather than the contract itself. And this is a phenomenon that I've seen over and over again. For some reason, a lot of the semantics of the domain within object-oriented systems would not really be in the objects and their structure. But the semantics would be in some methods that contain code that would manipulate the state.
Mike Sperber: The code, by definition, is this opaque thing. You can't look into it, you can't analyze it, you can't generalize it to other aspects... In a bank, for example, you want to interpret a contract in many different ways, depending on what department of the bank looks at the contract. And if you just have a piece of code describe the semantics of the contract, then there's just no way to introspect and to look inside it.
Stefan Tilkov: So is that something that you do often? First of all, I don't know how true that is in languages other than, for example, the Lisps. The Lisp family I know a little bit, so I think the distinguishing characteristic here is that code is data, and that you can actually manipulate it using the same kind of actions that you would use to manipulate any other kind of data. Is that true for all the languages, or does it even matter? Or were you referring to something else.
Mike Sperber: No, no, no. I was referring to the object-oriented way. A contract would produce a payment, and the way that would work is you would call a method that said "Give me whatever happened today", and it would return the payment and would advance the state of the object. So that would, for example, be sort of a back-office interpretation of a contract. That is the interpretation that the customer sees. It generates payments.
Mike Sperber: Other departments - for example risk control - is going to want to know how that contract behaves on average, or if certain market scenarios happen. And for this, it would be very beneficial if we could look at the internal structure of the method that implements the semantics, but we can't. So the functional model that I've just described as the entire structure of the contract and its semantics is described by the structure of an object that I can look at. So you hear me say "object". It really is an object-oriented idea.
Mike Sperber: Of course, I could build the same kind of object, I could build that in an object-oriented language, and everybody would look at that and would say "Well, this is a really good object-oriented model", but I just rarely see that in practice; I see it happen much more frequently in the context of functional programming. What happens is the functional languages sort of enable you to do that more easily, and that gives you a little bit more free brain capacity to think these things... And it also gives you more succinct and more direct abstractions to think in terms of, that you can use. So if you will, the made-out building blocks to build the building blocks that you then use to do your domain model... So that gets us, I think, to the subject of abstraction, at some point.
Stefan Tilkov: What I think I hear you saying - and just using different words than I'll probably use - is that a functional programming language lends itself very nicely to building something like an internal domain-specific language (DSL)...
Mike Sperber: Yes, and we do that all the time.
Stefan Tilkov: ...which is a very natural thing. You can, of course, do the same thing in an object-oriented language, but it's less idiomatic and it's a little harder, and I would completely agree with that. And I agree that a program that heavily relies on this idea of an internal DSL really looks and feels and behaves different than one that doesn't.
Mike Sperber: Yes.
Stefan Tilkov: Okay, we have agreement. Very good. So elaborate a bit - before we go to the other part - on the stating. I'm not sure I understood that. What you were saying was that each action or function that you apply to the contract would give you a residual contract. What do you mean by that? How do I have to think about that?
Mike Sperber: It's kind of hard to explain in audio, but I'll try. Imagine you have a contract that says "I will pay you 1 Euro each week, for the next eight weeks." Imagine somebody at the bank manages that contract. They look at it today and they say "Oh, it's payday. I've got to give Stefan his due today..." And that means I generate a payment. At the same time, there is what I call the residual contract, which says that there's only seven payments left. So I could just treat that as a new contract that sort of comes out of the old one by removing that first payment.
Mike Sperber: So as the contract's lifetime progresses and time passes - we started with a contract that has eight payments, then we get a contract that has seven payments, six, and so on; it goes on to zero at the end. But each stage here we just represented as a new contract, whereas the object-oriented systems that I've seen would typically just say "We just have a state variable that says what day it is, and each time you asked it what's going on, what's going to happen today, they would look at the same eight payments, because these were in the original contract. They would look at the same eight payments and say "Well, this payment is over, this payment is over, this payment is over and this payment is over. Oh, and the next one is that one."
Stefan Tilkov: Okay, I understand this concept. So how would you persist this to some sort of external data storage, like a database? Wouldn't you have to create something at the database that looks eerily similar to the state and the object-oriented program object?
Mike Sperber: Well, it is state. We just deal with it differently. But this is really easy to put into a database, because you just find a serialized representation and stick in a field. So that's another major difference; I'm not sure how much that has to do with object-oriented programming, but when we put things in the database that have that kind of structure, we don't try to reflect that structure in the structure of the database schema. That would be pretty much impossible, because as I said, you have what we call combinators; so you build a contract from two smaller contracts, for example... So there's a kind of recursion in the data definition for that, and your typical SQL schema description language just doesn't have that facility. So we just serialize it, stick it into a database field, and databases are perfectly fine dealing with that.
Stefan Tilkov: Okay. That's a very significant different to the approaches I know. Okay, so I agree this has nothing to do with object-oriented versus functional...
Mike Sperber: Yes, this has more to do with enterprise database systems.
Stefan Tilkov: Maybe, yes. And it's a bit like "What is the common way of doing things?" I'm sure you will find object-oriented enthusiasts who will strongly agree that this whole object-relational mapping thing has been and always was a bad idea... But still, it's sort of a mainstream way. So many people - not all, but many people building object-oriented programs, specifically enterprise programs, actually use that pattern and have a strong relation between the relational data model (or whatever database model it is) and the objects represented there.
Mike Sperber: Yes.
Stefan Tilkov: So what you're saying is that, at least in the functional world your inhabiting very different approach to that... And with that I would completely agree. Going back to our initial discussion, that is a very significant architectural difference. So if that is the consequence, then things are different. I'm not saying that they're better in your version of things, because I've seen lots of problems with serializing programming language state to databases... But that's a different discussion, I think. That's more of an architecture discussion than a programming discussion.
Mike Sperber: Sure, yes.
Stefan Tilkov: Let's get back to what word you used - and you used it as if everybody know what that is, which was the word "combinator". Can you elaborate a bit on it? Because that seems to me as something that you think is very significant and makes a lot of difference in the way we program. Not just in that example, but in many examples.
Mike Sperber: Yes. We look for that, always.
Stefan Tilkov: So what's a combinator?
Mike Sperber: What's a combinator - a combinator is a function that takes two A's and produces an A, where A is some domain, object, entity, value or something like that. So that's sort of the purest form - you combine two things into a larger thing... And the significant thing is that really the same kind of thing comes out as goes in. Here's a significant difference in practice - if you have any enterprisy database-based system that you look at, it's typically going to have a hierarchy of domain entities. So you have an insurance company and it has some building block for a contract, and then it has a contract, and then it has a portfolio, so you have a hierarchy of terms... And usually, what happens is that when you want to combine things from a lower part of the hierarchy, you will end up in a higher part of the hierarchy. And this never loops back.
Mike Sperber: If a portfolio is at the top of your hierarchy, there will typically not be a way to combine two portfolios into a portfolio or into something larger. So this hierarchy or this pyramid will have a tip, at some point... And that, in our experience, first of all creates a proliferation of domain language, where you have a lot of words that don't really describe fundamentally different things. Also, it creates a very rigid structure. And the day always comes when a customer wants to combine two things that can't be combined within this hierarchical taxonomy.
Mike Sperber: So there's general things that we try look at in functional programming - we try to reduce the number of different domain entities that are in our system, and then we always look for combinators to combine two of those things into yet another thing. Then I can combine that thing with another thing, and I can do that indefinitely. So that leads to structurally simpler models, first of all, and it also leads to much more flexible models, because typically -- well, this restriction doesn't exist. And usually, what also happens is that the domain model will strongly suggest combinations that don't yet exist in the customer's idea of the domain, but that could... Just the language of enterprise database schema thinking just hasn't been able to express it.
Stefan Tilkov: Can you walk me through an example of where this hierarchy occurs, so that we can contrast the two approaches?
Mike Sperber: Yes. My favorite example occurs in -- we do a lot of business writing systems for semiconductor fabs, for systems that control semiconductor fabrication. Making a semiconductor means that you go through a lot of steps; the microprocessor goes through about a thousand production steps. And what you can have is within those one thousand production steps you might have a subsequence that if you started, it has to finish within a certain amount of time, otherwise some chemical process will destroy your wafer.
Mike Sperber: So the way to think about it - the way that traditional systems in that space work is... So this sequence of steps is called a route, and a route consists of steps, and then these so-called Q-time zones that describe that subsequence that needs to finish in a certain amount of time. So there's a couple of steps, a Q-time zone, a couple more steps, another Q-time zone, a couple more steps and another Q-time zone.
Mike Sperber: So now you have two entities that are hierarchically arranged. So a route consists of Q-time zones, but a Q-time zone can't consist of routes. So to us, when we started writing a functional model for this, it strongly suggested to us that we should be able -- so when we think of "Well, what does a Q-time zone consist of?" and the traditional model said "Well, a Q-time zone consists, again, of production steps...", but the funny thing is that we can have production steps in a route and a Q-time zone, so that really suggests that those things should be the same.
Mike Sperber: So a a Q-time zone should not just consist of individual production steps, but really that a production step should either be an individual operation - you know, put something into a machine, take it out, apply some chemical process to it - but should also be able to be a a Q-time zone. If that didn't mean anything to you, that model immediately suggests that we should have Q-time zones nested inside of Q-time zones. Or from a combinator perspective, that we should be able to combine multiple Q-time zones into one.
Mike Sperber: So that's an idea that came out of the programming language representation of these semiconductor fabrication routes, but that weren't present in traditional models that customers had of that route. So we went back to the customer and said "Well, our model suggests that Q-time zones should be able to nest. Then we could have a combinator and we would be able to build much more elegant software. But does that make sense in your domain?" They said, "Yeah, absolutely." If you think about it, it does make sense that you might have sub-sequences that have their individual constraints on time.
Mike Sperber: So that's my favorite example where we would gain new domain knowledge almost, just because we expanded what we could represent in our software model from what previous models were able to do.
Stefan Tilkov: Okay, so I think there are two ideas in there. I may be wrong, but one of them might be achievable if you weren't using functional programming. Let's say if I analyze the domain, I recognize this same relationship that you recognized, that there is this structure, and I simply decide to arrange my domain entities in a composite pattern that allows -- I have a super-abstraction of some kind that's the generic step (or whatever), and generic steps can nest, and obviously these others can nest as well... So I might discover the same relationship, and if I went with this model to the domain experts, they might agree in the same way, and I will have helped them to get a better system.
Mike Sperber: Yes.
Stefan Tilkov: But I do think there's a second thing that you don't get as easily in the object-oriented world, which (I'm just guessing here) seems to be that this combinator isn't really specific to this particular scenario. It seems like the combinator approach is something that you seem to see and apply everywhere. It's like the essence of this could be applied to everything, and it's only complete if it's this way, because you see that there's something missing there. It's like a missing element in the periodic table, or something; it suggests there needs to be something here, and then it's easier to find. Is that a fair summary?
Mike Sperber: Yes, that's a beautiful summary... And it goes beyond that, in that if you have a combinator, and that combinator is a function - that maybe gets us to the next subject - we can then look for mathematical properties of that combinator.
Stefan Tilkov: What are some examples of that?
Mike Sperber: Well, from school we might all remember associativity. We know that addition and multiplication are associative operators; it doesn't matter which way you put the parentheses. So the next step after we've found the combinator is always to look if our combinator is associative... That means we can learn something about the domain, it's a particular way of thinking about the domain, and it also allows us to use the combinators in more flexible ways. Sometimes that is just more domain insight, and sometimes that's very practical things.
Mike Sperber: For example, when you do a lot of aggregation in big data systems and you do that in a distributed setting, then associativity is exactly what you need to do in order to distribute your computation in a tree shape.
Stefan Tilkov: Yes, parallelized. So when you say those are functions, what are typical names for those functions?
Mike Sperber: That are associative?
Stefan Tilkov: No, just the kind of combinators that you find - what are names that you come up with? Or is it always the same names that you use for those?
Mike Sperber: No, it's not always the same name. It really depends on the domain. Sometimes the combinator is just going to be "and", where you have two things. In the financial contracts, for example, there's one combinator that is “and”; so you have two contracts, with certain rights and obligations, and “and” will just combine those.
Mike Sperber: However, in other settings... One classic setting is for example with images. Again, in an object-oriented setting, you tend to think of images -- well, usually the way images are done is you have a canvas somewhere, and that canvas is your object, and it will have methods that will draw pictures on that canvas, that will mutate the state of that canvas to make some shape visible. And again, in functional programming we think of the pictures as objects, and we make pictures by combining smaller pictures.
Stefan Tilkov: Well, you would do the same thing in an object-oriented system as well, I think. You would have an image object, or a picture object, and you would merge pictures...
Mike Sperber: Yes, well - you've learned from us...
Stefan Tilkov: Well, maybe yes...
Mike Sperber: I think this really is, again, just a continuation of that idea of object-orientation, that we have objects to represent the things that we recognize in our domain. You can combine pictures in different ways. There's different combinators that we can imagine, so we can put two pictures on top of each other, we can overlay two pictures, or we can put them beside each other, and then actually we will have different combinators. Then they have to be called something like "beside", "overlay", "above", things like that... And that would reflect what goes on in the domain.
Stefan Tilkov: Right. So in that object-oriented version of the image library I would have a super-class- the basic image interface/class that sits on top or above all concrete implementations, and then that interface would have all of the operations that I can use with images. In your functional model those would just be free functions, that can be applied to things, to the left and to the right of that combinator, to yield something new, that will be the same thing...
Mike Sperber: Yes, yes.
Stefan Tilkov: So again, it's possible to do the same thing in the OOP system...
Mike Sperber: Absolutely.
Stefan Tilkov: ...but it's maybe more common in yours. Okay. And I understand, I can see how if you give them names that you already know the semantics of (like “and”), then it's kind of obvious that associativity is one of the properties that you should have if you call it “and”.
Mike Sperber: Yes, yes.
Stefan Tilkov: So what about the other properties?
Mike Sperber: Associativity is always the first that we look for. We look for a combinator, we look for associativity, and then we sort of go up a hierarchy. The next one up would be to look for what's called a neutral element. Functional programming lends itself to performing algebra, so we look at algebraic structures for inspiration. There's an algebraic structure called a monoid. A monoid is just something where you have an associative operation that has a neutral element. A neutral element is, for example, when you do addition, then that would be zero. You add zero to a number, you get back the same number. So it's neutral.
Stefan Tilkov: So is zero the monoid then, or...?
Mike Sperber: No, the zero is the neutral element of the monoid. In algebra usually you have terms like monoid, or semi-group, or group, or something like that, and what that usually refers to is you have a set of things or a type of things or a universe of things, and you have some operations on those things. As I said, the first operation that you always look for is a combinator - something that will take two things and produce another thing.
Mike Sperber: And then you will look for equations that describe how that operation works. For example, a simple equation is a+0=a, for any a, for example. That's really what makes up the monoid; it says "Well, you have a set, you have a combinator, you have one particular element, and you have two equations that describe what a neutral element is." And another one for associativity, of course.
Stefan Tilkov: Okay.
Mike Sperber: And if I can point out one more thing, it's that sometimes you will come up with a combinator that is not associative, but that could be, if you twiddle with it a little bit. That's usually a sign that you should actually change your combinator to be associative. So that really is a powerful design guide, if you will.
Stefan Tilkov: Okay, understood. Are there other relevant properties that will help you?
Mike Sperber: Yes, there are lots of them. Whoever has been in an algebra class, in math, might recognize that there's a hierarchy going on. So there's a semigroup, which says associativity, then there's monoid... Then you might also look at groups, but groups are already considerably rarer than monoids are, for example. But you might have a commutative monoid, which is also very useful, where it says a+b=b+a. So that's how you go up in the hierarchy.
Mike Sperber: The next level that we tend to look at -- so I have this next rule, after "Look for the combinator", it's "Look for a functor." Everybody has seen a functor, I think, in a modern object-oriented language - you have a collection of A's, a list of A's, and it will have a map operation. Of course, they come from functional programming originally... And the map operation says "Well, I have a list of A's, and I have a function that turns my A's into a B", and that will apply that function to each element of that list and give me a list of B's back. So it will sort of apply a function inside a container, inside something that contains A's.
Mike Sperber: That's a very general idea. If you look around, for example in the Java class library, you will find this map operation in classes or in interfaces that ostensibly have nothing to do with each other. But it's always the same idea. For example in Java optional and stream both have a map method, and you recognize very related signatures there. So this idea also has a name, and that name is called functor. So that's the next thing - look for a combinator, and then once you've found the combinator, you look for a functor.
Stefan Tilkov: Wait, wait, wait. I need to ask. So I'm familiar with the map thing; I think everybody who's used a modern programming language is that. They don't even have to be functional programming enthusiasts, because every mainstream OO language has those things as well. But you seem to, again, be suggesting a different thing than just using that as something that's in your collection class library, right?
Mike Sperber: Yes, absolutely.
Stefan Tilkov: What you're saying is you're discovering those things at the domain.
Mike Sperber: Yes, exactly.
Stefan Tilkov: So help me find a map operation in a business domain. I understand the generic map operation that's sort of defining the context of lists or data structures, or any sort of data structure where there's set semantics...
Mike Sperber: Yes, so it kind of surprised me too, because as I pointed out, it's very generic... I just briefly wanna hook to your question, which is too look for equations. This map - it's not just a type signature that map has, it also needs to fulfill two equations that come with functor. And one of the simplest equations is one that says if you apply the identity function, if you map using the identity function, you get the same thing back.
Mike Sperber: So by having said that, how do you discover a functor in your business domain? Well, the way to do that is -- so you've got a colleague, Lars Hupel, who has a great talk; he said "Better a type parameter than no parameter", or something like that. And that's what you need for a functor - you need a type parameter. So what you do is you look in the type definition for your business domain entities or object and you look for something that would be a good candidate for a type parameter. That might be something like the currency if you're doing a deal and you finance things. That suggests that it could be a different thing.
Mike Sperber: I always do a domain example where we model animals on the Texas highway, and it's supposed to demonstrate this idea of functional programming, where you have state by just always producing a new object. I never thought about it in these terms... And then somebody said "Well, always look for a functor. Where is the functor there?" The idea with these animals is they all have a weight field; they all have a certain weight, and then there's a function or a method on these animals that feeds them.
Mike Sperber: What you then do is you stick a type parameter there, you stick a generic there where the weight used to be, and then that is immediately a candidate for "Oh, okay, now I've got a type parameter. I could try implementing a map operation", and that turns out to work really well. Then you look for what you can do with that. It will give you something else, namely that where there was a concrete type before in your business domain representation, it now has a type variable. So it doesn't say what it is anymore.
Mike Sperber: If you have an operation that doesn't put a concrete type there, you will have learned something, namely that that operation will not mess with that particular field. So that gives you insight again. Maybe a small morsel of insight, but insight nevertheless about your domain object. So you just stick a type variable there, see where it takes you, and in surprising ways it might actually work out... And it often does.
Stefan Tilkov: So it's taken you until minute 50 to completely lose me, but you have now, so you need to catch me up.
Mike Sperber: Okay, ha-ha!
Stefan Tilkov: So what is it that you're saying? Walk me through that animal example again.
Mike Sperber: The animal example - yes, let me do the complete example; it's not that complicated. So on the Texas highway there are two kinds of animals. One type of animal is an armadillo, and each armadillo has two properties - it's either dead or alive, and it has a certain weight. There's also rattlesnakes. Each rattlesnake has a certain length, and also a certain weight. So they have a field in common.
Mike Sperber: So in the initial version of that domain model, that weight field would have type double, or int, or you would define a type using units, or something like that. Or maybe not units; you would just stick an int there, and say "That means that many kilograms." Now, this idea of "Look for the functor" suggests that you take out that int type and you put an A there. You just stick a type variable or a generic there, and then you change your type to no longer be animal, but animal a. Does that make sense?
Stefan Tilkov: That makes perfect sense, yes. I parameterized my class or my system by--
Mike Sperber: It's rather arbitrary, but weight seems a little bit more important than the other things in that domain model, so that's a good candidate for sticking your generic thing there. So I get immediate advantages from that, somewhat surprisingly. As I said, my students took me there... So one of them is that a DDD expert is going to look at this and say "Well, you stuck int there, and I don't know what that int means. Does it mean kilogram, or does it mean gram? What's the unit here?" So I could then immediately stick a type there that will be more specific about the unit. So that's an immediate benefit that is not abstract at all. It also allows me -- for example, there will always be a method that would feed the animal, and that would mean it would affect the weight. There would also be a method that would run over the animal with a car.
Stefan Tilkov: The poor thing...
Mike Sperber: Poor thing, right? So again, because we're doing pure functions, there's a lot of things that we can learn from the type signatures of the functions that we apply. In this case, the feed animal function might have type "animal of weight to animal of weight", where weight will be a specific type, if that makes any sense.
Stefan Tilkov: That makes sense, yes.
Mike Sperber: Okay... Whereas the run over method -- it just so happens running an animal over does not affect the weight... So it will have a type that says "Animal of A (where I don't care what the A is) will go in, and animal of A will come out." So that means looking at the type signature I already know - just looking at the type signature, not looking at the implementation, and with no documentation at all - that running over an animal does not affect its weight at all.
Stefan Tilkov: Because you don't depend on the type parameter.
Mike Sperber: Yes, because the function doesn't know anything about the type. Whereas the feed animal - it might not, but it probably does, because it says a concrete type there, for the weight. Moreover, of course, that feed animal function can be implemented with a map function that we've just defined. So that would be a simple example of how to look for where the value -- where you derive a small value from finding a functor. So that's another worthwhile thing you can do... And again, it certainly leads to even more surprising results when it's successful than finding a combinator.
Stefan Tilkov: Okay. Other things to discover in this regard?
Mike Sperber: So once you have a functor, you have a type variable - so you start with looking for the type variable, as Lars says, then you look for the functor, and then there's a hierarchy that goes up from there. That culminates - you always listen for the m-word, the monad word... So for a monad you need a functor. You start with a functor, and then there's additional properties and equations that you discover, and ultimately you end up with a monad.
Mike Sperber: In between there are several other terms. I don' think it makes sense to discuss what they are, but they are useful... There's something called an applicative functor (or we sometimes just say applicative), there's something called an arrow, and there's various variants of that in between.
Mike Sperber: The cool thing, again, about these, is that each of them supports specific operations, like map... For example, a monad will have an operation that in old-style functional programming we would call bind, but these days it's often called flatmap; you find flatmap in the Java standard library. Again, in different places... And it also has some equations that describe how flatmap should behave.
Mike Sperber: Based on these simple operations that are immediately connected with these abstractions, you can build libraries of more useful, but very general abstractions. So again, that gives you tools to think about your domain. So if you discover a monad in your domain - and it does happen, occasionally - then immediately it raises the question "Oh, what's the flatmap operation?" I mean, usually you find the monad by finding the flatmap operation... But monads have a join operation. The typical join operation on lists would be if you have lists of lists of A, if you have nested lists, you can just append all these lists into one. So you take the lists of lists of A, and you can get from that a simple list of A, with no nesting.
Mike Sperber: But above all, it raises the question "What does it mean to be a list of list of A's, or does that concept even map to something?" And when you find that in your domain, you think of "Well, what does it mean to be a something of something of something?" One classic example, again, is of images. If you generalize images using this idea of looking for the functor, you generalize over what's in the pixels, and put the type parameter there, where the color of the individual pixels is... And you get up to monad; when you get up to monad, you raise this question "Oh, but now I have images of images, where there's an entire image in each pixel, if you will." That, again, leads to useful ways to thinking about your domain.
Mike Sperber: What's important really, I think, is that these things - I can write them down in a functional language because of their abstraction capabilities. So I can define what a functor is, at least what the operation's type signature is... For example, in Java I can't write it down, so I have to put it in a comment. Having these things at my fingertips in the language really not just enhances what I can do when I program, but it also enhances what I can do in my brain.
Stefan Tilkov: So if I understand your argument correctly, what you're saying is there is a different domain, a rich domain of the mathematics or the patterns underlying functional programming/modeling, and if you bring that knowledge to a different domain and merge the two of them, you actually can see where there are interesting things that could happen. I find that very convincing. Very good. So is that the same thing as what you call denotational domain-driven design, or is that a different beast?
Mike Sperber: It's the beginning of that idea. The general idea of denotational domain-driven design is that you look for a representation of the entities in your domain, and then you look for mathematical properties of those objects. So denotational domain-driven design puts a heavy emphasis that you try to come up with a representation mainly with the methods of mathematics... Given the fact that mathematics really is the primary tool that for many centuries humanity has had for modeling the hard facts of the world around us.
Mike Sperber: You don't necessarily look to it with an eye of creating a concrete implementation. It just so happens that because functional programming languages use mathematical ideas in their structures, you can often implement those ideas anyway. Coming back, for example, to that classic image example - a cool representation you can use for images is just a function that will translate coordinates into a color, as opposed to an array of pixels, or something like that. So that's an idea that's very mathematical; it's not geared towards producing pictures on the screen at all, even though you could make it do that.
Mike Sperber: So if you do that, if you map it into the world of mathematical constructs, then usually we have at our disposal this rich set of things that we can look for, this rich set of properties. These include things like functor, or monoid, or something like that. But denotational domain-driven design means you look for that representation, that representation is called a denotation, which is why it has that name... And then you look for properties.
Mike Sperber: Denotational domain-driven design adds one more element, in that a lot of mathematical abstract structure already has a lot of properties. To give a concrete example, you have the option type in Java. It's an option of A, and it might have something in it or not. So that A might form a monoid. You might be able to combine two A's into one. So if the A's form a monoid, then you can make option of A also into a monoid. So the structure will sort of fall out. You just say "option" and you immediately know what its structure is like as a monoid. So there's a lot of structure there.
Mike Sperber: So functions have a lot of structure, just compound data where you create two polls have a lot of structure... And in mathematics we have a library of the structure that these various mathematical constructs have, that we can immediately draw upon. So we don't need even need to look at "Well, how can we make a monoid out of this thing?", instead we just construct our domain denotation in such a way that the monoid will be obvious and implicit in the construction. That really dials the abstraction one level higher, and it will often allow us to discover even more stuff about our domain by mapping these abstract things back into the domain.
Stefan Tilkov: Very cool. Okay, so I think we've reached our time limit... I'm sure we could continue for about 25 hours very easily, but we won't. Instead, I'd like to ask you - are there some resources that you can point our listeners to? What's a good way to start, if one wants to find out more?
Mike Sperber: Yeah, so if you really want to start out in functional programming then I've got a shameless plug. If you google for my name, you will come to a page called Dein Programm, which is a program I've been running for many years for German speakers.
Stefan Tilkov: We'll add that to the show notes, of course.
Mike Sperber: In a former life I actually did a lot of research on teaching how to program, and surprise - that ended up being related to functional programming as well... And there is a lot of free material. Particularly, there is a book I'm currently rewriting, and will hopefully finish in the next couple of months; you can always get the current PDF for free from that website.
Mike Sperber: There's also the racket system that I mentioned earlier. Originally, it was conceived as a system for teaching how to program, so you can download that and it comes with special teaching languages for functional programming... So that's a great way to get started. And it also has a lot of links. In general, after you've looked at the material there, you're probably going to be interested in a particular functional language, and then really a not too bad approach is to just look at the homepages for any of those languages, which will typically have at least a tutorial and some more comprehensive documentation on what goes on there... And there's lots and lots of books.
Stefan Tilkov: What about the modeling side of things?
Mike Sperber: Well, good point... So there's not as much there as I would like. A lot of the modeling folklore that I talked about in this podcast actually comes from scientific papers. A lot of the knowledge that we have on how to write functional programs was really driven by the research community, and published at research conferences, and in research journals. That unfortunately means that a lot of books on functional programming focus on programming in the small, not so much on real-world domain modeling.
Mike Sperber: There's a wonderful book by Scott Wlaschin called Domain Modeling Made Functional. I think that goes in that direction (which I like), which shows how to link the concepts from domain-driven design, and do that in F#. But for the higher-level modeling stuff that I've just described there's no book yet, so I guess I'm gonna have to write one after the current one is done.
Stefan Tilkov: You have to. Very good. Thanks a lot, Mike. It was awesome to talk to you. I learned a lot, which is always great.
Mike Sperber: Yes, it was my pleasure, too. Thank you very much.
Stefan Tilkov: And thanks to our listeners for listening. Until next time. Bye-bye
Mike Sperber: Bye.