Transcript
Sven Johann: Welcome to a new conversation about software engineering. My guest today is Peter Elger. Peter is the CEO of fourTheorem, and he is the author of the book AI as a Service, which is about how to leverage cloud-based AI services and cloud technologies themselves to build AI-powered applications. Peter, welcome to the show.
Peter Elger: Hi, Sven. How are you doing? Thank you very much for having me on. I appreciate the opportunity to talk to you and your audience.
Sven Johann: Yes, I am appreciating very much to talk to you about a topic I was kind of blind for until last year, or maybe half a year ago. I mean, not AI in general. We see all those crazy things happening in AI - self-driving cars, Google AlphaGo, the Boston Dynamic humanoids, and stuff like that... But I'm in a (let's say) "normal" business; I work with retail finance, companies like that... What are typical use cases for those businesses when it comes to artificial intelligence?
Peter Elger: It's a great question, and as you say, we've seen rapid and exciting progress in the field of AI over the last few years, really driven by the kind of increase in compute power that's enabled new algorithms and approaches to really come to the fore. And if you go to AI conferences, you'll hear lots of really exciting talks about robots, and machine vision, and all of those great advances... But as you say, for typical normal businesses, those aren't the use cases per se. Really what I'm interested in - and I guess you, and maybe a lot of the audience are interested in - is "How do we leverage this technology to make our jobs better, to deliver better solutions for our clients, without necessarily needing to go and get a Ph.D. in machine learning?" It turns out that that is something that is achievable these days, basically through the use of the increased commoditization of AI services.
Peter Elger: I'll give you an example of some use cases that we've done. One great example would be around KYC (Know Your Customer). That can be a laborious task of having to manually read passport documentation, utility bills and the like. That problem is now certainly very amenable to an AI-based solution, so that almost 90% of that can be automated away.
Peter Elger: Other typical use cases would be things like sentiment analysis, things like prediction or forecasting trends... We're also using it to actually operationalize custom models in the agrotech space, as well as a number of other areas.
Sven Johann: Nice. So utility bills means I never have to fill out my travel costs; I just make a photo of my bill and that's it.
Peter Elger: Exactly. So if I can just take a photo of my bill and then I can have that automatically ingested and checked, and then fields extracted automatically, that's a great case for AI as a service.
Sven Johann: And basically everything you mentioned is available as a service. I don't have to build those AI applications myself.
Peter Elger: That's correct. If you go and look - and I've done this exercise a couple of times now over the last two years... If you go and look at the major cloud providers, if you focus on commodity, you'll see that there's been a growth in the range and scale and scope of the AI services they're all offering. And by major cloud providers I mean Amazon, Microsoft and Google. Apologies if you're using other clouds, but that's really where the arms race in AI is at the moment.
Sven Johann: And probably Alibaba, or something.
Peter Elger: Yes, of course. Sure. But what you'll find is that those services that they talk about break down broadly into two categories. One category of services I would classify as really API-driven and true AI as a service, and the other category would be support for operationalizing AI. I'll explain what I mean by that... Under the AI as a service type category you would have things like natural language processing, so understanding text, text classification, voice-to-text, text-to-speech, personalization engines, text processing, natural language translation, speech recognition, image recognition, and predictive analytics or forecasting using machine learning. And those types of things - those are all very useful in the business - can all be accessed through an API. So either a synchronous API pattern, or for longer-running jobs through an asynchronous type of consumption model.
Peter Elger: The other classification of services is really around the training and operationalization of AI. Things like the ability to run your custom TensorFlow models as a service... I'm talking about things like Amazon's Elastic Inference service, SageMaker, and support for deep learning machine instances, support for TensorFlow out of the box in containers, and so on. So all of those help you to accelerate training in a scalable way on a pay-per-use basis, and then accelerate your delivery of your models to your audience or to your customers.
Sven Johann: You mentioned a few services, and then you said they are tooling for training. I'm just wondering, text-to-speech, or speech-to-text - I can use this API out of the box, right? I probably throw in an MP3 file and I get text back. Or I throw in text and I get an MP3 back, something like that.
Peter Elger: Absolutely. Things like that, that you would consider as commodity, as solved problems. Those are things you don't want to do yourself, unless it's an incredibly specialist use case. And what we're seeing really is a growth in commoditized services. As new AI services are researched and developed, they follow the commoditization path that we've seen with all technology... And why shouldn't they, because they're used in a business context. So really, what you need to be considering when you're looking to apply these tools is "Is there something off-the-shelf I can use? If it doesn't meet my needs exactly off-the-shelf, can I tweak it for my particular domain?"
Peter Elger: An example would be natural language processing. If I want to classify some text as being relevant to a certain area, that may mean I need to do an additional piece of transfer learning on that model, and I can do that purely in an API-driven context. I don't need to necessarily understand the guts of what's going on under the hood with that model. I just need to provide some data that's specific to my use case, and then I can cross-train that model so it works more effectively in my context. But all of that can be done through a set of commodity APIs.
Sven Johann: I've lately seen it that there's a chatbot, and obviously a chatbot is not a generic thing. You have to train it. But the question is -- to me, it was totally unclear how does such a training look like? Do you formulate the typical sentences you expect? Let's say when someone from an insurance or bank just says "Hey, guys, I am wondering what my contract is doing." Somehow you have to teach your service what's the input and what do I want to get out of what the customer says or writes down.
Peter Elger: Yes, I guess with chatbots -- the way that you would use a commodity chatbot isn't really training per se. What you're really doing is supplying a set of intents and thoughts to a pre-trained chatbot. So you're saying "If I get a date field or I get this particular keyword, then I know that I'm roughly in this space, and I will respond in that way."
Peter Elger: So what it does over and above just a rule-based approach is that -- let's take the canonical example of ordering something online, or scheduling a meeting, let's say. Then I should be able to say to the chatbot "I'd like to schedule a meeting." The chatbot should respond "Great, when do you want it?" I should be able to say "Next Wednesday." And then it should execute that and put that meeting in my diary. That kind of thing is very simple to do with commodity chatbot services.
Sven Johann: But what do I need to know upfront if I want to do that? How do I define those requirements? Because in a normal application, I just say "If you click on that button, then the following thing should happen, under certain circumstances." And if I have a chatbot or something I have to train somehow -- I mean, you said it's not training... What did you say it was?
Peter Elger: You're basically giving a set of intents and expressions.
Sven Johann: Intent, exactly. But you kind of have to sit there upfront and think about several combinations...
Peter Elger: To an extent you do, but generally, as you say, it's looser than an absolute set of requirements. But the way to go about that really is to say, if we're taking the specific case of a chatbot, "What is it that I want the chatbot to be able to do for me?" Do I want to order pizza, or do I want to schedule meetings, or whatever that list of things is, then really what you want to do in terms of defining your requirements is "These are the output functions that the chatbot can achieve." And then your test cases - you can put a myriad test cases against that, as long as your test cases then map through, and the execution of what you want to happen has occurred, and you can measure that externally, obviously, by stubbing out external interfaces, and so on. Then that's really the approach to testing and regression testing, and requirements against the chat type interface.
Sven Johann: Okay. I think testing is another interesting topic, because you probably cannot use JUnit and have an equals statement...
Peter Elger: Not quite that, no. Exactly. But again, if you're looking at the chat case, then really what you're saying is -- let's take scheduling a meeting. I want this to schedule a meeting; here's ten different conversations that should end up in a meeting being scheduled. And then by playing those conversations in, I can then check that an API call is made at the end of the process, and that meeting is scheduled as expected. So it can be done, but you're right, it's not really JUnit kind of level testing.
Sven Johann: So it's more that I get a probability back that the person answered -- I think the person answered "tomorrow", "three o'clock", and the chatbot can say that with a probability of 90%, 100%, 70%, or something like that.
Peter Elger: Yes, but your test cases should be the same each time, because if I push my test case in and that's the conversation, it says "tomorrow at three o'clock", then I should be able to confirm that I have in fact scheduled that meeting at that time.
Sven Johann: In the current project we had a recommendation engine, which is obviously also a nice case... And there, the testing was quite interesting. You look at several customers, and you look what they already bought in real life, and then you remove those items from the list and you check if they would be recommended.
Peter Elger: Yes, exactly. You're certainly dealing with a more probabilistic outcome with a recommender than you are with that kind of chatbot that we've just described.
Sven Johann: And a recommender - you probably have to give the recommendation also intent, or some sort of data.
Peter Elger: Yes, of course, you need to feed -- they become more accurate... Well, accuracy depends on the user, at the end of the day, but they certainly become more accurate the more data you can feed in... So that should improve over time, as you get larger and larger datasets.
Sven Johann: So usually, in the beginning, it could be that it's not so great, but over time you get better. Yes, that sounds pretty cool. So when I want to use those services, I don't need to learn multi-layer neural networks with back-propagation, or something like that.
Peter Elger: There's been a lot of recent hype about the technology... It's been around for quite a long time. The research has been ongoing for a long time, from (I suppose) the early '50s... There were attempts to build automimic neurons in hardware, and there were various false starts. You may have come across first the AI winters, where funding for AI just dried up because there was really an overpromise of what the technology could achieve, and an under-delivery.
Peter Elger: I guess if you look at the history, the under-delivery was due to a lack of compute power. And as Moore's Law has kind of progressed over the last few decades, that compute power has become available. So what we couldn't do with AI a couple of decades ago, we now can. That's why we're now seeing this increase in the technology... But it has been bubbling away for a long time... Which means there's an awfully large body of academic and applied research to go through, and it's quite confusing to try and understand all of that. You really need to be deeply embedded in the whole space for a number of years to become an expert... And I suppose the message of the book is that you don't need to do that. A higher level understanding and a focus on the business problem can get results rapidly by using commoditized services, rather than needing that kind of Ph.D.
Sven Johann: Yes. Now I'm happy, because we have all those commodity services, and text-to-speech, speech-to-text. That is something which is really easy to use. Basically, you don't have to know anything, I would assume. You just throw in some data and you get some data back. In other cases, like a recommendation engine or a chatbot, maybe also image recognition, you have to tell the system what you probably expect to come back. So there you need to do a little bit; like what you said with the intent and feeding data...
Peter Elger: Generally, what you do with the image recognition services is to say "Find X number of labels." So you're saying "In this image I only want you to label five things that you find in the image, and I only want you to return things that you find in the image with a confidence interval of greater than 95%", let's say. So you can kind of tune the accuracy in the sense that if I want just Getty's, then I reduce the confidence interval; if I want to be very sure of what the API is detecting, then I can ask for a very high-confidence interval and less labels in the image.
Sven Johann: When I look at bills, for example, for my travel cost, I just have my application and I say "Make a photo", and then upload the photo, and I want to see everything with a probability higher than 95%... But I still need to understand what is really the cost, what is the tax...
Peter Elger: What happens there - actually, this is a very interesting piece of commoditization around this... We did some work for clients about two years ago, where we were doing some KYC work, which was scan a bill and figure out what the fields were. Particularly, this was get name and address, get a meter number, and so on out of a utility bill. Now, we had to do that by using -- we took an open source library that did the kind of image to text (OCR), and then we had to do some math to figure out how the text blocks are related to one another, and identify those text blocks, then run them through a classifier, and then extract the data.
Peter Elger: Now, about a year later, all three of the main vendors released their own form recognition services. Basically, the work that we'd done within a year was commoditized, so we wouldn't have worked in that way had we to do it again; we would have just used the off-the-shelf service. So often just by waiting a while, you can actually get a service that you won’t have to produce and save yourself some time and effort.
Sven Johann: Okay. That means that I basically get two tools in one. So I get image recognition and a classifier, which understands tags, or address, and really finds the value of the tags, or the value of the address in the image.
Peter Elger: Because form processing is such a common business problem, there's value in the commodity providers providing a service for it. It's where you start to get to the edge cases which are much more niche where you have to consider how you do a transfer into that particular domain that you can't just use a commodity.
Sven Johann: Yes. Half a year ago I spoke to a general manager from AWS AI Services. It was at a conference, and he said -- I was like "Wow, this is so cool. This is exactly for me. I just use it, I don't need to know much." But he said "No... You should really understand AI if you use our services." I'm just wondering to what detail do I need to understand it... Because obviously, if I use the cloud, I somehow need to understand how the cloud works, but I do not need to understand how AWS or Google is doing scheduling, or automatically setting up instances. I'm just wondering to what level should I learn AI, so that I can use it safely?
Peter Elger: I think you should understand the basics of what each service is doing under the hood, so you can kind of connect it up to a mental model. One of the important things, of course, is understanding the cost control around how much a particular service is going to cost you when used at scale, or will cost your clients when used at scale. That's obviously an important thing to understand... But I'd kind of liken it to an operating system, in the sense that there are not many developers working today that actually understand the guts of operating system mechanics and how an operating system really works. But most developers understand it at a kind of higher level. They know there's a task scheduler, and there's I/O drivers, and all this kind of stuff. So they have a rough idea... And I think that it's that kind of level of understanding you should get to you with AI - you have a rough idea of what's going on under the hood, and you can understand roughly what the cost model is going to be for consumption of any service.
Peter Elger: Don't get me wrong, it's a very laudable exercise if you want to go off and understand this stuff very deeply and become an expert in it; absolutely go for it. But if your goal is more provision of business value through these services, then a high-level understanding will generally suffice.
Sven Johann: So I want to use the chatbot or the recommender, and the documentation of the cloud provider should be enough. I don't have to buy a big book on --
Peter Elger: It should generally be enough, yes. Of course, it's good to understand maybe a little deeper "What do I think this is doing under the hood?" or if you can understand roughly what it's doing under the hood for your own sake...
Sven Johann: Yes. I mean, it's always good to dive deep and basically -- I like to understand how things actually work in absolute detail, but it's a big difference if you do that for fun or if you really have to. Because if you don't, then you jeopardize your product, for example, the things you are building for a bank, or something like that.
Peter Elger: Another example would be how many web developers actually know in detail what happens when the applications that they build load and execute.
Sven Johann: I don't know, but...
Peter Elger: I mean in terms of how the TCP protocol operates, and so on... Probably most don't. Most don't need to know that, because they're working several levels of abstraction above that. I think it's a similar analogy here as well; you're working at the business level, or a few levels of abstraction above the service. And that's okay, because you can still do useful work as long as you understand the parameters and boundaries of the service you're using.
Sven Johann: In the case of web development, not everyone needs to know it, but they should at least (in my opinion) always have a very senior person which understands everything to a very detailed level, so that you can catch possible problems early enough.
Peter Elger: Well, I'm one of those guys like you, that likes to understand everything at a detailed level... But again, it's a matter of time as well.
Sven Johann: Yes. I don't have to; it's just out of interests that I...
Peter Elger: Exactly, yes.
Sven Johann: That's pretty cool. You mentioned quite a lot of services already, and I have the feeling the arms race, of course, is still ongoing, and every month or every two months a new service will be launched by any of those providers. I think that's also pretty fascinating.
Peter Elger: Yes, as I said, I think it is a bit of an arms race at the moment. Some of it is marketing hype, obviously. Some services are way too thin, but some of them are genuinely game-changing, depending on your situation. One of the other things I'd like to touch on as well is the whole idea of how you operationalize AI. It's one thing to consume a service - that's great - but there's a kind of middle area in between, which is... Let's say that I've done some research and I built a model for a specific domain; how do I get that into production? And again, commodity AI as a service can help with that, because I can go and learn my own TensorFlow models and build a big cluster, or something like that... Or I can just use a service to execute my model. Amazon has something called SageMaker, Google and Microsoft have something very similar, where basically you can take your model and put it into context and have it execute, and have the cloud provider take care of a lot of the non-functional concerns around scaling and so on.
Peter Elger: Let's bear in mind as well that it's not just operating and scaling the model; there are all of the non-functional concerns around "Well, how do I get data into the model, effectively? How do I get my results out? How do I monitor what's going on? How do I alert when it fails? How do I do a new build and get a new model into production?", because you don't just build a model and leave it for the next five years. Just as you are with code, you're constantly improving the parameters, the model, so you need to keep updating your core service, so that requires a CI/CD pipeline for your AI service, in the same way that all of your other pieces of your system need CI/CD. So there's an entire workflow and dev ops practice around this that you need to adopt to be effective.
Sven Johann: But what is not quite clear to me is when I use AI as a service, and let's take the chatbot... I mean, it can be the chatbot or the recommender, where I really have to do something - not much, but I have to do something; obviously, I try those things out on a testing environment from AWS, or Microsoft, or Google, and then I have to get my intents, or my data, or all that - I have to deploy to production. That's how I understand it. So whatever I'm doing on my testing environment, I theoretically can use the web interface, but it's better to use the API and propagate what I'm doing with the API in terms of putting in data, or intent, or whatever...
Peter Elger: Oh, absolutely. All infrastructure is code. That applies here in the same way that it applies on all on-cloud work. Tools like Terraform, or serverless framework... We've been using the CDK, which is a more imperative way of doing infrastructure as code... But yes, absolutely; all of these things or environments - they all need to be committed into GitHub, and all of your parameters around this need to be version-controlled in the same way as you would with any service that you're deploying.
Sven Johann: Yes. The web interface, those services offer - they are just there to get everything known, and get a little feeling for it... But once you have it, you have to put it into code.
Peter Elger: Absolutely, 100%. Yes.
Sven Johann: You mentioned cost... Usually, when we talk about cloud, we are all very happy until the first invoice comes, right? Once the first invoice is there, we are maybe not so happy with the cloud anymore. We obviously have to do something to really leverage the cloud, and the elasticity, and stuff like that... But for AI services - I would guess it's a pay per use model. Or does it depend? Probably it depends.
Peter Elger: No, it's a pay-per-use model. I mean, for the API-driven service it's pay-per-use. If you're using those at scale, there's a cost, obviously, so you need to consider the same things as you would with any cloud platform - putting in billing alerts, making sure you're in control of the costs, making sure you've got a good estimate of what the cost base will be before you go to production, and monitor that as you would with pretty much any service.
Peter Elger: Where you can get a lot of costs is in training. Training is a fairly computationally expensive thing to do, so if you're going custom, you can definitely leverage that scalability and compute power, but keep an eye on the costs, because that kind of mounts up fairly quickly. Running models is less computationally-expensive generally than training, but inference still comes at a cost, so you should be looking to optimize that inference, and there are a number of additional services you can use to help you optimize that.
Sven Johann: What is an example of cost? Let's say I have speech-to-text, and I make an API call. The question is "How much does one API call cost?" and is it only on the calls, or is it also the megabyte I'm throwing in?
Peter Elger: I'd have to go and check the specific pricing there. I don't know it off the top of my head, but it's typically going to be on execution time of the model, I believe. Each one has their own pricing model, so I couldn't tell you off the top of my head
Sven Johann: Okay, but the main message is...
Peter Elger: Understand the pricing model...
Sven Johann: Understand the pricing model, exactly. What I like with API services and their pricing model - usually, if you were very successful because you have lots of requests, because it's a popular functionality, then the cloud provider is also successful. And if you are not successful, they are also not successful.
Peter Elger: That's right, yes.
Sven Johann: I think that's quite nice. One question - we talked about deployment, operationalize... We had testing, we had how to develop it, but when I'm a product owner, how can I actually write requirements? Because I assume it's hard to write down. One example I recently heard -- when you go back to software architecture and you look at certain qualities of a software architecture, you can simply write "This endpoint should respond in so-and-so many milliseconds up to a load of 500 requests per second", something like that. That type of scenario doesn't really work for AI. If I have a self-driving car, I cannot come up with all those scenarios and say "Yeah, when a kid jumps in front of the car, then break. Exception is if you're too close to the child, then do something else; run into another car", or I don't know.
Peter Elger: Those are almost impossible to write. And the reason they're almost impossible to write is because those are moral decisions that you're talking about there. You've gone right up the chain to human morals.
Sven Johann: It was just an example, but let's say if I have a recommender or a chatbot, I think it's hard to write down "if that happens, then do this." It's too complicated.
Peter Elger: Well, I think it depends on the service. So you're right, in the sense that if I feed it a picture of a cat, it should tell me that it's a cat. That's a fairly straightforward requirement. But as you say, there are some services where it's not so easy. Recommender systems are one. The recommendations will change based on the increased amount of data that comes in, or forecasting future events - again, it's all dependent on the quality and the volume of data that the model has to work with. So in that sense, it is difficult to put hard requirements around that. I think you're probably more looking at KPI-driven type requirements, which would be at a high level. "What's my business goal? Well, I want my forecasting to be 20% more accurate than my current forecasting model."
Sven Johann: That would be one thing, but another thing would be we have recommendations, and probably with the recommendations we want to make more money. So of course, a hard KPI would be to see if we sell more items, for example. But when I just get started, I'm just wondering "How do you know it's enough? This is a good enough solution to get started, and get started to measure real KPI's."
Peter Elger: Yes. Again, it depends on the service. It may be that you can't, until you've built up a set of results. So if you're new to doing recommendations, then maybe what you need to do is to run some experiments, because it's non-deterministic. So that would be "Am I gonna A/B test a couple of recommender engines, or recommender services against each other, and see which one gives me the better results?" What do I mean by better results - well, you probably mean more sales. I think certainly for those types of systems there is a certain amount of in-production experimentation if you're new to it. That's what Amazon do themselves, if you want to take that case - they're continually optimizing their recommender system if they wanna continue to drive more sales. As you say, it's not a deterministic problem.
Sven Johann: Yes. Alright. I think when it comes to API-driven services, my questions are answered. Now let's dive into another set of questions; you mentioned training services, or can you say "general AI platforms", something like TensorFlow, or SageMaker?
Peter Elger: Yes, there are services - and again, across the three large vendors - which are really aimed more at a lower level, in a sense, at the data scientists or machine learning experts, to help them with training. And then for guys that are operationalizing those services as well. SageMaker is an interesting one. It helps you deploy and train models as a service, and you can go a lower level again and just use instances that are preconfigured with TensorFlow, or Keras, preconfigured images... They just make life a bit easier when you're doing training. So there's a number of options there.
Peter Elger: Generally, we would go with managed services, if possible. In other words, if there's no good reason not to, we would go with something like SageMaker if we're operationalizing on Amazon. So the typical workflow there would be that we would work in conjunction with a client who may have done some research, built a model which they've validated - not at scale, but they've validated it in a particular context... So then we would help them to bring that on to production, and then put all of the bits around that, such as the deployment pipeline, such as the monitoring, alerting, all of those kinds of things that are required in order to turn a piece of research into an actual platform that delivers value.
Sven Johann: Yeah, you mentioned the trade-off - can I use already an API service for that? And if there isn't a service, I have to use SageMaker or TensorFlow, or whatever Microsoft has?
Peter Elger: Yes, what you need to do is to check whether there's a commodity service there, or if there's a commodity service that I can transfer into my specific domain. So a good example would be natural language processing. One can get standard services that will do named entity recognition; in other words, they'll recognize things within text. But where they tend to struggle is when you go to a specific domain. A good example would be the medical domain. There are words in there that just a standard NLP engine wouldn't understand, but it can be cross-trained into that domain, to then understand the lexicon of terminology within the medical domain. And you can do that, again, API-based, within your specific business domain.
Sven Johann: I think that's a really interesting case, because the people I know who are in NLP (natural language processing), for them it's the totally normal business case that you give a doctor an iPad, and when a doctor visits the patients, the doctor just speaks to the patients, explains everything and then says "Okay, put that into the record of the patient and send an email to the patient, and also to the doctor at home from the patient." It seems to be a normal thing in Scandinavia to do that, and I'm just wondering, how do you combine a service like SageMaker and speech-to-text? That's a bit unclear to me how I would approach that problem. If I have an API and I have TensorFlow, how can I bring those things together?
Peter Elger: That's really about how you then assemble your pipeline. If you've got to do multi-phased work on some input data, or maybe multiple input channels, then that's about how you architect your ML-enabled data pipeline. One can imagine having a number of inputs coming in at one end, which could be image data, it could be text, and they could relate it in some way... I could imagine feeding an image through a recognizer and getting some results out of that; that would give additional context that I could then use with the text to feed into a classifier, to then get a response on the back of that.
Peter Elger: It's really quite fascinating as you start to combine these services together into ML-enabled pipelines. Quite a lot of interesting stuff, and you can have custom steps or commodity steps within that pipeline; it depends on the problem context.
Sven Johann: I understand the delivery pipelines, but how does this pipeline look like?
Peter Elger: Let me give you an example from the book, actually. Hopefully, I'm going to get a bit more concrete. In the book we take the example of "How do I manage a product feedback?" Let's say that I'm a large retailer and I've got several departments - pet, beauty, whatever. And I'm getting feedback through web forms, email, various social channels like Facebook and Twitter... How do I take all of that feedback, which could be pretty large over the course of a day or a month (there's an awful lot of feedback coming in), and figure out what's positive, what's negative, and then route it to the appropriate department for processing? Well, you can do that with off-the-shelf. I can feed data in in one end, and put that through a pipeline using something like Kinesis, or Kafka, or some queuing technology. That can then be fed through a language translator. Let's say it's in multiple languages, so I can do translation into (let's say) a base language like English. I can then feed that forward into a sentiment analyzer, which is commodity, and discard the positive, keep the negative, because I'm interested in handling the complaints only. Then I can feed the results of that into a custom classifier, which understands my context. And from there I can then say "Okay, I've got a negative sentiment from the beauty department", so now I can forward that piece of data on to the beauty department and say "Hey, you need to address this."
Peter Elger: It's a contrived example, but it is a real business problem. The space of social CRM and processing of that is a real thing, and the point is that that can be built using commodity AI, with a little bit of cross-training. Without too much effort, you can assemble that in less than a week, quite easily... Which a few years ago would have been a pretty large development effort.
Sven Johann: Yes. But if I want to do that, obviously that is not a beginner's task, just to read a week -- let's say if you're someone like me, it's not that I can just get started, learn about an API... I really have to put in a lot of effort into understanding something like SageMaker or TensorFlow.
Peter Elger: If you want to go down to that lower level, then yes, you need a bit more understanding. It's why we would suggest commodity is the way to go to begin with, and then look at custom if you need to.
Sven Johann: Yes, adding the custom step is a big step, I would assume. The commodity is easy, but the custom part - for that one you need significant knowledge, I would assume.
Peter Elger: Yes, you may need, absolutely. Depending on the domain. But this is where to deliver a solution you're looking at a cross-functional team; you're looking at some data science, ML experts that can build that custom piece, and then the people that are expert in assembling pipelines, handling data, infrastructure as code, dev ops, all of that kind of stuff. So you need that cross-functional team that delivers a solution.
Sven Johann: Yes. And what about cost? When I use something like TensorFlow - I think you said it - I pay how much it costs to compute your model, or...
Peter Elger: TensorFlow is the open source library from Google, that allows you to build networks and train them. So in that sense, you can run TensorFlow on your laptop, if you wish. The only problem is that if you want to train a network of any size, it'll take weeks to run on your laptop, because it's compute-intensive. Therefore, the way to do that is to use GPU-enabled instances or containers that have the associated horsepower available to allow you to scale up that training. Then you're really looking at the cost per instance or container hour for execution. So it'll get your training done quickly, but you just need to be aware of the costs.
Sven Johann: I would assume that training on a GPU is already a service. TensorFlow as a service, so...
Peter Elger: Yes, this is essentially what SageMaker is.
Sven Johann: Okay. Another AI platform I was thinking about, but I'm not sure if it really exists - we have very specialized packaged solutions, like SAP or those business packaged solutions; the CRM system, Salesforce, an Oracle database, things like that... And an Oracle database is a wrong example, but I think SAP is a good example, that you buy something and you can modify it and adapt it to company-specific needs. Does this already exist for AI, that companies offer their very specific platform for a certain problem? And it might not be perfect for your problem, but the company offers professional services and consulting to adapt their solution to your problem?
Peter Elger: Yes, certainly there are a lot of companies that specialize in data science and machine learning, and we're one of them obviously... We see ourselves more on the operational side. We do do our own research into machine learning, and maybe we can touch on that later... But yes, certainly there are companies that do that. There are also companies providing data science type platforms for exploration and investigation. Those are quite interesting, so it's kind of like Jupyter -- are you familiar with Jupyter Notebooks, and that kind of thing?
Sven Johann: I only know the name.
Peter Elger: It's collaborative platforms for data science investigation, and there's a number of interesting companies in that space. If you check out the Gartner Magic Quadrant you'll find a bunch of them. So those platforms are growing... Again, I would imagine that those become subsumed by the larger cloud providers as part of an additional offering on top of their existing services... Collaborative data science platforms.
Sven Johann: Okay. The last option I wrote down was building your own AI from scratch. I can assume - or I would assume - that maybe something like TensorFlow or SageMaker is not the right thing to do if you start a company like Boston Dynamics...
Peter Elger: That's pretty specialized...
Sven Johann: To me it sounds like this is something usually you don't do. You only do it if you have very specific skills...
Peter Elger: And lots of money. It's kind of capital-intensive, and you're not gonna do that without a significant war chest the way it is. What they're doing is -- okay, it's applied research, but there's a certain amount of fundamental research pushing the envelope there... Which tends to be very open-ended and tends to have no specific timeframe as to when you're going to have something that you can get to market and make your money back on... Although it's clear from companies like that that as they begin to solve all of those problems, the value that they create is going to be absolutely enormous. So you're going to win an entire market with that investment. It's just development.
Peter Elger: We do quite a lot of work in the space of converting monolithic, older platforms to microservices and to next-generation cloud. And the reason that's a space that we do some work in and is a growing space of interest is that many companies recognize that their technology investments of the past are now not necessarily fit to run in an efficient way on the cloud. Our belief is that serverless will become the de-facto standard for enterprise development, and that legacy workloads will shift to those consumption patterns over time.
Peter Elger: We actually have a research project which is accelerating at the moment, which is how do you apply machine learning to the process of software transformation itself, which is a very interesting area to look at... Because if you can make that process more efficient by 20%, 30%, 50%, that represents a huge cost saving for people that are undergoing that transformation.
Sven Johann: For me, when I hear that, it's not -- one thing is costs, but to me, doing those things quite often is also a very high risk, because if you have a system which is 40 years old, or maybe even 20 years old, it's very hard to move that to a new platform and still everything works, and all the strange things which happen along the way... There is a huge risk to transforming those systems. I'm just wondering - for a big set of humans which are experienced in transforming that stuff, what would make our life easier there? Do you find out what's the easy part to transform, and you just move the easy parts to serverless workloads? And the really tricky part, you say "There is a very hard part, maybe 20% of the application, and that I leave to the humans, or maybe humans in the loop", something like that.
Peter Elger: Yes, just to be very clear - we see this entirely as human in the loop. It's not automated transformation. What we're trying to help with is the process of pattern recognition of services within code. If you think about it, when you do this breakdown, it takes quite a lot of person-months even to identify service candidates, trial them out, move forward, rinse and repeat that process. And the reason is because there are -- you could be talking about a million lines of code, of which significant fractions nobody in the company even knows what they do anymore, or why they were written the way they were, because that knowledge has left the building.
Peter Elger: So the ability to take a lot of data from the running system and figure out how it works and figure out "Okay, this looks like a candidate service, and this looks like a candidate service" can optimize that process, because then the human in the loop is saying "Okay, that looks like a service. Okay, let's try extracting that. Now let's rinse and repeat again." So you're accelerating the process through pattern recognition, which is what the human is good with, not necessarily automating the extraction process, which I agree is a next-level problem.
Sven Johann: Yes. Awesome. Basically, in five years or something, when you have enough data, then it worked like a charm...
Peter Elger: Well, we'd hope sooner than that. I guess the reason I was talking about is really that's an example of when should you go and build something custom, and there's no solution available to do that today. But it's something that has value to us as a business, and therefore it's something that we're pursuing as a means of accelerating and providing better value to our clients. We're pretty excited about that project, I have to say.
Sven Johann: We mentioned the human in the loop - we never talked about it, but I think that's also an interesting thing, in my opinion. You don't have to automate everything, but if you try to help the humans to do their work more effectively, that's already a cool thing. My travel bill, if I just upload my bill - not everything is from my hotel and not everything is perfectly recognized. At least if (let's say) 80% is recognize and I don't have to type everything, then I'm already...
Peter Elger: You're winning. Exactly. And I think you're right, the key thing is to recognize the boundaries of where AI services are as good as or better than humans at the moment, and not to try and overreach those boundaries. So I think human in the loop is in a lot of cases a very valid approach.
Sven Johann: Yes. To wrap this up, I have no questions anymore... So the question is "Did I forget to ask anything important?"
Peter Elger: I think we've covered a lot of ground today, actually. It was a very interesting conversation. I suppose the thing I would say is for people that are looking to get into this field, the first step would be really to just go and have a look at what's out there. Go and see what's available. You might be surprised by the development, the range of services that are available and what you can achieve with them. The second thing to bear in mind is there's rapid innovation and change in this space, so try and keep on top of those developments because there will be many services changing over time... So make sure that if you're going off to build something yourself, be careful, because you might find that it becomes available as a commodity before you're even finished. That would be the two things, really.
Sven Johann: Alright. Thank you very much, Peter. This was another conversation about software engineering. Thank you very much.
Peter Elger: Thanks very much, Sven. I appreciate your time.