Conversations about Software Engineering

Transcript

00:00:00: hi and welcome everyone to a new episode of the case podcast conversations about software engineering it's my pleasure to host this session today my name is Alexis I'm with Sven Yohan and Heinrich Hartman and today I'm very happy to have Mercon or vuckovic with us Miracle is a Serial entrepreneur and investor he started his career as a software engineer became freelancer for application performance before he co-founded concentric and voting from code Centric he spent our Center device he spent off in Stana which was afterwards bought by IBM

00:00:39: and of 2020 and turns out that just being an investor was not sufficient for miracle seemingly so you started there's zero Miracle warm welcome to the show great to have you is there anything that you would like to add by yourself to your introduction

00:00:58: yeah hi everyone and thanks for having me on the show know I think it's a good it's a good summary i i in between instant 0 so open a few restaurants and invested in all of oil farm so a few things that were outside of software and but I'm glad being back

00:01:22: and why is it

00:01:24: is it olive oil okay because I thought it's it's olives and you said you know you you said oh which kind of Olives should you buy and then it's no it's really oil and the olives for olive oil that we use you can't really eat so it's really for oil it's different okay we also sell olives in glasses for to eat but it's actually not our lives it's color matter olives so our from our Island of Lesbos okay okay

00:01:56: so what makes a good olive oil for you may go

00:02:00: at the end of the day it's I mean I had no idea of olive oil right so at the end of the day it's really the olives it's that simple right and and harvesting them at the right time and sorting out the bad ones and then the only thing you do is you press them right and you have to be careful with the temperature that's the cold pressed it's somewhere I don't know exactly between 20 24 degrees Celsius let's the temperature and then you press them and that's it done there's no nothing you add to the oil there's nothing you can really do so it's really about the olives

00:02:36: but as a consumer like what do you watch out for if you are like shopping all of all sorting olive oils

00:02:42: I think one of the things I learned is that you should look at the origin of the oil and the more specific it is

00:02:51: the better is it for you for the consumer so for example if you have all of all that says olive oil from Italy that doesn't mean a lot yeah probably olives are bought from everywhere around the world but if you have an olive oil from I don't know a specific region like in lesbos it's it's if you say it's all of from last boss the olive has to be from Lesbos right so then you know that it is from a from a special region from from local farmers and I think that's the best thing you can do basically look at a very specific region and then it's it's it's it's it's a taste alright I think there's olive oil good olive oil Italy in Spain everywhere but of course ours is the best always looking at the best performance exactly exactly can we asked like other any like transfer learnings you have from the olive oil making

00:03:51: kind of transferred to software or is it more recreational activity for you I can I can generally say that my whole experience in restaurants Olive Farms and everything the thing I learned is that everything I learned before can be applied to that business because it's actually the margins the way you work are so different right it's more about really low margins the whole food industry is in Germany has very low margins right and restaurants but also if you sell in supermarkets so it's all about cost control and very detailed cost control right where in our businesses in consulting or software

00:04:34: I mean the margins are pretty high and therefore you can deal with cost totally differently and therefore you cannot manage companies very differently right in terms of salaries and terms of benefits in terms of how you operate so my learning was that everything I learned before is not is not a good way to approach a food business

00:04:57: yeah so you had to unlearn probably a lot of things exactly and also I lost a lot of money okay that means re-education okay that's actually actually interesting like when you when you explain the the process that actually all the quality is in the in The Grapes of the olives then

00:05:27: that doesn't like one one by one apply to software at all right because like with software you have

00:05:37: you have a very high change ratio that that you can have and with olives it's like

00:05:43: the trees are how old like unless post there are 100s of years old actually that's all so something it's a lot of trees and they are pretty old exactly right and you just have to wait and you can do a lot right it's also very dependent of the weather is the overall Olive also the prices of olive oil is highly dependent especially because of global warming the past two years the price almost doubled

00:06:11: because Spain had such a drought and Spain is the biggest producer of olive oil that the overall olive oil Market just exploded right from like around five euros per liter to 1160 I think was the high and now it was a good year so the price is going down again right

00:06:32: and it is positive and negative because for the farmers is actually pretty good if the price is high because the farmers get more money for their oil but for the consumer it's it's I mean a good good olive oil these days is a north of 20 euros a leader right everything that's below it's probably not good olive oil because just the production costs is around 11:50 or what's what you pay for the farmer right so if you calculated it's very hard to be below 20 euros it's actually interesting coincidence because yesterday I was watching a report by some Spanish television company they interviewed all the farmers in Spain because last year they had this drought and then they had the flooding and now they have drought again but in some regions in Portugal just this week there is there is flooding again and so they are looking at different types of grapes and I learned that

00:07:33: are over 700 different types and they are and thinking about how do you say mixing the genetics and they are using large language models to actually artificial intelligence to to find out which Graves to to cross and they are also building up in iot network at the moment to like have sensors under each tree to understand what they can do in terms of most efficient watering so to save as much water as possible but I have to say one thing that's true for olive trees and all for a wine grapes so unless for example the trees don't get water

00:08:22: and what happens is if you have all trees who never get waters the roots go very deep and they can survive droughts if you water trees

00:08:32: they don't build out those roots and they can't survive droughts anymore right so it's it's kind of a complex topic of industrial production and watering and getting more grapes but then if there's a drought you don't have water the trees can survive anymore right but naturally all of trees would survive a drought if they are not watered right so it's

00:08:56: it's a it's an interesting thing to to look at these

00:09:01: trees and and grapes and all these things

00:09:05: sounds like a sustainability topic yeah um

00:09:11: actually wanted to to start with interviewing you a little bit about the past path that you've that you've come until now with with their zero and I was thinking is there is there common theme that you would say that

00:09:33: takes you from the very beginning your interest in a software engineering

00:09:39: towards your time as an investor and then that lets you into founding their cereal

00:09:45: I don't know if there's a common theme I think the common theme is that there are waves of Technology Innovations right when I finished studying that was 99 I came into the internet bubble right and and what that was that was really good for me right I was young I came into projects where we build the First online banking for a German bank and by doing that I also experienced firsthand all the problems that you got right I remember we had we actually did the online brokerage for the rest of bank and that and they did the IPO of tea online if you remember

00:10:28: and and this called implementation yeah it was korba so it was a technology for from today's point of view but especially

00:10:39: I remember when we started it and and hundreds of users and thousands of users came everything crashed right and we had this black box and we didn't know why it crashed and that's how I came into the performance and troubleshooting space right because I was I was there I was the developer I had to figure out how it works so I figured out tools and implemented things like blogging to figure out where where it crashed and then because I did it and I was one of the first at the time because it was new

00:11:08: I went to the second project where they they knew I was inside of IBM and then you oh there's this guy who troubleshooted this dress now Bank thing maybe he can help us with the Tuohy portal right and then I went there and then I went into the next one and so I think it's always really cool to be one of the first in a new area or topic because you can just gain experience which you just can't get

00:11:34: that fast if you're not inside that way right and then there were other ways as you as you know there was the mobile wave then the whole Cloud wave microservices and I would say now we have the AI Jenna I weigh which is moving very fast and whenever you are in the middle of such a wave

00:11:52: there's a ton of opportunities right

00:11:55: Everything Changes the developer tools change the infrastructure changes the processes normally change the way you work Changes Everything Changes right and whenever like like was Jenny I will be the same right it will be monitoring tools for Johnny there will be security tools that will be new tools new developer tools

00:12:14: and I think that's kind of a topic that is is part of our industry not only myself and I think I'm I'm very curious right I love to investigate these new things and then I start thinking okay what how can I basically monetize it and because I I am in the performance space for now 25 years or longer that's that has always been my passion right so I always think like okay what does it mean for troubleshooting what does it mean for monitoring and that was always a common theme when I found it code Centric the initial code Centric was all about performance and troubleshooting right and and then we did all so software engineering and programming but and then istana was an APM tool for microservices because I saw there are these things called Docker containers right that works totally different than before so we need something else different agent technology and and now I I saw the opportunity was open Telemetry which is -0

00:13:14: a new standard a lot of open sourcing in that space a new things coming up llms and yeah so that's a common theme I love

00:13:24: I love the change in the industry and I

00:13:28: I basically figured out that every time everything changes right from a database technology to

00:13:35: I mean I'm not sure if I'm the older guys here but the when I started it was a web sphere application server with the db2 right that's not how you program these days right but so it's changes yeah yeah exactly exactly and and on the AI x and hpu x and Linux nobody used write open source was weird right nobody used open source that time right and then Linux started and then other open source I mean you know it but so if you just look at the past 25 years it changed a lot right and and that's the fun part and I think that's all so the part where if you are eager to learn I think that's the biggest thing as a software engineer or as a founder

00:14:20: you have to be open to learn right and and unlearn all the things that you learned before right it's it's it's crazy but just think about an idea I was an eclipse he was there for a long time right and then if you use something like IntelliJ you feel like you start from scratch right because all the shortcuts do not work and it feels weird but then after a few weeks months you get more productive so it is this thing that you have to

00:14:47: kind of love also this pain right of starting from scratch all the time and and figuring out things and going through this well of not not being as productive as before but then coming out even more productive

00:15:02: so the common theme would be you love to unlearn things or actually love to learn how to do the things in a different way or look at it in a different perspective

00:15:13: yeah and I I just love learning right I think that's really and I kind of it sounds weird but I kind of like the pain right it is really the situation I don't know if you can second that but if you if you do something and you have totally understood how it works then it gets boring right I mean I'm just not the guy who wants to build the tents spring application in the same way there are people who love that right to who are really good at that but I'm just not good at it right I'm bored and then that also leads to problems right because sometimes I would probably use a new technology though it doesn't make sense right it's also common Siemens software engineering that people just experiment to not get bored but it's it's the nature of

00:16:03: of being curious and and wanting to learn something

00:16:06: at the skill set of software Engineers evolving like from the 2000s area to now specifically even comes to like performance and debugging systems I think like in the 2000 like full stack understanding also understanding how the operating system works kind of being able to reason through the layers has been like a critical skill set that I think

00:16:32: it really sets you apart and now it feels like with containers and kubernetes and more and more like complexity along the stack the skill set is becoming a little bit more horizontally focused so how do you think about this maybe all so in this context of the current wave which seems extremely like high level and extremely wasteful in some sense but comes comes with its own challenges I'm I'm really a big believer in fundamentals

00:17:03: so I do think that learning Frameworks and these things is nice but the I really always I mean I learned compilers and operating systems and I think those things are really the fundamentals that help you

00:17:22: whatever you do is specially in performance right because at the end of the day everything comes down to a few fundamental principles and they actually if you look back in 99 until today they haven't really changed right I mean there are a lot of fundamentals which are

00:17:38: still pretty much the same and if you understand them then it is much easier for you to to deal with problems right and to

00:17:48: and I haven't programmed for I don't know probably let me think 15 years right so really hardcore programmed and but I can still understand mode of the things because I can break them down into some fundamental principles and I can understand what I can understand what it means right I understand what a container is I understand what kubernetes does I understand then what the fargate does if it's serverless or understand new serverless technologies how they work because I understand fundamentals of databases and and file access and and and and and these things so I do think if you understand fundamentals it is much easier to understand larger Concepts too right and my son just started studying at in Munich and and they do assemble and and micro controller programming I think it's a super cool idea right yeah really good that they understand what microcontrollers are what registers are

00:18:48: and and I think that's I thought they wouldn't do it but I really think that's good right they still do it assembla C compilers and I think that's the right thing to do right if you have those fundamentals you can learn everything

00:19:06: yeah Gunda Duke he is also talking about this constantly that you don't want to have

00:19:14: you know the latest trends at University

00:19:18: because the trend is gone in five years and then you know the trend stuff

00:19:24: I also think it's a different topic but that's my biggest kind of when I think about Ai and and AI coding or supporting a large part of the coding I really think that it can be hard for some developers because they and they will be even further away from the fundamentals right you just type in some prompts and some code will be generated and depending on not I depending on this really evolved we will have a generation of developers who have no idea of code many even anymore and that that can be really challenging and in situations where you have to troubleshoot where you have to understand what is happening right so so I think that's something why I really think it's good to still do the fundamentals even though AI could generate 80% of the code right that doesn't help you in situations where you have to really deeply understand what's going on

00:20:19: at the funny thing is there is there is actually a study not sure who did the study I need to look at up but I mean not surprisingly the study found out that if you

00:20:31: if you are bad developer consider the bad developed by any use AI at you as your coding assistant you produce just more bad codes and if you are good developer whatever that means in that study you produce

00:20:47: good code now

00:20:49: so yeah it's you just cannot abstract the way a lot of things it's just you have to know how it works absolutely interesting it's interesting I had this discussion once with a with a Netflix engineer unfortunately forgot his name he said

00:21:09: the engine is these days and it was in I think 2015 is that the engineers these days they they don't actually know how the Chrono works and how it actually addresses the CPU cycles and that they are messing up completely because of the wait time that they implying and I was like hmm

00:21:34: we're already talking about like the the rise of containers I think it was the early days of Docker swarm but I might be wrong

00:21:43: I know that that Google was already working on the predecessor of kubernetes and I was like well

00:21:53: the people are so focused on their containers and and what happens in their containers

00:21:59: and then you say well we have container orchestration and

00:22:05: to me container orchestration in the meantime even more than back then filled Slide the new applications rather application service who try to do everything for you and Abstract away all the the system limitations

00:22:20: and the Java application servers I was always thinking okay but what really happens in the background because when the jvm does the full full stop then

00:22:32: yeah like what happens right what happens to my user requests and and it was like yeah but you have to understand it

00:22:42: what would you say what level level of depth

00:22:48: is the the basic that you have to understand and what is it that you should be able to rely on the on the platform to tell you a certain signal

00:22:57: like to be more specific

00:23:01: I was saying well if I'm using the kernel wrong inside a container

00:23:06: the question is how would I even know without like doing kernel debugging and how would I do this on a container orchestration

00:23:16: I mean it's it's it's it's really hard to say right again I'm I'm pretty curious and in my in the observability says for example there are technologies that are evolving right now called ebpf which essentially is some sort of Kernel debugging right so you are inside of the kernel and kind of a virtual machine inside of the kernel where all the events that happen come in like a call of a function right or a call to a file and or to network right and and because if you do it right if you filter it right you can basically create an agent which then understands how things can how containers communicate over a network on the machine right because you are investigating the messages inside of the kernel so I I like to read those books and understand the technology because I want to understand what does it mean for me what does it understand from what does it mean for my industry I I would try it right I would try it out and I think this

00:24:16: curiosity would help every developer right understanding better what you can do but then there's a topic that there is a lot of things I have absolutely no idea of right because you can't

00:24:30: go deep on everything right that's that's all the reality and you can go really deep on the Java virtual machine you can go really deep I mean again I'm not programming for 15 years but I was really good at optimizing garbage collector and Heap sizes and back then there were probably 100 different parameters how you depending on the garbage collector algorithm how you how you size the different generations of the Heap and all these things I loved it right and but not every developer probably would do that right so you need some like the t-shape right you need some I think it's always good if you are really good in some things and then have a very broad understanding of a lot of things right I think that's this this thing that's I think called t-shape I think that's that's pretty good but you can be deep on everything I mean there are some people I'm not that smart I'm really sometimes talking to

00:25:30: people and I'm like oh my God he's deep in everything right but I think I can do that so I always focus on certain topics which are in my interest so performance optimization monitoring and I try to catch up with everything that's happening there right just because I

00:25:49: I also enjoy it and also I have a fundamental knowledge of a lot of things so it's easier for me to stack up more knowledge right but then there are areas where I probably

00:25:59: have absolutely no idea right like mobile phone Development I've never touched right I know nothing about mobile phone development right essentially so

00:26:10: that's that's a topic I never really had interest right as an example so but others other things I'm really interested in so

00:26:22: I don't know if there's good or bad right but I for me it always helped to be good in a few certain thing things right and and not being so good in others right

00:26:36: I had to look up the name of the guy because I think I know the Netflix guy just as a little input you probably mean meant and crack yeah yeah yes I'm really good I've read all his books right he's got some really good books about hot core performance analytics yeah and

00:27:01: I I love those books right and and yes also some really good methodologies right how to profile things how to analyze things so a lot of good fundamentals which you can learn from yes but Brent is a it's a rockstar Rockstar in that then again if you have people like him or Marcus laga Green it's well you very soon very quickly understand that they are experts in their field and

00:27:31: I rather refer to them instead of trying to go as deep as they are in terms of understanding and by the way that's something if you if you are young engineer or something I would say what what does the difference between brand Greg and I don't know someone at the normal company in Germany for example right and the big difference is that he had the chance

00:27:51: to learn and environment which had extremely crazy requirements right Netflix right and and it's kind of luck also being in those kind of environments if you would be one of the first Google engineers and you would have to scale that you would probably have learned things that you can learn if you are working at the bank right just because you don't have that requirement to handle one billion users and or have a Netflix which streams 100 million and should react in 10 million seconds right I mean these are just requirements that

00:28:30: probably require you good to go very deep in certain topics right which you

00:28:35: just don't have to in others right that's

00:28:39: yeah

00:28:41: I just

00:28:45: Brendan moving to Netflix I think was one of the best things that happens to Linux observability in the last 10 years he was kind of working on the Solaris kernel for a long time I think he had big contributions to dtrace and when he moved to the Linux there was for the first time or what the hell is this going to to be here and yet this great talks on how broken all the Linux tools are and then he mapped out like observability in Linux and identified also ebpf as the technology that is going to bring that advantages and then we saw like BPF trace and this kind of the trace like tools also break in to the notes man and kind of really Advanced the industry quite a bit and every thought also at Netflix by the way he was one of the initial D Trace developers on Solaris right I think that's oh yeah that's a team from solar is going to Netflix and yeah but you can see I like having this deep knowledge

00:29:45: on a Solaris operating system then moving to something like Netflix or company electric then you can apply your knowledge right and and that's yeah I mean it's it's also luck to be at the company like Solaris and then being at the company like Netflix right yeah exactly and this is a little bit of my learning I came from a company that was very performance focused and was also running Solaris for this reasons and then going to a German e-commerce company it was like

00:30:14: there's there was really no market for for the trace there was really no market for like detailed Cisco profiling or stuff like this we only had

00:30:25: have one or two applications which are really performance critic critical and there are some Engineers have figured this out but the average engineer wouldn't touch the current at all wouldn't go so deep it will be more okay I'll be shaving of 100 milliseconds here I'll be shaving of 100 milliseconds there rarely some garbage collection issues but it's more like higher up the stack and I think this question about skill set proficiency you can can ask this about the personal career where it's like okay how do you want to develop and also how do you like complement your day-to-day skillset with just like deeper skills which will indirectly inform but there's also this question about like engineering a platform so as a steward of the overall developer population or as an owner of the platform how do you develop the tool set in a way that also shapes the skillset and supports the skill set of these like are you offering trainings in certain areas are you buying

00:31:25: tools that go in a certain area and I was like

00:31:30: thinking about like for example continuous profiling or more advanced tools that kind of go into into the kernel and that context and I think what we are seeing in is that they are at the moment not really crazy successful it's really the stuff that is high up the stack and then there's a question how low are we actually getting and at which point you switch the tool and so on so having developed at least two observability solutions like what is your take on this how you anchor the stories and like what is the the conscious choices that you you make okay we're not going there

00:32:08: I mean I can just give you a rough estimate from how the market in terms of Revenue

00:32:15: yeah at the moment overall right and then we can discuss why right so if you look at the market probably 50% of the observability space is metrics

00:32:28: and dashboards then from the then around 30% is logs

00:32:34: and then around 10% is traces and the last 10%

00:32:40: is the rest and that includes real user monitoring synthetic monitoring profiling everything right everything so so it is really a lot of metrics right a lot of logs

00:32:55: a bit of tracing I'm a really big Trace fam alright I love tracing but the reality is and we see that with our inside of our customers we saw that it in Stana and there are no revenues from from companies like data doc

00:33:11: it's always the same shape basically right it is

00:33:15: developers come in

00:33:17: or srees they first build some dashboards with some metrics right and then if they want to debug something the first choice is normally logs I lock something somewhere and then I put in my own messages that I can find and and then if you are already advanced you use tracing which is if you talk about low level kernel tracing this is the different this is like the depth of paper from Google right this is microservice tracing what I'm talking about so very high level still giving you ideas of which service of wrong then profiling almost nobody uses right it's it's already a tool that's not used a lot and and

00:34:00: it looks like and I'm saying this it looks like and I had to learn this also in the past 10 years because my my way of thinking was always I need to provide these deep insights but it looks like that

00:34:13: 90% I would say of the developers are not interested in that and they also don't need it right and

00:34:21: that's that's just the reality of the market in that's what I learned in the past 10 years and you really have to provide I would say these basic use cases in a very easy way and a very accessible way

00:34:35: to be successful right I would say data doc who is the I would say the most successful company in this space at the moment right it's changing over time right if you look at the past 20 years there was always but at the moment I would say it has the largest market cap it's it's the fastest growing company so I would say they are the clear number one at the moment so if you look at them

00:34:58: they really managed to give you I don't know

00:35:03: 80% of the value with 20% of the functionality in a

00:35:08: pretty easy way right and and that's seems to be the winning way of doing it that's and and then there are others which are very very deep right and provide you with very deep profiling tools for continuous profiling tools for example which for some use case if you are a database developer right you probably need that right because you still have to but the normal applications these days it doesn't seem to be a problem anymore right because and and let's face it resources are super cheap super cheap I mean if you look at what

00:35:42: cpu's or or a storage S3 costs it's literally nothing right so

00:35:51: I mean I want stories with mind blowing for me and you you would probably get that but I I look at these days how we for example search and logs right if you store logs we store logs and S3 and if you search for them we literally there's no index

00:36:10: we go through all the logs and look at that

00:36:14: right even if they are billions of them and and the way you do it just like a little bit like a mapreduce right you use a function spin-up you take you cut the the locks into 100 buckets and then you have a function for each bucket and then you merge the result together right but like looking 20 years back it would be absolutely ridiculous doing that right because this was so slow it was super expensive doing this so you use something like an index right to to index these things have that index in memory I'm just saying

00:36:47: things change and something that was super obvious that if you don't have an index on a database you can't really be fast it's not true anymore right and and the reason for that is that resources have become so cheap

00:37:01: that you can do it differently

00:37:04: I was say I had to laugh my previous company

00:37:09: they co-founded elasticsearch and at that time let's search was a search engine yeah and then the whole lock station kibana thing showed up and we were all like yeah I mean in next space logging that's the thing to do it's so cool we were wolf

00:37:26: and then that was 2012 when elastic search has been founded

00:37:32: and then another part of the company you probably know those guys guitar Schmidt and they found it to me and they said no index no more experience yeah exactly exactly it's a stupid idea you know don't do this press yeah exactly who has all we talked about this butcher farming he is also a farmer now yeah exactly yeah

00:37:57: but you can see how things change right I think that's exactly in a space where you you go this is something that is 2012 probably it was pretty hip until 15 16 17 whom he always founded 1516 and then it took a while so probably you have like one technology and then also when it's in production I mean we use elastic and it's done at the beginning right and at some point it becomes horrible right because if you have to scale the cluster you have to re-index rebalance things it costs time it's it's a heavyweight process it doesn't scale as you like alright so you then figure out oh okay index maybe it has also some downside right and then you see something like whom you which is index free and you say oh this is cool you can scale it up easy are right it's and and then next day lambdas come up right and and there's a new serverless thing and you can do it again differently right

00:38:57: so I talked to Creston a few weeks ago and he told me that he is like researching and and and look any said okay these days you could build a home or 10x faster and better and and because there's this new technology and there's this cool new thing so

00:39:17: this set this is the cool thing about our industry right overall that every five as I said initially every five to ten years

00:39:25: things change and that's a huge opportunity because you can reduce costs you can be faster do something different right that's that's the cool thing

00:39:34: at the same time I see a lot of Engineers when they are in trouble they stick to the tool belt that they always had

00:39:43: and most of them don't update their their tool belts and

00:39:49: when you just said like 50% of the observability market is metrics I was thinking yeah of course because we know which metrics we want to look at right

00:39:58: exactly

00:39:59: and because we are also good one thing I learned is we all good looking at charts

00:40:05: if you have ever done it if you put the dashboard in your I I know that exactly if you are Developer you know exactly what I mean right you put a dashboard and you look at it every day and then you look at and you think oh something is wrong you see it directly because your brain knows what normal looks like and then if something is even if somebody else would look at it they would just say you are you how could you do that right because you are really good at looking at charts and and the chart is actually

00:40:33: a representation of probably millions of data points in a visual representation and our brain is just really good at looking at these visual presentations yeah and and also by the way one thing that's not that obvious because that's one of the reasons why at the moment llms are not really good at these things because they would look at the data

00:40:54: and the abstraction the chart works much better right and I think at one point if we train models to look at charts not at the underlying data I think they can maybe become as good as as as we are but we have looked at different algorithm most of the algorithms take the chart and transform them into Data again and then look at the data so but I think you should really just look at the chart right because you see anti-patterns and patterns very good in these trance right I think that's why metrics and dashboards work so well because we are really good at looking at millions of data points by just looking at some nice colored bar charts or something like that

00:41:37: not a very interesting aspects about dashboards and they didn't get a lot of love these days I feel in at least in the conference realm is that they are really good way to communicate conceptual models about software like working on a Linux kind of use dashboard for all the resources are like

00:42:01: structured I learned we don't have error metrics for a lot of the stuff that we actually run so it's actually like fairly poorly covered with Telemetry part of their and then similarly like producing a postgres dashboards what are the components you have to be aware of this I mean building this I learned a lot but also looking at charts that others build which were really principled really helped my understanding of the architecture and the operational concerns

00:42:27: it's very significantly absolutely but I also agree with Alex that it's kind of also because we do this because we are just used to it

00:42:36: right but there's one thing I mean you mentioned I also did Santa device right and and and I learned a big lesson there because I sent a device was something like a file system shared file system in the cloud something like Dropbox right and I at one day I came to the conclusion that I hate folders

00:42:56: yeah but because I thought folders is a very stupid concept right because you have a file and it's one folder and I said yeah but I sometimes have filed which I want to have in six folders right because it is tax relevance so I wanted in the text folder but it's also relevant for my house so I wanted in my house folder and it's relevant for my bank account right and so I said it's much

00:43:18: smarter to just tag them right just have a file you put it in and then it's tagged probably even outer tagged and then I can just use the text right so I built this tool and what I figured out is and there is this book called don't make me think yeah at the same when when Windows removed the Explorer if if you if you are so used to structuring things with files

00:43:43: it is super hard to change the behavior of users right same for if for example I use Google mail for 15 years I don't have a single folder right I can find everything with the search sometimes a text something but then there are still the Outlook users who have these large folders and every email gets into a folder

00:44:02: it's just I think changing behavior of users

00:44:06: is super hard and building a company on the idea that you will basically retrain the way a user works is pretty dangerous because there's a high chance that this will fail right not because it's a wrong concept just because it takes probably decades to retrain people right and that was the point that was trying to make because just because there are new technologies and I'm actually looking at at home for example doesn't mean that that

00:44:46: people will adopt the product because

00:44:49: if they have to behave differently in order to to use the banner to utilize the benefits then

00:44:58: it will be a very very steep learning curve like a steep entry point and that's

00:45:06: that's really something that I'm wondering these days a lot about it's like this this willingness to change is

00:45:15: is based on the understanding that

00:45:19: I cannot stay the way I am

00:45:22: I I have a very simple pattern right and you can tell me if I'm wrong but I always say change only happens with pain

00:45:32: right if there's no pain you will not change you don't change and this was when I was at concentric we introduced agile right and if there was no pain with your waterfall model you would never go with agile right if I mean real pain right you have to experience it then you have to solve and it's the same here right if you for example with whom you if you don't have the pain of very large data sets for example but I I don't think I mean I know that that whom he had very very large clients at the beginning right like I think Bloomberg was one of the biggest customers and the referenceable one and and they they had I don't know

00:46:12: billions of billions of logs right and and then as I said the whole thing of index fails right because you had to rebalance all the time and it was not working so they had to find something that was working for them and and so whom you had a solution so they changed because they had the pain

00:46:30: of having too much logs for for their Index right if you have a logging system and elastic work with an index why would you

00:46:39: do it differently but you don't have the paint to change right then you probably do something else first because that's the pain right I think that's kind of the way I think right by the way covid was the best example right

00:46:53: a nobody ever wanted to do home office and everyone said we can do it we don't have vpns and then then it was stay at home and even the most traditional German companies managed to become a remote only company in whatever a week or a month right which was exactly crazy if you think about if you would go to one of those companies before covid you said you you have to implement remote the it would say oh no VPN that you know no way we can't do this so if there is pain they all left of exactly so if you if you have pain which in covid was obvious right oh my employees at home they can't come then then you're ready for the change right and and I think that's that's what I learned in the past if there's no pain

00:47:43: there's no change

00:47:45: all right

00:47:46: let's say if the pain is not felt yet

00:47:51: because it could be that there will be pain in the future and you you see it and that's basically the same like smoking right you know that it's not great for your body but

00:48:03: if you don't feel the pain right now that you will have afterwards

00:48:08: you might change only when it's already too late and that is that is a big question as I said I'm wrapping my head around that these days

00:48:19: quite a lot because

00:48:21: the thing that I see necessary for our industry

00:48:26: is the speed of change

00:48:29: for a lot of companies the necessity to change very quickly

00:48:34: we'll become a life saver

00:48:38: or like a matter of their Extinction

00:48:43: and I don't think that the

00:48:45: every company has has understood yet like how fast they have to move in the future how fast they have to iterate and innovate

00:48:56: and that's a pity

00:48:59: it's true but something else and that is that you I always overestimated

00:49:07: the need for change all the time right I was always like always and always under or there are a lot of Industries where

00:49:18: slowness is not such a disadvantage right and that's what I want to say it's like for example I was in I was working for insurance companies right now I always said oh if you don't get become 100% online company then you will be gone right turns out highly highly regulated market I think

00:49:39: almost no insurance companies really digital yet

00:49:43: and they still also Vive and have better and better results a kind of trees another example right I I had Jax keynote in 2014 where I said if if the German car companies don't become software companies and electrical they would be gone in five years so 2019 now it's 2025 they still have shity software systems in the cars they still don't really have electric cars but still they have record earnings right so it's I know that there is a chance but I I think I always still even if you ask me to today I would say oh there's a high chance that they will be gone in five years reality is probably in five years

00:50:25: they will still be there still be very powerful because they have a brand name and they will adopt

00:50:32: they could have been 15 years faster but

00:50:35: it

00:50:37: I I agree with you totally I think we have to change radically everywhere right and be more digital and becoming rapid and change I'm just saying it's it's also a little bit learning from me that a lot of people are just not that fast with change and and you can see that

00:50:57: that helps also companies right because the customers also don't change right you can see that people are still buying diesel right and and other things though everyone should know that this is all technology now right

00:51:09: and talking about old technology and new technology

00:51:15: you just mentioned the ways that are ongoing and Ai and the rise of open Telemetry so how do you explain the benefits of this to your new customers now if they don't feel the need to change

00:51:33: that's a very good question but the really positive thing I'm seeing at the moment in the market but also talking to customers that they

00:51:41: understand this without me even telling them so so the option the adoption rate of open Telemetry is really high it's it has become a think a few weeks ago it become the number one project on cncf it overtook kubernetes in in the number I think of commits or something so it is a very active project and and I think there are many reasons why people experience that number one is

00:52:07: lock in with vendors and the way vendors use that lock-in to negotiate prices right and so I mean there was this big post on I don't know where I read it on X somewhere that I think coinbase paid 60 million dollars a year to date a dog which was like I was like what 60 million for for observability a year right and that was kind of the beginning of a discussion is that a fair price what should I pay and but once you are locked in was proprietary technology it gets really hard switching so open Telemetry for the first time in

00:52:45: the in the observability history basically is an open source platform with agents that allow you to get all the data you need vendor independent right and I think that's what a lot of customers are understood and I always say it's probably the last agent you will ever install right because that's the only one you will need

00:53:08: and there was a second thing which they did pretty well and this is the standardization of the metadata of

00:53:15: telemetry which is the semantic convention and that is a super powerful tool you have to understand a little bit how this works but I always explain if you give 10 developers the task to name the tag of a host name right so there's a host name and you give 10 developers they will decide for the tech name there's a high chance you that you at least have eight different

00:53:40: versions of the name of the tag right server name server underscore name host name host underscore name host dot name right and what open Telemetry did they standardized that tagging system and so now it's host name

00:53:56: and what does it mean it that's why it's called semantic now you have semantics right now you get attacked and it's not any more a piece of text it is actually understanding for a tool that this is a host name and now you can do things with it right you can map it to a host map right you can correlate things CPU of that host with something else right if a lock from that host comes you know that it actually happened on that host right same is true for kubernetes pod names so the semantic convention

00:54:27: Ah that's actually why I found that they're 0 right I saw the semantic convention I saw like oh my God this is so cool right because now I can get data from different sources it could be from a cloud vendor from a second party tool that I'm using but I can still understand everything they sent me because I have hundreds of specified tags

00:54:51: that I can now semantically understand right and and this is what we build the tool around having a context having a semantical understanding of the data and that makes it super different right now a log is not only lock it has a it has a resource attached it has it has it has contacts it has a connection to a trace to a server to a pod to kubernetes Cluster to cloud and it makes

00:55:18: the usability totally different so I I think it's a really really good standard

00:55:25: it took five years to get there right like good standards always take some time but now it's pretty stable at least for the court Telemetry data for for logs trace and metrics now there's also profiling and and rum this is not yet stable right we'll take some time but for the core Telemetry that I think it's it's really good

00:55:50: up so then I guess most of the Big Industry players have

00:55:56: staked in open Telemetry and send developers to work on it

00:56:02: absolutely I think the big cloud vendors did AWS Microsoft Google though the option is lagging a bit right and and it's always

00:56:12: outside I mean for vendors it's also always a thing of I would say the innovators Lemma right think about you you are AWS and you have a lock service with the own logging system and that's probably I there's no numbers on on these systems but I guess that actually not data dock is the one that's the largest probably it's one of the cloud vendors right probably adult AWS is bigger on observability in terms of Revenue then

00:56:43: a lot of the vendors and so

00:56:46: they are also very careful

00:56:49: in adopting those standards because that gives you the ability to go to something else right so so I think the the innovators the level that you basically turn your own revenue is always a problem in adoption right but but you can definitely see that all of them are adopting it all of them spend a lot of resources in the open source Community right AWS the vendors datadog elastic grafana honeycomb all the vendors right they they spend a lot of time there a lot of resources and I I think

00:57:26: I don't think there's a way around open Telemetry anymore right so if you if you start from scratch these days I think it would be a mistake not not going on open telemetry

00:57:37: can you talk a little bit about how you position Dash 0 like overall in the landscape I think that semantic conventions definitely super interesting technological innovation and opens up new possibilities but like if you zoom out a little bit you see maybe data dock you see grafana you also see installer positioned and I'm sure you kind of saw a gap where you have a value proposition that is that disappearing can you talk about this a little bit absolutely so I I have a very simple analogy how to explain it and and the framework we try to solve the problem and I always describe if you do troubleshooting with an observability to which is essentially what you at one point want to do right there problem and you want to troubleshoot it is a bit like finding a needle in the haystack right that's kind of you have a lot of data and you're finding good data that shows you that's called the root cause right what's the root cause it's the needle in the haystack and I think what most of the vendors did including myself in the past

00:58:39: we made the haystack bigger

00:58:41: more data more data right more data and and and we did not really help solving the problem to be honest I think out there and there's a lot of talks about Ai and things but if you

00:58:52: if you look at most of the vendors it's it's still very hard for the developers to find the needle in the haystack right and so

00:59:00: we we came up with the framework how we want to tackle that and we call it Rift Rift and R stands for remove

00:59:09: that's a big thing I think we should make the haystack as small as possible Right everything that you don't need you should throw away as soon as possible Right which is kind of against most of the pricing models that are out there right that's that's one of the big issues but I do think that you should not keep logs or trace it that you don't care right and as an example we just introduced a feature we call it the Telemetry spam filter which works very much like a span for you know may if you see a log and you say oh this log is useless right or you see a trace and it's just a ping to a service which just looks at the house you can click on it and say spam and we will create a filter and all those traces will not collect be collected anymore right that's really nice so a very simple I think to support users which is a little bit I can tell you as a vendor it's a little bit I even now that we are early it's a little bit scary right because people pay for the number of locks and traces and now you have people already paying you say oh

01:00:09: I release this feature there's a high chance that I don't know people remove 20 30% of the data so I make 20 30% less money right that's

01:00:18: that's number one so it's R remove data and I think you can do it

01:00:22: again think about at the moment it's a very simple feature users click regenerate the rule but now what we are doing we train a model in the background with all the user-generated rules and similar to spam filter in your Google mail I think over time we will learn what is actually spam and not and we can automatically remove the spam for you right so removing data sampling data is a big topic for us

01:00:51: because I think making the haystack smaller is in any sense good you have less data

01:00:57: less less things that you I mean again the male if you would have all the spam in your inbox and you search for something you get all the spam mail in your search results right but you don't want so finding the needle in Haystack is easier in small Haystacks right and then the I and Rift is improved the data

01:01:17: and

01:01:20: it there we have multiple things that we are doing as an example we released our lock AI if you send us an unstructured block we structure it for you

01:01:30: and in this case we really use an llm so we use an llm to analyze the log think about you send us a

01:01:37: an engine X log file and you know the first three is the HTTP status code so you get something like 200

01:01:45: URL

01:01:47: and then a number like 7,000 with is the response time in milliseconds

01:01:52: if it's unstructured it's just text yes you can search for 200 but what we would do is we will actually understand that the 200 is the HP status code

01:02:00: and we will translate that in the semantic convention hdb dot status under so code to 200 and now what you can do you can just filter

01:02:10: by saying give me everything that's an error and we know 200 it's not an error 200 is okay and and we can filter it out so by adding more semantics to the data structuring it or for example at metrics just figuring out what kind of a metric is it is it is is it milliseconds or is it a number account and by adding that to the metric that context for example if you put it on a chart we can select the right chart for you because we know if it's milliseconds it could be a different chart than if it's a gosh right and and these things we add so this is the improved part and we are ending more and more functionality to understand patterns to to do these things that's the second part where we want to really distinguish ourselves from the rest

01:03:00: and

01:03:01: sorry if you have a question just a quick tangent here the thing you're doing with the log AI it sounds incredibly expensive right if you are looking at all the logs is it like yet you are generating a rewrite rule or like a like a regex that is parsing out the fields and 100% correct we also had a few posts around that also when you look for AI Engineers there are different types of AI Engineers so you have a Engineers who can do something with python and and create you or use the llms but then you are I can see that you work with performance then if you have billions of marks you can't call the open iapi for each lock and say hey look at this lock right it's impossible it's too expensive to do that we actually do we generate reqction

01:04:01: over time right you you figure out oh L&M works well but then you figure out oh but I can't really apply it in production because it would be just too expensive and can I will let you go further in just a second but just yesterday I listened to the podcast and for the first time the insanity of the inefficiency of llms really I understood this right because like what you're actually doing is you have this kind of 10 billion of 100 billion of neurons which is this gigantic amount of memory that gigabytes of memory that you need to load and then essentially you are visiting all of this memory points monsters to a multiplication

01:04:37: and then you are arriving at the very end at one token and every time you are ingesting a token you are all doing this again and the big part of the cost is really reloading all the stuff like gigabytes and gigabytes that into the graphic card and then comparing this to okay I have like a very good expression that I'm executing on the CPU it's like so many orders of magnitude of efficiency that is standing in between this and this trend of can let me throw an llm and it it's surprisingly effective but you kind of begin to understand why it's so I've ordering expensive because of what you what you are actually doing is just so it's pretty insane absolutely absolutely but there are I mean if you are interested I'm I always because I'm also performance guy there is this idea of liquid neural networks I'm not sure if you heard of it now and MIT research and they are basically reducing the number of of kind of neurons you need

01:05:35: incredibly right so I I would say I can see that over time similar to what I said with index or something today you can call an llm for each lock maybe in two three years

01:05:49: it will be possible Right but I agree with you the inefficiency is I heard the chief scientist of Facebook he compared the current model

01:06:00: and the human brain is around 20 watts yeah and none of the llms has the power of a human brain but they consume I don't know what it was right for for one request how many megawatts you need and then he compared like 20 watts

01:06:15: against this the the power that you need to use an llm it's so the human race magnitudes more efficient than anything that we have today right that that's also an interesting comparison yeah sounds like the need of performance engineer

01:06:31: but I think it's also normal if you I think it is super normal that

01:06:37: things are very inefficient at the beginning

01:06:40: and then you can see orders of magnitudes in performance gains over time because then you and I I do think that we will see that here right I already happening it's already happening and I can see that in the in the future probably you can actually do that for every log right but at the moment as I said we we need to build something that works on large amounts of data and and that's at the moment it would be super inefficient thank you you stopped at the eye of the rift strategy and but you know I thought I just wait until the end of the roof but now we we interrupted in the middle of

01:07:22: when I look at reviews

01:07:26: you know I think if there is a there is some methodology to you know to to have spam fit us

01:07:34: I would immediately think okay and now I can lock more you know I I don't have to you reduce with a spam filter that's the price but very often we are limited you know people say oh you just cannot lock as much as deep because the cost is too high but now I hear okay with the same course I get more or higher quality because I can add more logs

01:08:01: we are not there yet but finally I would say yes finally if our system would perfectly work right and as I said at the moment it's a manual thing where you reduce but think about a world where you could log and send whatever you want and the system will filter it for you and figure out the things and also reduce the things because sometimes it's also you can reduce things by creating a metric so for example maybe you have thousands of logs that are all the same and then that time you could say okay I drop all the logs I keep only one but I have another metric which counts how many times I have this lock and you still have a lot of information you have one lakh and I can tell you and by the way this thing came up 100,000 times in this time frame right so I think if you have more and more intelligence on the art part but in we are at the beginning here then you could send as many data as you want and we would cool only keep the relevant data right which is

01:09:00: it sounds easy but you can imagine it's actually pretty hard because you somehow have to predict which data you would need to find the problem right but I I would estimate that today if you lock things and if you look at data and I I see it with a lot of customers there's a lot of garbage I mean obvious garbage in the data which you can really drop

01:09:23: I mean I remember a long time ago I wrote a logger my own logger and the logger worked you know it looked very details

01:09:34: but it you have lots of very detailed statements

01:09:39: but they are not locked yet

01:09:41: there was kind of a buffer in between and if you have an exception

01:09:46: all the things from the buffer are written into the lock if you don't have an exception if the thing runs through

01:09:53: you know no logs and that would be something which you know would be really cool to have you know I don't want to write this myself and by the way open till again coming back to open Telemetry the cool thing about the system is that you have the collector

01:10:08: and The Collector is in different levels right it can run inside of a pot it can run on your inside of your cloud and it can run on our side and so you have options where to drop things which is the interesting you have these things like tail sampling where you sample at the end but you could also drop metrics as you said you could have a buffer

01:10:28: inside of the process or or next to the process keep data there and then depending on some status codes you could drop it there right that's why what we what we produce with the spam filter is something called OTL

01:10:42: which is the language that you can use inside of the collector to decide if you keep or drop the data and you could deploy that ottl that rule either on our side so the data is already produced and sent to us but you could also use the same rule inside of the pot

01:10:59: and then it will be never sent to us we will never see it and you drop it earlier so I I totally agree with you but which shows how that it's a more complicated topic which can become pretty complex but if you write a system that can do that efficiently and smart I think that it's super powerful right and the big a big benefit in terms of price and cost but also a big benefit in reducing the hay

01:11:25: and finding problems more easily

01:11:28: might actually address and like a knot that I had in my head when you said well we have an automated automated spend detector I was like hmm and what is

01:11:41: that spam detector detects something that after the next application release is not spam anymore or it detects a new lock line that after the release is actually relevant so what you just said is if if it's configuration within the collector we could also like Mark these as don't don't recognize these for like 30 minutes or an hour and activate donate yeah it can become pretty sophisticated but I do think that removing data is a big part and I mean actually categories of tools if you know crippled

01:12:23: cripple is now I think they posted that 300 million our companies so it's pretty big company billions of dollars worth

01:12:30: that's a company that basically builds a tool that sits in front of observability and logging tools and removes data yeah I saw there the whole the whole kind of concept was oh and they started with Splunk so they said hey I think they are all worst blank employees right and they saw my customers actually sent

01:12:50: millions of dollars of data into Splunk and a lot of that is absolutely useless so we and Splunk will never build something to reduce the data because why why would they write and so they built a company where the only use cases by the way we save you 30% of the cost of slug

01:13:07: and you pay only 10% so it's a good deal for both right so that's that's a whole category and they're now like four or five different vendors out there that only do that that's called observability pipelines these days but it's essentially removing data

01:13:22: it's interesting I thought it's not on Vogue at all anymore I mean people are using Splunk since I would say decades but I don't know like how long is flying actually is around I've used it in 2012 I think they are using it for to connect observability data with business metrics

01:13:44: every now and so so which

01:13:49: I wouldn't say it's pretty reliable but anyways but like if you want to do that of course I mean your stretch the fields are because or works because you want to have proper data that is reliable so improved data yeah and you want to make sure that you can base on on that the same goals for the data that I used to for example train my AI models yeah

01:14:17: that reminds me of The Medallion architecture that that databricks is advocating for like improving the quality of the of the data so that you can actually make something of it but

01:14:32: when you go on with the rift yeah maybe that's not your mind to to elaborate like whether you already Target llm based applications and I can base tracing yes so just shortly the Ft stands for filter and triage and filter essentially again coming back to the haystack filters away all the data that's irrelevant for your troubleshooting case so you make the haystack even smaller so remove data improve the data and if you have improved that you can better filter the data example coming back to my example if you are understand that it's actually a status code and you search for an error you can remove all the 200 because that's probably not what you are looking at but you keep the 500 and the 400 right and then and then the last is triage which we also released this week and it's essentially then pinpointing you to the relevant data

01:15:28: and and what we do there is it's it's really not easy to explain you have to experience it we we designed to feature and we were not sure if it's working but once we had it it's mind-blowing so essentially you mark something in the chart and you say oh this looks

01:15:46: suspicious and what we then do is we take the data that you marked inside of the bubble and compare it

01:15:53: to data outside of the bubble right that you marked and and we have different algorithms for that and then what we what we then do is we show you the differences so think about you mark

01:16:06: 10 errors then we will tell you oh by the way for that 10 errors it's always that customer ID

01:16:12: that in there and in the ones that don't have an era it's not that customer ID right that's kind of the thing we do it's not AI or something it's really just comparing large amount of data statistically and giving you the result and it's remarkable good at doing that and just one point what I want to say that's why we don't call it root cause I read the block from Google which essentially explained why if you if you search for something you should not present only one result and that that was kind of mind-blowing for me it and this happened a few weeks ago that I understood it so they said people always try to give you only one result right if you say root cause they want to say this is the problem right but the way Google learned is

01:16:58: it is good if the first result is the problem but if you if it's not the first result and you present a second or third and a fourth and the user finds the result in that once they are still happy if you only present one result so think about Google search would only represent the first

01:17:14: hit

01:17:15: in 90% of the case it would be exactly what you were searching for you would be happy but if in 10% it's the wrong result you would be very unhappy with the experience right and so so what we do is we don't show you only one we really show you a ranked list of problems

01:17:31: and you can essentially look at all of them right

01:17:34: I mean the it's I find it interesting to say the root cause we mentioned Netflix lots of Netflix guys who who were in the chaos engineering space there they always talk about there is no single root cause in the complex adaptive system it just doesn't exist right there are always more things and I thought it's always be an S but

01:18:03: yeah but I think that's a really cool feature I I heard about this idea a long not a long time ago maybe two years ago it's a long time and software at it but I think Alex Hidalgo he also wants you know was thinking about this and I was like hi this is what what you want to have so it's it's good to hear yeah yeah I think you make it easy right for the user that's something that makes it a little bit tricky right like

01:18:31: again like the span filter just one click and make I think that's kind of playing the learning which I said before right you can have very complex technology if you don't make it usable right so maybe a spam filter is just super simple

01:18:46: and it works in 80% of the use cases and you just ignore the other 20% it's still helpful right that's that's kind of the approach that we are taking here

01:18:55: are you allowing to connect the trace data for example or the data generally in their 02 for example business metrics

01:19:07: it's it's a big topic not yet not yet I think it's one of the things that I really want to do and and and we have a clear opinion also how we want to do it but we don't have it yet no and I really want to connect a lot

01:19:24: lemon tree and based on all the Telemetry right I can think about you see a span a call to something like I don't know a call to a checkout service and that includes all the data what's checked out the revenue and then I really think about having just one click and you create the metric like Revenue

01:19:47: right

01:19:48: and now you have the metric on the dashboard but it's actually connected to all the spans

01:19:54: which allow you if you have a drop in your business metrics so Revenue drops you can directly connect it with the data

01:20:03: to find the root cause for example it could tell you but by the way these fans now have an arrow because the database is broken right so you have a connection between your Revenue drop and the database problem right away so I do think that connecting business metrics with the underlying open Telemetry data makes a ton of sense right ton of sense and that's something we don't have that yet we also have them started building that but I'm

01:20:30: I'm really I mean this would be like the thing I want to build yes

01:20:35: yeah one thing that really impressed me when I looked at the digitally row demo was how opinionated it was and how much like value was built in so if you are onboarding you are directly kind of seeing like things in a perspective that is helpful and yeah for example like the service graphs are build out and then you have the red methodology to look at the apis and the use methodology I can kind of saw some of this with the resource kind of metrics you're correlating so I think that is a very important tenant for specifically like the onboarding experience like zero to Value but also 0 to how much value if you have like you are selling a Swiss army knife then it's really like okay you can probably kind of engineer a lot if you are investing like a few months of time but I felt like the out of the box values pretty good and also given that it's a really new product I I was

01:21:37: of the features have it was really like okay I remember running into this problem and we talked to a PM another year later we have it and now it's like well you can already add attributes to the table and then that's just there so can you maybe talk a little bit about your kind of product management strategy behind it and like you're kind of approached like apparently you didn't follow the release early and iterate strategy but but to put some more color into this I think is I'm not really sure if you classify as a technical founder but technical Founders struggle a lot with exactly this value proposition and the product management aspects I think let's there are multiple aspects of it and I will try to explain them first why I'm a big believer in releasing fast and iterating with customers but I didn't do that here for the first iteration because we are in a red Ocean Market

01:22:38: if you start in a blue ocean market so if you don't have too much competition in it's not an established Market you can start very early with an MVP and people will just love it and start but in our case if you have a data if you have a graph I if you have other competitors if you start with the crappy product

01:22:56: people can compare it with existing tools and they will not accept it right that's just my experience I have so I said we have to start with what I call a minimal lovable product it should be minimal but at least it should be lovable right and then coming to the topic of yes these little things like I love the thing that you can take every tag and make it a column in the table in these things right and make everything filterable and I think that's a culture I think

01:23:24: I really believe that the only way you can achieve that level of details and and functionality is by making sure that the whole organization cares

01:23:36: and is specially the Developers

01:23:39: because I I saw this a lot also in the Sun and other companies that if you have product managers if you have these designers at some point developers don't really feel responsible anymore for the quality what they are building because it's a I'm just doing what I got in figma right

01:23:58: I and then sometimes I really had this moment at instana I have to share this right the publicly where I came across Ben Ben was working a program he's not the CTO was our best programmer and I looked at the screen as I saw this looks like in Stana but it's different right

01:24:15: and I looked at it said what is this right is in the future and they said no I built my own dashboarding system because I I don't want to use the dashboard against all right and I was like why yeah he said because it it it's clunky it doesn't work and I like what the fuck right why don't you fix

01:24:33: this thing inside of instant for all of our customers because they can build what you did right and and so I I was really mad but I felt that it's actually a culture problem because somehow

01:24:46: the develops had the feeling that they can't change this thing that they can build the product they love

01:24:52: and they were just doing what the prompt managers were telling them which

01:24:57: we're not developers and couldn't feel the pain probably right and so you have to establish a culture where

01:25:04: you kind of

01:25:05: you have to really ask for it right that's that's a different I think if you just say hey we are open cultural blah blah people developers are just not not all the lot of developers are not that vocal right

01:25:19: and then you really have to make sure that you have a process and you have meetings where you ask for feedback where you ask like is that something you want to have what would you change and you have to integrate them in the whole process and that's what we do I think pretty well at the 0 that we integrate the whole team in the process of Designing features and making sure that we like it alright everything is keyboard usable because yeah people developers love the keyboard and they and we look at that it's really working well right that you that and and it's little details

01:25:51: and and the only way you can do that is by making sure that everyone in the organization cares right including myself right I look at things I test them I

01:26:01: yeah that's that's kind of so I'm thinking and then the other I have to add is that we have a very high ratio of designers to Developers

01:26:11: so you have a lot of like designer and product managers as well but just we have really good ones but we have much more designers and product managers so our ratio is around 6 to 8 Developers for one

01:26:26: designer right where I think the average in the industry is 40 developers on a designer right and the Really best ones is around 1 to 8 and I think this ratio of 1 to 8 1 to 6 is really really good right and you have to find people that are

01:26:42: designers and ux people who also understand the domain right it is very hard to have a kind of web designer who doesn't know what a container is in a pod and kubernetes every designer has to learn from ql so they understand query language and they have to kind of understand the concepts and you have to give them time to learn them that they become really good at designing and working with the developers speak the same language so I think that's a combination of how we try to make sure that the product really is opinionated as you said right yeah that's all so a point but also kind of lovable and and cares about details

01:27:21: yeah I mean I was impressed so thank you for for doing this and yeah I think this

01:27:29: um if you start with the right culture I think it's kind of self-propelling but if you are like having a different culture than kind of changing it to the place where you need to be to build an amazing product that's that's a much harder Journey but I like the focus of ux design and then I think observability and Telemetry is special since it has so much of a dog fooding aspect to it right a lot of the developers they kind of they understand the product on a deep level and Converse the product person coming in

01:28:03: has always the struggles so maybe like keeping the product management more in the technical hands but having a lot of designers who are then making this really compelling and making it smooth I think that's very very interesting so something I always tell my my team is like it's it's a really rare case where you as a developer are also primary user of the tool that you are building right I mean that's kind of big privilege right if you build an online banking you also kind of a user but not the user but in our case our developers are like the perfect fit

01:28:37: and we as the zero also perfect as a customer for our own tool right so it's like we are building a tool for ourselves somehow

01:28:45: nice yeah okay I'm feeling now you know I I currently use a product which can do everything it's in the Gartner quadrant it's you know very right up

01:29:00: and the problem is nobody is using it because it's too complex I think I'm the only person because I say because I'm the owner of the topic

01:29:09: but it's

01:29:12: I think that simple to offer simple tooling you know easy to use that so important but also it's also again coming back to my initial this is probably a good way of finishing up this thing right coming back to the initial idea was that things are changing and there's another thing that I learned and I like to compare it I in 2001 I was attending a conference in Boston called the server side conference I don't know if you remember the service I was a big website for Java Technologies and I had breakfast which became info queue later I know yeah right yeah exactly and so I was sitting at the breakfast in the hotel with the with the young guy in the white T-shirt and it was it had the teacher had jira on it and it was Mike Cannon Brooks and he released jira one at the conference right and I learned about jira and every open source at that time axilla

01:30:12: was the big thing right and and jira was so much nicer right the usability was nice it was so it's such a great tool right and every developer wanted to use it

01:30:23: right and that's a story about one two 2025

01:30:27: I think if you I tell my sorry to say that if I tell my developers we will use jira they will all quit their job right because it has become a bloated large Enterprise tool and they do billions of dollars of Revenue it's a great tool but it's an Enterprise tool now right and I think that's kind of what happens and now you have linear

01:30:46: which is the smaller good usability

01:30:50: opinionated tool which

01:30:54: is nice but the reality is if they are super successful and they will grow they will get more Enterprise customers

01:31:02: and they will add more features for them because they spend millions of dollars with you and then 10 years

01:31:07: it will be the next jira right that's just and it's the same in observability right you can start at the cool kid on the Block

01:31:16: and then over time if you get more Enterprise customers and more features you will get more complicated and I haven't found a solution for that to be honest I think it's just

01:31:26: here the way

01:31:28: things work right it's if you if you are successful and you grow into Enterprises I think there it's it's really hard to keep

01:31:37: your opinionated focus on value on the core use cases yeah and and it just becomes

01:31:44: an Enterprise tool and and then the smaller customers and the cool kids will not use it anymore right that's just

01:31:51: I don't know if you have an idea about the solution for that but I think it's kind of a little bit it's not even the innovators dilemma it's it's it's it's really lever of becoming successful and and Catering the large Enterprise of the world who at the end give you the budget that you need to grow as a company and that's kind of the way I see it and and that's again if you are in the magic quadrant in the top right corner

01:32:14: you are probably in the Enterprise space and then your product gets more complicated right that's that's just the reality of it

01:32:23: yeah I think so the nice thing sorry is just to have somebody break into this market right with observability it's really like this you have like established player which also like data like examples that just started extracting a lot of money and they have so much advantage that it's really hard to catch up and I think we are seeing something unique where you were able to position the product which is already competitive at a very early stage and you were able to kind of Leverage some of the advantages most notably open Telemetry on the technical side I heard your click house user which I think it's also a big step up from a company we had to write our own time series database that was a little bit of a drag basically was half the engineering team that was keeping that alive so it's it's extremely good to see that there are new players there's obviously kind of the risk that IBM will grab it or a Splunk rabbit or Cisco Revit I don't know

01:33:23: so as a like

01:33:26: markets participants I hope you stay independent for a while but I think that's that's just an example of a development where it kind of gets interesting again and maybe another situations there's also hope but it's like it's never clear if you will see this absolutely and I can tell you that I this time I really want to stay independent also because I learned that selling the company and not having the company anymore is not enjoyable right so yeah I I'm really enjoying the way and I I want to take it a little bit further this time yes nice hopefully not becoming art I would love to become the next year to be very fair right I mean it's a super successful I think what my kin Brooks is not the richest Australian right they are they are super successful with what they have build but I I would love to build this tool first to be a really super usable lovable observability tool that solves the problem that I mentioned

01:34:26: and and that would be fun yeah

01:34:29: I would like to go back to your previous lesson and regarding the tool set

01:34:39: meter statement that I took from that is basically that you said well if it will become successful it will grow more features because it wants to attract even more users and then it's not clean and slim and focused anymore so

01:34:57: the main thing that I get from that is actually it's totally normal that tools cannot make

01:35:06: more than a few of the ways that you that you define earlier because at a certain point in time they grow so big that they can hardly make the change

01:35:19: exactly and then it really depends how long you survive it really depends in on the integration level into the stack of the customer so how painful is it to remove

01:35:28: to say sap survived so long

01:35:31: because you

01:35:34: yeah but yeah but you also can't remove it right it would be just too expensive to remove it so you can you can basically become such a enterprising DUI is maybe not that good you have thousands of tables which are complicated but but it's not replaceable easily right but if you are a tool like I don't know task management I'm not sure if it's a good example right but if you are a tool that's not so integrated right and it's easier to remove I think you there's a much higher chance that you will get disrupted and in the observability space you can see that right

01:36:10: when I started it was Wiley interest scope right while interest scope I don't know if exists anymore right and and the founder of New Relic afterward Lucerne right and you really knows pee play they're still around but who knows right and then he makes right I work with them and Dynamics got bought by Cisco and then now they bought Splunk and Splunk bought signal effects so I you know what I mean I think it's a it's a constant wave that every five to ten years the tools that were there ten years ago I thought do not exist anymore or there is a new way of starting so yes so to answer your question I really think that's the case not only in observability it's the same insecurities in developer Tools in

01:36:56: I mean you know you know which tool set you use 20 years ago not So Much from the application server stack and ideas and everything

01:37:06: most of that became again

01:37:09: eclipse

01:37:10: I remember when I used the first version it was so small and nice to use and then over time it became bigger and bigger and bigger and more you know what I mean and and then it was like the visual age for Java that I hated before exactly yeah so it was said and ugly before exactly yeah I mean don't want to be not the same guy right Eric gamma build H for before and then he build eclipse and then product managers around him if I remember correct yeah so that's the I think it's just natural that things

01:37:47: I mean I would love to say that I have the formula to prevent that to become big and stay nice and and it's it's hard

01:37:58: let's let's assume for a second that someone made it through the one hour 38 minutes to listen to our conversation and they are Developer that

01:38:11: I had an idea for a little software a little tool that actually solved a problem of their team but

01:38:17: they don't know how to start a business out of it what would you recommend to that person

01:38:24: or should they started or what to consider actually

01:38:29: I mean first of all I can say I did more than 80 Angel investment and invested mostly in developers write as Founders and I saw very different types of people people who are very introvert had no idea of business and they totally succeeded right so what I can just tell you as a paradigm is that there is not such a thing like a good

01:38:54: startup founder that is an extrovert or you can't you can't

01:38:59: see that I know that a founder of uipath I met him very early Daniel Danes who's now a billionaire and the nice company when I met him the first time he was really like this nerdy couldn't speak English very well right and and I know he took lessons afterwards in English and also from actors to be on the stage and how to present himself I I read the same about Martin Fowler right that he did a lot of acting lessons to to feel comfortable on stages and so what I want to say is I think being an engineer is is kind of a gift in if you want to build a company around software because

01:39:39: the chorus the product right and everything else

01:39:43: you can find good people who will help you with that right who will help you with Marketing sales building a great company so I love to invest in good Engineers so my recommendation would be if you have an idea for something that really solves a problem

01:39:59: just go for it right I mean it has become so easy right these days think about 20 years ago right you would have to buy servers to build something these days you can start in the cloud as a startup you get like 100,000 dollars of free edws or free Google Cloud so you can basically for one or two years you can work for free in the cloud you have all the infrastructure you have all the tools

01:40:23: it is so easy to build a software company these days and then there's a lot of money out there investors who are looking for you there's AI I mean I just love the space and the time at the moment I just think you can you can just do it right

01:40:38: with the only thing I can tell you is saying from myself it is much easier for your young to do that

01:40:46: but then when you become older not because you're older means that you have probably kids you have a house to pay and the risk for you is much higher when you are 24 out of University and you lived in the dorm anyways and you need 500 bucks a month for a living it the risk what you are losing is so much lower than if you you know what I mean so it's it's always so I can't do a general recommendation because it's always there's a high chance that you will fail right if you build something like that and then

01:41:21: again if you are 24 and you spend two two years in building a software and you fail what have you lost nothing right you gained a lot of knowledge a lot of experience but if you are 50 like myself almost 50 and you have a house and kids and and you you waste three years of your life and

01:41:37: maybe you try harder then just yeah maybe maybe you do but it's not always that I don't have the feeling that money problems are actually a good motivation to become better they are normally I can say for myself record Centric when you when you have some problems not knowing how you pay the next mortgage or something that's not not nice right it's really not nice so I can't recommend that right so that's why I also recommend BCS that they take care of that right if people like who are more established found a company they should make sure that they are financed in a way that they can spend get a good sailor and have that security somehow because it's something different if you are again 24 and and 48 the double the age is

01:42:29: you are in a different setting right so but this set I think the opportunity for software companies is just has never been so good

01:42:38: um so my recommendation just go for it right okay and like legal Consulting and things like that they could for example get from a VC I mean you've been moaning about the bureaucracy topic quite a quite a bit yeah that might actually turn people away from founding

01:43:03: absolutely so there are two learnings and I can't give legal advice here right to but I have to say that

01:43:13: I think the risk is very low right the number of people who founded the company and are a general manager and end up in jail or anything is almost close to 0 if you don't do anything really illegal which you should I mean I mean not

01:43:30: doing something where you didn't know that you have to do it I mean something really bad right then like like why are caught bad right then then can end up in jail but if you if you do normal business and you do everything and you have your text consultant right and maybe a lawyer or something then I don't think that the risk is really high yes there's some bureaucracy yes agree but yeah you can it's handled bill right it's it's not that that big so the risk is not that high right but if you would read a book what the risk for a general manager are you would probably never found a company right because it's

01:44:08: there's a lot of risk but the reality is that

01:44:12: nobody will end up in jail because you file the text too late or whatever right that's just not

01:44:20: the case yeah

01:44:22: Nicole

01:44:25: it's been a pleasure I have to say

01:44:28: and time flies

01:44:31: is there anything that you would like to say as the last words for the podcast

01:44:39: no thank you for having me it was really fun I appreciate all your knowledge in this space to be honest what's really deep discussion from Colonel level up there that's that was really fun and yeah looking forward to hear more episodes in the future from you

01:44:57: thanks a lot and yeah I mean that's words from you Sven or Heinrich

01:45:03: thank you Miracle this was amazing yeah yeah I you know I had a lot of more questions but I in in the middle I thought if we go deeper and deeper and deeper you know we end up with the six hours or something like that very nerdy long-term

01:45:24: yeah I mean

01:45:26: the good thing is actually that Sven and Tyler and myself can keep the questions because we know that Ben Blackmore your CTO also said that he would love to join the podcast conversation and we're really looking forward to this and maybe go even deeper but let's see absolutely and been really is the best developer I've ever seen so he's

01:45:56: a pounding he's he's amazing he was he the one with the dashboard too yes

01:46:11: so then I say things a lot to you guys and thanks a lot to everyone who has made it through the podcast so far thanks for listening and yeah hope to see you on the next episode have a good time

01:46:24: thank you cheers bye-bye bye