Developer Tools And Architecture: Mike Hall Interviews Erik Meijer | GOTO Conference 2013
Developer Tools And Architecture: Mike Hall Interviews Erik Meijer | GOTO Conference 2013
🚀 Discover how the reactive framework is revolutionizing software development by handling both synchronous and asynchronous data streams. 🌐 Learn about its language-agnostic design and the impact of its open-source community. 📈 Don't miss this opportunity to understand the future of developer tools and architecture! 🌟 #reactiveframework #developercommunity #softwarearchitecture #distributedcomputing #open-source
The Interviewer
Mike Hall
Interviewer, UGtastic
The Guest
Erik Meijer
developer tools and architecture
The Conversation
Mike Hall
Interviewer, UGtastic
Hi, it's Mike again with UGtastic. I'm sitting down at GOTO Chicago with Eric Meyer. Eric Meyer is, well if you've worked with Microsoft technologies over the last decade, you've worked with something that Eric has created or been a part of, from C# to the Azure cloud platform. Now you're away from Microsoft and you're working a lot more on the open source world. Well, first off, thank you for sitting down with me, but can you tell me a little bit about what this platform that you just spoke about, this reactive framework, what is that and what is that? Okay, so I can try to explain that in general terms because that I think is also where you mentioned open source, where the kind of the value of open source comes in. So the thing is if you're trying to compose software, in my view, there are four different kinds of effects that as a developer you have that you have to take into account. So let's look at the simple, you know, when you're doing, when you're programming Java and you're calling a method, typically the method returns one value and that method is actually synchronously. So you call, you know, two string and, you know, that call blocks and then returns a single string. Right. Right. So that's kind of the simplest synchronous call that returns one value. So input, wait, output. Yes. Now the second thing that what you do is that if you make a call, synchronous call, and what you get back is a collection of values, like an iterable in Java. Right. So you make a call and now you get back an iterable, but that iterable represents multiple values, but also they are synchronous because you, you call next, next, next, and each call to next is blocking. Right. Okay. But the, the, the, the important thing is that there's two, you know, ways to compose your program. One where you have a single result and one where you have multiple results. And then, you know, when you have multiple results, you loop over that and, and so on. But those are for, for synchronous programs, those are the two kind of main, um, ways to compose your program. Now let's go to the asynchronous case because that's the very important now that we're moving to more kind of, you know, distributed world. When you make a call, you don't want to be blocked because there's like latency, errors, whatever, you know, we all know the kind of fallacies of distributed computing. So what you want to do is you want, you want to make these calls asynchronous. Right. Now, and there's a lot of, uh, talk these days in JavaScript, in many languages about how to deal with asynchronous calls. And in my opinion, when you have an asynchronous call that returns a single value, um, what you do is you return some kind of future or,
Erik Meijer
developer tools and architecture
a promise. A promise represents a value that may not be available yet, but will be at some point in the future. It allows you to handle the result of an asynchronous operation in a more structured and predictable way. When you have an asynchronous call that returns a single value, you can return a promise that will resolve to that value once it's available. This allows you to chain multiple asynchronous operations together in a way that's easy to read and understand.
Mike Hall
Interviewer, UGtastic
And what about when you have multiple results? How does the reactive framework handle that?
Erik Meijer
developer tools and architecture
In the reactive framework, multiple results are handled using observables. An observable is a sequence of values that can be subscribed to. When you subscribe to an observable, you receive each value as it becomes available. This allows you to handle multiple results in a more declarative way, without having to manually loop over them. You can also use operators to transform and filter the values as they're emitted.
Mike Hall
Interviewer, UGtastic
So, can you give an example of how you might use the reactive framework in a real-world scenario?
Erik Meijer
developer tools and architecture
Sure. Let's say you're building a web application that needs to fetch data from multiple sources, such as a database and an external API. With the reactive framework, you can create observables for each of these sources and then combine them using operators to create a single observable that emits the final result. This allows you to handle the data in a more efficient and scalable way, without having to manually manage multiple asynchronous operations.
Mike Hall
Interviewer, UGtastic
That's really interesting. Thank you for explaining the reactive framework and how it can be used to handle asynchronous operations in a more efficient and scalable way. It's great to see how open source technologies are being used to solve real-world problems.
Erik Meijer
developer tools and architecture
Thank you, Mike. I'm glad I could help. If you have any more questions, feel free to ask.
Mike Hall
Interviewer, UGtastic
You know, promise or whatever, how you want to call it. And then .NET, that is, they use the type Task for that. So a Task represents a computation that is asynchronous, that has a single result. So you make a call, you get back this thing that represents, you know, a promise that, you know, there will be a value at some point. Um, now what Rx is, is that it complements, this picture by having, what happens if you have something that returns multiple results asynchronously. So say you make, um, the most natural examples, maybe Twitter, you make a call, you send a query to Twitter, and what you get back is a stream of all the tweets. Right. And you're not asking, give me the next tweet. No, Twitter will kind of, you know, notify you when the next tweet. Just pulls up a buffer, kind of. Well, it just notifies you with events or something, or like your, and even on your local machine, if you're doing, um, events processing, like your mouse moves, or so on, you're not asking the mouse, give me the next mouse move. Right. You get notified, so you get a sequence of, of collections. Sorry. Probably shouldn't drink, uh, soda. It's okay. Um, now, so, so, so what Rx is, is it, it's trying to kind of, you know, deal with asynchronous computations with streams of data, asynchronous data streams. Okay. And, and, and, and if you look at languages like Erlang, they have embraced that, that same paradigm for a long time, right? Where you do message passing, you have two different agents that send messages to each other, um, and, and so what this is trying to do is to define an interface that you can put on, on different services, and then you can glue these things together using this interface. Okay. Uh, a lot of the examples you gave were in, in .NET. Is this a .NET only thing? Ah, that's a very good question. So one of the, um, things is that since this is an, an abstract interface in the sense of abstract, it's a conceptually abstract, there's no language dependency. So we, we, we, um, on the open source side, it's like, you know, rx.codeplex.com. Okay. Where we open source this. We have versions for JavaScript, for .NET, and for C++. The C++ one is not as mature. Right. Um, then Netflix has written a Java implementation. It's a Java implementation. Um, GitHub has written an Objective-C implementation, and in Dart, uh, Google has the Dart streams library, which is also, um, based on these principles. Okay. And then other people have done, like, Python, and, and Ruby, and so there's, like, you know, again, this idea of having asynchronous data streams is not tied at all to a single language, and that's exactly also what I want, because you don't
Erik Meijer
developer tools and architecture
Understand. So Rx is a way to handle asynchronous data streams in a more functional and reactive way. It allows you to work with streams of data in a more declarative manner, rather than in a procedural manner. This can make your code more readable and easier to maintain. Rx provides a set of operators that you can use to transform and combine streams of data, and it also provides a way to handle errors and manage the lifecycle of your streams. Overall, Rx is a powerful tool for building responsive and scalable applications.
Mike Hall
Interviewer, UGtastic
That's a great explanation. So, can you give me an example of how Rx is used in a real-world application?
Erik Meijer
developer tools and architecture
Sure. Let's say you're building a mobile app that displays a list of tweets. You can use Rx to handle the asynchronous data stream of tweets from Twitter. You can use the `Observable` class to represent the stream of tweets, and you can use the `Subscribe` method to start receiving tweets. You can also use the `Map` operator to transform each tweet into a more user-friendly format, and the `Filter` operator to only display tweets that match certain criteria. Finally, you can use the `Dispose` method to clean up the resources when you're done with the stream.
Mike Hall
Interviewer, UGtastic
That's a great example. So, can you explain how Rx is different from other asynchronous programming models?
Erik Meijer
developer tools and architecture
Sure. Rx is different from other asynchronous programming models because it provides a more declarative and functional approach to handling asynchronous data streams. Instead of using callbacks or events to handle asynchronous operations, you can use operators to transform and combine streams of data. This can make your code more readable and easier to maintain. Additionally, Rx provides a way to handle errors and manage the lifecycle of your streams, which can help you build more robust and scalable applications.
Mike Hall
Interviewer, UGtastic
That's a great explanation. So, can you give me an example of how Rx is used in a web application?
Erik Meijer
developer tools and architecture
Sure. Let's say you're building a web application that displays a list of products. You can use Rx to handle the asynchronous data stream of products from your database. You can use the `Observable` class to represent the stream of products, and you can use the `Subscribe` method to start receiving products. You can also use the `Map` operator to transform each product into a more user-friendly format, and the `Filter` operator to only display products that match certain criteria. Finally, you can use the `Dispose` method to clean up the resources when you're done with the stream.
Mike Hall
Interviewer, UGtastic
That's a great example. So, can you explain how Rx is different from other asynchronous programming models in a web application?
Erik Meijer
developer tools and architecture
Sure. Rx is different from other asynchronous programming models in a web application because it provides a more declarative and functional approach to handling asynchronous data streams. Instead of using callbacks or events to handle asynchronous operations, you can use operators to transform and combine streams of data. This can make your code more readable and easier to maintain. Additionally, Rx provides a way to handle errors and manage the lifecycle of your streams, which can help you build more robust and scalable web applications.
Mike Hall
Interviewer, UGtastic
Do you know if you're on the server side, using Java, and how to use Java? And I, on the client side, use JavaScript, and I still want to, kind of, exchange events with you. Right. So, um, that's, I think, very important that, that it's not tied to a particular language. Okay. So it's a little bit more like a protocol? It's a little bit more like, like, like, like a protocol. Okay. Um, because the, the, the, what we call the Rx grammar, or the Rx design guidelines, so if, if you and I are exchanging messages, um, what the protocol says is, like, what is the, you know, how are the messages represented, what is the binary format, or what is the kind of, you know, what is the state machine that we use? So Rx has a very simple state machine like that, so I can send you a number of messages that are ordered, and then I can terminate that stream successfully, that means you don't get any more. I can say something went wrong, and then, you know, so that's really the, the, the, the protocol that, that there is, but it, it, it makes certain kind of assumptions, so, but, what, what is, you know, I, I'm, I'm not so sure if, if it's really relevant to make a distinction between a protocol and an API, because you can also say, if you have an API for files, right, you have to, you open your file, you write, and you read, and you close the file, that's kind of a protocol, too. So, for me, I, yeah, I don't, it's somewhere in, in between, it's a, well, I, I think, I, I like to emphasize what, what is the same instead of what's different, so there are certain ways that you should use, or expectations of, in what order you make calls, and, and whether you want to call that a protocol or not, I mean, that, I'm fine with either way.
Erik Meijer
developer tools and architecture
Well, is it kind of like if, if you start to use the word protocol, it becomes kind of rigid, and it becomes dogmatic. Yes. And then, then people. Because protocol is very formal, and it's like, everybody has to do it exactly this way, and if you don't do it this way, you're not, you're not reactive. Yes. That's the next thing you're going to say, this isn't reactive enough, this isn't reactive. Yes, and, and, and so I'm trying to be very easygoing with that, because, for example, you know, if you have an existing service, for example, I, I, I'm, in the past weeks, I've implemented a wrapper for Loggly, which is a, is a, a logging service, so what you can do is you can send it values, you know, events, you know, whatever they're just like event streams and then you can query them later so it's like now what
Mike Hall
Interviewer, UGtastic
What did you do to wrap that in a reactive shell?
Erik Meijer
developer tools and architecture
I wrapped it in a reactive shell so that the underlying thing doesn't know anything about RX. It's just whatever the REST API that the locally guys and exposed. Now, by making your client library talk RX, there's no difference between you know, locally or your mouse. Now you can kind of mesh together things that come from locally with your mouse or and you can visualize it. So it allows you to compose. By exposing all these things with the same interface, then you can compose them, but you don't force the interface kind of all the way down. So that's kind of bridging the protocols right. You can always you know, and one of the things I think about that's interesting for as a is more of a I'm a business developer. I'm going to implement just an implementer. Um, is thinking about the log not as just a place where things go to die, but that the log is is something that that's alive and can feed back into the application and provide information for back into the app. Not just oh this is a thing that I go to figure out when something went wrong yeah. So, so this is it's kind of interesting if you if you look at and and again you have to be very careful that you don't abstract too much because then everything looks the same, but if if you look at the UI for example right, what is a UI well, I'm sending things into the UI and maybe to change the back color of a button or to kind of you know make a button disabled and out of the UI come a stream of events like mouse clicks and mouse moves and and button clicks and then my my code my UI code transforms that stream of of UI events into other events that it sends into the UI. So there's this loop where there's the UI and then my kind of event processing and they are kind of coupled in a loop now. If you look at something like locally for example there's no difference there because my program sends events to lovely and then can query those events and then go back so you get the kind of same loop so and suddenly a lot of these things now look very similar right and because you're observing you know like it's like when you write UI code it's like you're observing the UI and based on what you see you're kind of doing something with the UI so you know what is your logging system you're observing your running code and based on what happens you can kind of you know do something you respond
Mike Hall
Interviewer, UGtastic
to that yes and like when you say yes exactly and when you say logging you know that goes there to die yeah then why do you do it
Erik Meijer
developer tools and architecture
you do the logging to observe the behavior of your code such that you know intercept when something you know unexpected happens or yeah
Mike Hall
Interviewer, UGtastic
and well in a way it's thinking about what my relationship is to the application and then starting to abstract what i do where where i run my app and then i watch the logger and i'm watching messages fly by and then i'm like oh that wasn't right then i go do something well okay now i can start to abstract that thing that i did and realize that that stream is data yep and i am the observer watching that data and then i'm going to react and and i think that's just a way to kind of um realize what's going on with the reactive yes that's that's that's how i just processed it now yes and that's exactly right and then what what i'm trying to do is that you can automate yourself away yeah so you write then now what what the next step i would say to you is that now write some code that will do what you what i did yeah and then and then now you can kind of you know create more value for your business because now because once you can automate it really you know a human doesn't need to be there or one thing that i'm also trying to do is that there's many things where the data like with with these logs for example i think this is a nice example there's so much data in the logs but you what you have to do is you have to turn that into something actionable for humans right and so i think that that's one of the things that i'm trying to make easy where you can take massive amounts of data and then you know use the computer to kind of you know munch that such that at some point you get something that's useful at the human scale right because we cannot deal with millions of events per second i mean even nobody can do that right because we can maybe deal with like you know one event or alert every five minutes right so what you want to do is that's kind of you know you want to boil down all that data into something that's consumable by humans and the rest should all be automated uh to take a step back a little bit so we talked about how this is a this is a uh not to use word protocol but this is a way that many different types of platforms and languages can interact with each other uh so i'm assuming you have to share the this these concepts with many different audiences uh you you Do you have a Microsoft audience who's implementing in .NET, a JavaScript audience who might be running Node.js or browser-based applications, or a Java audience? How do you share this message? Do you use Microsoft examples of JavaScript, or do you tailor for what they do or for Java?
Erik Meijer
developer tools and architecture
When talking with JavaScript people, you should use JavaScript examples. One of the things that I really enjoy now is that it's in some sense much easier to embed myself in a different community. For example, Netflix did Rx for Java, and while the underlying ideas are the same, there are very big differences in the Java and C# languages that affect the concrete design of the API. In .NET, there's this notion of extension methods where you can take any type and add instance methods to that type independent of the type itself. For example, if you have an array of int, you can add a new method like conceptually to an array of int. In Java, you cannot do that, but Java 8 has virtual extension methods on interfaces that are slightly different. The design of Rx for Java has to stay within the constraints of Java. When working with people who are doing an Rx implementation for their language, I try to make sure that it feels natural for their environment but still embodies the same underlying, more abstract mathematical principles. For example, in JavaScript, there are already functions like map and filter. When you do Rx, which is the push-based version of that, you're not going to call them select and select many; you call them map and where because that's natural. That's kind of what I'm really looking forward to. One of the things that I'm really looking forward to is to work with all these different language communities to make this thing feel natural for what they're doing and then still being able to interoperate between machines or between languages. The protocol should be consistent, but the implementations should feel natural.
Mike Hall
Interviewer, UGtastic
We did the JavaScript version in the beginning in JavaScript. The casing of identifiers is opposite of C# so in C# it's like you know, two strings start with uppercase. I always forget whether that's camel or Pascal case. I always have to do we look at that up on Wikipedia. Whereas in JavaScript, it's different, right? Right? So what we did in the beginning is we used the .NET naming conventions, and then all the JavaScript people would go like, and then I was like, oh man, that's bike shedding, but then I realized, no, I'm wrong because I find it annoying too. If you have to go the other way, exactly, so we changed that. Now, you know, then Jafar Hussein from Netflix said, oh, we should change the names too. So my thinking has evolved very much that now it's like okay, you do what's natural for the language and then, um, you know, make it still conceptually the same and then you pass the kind of same test. So there's there's a whole bunch like I don't know, like 3000 unit tests and that specify the semantics. So it's really maintaining the semantics across languages but with different implementations. And and I must say it's it's that is the hardest part is that some things don't translate so you have to which is also nice right like for example what I mentioned with the extension method. So you cannot just take the C# um implementation and just plug it into a different language. So you have to really work hard on that to to make it natural and to make it possible in a different language, okay? If I'm if I have some pet language and I'm looking at I'm very interested in this reactive stuff and I want to implement it in in my let's just say Python. Let's imagine Python doesn't have anything and where where would I go to learn about um or or talk to people and understand what how to implement it what what the the underlying principles are okay? So that that that's that's a very good question. So there's um there's learn rxrx there's learn rxrx dot com so that that's uh one of the .NET early users that has written a book. There's also Paul Betts and Jesse Liberty have written a book, a very thin book on on Rx, um which I like because it's thin. And then there's the Rx guidelines that give you a little bit more of the semantics okay. Um and and and so I would say you know, you start playing with the .NET version or the Java version um and then you know you you go from there. The other thing is that in some sense the JavaScript version that we have um for .NET is a little bit like a reference implementation because JavaScript doesn't have threading and and so so
Erik Meijer
developer tools and architecture
So, if you're looking to implement reactive programming in Python, you can start by learning about RxPy, which is the Python implementation of the Reactive Extensions. There are several resources available online, including the official RxPy documentation, which is quite comprehensive. Additionally, you can look into books like 'Reactive Programming with RxPY' by Antonio Cangiano, which provides a good introduction to the subject. There are also many online courses and tutorials that can help you get started. It's important to understand the underlying principles of reactive programming, such as observables, observers, and operators, to effectively implement it in any language.
Mike Hall
Interviewer, UGtastic
That's great. Thank you for the detailed explanation. So, in terms of maintaining semantics across different languages, what are some of the challenges you've faced and how have you overcome them?
Erik Meijer
developer tools and architecture
One of the main challenges in maintaining semantics across different languages is ensuring that the behavior of the code remains consistent, even when implemented in a different language. For example, in C#, we have extension methods, which allow us to add methods to existing types without modifying the type itself. However, when we try to implement this in JavaScript, we need to find an alternative approach that achieves the same result. One way to overcome this challenge is to use higher-order functions and closures to simulate the behavior of extension methods. Another challenge is dealing with differences in language features, such as threading and asynchronous programming. For example, JavaScript doesn't have built-in support for threading, so we need to use libraries like RxJS to handle asynchronous operations. By understanding the underlying principles of reactive programming and using language-specific features to implement them, we can maintain consistency across different languages.
Mike Hall
Interviewer, UGtastic
That's very insightful. Thank you for sharing your experience. So, in your opinion, what are some of the key takeaways from your work on reactive programming?
Erik Meijer
developer tools and architecture
One of the key takeaways from my work on reactive programming is the importance of understanding the underlying principles of the subject. By understanding how reactive programming works, we can effectively implement it in any language. Another key takeaway is the importance of maintaining consistency across different languages. By using language-specific features and higher-order functions, we can simulate the behavior of reactive programming constructs in different languages. Finally, I believe that reactive programming is a powerful tool for building scalable and responsive applications, and I'm excited to see how it will continue to evolve in the future.
Mike Hall
Interviewer, UGtastic
In many cases, the implementation there is simpler than the .NET version, which is highly optimized. It's very mature code. It's like you know, so because it's written for performance rather than clarity, you know, looking at that code may not help you that much, okay? So, but the common reference seems to be JavaScript, yeah? So, JavaScript is probably the kind of thing that that's the most approachable, okay? And then the concepts are not that difficult. I mean, that's the things like well, I think you have to be careful that you know. I talk a lot about duality and category theory, which might you know scare people off, but this has nothing to do in some sense with that. That's one way to explain it. Another way to explain it is say you're a JavaScript programmer right. JavaScript already has a notion of events. Everything in JavaScript is event-based, right? Now, one way to look at Rx is to say well, but events in JavaScript are not something that are first class, so I cannot have an array of events or I cannot have a function that returns an event right? Right, just it's a byproduct of the environment exactly. And I think yes, and what Rx gives you is it makes events into things that you can manipulate and and pass around. So, it's one way to that Rx is sometimes also called it's like first class events, so in that sense you know, it's very familiar to JavaScript programmers because it it takes these concepts that they already know and another concept that they already know like arrays. Arrays are first-class things. I can have a function that takes an array. I can have an array, and now you you allow the same for events, and the power of that is that you can now decouple the source of the event, say a button that fires events from the events itself. So, for example, I can now create an array that has the values and then make that into an event source for example to test. So, I don't have to have a button that fires the events, but I just put the values in an array and say to event, and now that thing serves as an event source, and so that's the kind of trick. So, it's really conceptually not that hard, um, but then of course, the subtlety is once you make that first class, you have to kind of be careful because of the asynchrony, okay? Well, thank you very much for taking the time to stay with me. I really appreciate it, yes. And you know, I would say try it out and if you yeah, we're always happy to accept you know contributions yeah, and you said it's on CodePlex, it's on CodePlex, and the Netflix version is on GitHub and that's uh Java Rx on. github the Netflix side okay and learn rx is is the place yes yes and so i i i you know you will probably put those yeah those links will be in there okay perfect Thank you. so much okay that's my edit point
Critical Insights
durable
"The reactive framework allows developers to handle both single and multiple asynchronous results, providing a flexible solution for distributed computing."
durable
"The framework is designed as an abstract interface, making it language-agnostic and enabling cross-language communication."
durable
"The open-source nature of the framework allows for a diverse community of developers to contribute and improve the platform."