Erik Meijer on Developer Tools and Architecture | UGtastic Archive
Hi, it's Mike again with UGtastic. I'm sitting down at GOTO Chicago with Erik Meijer. Hi, it's Mike again with UGtastic. I'm sitting down at GOTO Chicago with Erik Meijer. Discover how the reactive framework is revolutionizing software development by handling both synchronous and asynchronous data streams. Learn about its language-agnostic design and the impact of its open-source community. Don't miss this opportunity to understand the future of developer tools and architecture! -source
The Interviewer
Mike Hall
Interviewer, UGtastic
The Guest
Erik Meijer
developer tools and architecture
Mike Hall
Interviewer, UGtastic
▶ Speaking
Hi, it's Mike again with UGtastic. I'm sitting down at GOTO Chicago with Erik Meijer. Erik Meijer is, well if you've worked with Microsoft technologies over the last decade, you've worked with something that Erik has created or been a part of, from C# to the Azure cloud platform. Now you're away from Microsoft and you're working a lot more on the open source world. Well, first off, thank you for sitting down with me, but can you tell me a little bit about what this platform that you just spoke about, this reactive framework, what is that?
Erik Meijer
developer tools and architecture
▶ Speaking
Okay, so I can try to explain that in general terms because that I think is also where you mentioned open source, where the value of open source comes in. So the thing is if you're trying to compose software, in my view, there are four different kinds of effects that as a developer you have to take into account. When you're programming Java and calling a method, typically it returns one value synchronously (like toString). That's the simplest call. The second is when you get back a collection of values like an Iterable synchronously (next, next, next). Those are the two main ways to compose synchronous programs. Now for asynchronous programs, when you make a call you don't want to block because of latency or fallacies of distributed computing. When an asynchronous call returns a single value, you return a Future or a Promise (or Task in .NET). What Rx does is complement this picture for when something returns multiple results asynchronously, like a Twitter stream or mouse move events. Rx deals with asynchronous data streams, defining an abstract interface across languages.
Mike Hall
Interviewer, UGtastic
▶ Speaking
A lot of the examples you gave were in .NET. Is this a .NET only thing?
Erik Meijer
developer tools and architecture
▶ Speaking
Ah, that's a very good question. Since this is a conceptually abstract interface, there's no language dependency. On the open source side (rx.codeplex.com) we have versions for JavaScript, .NET, and C++. Netflix has written a Java implementation (RxJava), GitHub has an Objective-C implementation (ReactiveCocoa), and Google has Dart streams. on these principles. Okay. And then other people have done, like, Python, and, and Ruby, and so there's, like, you know, again, this idea of having asynchronous data streams is not tied at all to a single language, and that's exactly also what I want, because you don't
Erik Meijer
developer tools and architecture
▶ Speaking
Understand. So Rx is a way to handle asynchronous data streams in a more functional and reactive way. It allows you to work with streams of data in a more declarative manner, rather than in a procedural manner. This can make your code more readable and easier to maintain. Rx provides a set of operators that you can use to transform and combine streams of data, and it also provides a way to handle errors and manage the lifecycle of your streams. Overall, Rx is a powerful tool for building responsive and scalable applications.
Mike Hall
Interviewer, UGtastic
▶ Speaking
That's a great explanation. So, can you give me an example of how Rx is used in a real-world application?
Erik Meijer
developer tools and architecture
▶ Speaking
Sure. Let's say you're building a mobile app that displays a list of tweets. You can use Rx to handle the asynchronous data stream of tweets from Twitter. You can use the `Observable` class to represent the stream of tweets, and you can use the `Subscribe` method to start receiving tweets. You can also use the `Map` operator to transform each tweet into a more user-friendly format, and the `Filter` operator to only display tweets that match certain criteria. Finally, you can use the `Dispose` method to clean up the resources when you're done with the stream.
Mike Hall
Interviewer, UGtastic
▶ Speaking
That's a great example. So, can you explain how Rx is different from other asynchronous programming models?
Erik Meijer
developer tools and architecture
▶ Speaking
Sure. Rx is different from other asynchronous programming models because it provides a more declarative and functional approach to handling asynchronous data streams. Instead of using callbacks or events to handle asynchronous operations, you can use operators to transform and combine streams of data. This can make your code more readable and easier to maintain. Additionally, Rx provides a way to handle errors and manage the lifecycle of your streams, which can help you build more robust and scalable applications.
Mike Hall
Interviewer, UGtastic
▶ Speaking
That's a great explanation. So, can you give me an example of how Rx is used in a web application?
Erik Meijer
developer tools and architecture
▶ Speaking
Sure. Let's say you're building a web application that displays a list of products. You can use Rx to handle the asynchronous data stream of products from your database. You can use the `Observable` class to represent the stream of products, and you can use the `Subscribe` method to start receiving products. You can also use the `Map` operator to transform each product into a more user-friendly format, and the `Filter` operator to only display products that match certain criteria. Finally, you can use the `Dispose` method to clean up the resources when you're done with the stream.
Mike Hall
Interviewer, UGtastic
▶ Speaking
That's a great example. So, can you explain how Rx is different from other asynchronous programming models in a web application?
Erik Meijer
developer tools and architecture
▶ Speaking
Sure. Rx is different from other asynchronous programming models in a web application because it provides a more declarative and functional approach to handling asynchronous data streams. Instead of using callbacks or events to handle asynchronous operations, you can use operators to transform and combine streams of data. This can make your code more readable and easier to maintain. Additionally, Rx provides a way to handle errors and manage the lifecycle of your streams, which can help you build more robust and scalable web applications.
Mike Hall
Interviewer, UGtastic
▶ Speaking
Do you know if you're on the server side, using Java, and how to use Java? And I, on the client side, use JavaScript, and I still want to, kind of, exchange events with you. Right. So, um, that's, I think, very important that, that it's not tied to a particular language. Okay. So it's a little bit more like a protocol? It's a little bit more like, like, like, like a protocol. Okay. Um, because the, the, the, what we call the Rx grammar, or the Rx design guidelines, so if, if you and I are exchanging messages, um, what the protocol says is, like, what is the, you know, how are the messages represented, what is the binary format, or what is the kind of, you know, what is the state machine that we use? So Rx has a very simple state machine like that, so I can send you a number of messages that are ordered, and then I can terminate that stream successfully, that means you don't get any more. I can say something went wrong, and then, you know, so that's really the, the, the, the protocol that, that there is, but it, it, it makes certain kind of assumptions, so, but, what, what is, you know, I, I'm, I'm not so sure if, if it's really relevant to make a distinction between a protocol and an API, because you can also say, if you have an API for files, right, you have to, you open your file, you write, and you read, and you close the file, that's kind of a protocol, too. So, for me, I, yeah, I don't, it's somewhere in, in between, it's a, well, I, I think, I, I like to emphasize what, what is the same instead of what's different, so there are certain ways that you should use, or expectations of, in what order you make calls, and, and whether you want to call that a protocol or not, I mean, that, I'm fine with either way.
Erik Meijer
developer tools and architecture
▶ Speaking
Well, is it kind of like if, if you start to use the word protocol, it becomes kind of rigid, and it becomes dogmatic. Yes. And then, then people. Because protocol is very formal, and it's like, everybody has to do it exactly this way, and if you don't do it this way, you're not, you're not reactive. Yes. That's the next thing you're going to say, this isn't reactive enough, this isn't reactive. Yes, and, and, and so I'm trying to be very easygoing with that, because, for example, you know, if you have an existing service, for example, I, I, I'm, in the past weeks, I've implemented a wrapper for Loggly, which is a, is a, a logging service, so what you can do is you can send it values, you know, events, you know, whatever they're just like event streams and then you can query them later so it's like now what
Mike Hall
Interviewer, UGtastic
▶ Speaking
What did you do to wrap that in a reactive shell?
Erik Meijer
developer tools and architecture
▶ Speaking
I wrapped it in a reactive shell so that the underlying thing doesn't know anything about RX. It's just whatever the REST API that the locally guys and exposed. Now, by making your client library talk RX, there's no difference between you know, locally or your mouse. Now you can kind of mesh together things that come from locally with your mouse or and you can visualize it. So it allows you to compose. By exposing all these things with the same interface, then you can compose them, but you don't force the interface kind of all the way down. So that's kind of bridging the protocols right. You can always you know, and one of the things I think about that's interesting for as a is more of a I'm a business developer. I'm going to implement just an implementer. Um, is thinking about the log not as just a place where things go to die, but that the log is is something that that's alive and can feed back into the application and provide information for back into the app. Not just oh this is a thing that I go to figure out when something went wrong yeah. So, so this is it's kind of interesting if you if you look at and and again you have to be very careful that you don't abstract too much because then everything looks the same, but if if you look at the UI for example right, what is a UI well, I'm sending things into the UI and maybe to change the back color of a button or to kind of you know make a button disabled and out of the UI come a stream of events like mouse clicks and mouse moves and and button clicks and then my my code my UI code transforms that stream of of UI events into other events that it sends into the UI. So there's this loop where there's the UI and then my kind of event processing and they are kind of coupled in a loop now. If you look at something like locally for example there's no difference there because my program sends events to lovely and then can query those events and then go back so you get the kind of same loop so and suddenly a lot of these things now look very similar right and because you're observing you know like it's like when you write UI code it's like you're observing the UI and based on what you see you're kind of doing something with the UI so you know what is your logging system you're observing your running code and based on what happens you can kind of you know do something you respond
Mike Hall
Interviewer, UGtastic
▶ Speaking
to that yes and like when you say yes exactly and when you say logging you know that goes there to die yeah then why do you do it
Erik Meijer
developer tools and architecture
▶ Speaking
you do the logging to observe the behavior of your code such that you know intercept when something you know unexpected happens or yeah
Mike Hall
Interviewer, UGtastic
▶ Speaking
and well in a way it's thinking about what my relationship is to the application and then starting to abstract what I do where where I run my app and then I watch the logger and I'm watching messages fly by and then I'm like oh that wasn't right then I go do something well okay now I can start to abstract that thing that I did and realize that that stream is data yep and I am the observer watching that data and then I'm going to react and and I think that's just a way to kind of um realize what's going on with the reactive yes that's that's that's how I just processed it now yes and that's exactly right and then what what I'm trying to do is that you can automate yourself away yeah so you write then now what what the next step I would say to you is that now write some code that will do what you what I did yeah and then and then now you can kind of you know create more value for your business because now because once you can automate it really you know a human doesn't need to be there or one thing that I'm also trying to do is that there's many things where the data like with with these logs for example I think this is a nice example there's so much data in the logs but you what you have to do is you have to turn that into something actionable for humans right and so I think that that's one of the things that I'm trying to make easy where you can take massive amounts of data and then you know use the computer to kind of you know munch that such that at some point you get something that's useful at the human scale right because we cannot deal with millions of events per second I mean even nobody can do that right because we can maybe deal with like you know one event or alert every five minutes right so what you want to do is that's kind of you know you want to boil down all that data into something that's consumable by humans and the rest should all be automated uh to take a step back a little bit so we talked about how this is a this is a uh not to use word protocol but this is a way that many different types of platforms and languages can interact with each other uh so I'm assuming you have to share the this these concepts with many different audiences uh you you Do you have a Microsoft audience who's implementing in .NET, a JavaScript audience who might be running Node.js or browser-based applications, or a Java audience? How do you share this message? Do you use Microsoft examples of JavaScript, or do you tailor for what they do or for Java?
Erik Meijer
developer tools and architecture
▶ Speaking
When talking with JavaScript people, you should use JavaScript examples. One of the things that I really enjoy now is that it's in some sense much easier to embed myself in a different community. For example, Netflix did Rx for Java, and while the underlying ideas are the same, there are very big differences in the Java and C# languages that affect the concrete design of the API. In .NET, there's this notion of extension methods where you can take any type and add instance methods to that type independent of the type itself. For example, if you have an array of int, you can add a new method like conceptually to an array of int. In Java, you cannot do that, but Java 8 has virtual extension methods on interfaces that are slightly different. The design of Rx for Java has to stay within the constraints of Java. When working with people who are doing an Rx implementation for their language, I try to make sure that it feels natural for their environment but still embodies the same underlying, more abstract mathematical principles. For example, in JavaScript, there are already functions like map and filter. When you do Rx, which is the push-based version of that, you're not going to call them select and select many; you call them map and where because that's natural. That's kind of what I'm really looking forward to. One of the things that I'm really looking forward to is to work with all these different language communities to make this thing feel natural for what they're doing and then still being able to interoperate between machines or between languages. The protocol should be consistent, but the implementations should feel natural.
Mike Hall
Interviewer, UGtastic
▶ Speaking
We did the JavaScript version in the beginning in JavaScript. The casing of identifiers is opposite of C# so in C# it's like you know, two strings start with uppercase. I always forget whether that's camel or Pascal case. I always have to do we look at that up on Wikipedia. Whereas in JavaScript, it's different, right? Right? So what we did in the beginning is we used the .NET naming conventions, and then all the JavaScript people would go like, and then I was like, oh man, that's bike shedding, but then I realized, no, I'm wrong because I find it annoying too. If you have to go the other way, exactly, so we changed that. Now, you know, then Jafar Hussein from Netflix said, oh, we should change the names too. So my thinking has evolved very much that now it's like okay, you do what's natural for the language and then, um, you know, make it still conceptually the same and then you pass the kind of same test. So there's there's a whole bunch like I don't know, like 3000 unit tests and that specify the semantics. So it's really maintaining the semantics across languages but with different implementations. And and I must say it's it's that is the hardest part is that some things don't translate so you have to which is also nice right like for example what I mentioned with the extension method. So you cannot just take the C# um implementation and just plug it into a different language. So you have to really work hard on that to to make it natural and to make it possible in a different language, okay? If I'm if I have some pet language and I'm looking at I'm very interested in this reactive stuff and I want to implement it in in my let's just say Python. Let's imagine Python doesn't have anything and where where would I go to learn about um or or talk to people and understand what how to implement it what what the the underlying principles are okay? So that that that's that's a very good question. So there's um there's learn rxrx there's learn rxrx dot com so that that's uh one of the .NET early users that has written a book. There's also Paul Betts and Jesse Liberty have written a book, a very thin book on on Rx, um which I like because it's thin. And then there's the Rx guidelines that give you a little bit more of the semantics okay. Um and and and so I would say you know, you start playing with the .NET version or the Java version um and then you know you you go from there. The other thing is that in some sense the JavaScript version that we have um for .NET is a little bit like a reference implementation because JavaScript doesn't have threading and and so so
Erik Meijer
developer tools and architecture
▶ Speaking
So, if you're looking to implement reactive programming in Python, you can start by learning about RxPy, which is the Python implementation of the Reactive Extensions. There are several resources available online, including the official RxPy documentation, which is quite comprehensive. Additionally, you can look into books like 'Reactive Programming with RxPY' by Antonio Cangiano, which provides a good introduction to the subject. There are also many online courses and tutorials that can help you get started. It's important to understand the underlying principles of reactive programming, such as observables, observers, and operators, to effectively implement it in any language.
Mike Hall
Interviewer, UGtastic
▶ Speaking
That's great. Thank you for the detailed explanation. So, in terms of maintaining semantics across different languages, what are some of the challenges you've faced and how have you overcome them?
Erik Meijer
developer tools and architecture
▶ Speaking
One of the main challenges in maintaining semantics across different languages is ensuring that the behavior of the code remains consistent, even when implemented in a different language. For example, in C#, we have extension methods, which allow us to add methods to existing types without modifying the type itself. However, when we try to implement this in JavaScript, we need to find an alternative approach that achieves the same result. One way to overcome this challenge is to use higher-order functions and closures to simulate the behavior of extension methods. Another challenge is dealing with differences in language features, such as threading and asynchronous programming. For example, JavaScript doesn't have built-in support for threading, so we need to use libraries like RxJS to handle asynchronous operations. By understanding the underlying principles of reactive programming and using language-specific features to implement them, we can maintain consistency across different languages.
Mike Hall
Interviewer, UGtastic
▶ Speaking
That's very insightful. Thank you for sharing your experience. So, in your opinion, what are some of the key takeaways from your work on reactive programming?
Erik Meijer
developer tools and architecture
▶ Speaking
One of the key takeaways from my work on reactive programming is the importance of understanding the underlying principles of the subject. By understanding how reactive programming works, we can effectively implement it in any language. Another key takeaway is the importance of maintaining consistency across different languages. By using language-specific features and higher-order functions, we can simulate the behavior of reactive programming constructs in different languages. Finally, I believe that reactive programming is a powerful tool for building scalable and responsive applications, and I'm excited to see how it will continue to evolve in the future.
Mike Hall
Interviewer, UGtastic
▶ Speaking
In many cases, the implementation there is simpler than the .NET version, which is highly optimized. It's very mature code. It's like you know, so because it's written for performance rather than clarity, you know, looking at that code may not help you that much, okay? So, but the common reference seems to be JavaScript, yeah? So, JavaScript is probably the kind of thing that that's the most approachable, okay? And then the concepts are not that difficult. I mean, that's the things like well, I think you have to be careful that you know. I talk a lot about duality and category theory, which might you know scare people off, but this has nothing to do in some sense with that. That's one way to explain it. Another way to explain it is say you're a JavaScript programmer right. JavaScript already has a notion of events. Everything in JavaScript is event-based, right? Now, one way to look at Rx is to say well, but events in JavaScript are not something that are first class, so I cannot have an array of events or I cannot have a function that returns an event right? Right, just it's a byproduct of the environment exactly. And I think yes, and what Rx gives you is it makes events into things that you can manipulate and and pass around. So, it's one way to that Rx is sometimes also called it's like first class events, so in that sense you know, it's very familiar to JavaScript programmers because it it takes these concepts that they already know and another concept that they already know like arrays. Arrays are first-class things. I can have a function that takes an array. I can have an array, and now you you allow the same for events, and the power of that is that you can now decouple the source of the event, say a button that fires events from the events itself. So, for example, I can now create an array that has the values and then make that into an event source for example to test. So, I don't have to have a button that fires the events, but I just put the values in an array and say to event, and now that thing serves as an event source, and so that's the kind of trick. So, it's really conceptually not that hard, um, but then of course, the subtlety is once you make that first class, you have to kind of be careful because of the asynchrony, okay? Well, thank you very much for taking the time to stay with me. I really appreciate it, yes. And you know, I would say try it out and if you yeah, we're always happy to accept you know contributions yeah, and you said it's on CodePlex, it's on CodePlex, and the Netflix version is on GitHub and that's uh Java Rx on. github the Netflix side okay and learn rx is is the place yes yes and so I I I you know you will probably put those yeah those links will be in there okay perfect Thank you. so much okay that's my edit point
Critical Insights
durable
"The reactive framework allows developers to handle both single and multiple asynchronous results, providing a flexible solution for distributed computing."
durable
"The framework is designed as an abstract interface, making it language-agnostic and enabling cross-language communication."
durable
"The open-source nature of the framework allows for a diverse community of developers to contribute and improve the platform."