Fluentd And Log Integration Tooling: Mike Hall Interviews Kiyoto Tamura | RailsConf 2014
Meet Kiyoto Tamura, the maintainer of Fluentd, a log aggregation and data collector. Learn how Fluentd simplifies log management for companies like Slideshare and Nintendo. Discover the project's growth, its lightweight design, and how it's becoming a global community. Don't miss this opportunity to understand how Fluentd is revolutionizing log integration! 📊💻 #Fluentd #logaggregation #logintegration #RailsConf2014 #KiyotoTamura
The Interviewer
Mike Hall
Interviewer, UGtastic
The Guest
Kiyoto Tamura
Fluentd and log integration tooling
The Conversation
Mike Hall
Interviewer, UGtastic
Hi, it's Mike with UGtastic . I'm here at RailsConf 2014 and I'm sitting here with Kiyoto Tamura, who is a maintainer for the Fluentd log aggregation and data collector. Thank you very much for taking the time to speak . What is a log aggregator/collector?
Kiyoto Tamura
Fluentd and log integration tooling
Fluentd is a log aggregator/log collector written in Ruby and its key features are that it's really lightweight, it's a very small Ruby program augmented by extensible plugin architecture and it runs pretty fast for all sorts of logs. A lot of our customers collect data for like Apache Engine logs, Apache Nginx logs, syslogs, application logs and store them in like Elasticsearch, store them in MongoDB, copy the data, archive it, and so forth. We started a project two and a half years ago and the community has grown to be about like 50 regular contributors, thousands of users in the world including Slideshare and Nintendo. Yeah, Nintendo, that's actually, that's pretty cool.
Mike Hall
Interviewer, UGtastic
I mean, are you able to say how they're using it?
Kiyoto Tamura
Fluentd and log integration tooling
I'm actually not sure about the actual like, you know, how they use it, but I think it's part of their like online games that are going on now.
Mike Hall
Interviewer, UGtastic
Okay.
Kiyoto Tamura
Fluentd and log integration tooling
And basically like with all these companies, logs are very, very important. They want to centrally manage it and they want to do all sorts of operations on them. And before, like a lot of, like myself included, we would rsync, write a script here and there, but as like the number of your data sources increase and the number of storage systems you want to store and use increase, the complexity increases like exponentially. And we wanted 10 complexity, keep everything simple, and keep everything performantly.
Mike Hall
Interviewer, UGtastic
So how did you get involved in this project?
Kiyoto Tamura
Fluentd and log integration tooling
Yeah, so I'm the, so the Fluentd was originally created by my employer, Treasure Data. And I'm employee number one. And when we first created the thing, we didn't think it was going to be big, but one of our customers saw it and they were like, man, like we wish this was open source. And we sort of drew the inspiration from Heroku, who actually offers service as a pass.
Mike Hall
Interviewer, UGtastic
Okay.
Kiyoto Tamura
Fluentd and log integration tooling
But they also like open source a lot of stuff. That's really cool. Especially the stuff that runs on the client side. So we did the same. The community started to grow organically. Right now we have like a little short of like 1800 stars on GitHub and it's been steadily rising. Oh, wow. That's, so it's, it's a pretty well adopted.
Mike Hall
Interviewer, UGtastic
Yeah.
Kiyoto Tamura
Fluentd and log integration tooling
I used to, I sometimes do the self-deprecating so that it is the law collector that nobody's heard of.
Mike Hall
Interviewer, UGtastic
Right.
Kiyoto Tamura
Fluentd and log integration tooling
But it's actually pretty cool.
Mike Hall
Interviewer, UGtastic
Yeah.
Kiyoto Tamura
Fluentd and log integration tooling
No, 1800 people have heard it at least.
Mike Hall
Interviewer, UGtastic
Yeah.
Kiyoto Tamura
Fluentd and log integration tooling
Click that button.
Mike Hall
Interviewer, UGtastic
Yeah.
Kiyoto Tamura
Fluentd and log integration tooling
So the, the contributions is are, do you, it's open source, but how do you, how do you interact with the community?
Mike Hall
Interviewer, UGtastic
Do you have people like contributing patches or doing?
Kiyoto Tamura
Fluentd and log integration tooling
Yeah, yeah, yeah. So right now I think there are 18 people with commit rights. Three of them come from us, Treasure Data. It's a fairly, it's a very balanced community in the sense that we have commuters from like various companies. And although we started originally in Japan, we're, Treasure Data's founders are originally from Japan, starting to spread like a lot of the questions these days come from like all parts of the world. And as someone who constantly asks this question to the mailing list, I'm, I'm glad to see this adoption going global. And you know, it's always funny with some of these open source projects, like I have no idea what Fluentd means.
Mike Hall
Interviewer, UGtastic
What, how did, what's the name, where did it come from?
Kiyoto Tamura
Fluentd and log integration tooling
Oh, okay. So originally, we wanted to make various data sources talk to various data, like back-end systems more fluently. So the original author of the project, Kisada, who also wrote MessagePack, which is a binary protocol, he wanted to call it Fluent. And that's why our GitHub username is Fluent. But then, um, he also realized that's not very Google-able.
Mike Hall
Interviewer, UGtastic
Yeah. Like when you type Fluent, there's like all sorts of like language websites, right?
Kiyoto Tamura
Fluentd and log integration tooling
Right, right. So we just decided to add D, which, um, traditionally stands for daemon.
Mike Hall
Interviewer, UGtastic
Okay.
Kiyoto Tamura
Fluentd and log integration tooling
So that's like stats D, or power syslog D. Oh, okay, so it's the Unix D. Yeah, Unix D. So that, that's the, the story. Oh, okay. So now it's Fluent/Fluentd is the project repo. So if somebody was interested in, in, you know, log aggreg ation and collection, as we start to have more people running multiple servers and needing to aggregate data, um, what, what would they do to, to check out, like how, how do you recommend somebody? So our GitHub repo is a good place to start. Uh, so is our website, friendydontwork. com. And if you're familiar with Ruby, and I know not everyone is from the Ruby community, jam install Fluentd and you can get started right away. Is it like compatible back to 1.
Mike Hall
Interviewer, UGtastic
8, or is it?
Kiyoto Tamura
Fluentd and log integration tooling
Uh, so we officially, I think, support 1. 9. 3 and above, and in the newer versions we're trying to slowly push the community to 2 and above. Yeah, I'll tell you, yeah, because now it's 2. 0 is, is. Yeah, 2. 0 is the, and our experimental, um, bleeding edge right now is trying to, uh, work off of 2. 1. 1.
Mike Hall
Interviewer, UGtastic
Okay.
Kiyoto Tamura
Fluentd and log integration tooling
And Ruby has made a great performance, uh, progress over the past few years, and we want to take advantage of that. And as far as, as putting in the, the log aggregator, is it something that, uh, uh, a normal Rails developer could probably work through? Yeah, so that's the other great thing about Fluentd, like, uh, when we, when I tell people, especially not familiar with, like, writing Ruby systems code, is that, ooh, this is like system software. It's hard, it must be really hard to write plugins. Actually, I know a lot of engineers and programmers, or software writers, um, who learn to write Ruby, myself included, actually, partially, by writing, writing plugins for Fluentd.
Mike Hall
Interviewer, UGtastic
Oh, really?
Kiyoto Tamura
Fluentd and log integration tooling
And there's only, like, a tiny bit of code that you need to write, and everything is really well-attracted, so that you really don't have to repeat what other people have already built. And, yeah, so it's pretty easy to get started. And you said that earlier, it's only about 5,000 lines for the code. Yeah, that's the core. Yeah, so the core. So it's probably a good idea for people to do, like, what DHH recommended, and that's just go read the source code. It's not... Yeah, um, it is, uh, it is a pretty small program, pretty, uh, well-documented. Not in the code, but that is something that we're working on.
Mike Hall
Interviewer, UGtastic
Okay.
Kiyoto Tamura
Fluentd and log integration tooling
And, uh, there's a lot of documentation, and the mailing list is very active, like, when you post something, within 24 hours, someone's gonna respond.
Mike Hall
Interviewer, UGtastic
Great. Well, thank you very much for taking the time to... Thanks, Mike. Thanks. [Music] Thank you.
Critical Insights
durable
"Fluentd's lightweight design and extensible plugin architecture make it suitable for various data sources and storage systems."
durable
"The project's growth is attributed to its open-source nature and the global community of contributors."
durable
"Fluentd's performance has improved significantly with Ruby's advancements, allowing for more efficient log aggregation."