Dash Open 10: Moloch - Open Source Large Scale Indexed Packet Capture and Search System

Rosalie Bartlett: Hi Everyone, and Welcome to the Dash Open Podcast. Dash Open is your source for interesting conversations about open source and other technologies from the open source program office at Verizon Media, home to many leading brands, including Yahoo, AOL, HuffPost, Tumblr, TechCrunch, and many more. Today on the show I'm so excited to chat with Andy Wick and Elyse Rinne. Andy is a senior principal architect at Verizon Media. Andy is also the creator of Moloch, a large-scale, open source, full packet capturing, indexing and database system. Elyse is a software development engineer at Verizon Media. Elyse is also the UI and full stack engineer for Moloch. Welcome to the show Andy and Elyse! Andy Wick: Thank you. Elyse Rinne: Great to be here. Rosalie Bartlett: Before we start off our chat today, I was wondering if you could maybe tell us exactly what Moloch is. Andy Wick: Sure. Moloch is a full packet capture system. We use it here at Verizon Media to monitor our networks and capture the packets as they go by. We were using some commercial products beforehand, and they just didn't scale or have the features that we wanted, and they're just too expensive, frankly, and so we decided that, hey, we could build something that was simpler, cheaper and easier for us to deploy and we would totally control it. So we started with a small demo and it's grown into what it is now. Rosalie Bartlett: Are there any products out there that are similar to Moloch? Andy Wick: There's several different categories of products that are similar. There's many commercial products out there that are very similar, but because they're commercial, they're very expensive, they're from niche companies that charge a lot of money, and then there are some other open source products that do similar things as Moloch, but we feel Moloch has its own niche market where it's all based on Elasticsearch and it's all based on a distributed architecture that can scale for a big company like ourselves. Rosalie Bartlett: And Elyse, you joined the team many years ago and your focus is on UI. What are some of the things that you worked on when you first got started, and what are some of the things that you work on now, because you've obviously transformed the product quite a bit. Elyse Rinne: Yeah, so I was hired mainly to focus on usability and adding new features to the product. Originally there were some hidden features and things that users didn't even know existed. I spent a lot of time to create some user interface interactions so that users could find these things and actually analysts could do their job a little bit better. Rosalie Bartlett: What is the tech stack that Moloch uses? Andy Wick: We use a bunch of different things. I can talk about the back-end and Elyse will talk about the front-end, but on the back-end it's all C-based and we use standard libraries out there with pcap, some other ones out there that help us load the data from various sources, and we store it into Elasticsearch, and then from Elasticsearch that's really where the power of searching and aggregating the data comes from. Elyse Rinne: We use Node.js to create data and serve up data to the front end. The frontend is Vue.js, and we use that as our frontend framework, but originally we used Angular and found there were some limitations with that, so we recently switched to Vue and that's given us a great improvement in the user interactions. Rosalie Bartlett: When you think about the user experience and designing for that, who are some folks that you have in mind? Elyse Rinne: I try to make the product great for our security engineers, for our analysts who look at this data every day to find exploits in our system, to dive deeper and to figure out what went wrong. Rosalie Bartlett: What are some of the challenges with UI when it comes to showing big data? Elyse Rinne: That was one of the biggest challenges. Long lists render slowly, and that's one of the reasons why I decided to switch from AngularJS to Vue.js. The speed increase was phenomenal. It was almost 100% on our pages with large datasets. Our main page, the sessions page, a user can select anywhere from 10 to 500 sessions to display on one page. That needs to be responsive. It needs to be quick to load and a user needs to be able to dive into that data further, and if you don't have quick rendering, and if you don't have smooth scrolling, users are going to get frustrated. Rosalie Bartlett: Absolutely. You have built an incredible community and a very active Slack channel where you're constantly asking questions and troubleshooting. What are your thoughts on how to build and grow a thriving community? Andy Wick: I think the biggest thing is you have to have some time to do it, so you have to have support from your management that knows that you're not going to be spending all your time developing or deploying or designing, that you're actually going to be spending time working with your community, and that's really important. If you don't have that support from your management, it's hard to make that time to do that. And then you just have to want to engage with your community. Rosalie Bartlett: And Elyse, do you find yourself getting a lot of inspiration for different design changes based on the feedback and discussions you see on the Slack channel? Elyse Rinne: Absolutely. That's one of the biggest drivers of change. Specifically in the UI, for me, is getting that feedback from people. They will give positive and negative feedback. Both are really great to be able to move the product on to be even better. For example, we've had several people say, "Maybe this is not the best way to do this" and maybe they steer us in the right direction. Or conversely, they say, "This is an amazing feature. Here are more ways to improve it”. Rosalie Bartlett: For folks listening in today who are saying, "Wow, Moloch seems so cool, and Andy and Elyse sound awesome" how can they get involved? Where can they go to contribute? Elyse Rinne: Start on our GitHub. That's where everyone should start. Go download the project, run it yourself, and if you want to contribute, we have a contributing guide, so follow that. Ask us questions on Slack. We have a Slack, it's open, we have a lot of people there. Not just us, everyone in the community is willing to provide feedback and we would love to have people contribute. That's really what we're searching for now. Rosalie Bartlett: Andy, what is next for Moloch? What can folks expect to see in the future? Andy Wick: Elyse especially is working on a lot of new visualizations, so we're expecting to add lots of visualizations in the next version. We've been working a lot on making stability our number one priority, making sure it can scale to the next level, you know, full data center packet capture as opposed to just offices or smaller data centers. Elyse Rinne: I would add a plug in there saying that with specifically the visualizations, currently we're using D3, and I would love to have people help contribute ideas, code, anything they want to help determine where these visualizations go so that people who actually use the software can drive what we add in the next few features of visualizations. Rosalie Bartlett: And just a final question, Andy. You have been at the company for a little over 20 years. For folks out there who are maybe just graduating and thinking about, "Where do I go to work, what do I look for when looking for a career?"- what advice do you have for them? What are some of the things that you looked for and made you stay all these years? Andy Wick: There has to be opportunities for things that you love to do. It's key. If you're doing things that you don't love to do or you don't think you love to do, then you probably should look other places. There's the whole big fish in the small pond versus the opposite, and you have to decide what's right for you, but I've been lucky where I've been able to work on things that I'm passionate about, that I like to see improve, that millions of people use, and I've had great company support. Rosalie Bartlett: Fantastic. Well, Andy and Elyse, it has been so nice to chat with you today. Thank you both so much for making the time. Elyse Rinne: Thank you. Andy Wick: Thanks for having us. Rosalie Bartlett: If you enjoyed this episode of Dash Open and would like to learn more about open source projects at Verizon Media, please visit You can also find us on Twitter @ydn.

More Episodes: