Gil Yehuda: Hi Everyone and Welcome to the Dash Open Podcast. Dash Open is your source for interesting conversations about open source and other technologies, from the Open Source Program Office at Verizon Media. We're the home of many leading internet brands including Yahoo, AOL, TechCrunch, Huffington Post, and many more. My name is Gil Yehuda and I'm on the Open Source Team at Verizon Media. On the show today, we have Mujib Wahab, who manages a security team that's focused on anti-abuse and core security related technology platforms. With us today also is Henry Avetisyan, who is a distinguished engineer on the security platforms team and works with Mujib. Welcome Mujib and Henry. Mujib Wahab: Hello Gil. It's nice to be here. Henry Avetisyan: Hello. Thank you, nice to be here. Gil Yehuda: Thank you. I'm glad to have you here on the show. We're going to talk about security. Sometimes people don't talk about security as much, that it’s like a secret world. We actually share some of it. We've even open sourced some of it. This is The Dash Open Podcast. We like to talk about open source and let's lead with an open source project that came out of your team that is security related called Athenz. What is Athenz? Henry Avetisyan: Athenz itself, provides two major features. One is service authentication. What that means is anytime you bootstrap an instance, whether that instance is a container, bare metal, maybe it's a virtual machine, whether it's something you deploy in AWS within your Kubernetes Cluster, that instance has to have an identity and that identity is in the form of an X.509 certificate that allows that service, that instance, to communicate with any other instance within your infrastructure, whether it's within your data center on-prem, whether it's in public cloud, whether you have a hybrid cloud solution that allows that instance to securely communicate with any other service within your infrastructure using TLS. And it's not just TLS, it provides MTLS, which is mutual TLS authentication. So now the receiving server not only establishes a secure connection, but also knows exactly what service is asking for some resource from itself. Henry Avetisyan: That's the first feature. The second major feature is robust authorization. When we talk about security, authentication itself is not enough. That's only the first step. Once you authenticate, next you need to authorize and for authorization, Athenz provides a role-based authorization system. It could be based on your identity certificates in the form of an X.509 certificate, or we also support industry-based standards, JWT Access Tokens. Henry Avetisyan: So your clients can obtain access tokens and then they can actually pass those access tokens along with the identity certificates to the receiving server to request some type of a resource from that service. The service can verify the identity based on the X.509 certificate and then it can authorize based on the access token to make sure you are authorized to assume a specific role which has been authorized to execute the specific action on the resource and based on that, you can return the resource that you're looking for. Gil Yehuda: We now know what Athenz is. Why open source it? Mujib Wahab: Open source collaboration is key for security. We wanted to open source this security product so that we can collaborate with the many great minds across the world and take this product to the next level. Just to give you a little bit more background, we are moving from this several decades of a perimeter-based network model to a Zero Trust security model. Gil Yehuda: But let me just understand, there's the perimeter-based security model, which means that, I guess, if you secure the perimeter, the bad actors on the outside and everything on the inside is okay. As long as you secure the perimeter, you're okay. That was the old way. Gil Yehuda: So the new model is Zero Trust security. Mujib Wahab: Yes. Gil Yehuda: I trust that you'll explain what that means. Mujib Wahab: The Zero Trust security model is never trust and always verify. It has several core principles. A few of the core principles is that the traffic should be encrypted. It doesn't matter whether you're communicating within your data center or going outside of the wall, or coming inside of the wall. Since the threat may exist within your office, or external also, then you need to authenticate every single client that's connecting to you. Mujib Wahab: That's another principle. Authentication itself is not enough, as Henry said earlier, you also need to authorize to make sure that the authenticated principle, whatever action is trying to perform on your resource is being authorized or not, and we also need to make sure that we provide the least privileged access, not like wide admin access. Mujib Wahab: These are the few core principles and that's what Athenz enables you to achieve. Gil Yehuda: Athenz is a framework. It's open source, which means that anyone can use it. Mujib Wahab: That's right. Gil Yehuda: Anyone can find it. Mujib Wahab: Yes. Gil Yehuda: Where could they find it? Mujib Wahab: You can start with the athenz.io website. Gil Yehuda: Athenz.io and Athenz spelled with a Z. Mujib Wahab: Yes. Gil Yehuda: Okay, so Athenz spelled with a Z at the end dot IO and there is a website and there's documentation. Henry Avetisyan: That's the website. Mujib Wahab: There's documentation and there's a link to our GitHub. Gil Yehuda: There's a GitHub repo where you could look at the code. I guess Zero Trust people, we're not saying, "Trust our binary," we're saying, "Look at our code". Right? Because you're going to be running this as your security framework to authenticate your containers, your instances and you are a public, private, or hybrid cloud. Because we do and we show you the code so you can inspect it. And if they find an opportunity to improve the code? Mujib Wahab: Yes. Gil Yehuda: They could... Mujib Wahab: The whole purpose of open sourcing is to have everyone contribute. They may think of different other ideas, how we can improve the security. Gil Yehuda: Right. So they can just open an issue, issue a pull request, we'll look at it, we'll work with them. Mujib Wahab: Yes. Gil Yehuda: And we want to make it better. We want to make this kind of standard. If it works well for us and we're a pretty big company, we do quite a lot of work in this space, but if it works well for us, it might work well for others too. Yeah. That is the power of open source. Gil Yehuda: Henry, what's coming next with Athenz? Where's it going in terms of features to the framework or things that a Security Platforms Engineer needs to worry about? Henry Avetisyan: Well, right now we have integration with Kubernetes. Our Kubernetes Platform Engineering Team has open sourced the pieces that you need to integrate Kubernetes Clusters with Athenz. If you bring up a Kubernetes Cluster, you can actually use Athenz within your Kubernetes Cluster to make sure every single container within your cluster has an identity in the form of an X.509 certificate. We also have plans to open source the components for OpenStack and AWS. That allows any instance that you bootstrap within AWS also to come up automatically with an Athenz X.509 certificate identity. Henry Avetisyan: Obviously our OpenStack team is working on integrating Athenz within OpenStack and then hopefully they'll be able to open source that component as well. As far as functionality within Athenz, one of the major areas we're focusing on is compliance and auditing. As we deployed Athenz within brands and media, one of the major requirements was if you have a property that is under SOX compliance, they need governance, risk, and auditing capabilities. Henry Avetisyan: You want to make sure all those components are available within Athenz. I should be able to look at any rule, any audit history, to find out exactly why a specific change was made, who authorized it, and who approved it. So far, the major features we're working on right now is to provide all that capability within Athenz, so we maintain our login history, and we maintain a full audit workflow. So a single person will not be able to make a change to a role authorizing it himself, or herself, or somebody else, to have full access to resources, but instead that request is being approved by an external entity. Thus satisfying auditing requirements. Gil Yehuda: Operating this as an open source project, do you find that the speed of deployment is either faster or slower or the quality of code is either better or worse? How does this change the way you think about or you observe your code? Henry Avetisyan: As an open source project, I think one of the major benefits that I see is the quality of code. Because when you are submitting that PR, you know that the whole world is watching. Somebody else might look at that code and go, "What the heck are you doing here?" Gil Yehuda: Right. Henry Avetisyan: So you get to review that code and you think about, is this the best possible solution for this problem? In that sense, the fact that you are writing that code, if you're doing your own development, you might even sometimes skip testing, very bad practice, but you might skip it. Gil Yehuda: Right. Henry Avetisyan: But when you are putting it as part of open source, you want to make sure that you also have full test coverage. Whenever you're testing the framework that you're using, you want to make sure everything is tested. You're reaching for at least maximum 100% code coverage and that increases your code quality because not only you are writing your code, you're also writing your test cases. You think about the architecture of design to make sure if somebody else is looking at my code are they going to go, "This is a great code. I should be running this because I trust this guy who wrote the code." Gil Yehuda: So, what I'm hearing is that when you are working on a security related framework, quality is essential. You can't compromise on the quality of security and by moving to open source, that just changes the game, right? Henry Avetisyan: Absolutely. Yes. Gil Yehuda: It's unique to the way we do our operations and every other company is going to have their sort of way of doing it too. And I guess you get the best of both worlds. Mujib Wahab: If I can add a little bit more of what Henry said, is that, when we had a requirement that we need to have our workloads start deploying in dynamic environments, so you no longer have a legacy, we have authenticating those workloads. And we wanted to issue an identity for every single service deployed on the workload so that we can do authentication and authorization. That was our requirement. Mujib Wahab: Obviously we are all conscious passionate people, we don't want to go and reinvent the wheel, we looked at the open source to see, is there anything similar to meet that requirement? We couldn't find one, so we thought we should build one, but at the very beginning decided that, look if we are going to open source it, that means we need to have this interface. We don't want to have two different copies because it's always a challenge that once you open source, we end up having a copy in open source, GitHub and on-prem. Then it gets out of sync up some time and it gets unmanageable. So that was our principle from the very beginning that people get everything in open source, have all the interfaces, we'll bring it inside and interface with our local system. Gil Yehuda: Brilliant. A recommendation to people who are looking to build open source platforms. Think about it first and design for open source. We have a lot of users. That means that Athenz must scale quite a bit. Tell us about that. Henry Avetisyan: On average, we are generating close to two million certificates daily. Gil Yehuda: Two million certificates daily? Henry Avetisyan: Two million. Yes. We're using another open source component from Verizon Media called Crypki, which is our certificate signer, which is interfacing with your hardware security's module to actually sign certificates and your private key is stored in HSM. Henry Avetisyan: So we are issuing close to two million certificates daily and one of the unique features that Athenz provides is, all the certificates are very short lived. We do not issue certificates that are valid for six months, or a year, as you would get from a typical certificate authority. By default, our certificates in most cases are only valid for 30 days and our services that we provide automatically refresh them daily. So every single day you have a unique certificate for your instance being deployed, whether you're running in Kubernetes, whether you have a job running in Screwdriver, whether you have something running in Vespa, or if you are deploying your instance in AWS. Gil Yehuda: Vespa is another one of our open source projects. Vespa.ai for people who are looking for big data serving. Henry Avetisyan: Yes. Gil Yehuda: Gosh, we have open source all over the place. All of these things operate at really high scale. Like millions. Henry, what's next for Athenz? Henry Avetisyan: Last year at KubeCon, we presented our integrated solution within Athenz and our Kubernetes Cluster and there was a lot of interest in our solution. What we're planning next is how can we make Athenz part of the CNCF Foundation. We're looking for more companies who are interested in security, who have Kubernetes Clusters running within their infrastructure and they want to integrate with Athenz, make it secure, and make sure all their containers have an X.509 certificate based identity. But we're hoping that those companies can integrate with Athenz, become users of Athenz, extend its functionality, provide new functionality, new features, and provide pull requests. We're definitely encouraging everybody to take advantage of our solution and see how we can actually help them to solve their own problems. Gil Yehuda: So if you're listening to this podcast and security is important to you, Kubernetes is important to you, go to athenz.io, find Henry and Mujib, you'll find your contacts there and you could work with us to bring this to the masses and to bring this to the CNCF and to companies around the world. Mujib, Henry, thank you very much for your time. Really appreciate this exploration into the details of how security works in security platforms. Gil Yehuda: If you've enjoyed this episode of Dash Open and you wanted to learn more about our open source program and other technologies, please visit developer.yahoo.com and you can also find us on Twitter @YDN.