Rosalie Bartlett: Hi Everyone and Welcome to the Dash Open Podcast. Dash Open is your source for interesting conversations about open source and other technologies from the Open Source program office at Verizon Media, home to many leading brands including Yahoo, AOL, TechCrunch, and many more. Rosalie Bartlett: My name is Rosalie and I'm on the Open Source team at Verizon Media. Today on the show, I'm so excited to chat with Shaunak Mishra from the Yahoo Research team. Shaunak is a Senior Research Scientist based in New York. Welcome to the podcast, Shaunak. Rosalie Bartlett: Before we chat about the exciting research that you shared at KDD, I'd love for you to tell us a bit more about your focus at Yahoo Research in New York. Shaunak Mishra: Sure. I'm a Senior Research Scientist in Yahoo Research and I'm based in the New York City office. And I belong to what is called the ad science team within Yahoo Research. And at a high level, our job is to make our advertisers happy and also keep our users happy. Shaunak Mishra: So these are very challenging goals because we operate at a huge scale. It's basically the whole internet and we care about what users think about us or whether they are happy reading the content that we offer to them. And whether advertisers who want to show relevant ads to users also get something meaningful out of those ads. Rosalie Bartlett: That sounds like a very challenging problem because you've got two types of folks that you want to keep happy and it all relies on customization and relevancy. What initially attracted you to this type of work? Because it's very interesting. It's very unusual. Shaunak Mishra: The whole internet, a lot of services are free like Google Maps or even Yahoo Finance, Yahoo Mail, these are all free services and a lot of engineering effort and scientific effort is required to make the services sustainable. And the major monetization engine for all these internet companies is ads today. Shaunak Mishra: Ads are very relevant and that is what drives me. Working in the ad space can have so much revenue impact. It can be the backbone of a company and still drive meaningful services for users and the whole commerce ecosystem. Rosalie Bartlett: And what's it like to work in New York? That must be really fun. Shaunak Mishra: That's a good question. New York City is exciting, full of life and so much variety. And our office is located in a very nice location. It's close to the NYU Manhattan campus, close to Washington Square Park. And apart from all the great food that we have around, there are amazing people in the New York office, great scientists, great peers to work with. And it's just an amazing experience. Rosalie Bartlett: And I know for your research that you shared at KDD, you were actually collaborating with a research scientist from Sunnyvale, California. Shaunak Mishra: Right. That's another amazing thing about Yahoo Research. It's a distributed research organization. There are researchers spread across New York, Sunnyvale, Los Angeles, Paris, and Haifa in Israel. And we collaborate closely with each other because the spirit is to deliver on tough deadlines, collaborate with state of the art in terms of scientific achievements and collaborating with a lot of people, a lot of smart people, makes this possible. Rosalie Bartlett: And how do you sustain that collaboration? What are some things that you do to be able to be successful at that? Shaunak Mishra: Part of the sustenance is we need good feedback. We put in a lot of effort and we need the right feedback to keep us going. And in my experience, the impact or the feedback comes from two directions. One is internal, from the company. So if you do something which is very relevant for the company's revenue, it gets the company more revenue over time and keeps advertisers happy or pulls in more advertisers, the company also rewards you a lot. There's a lot of internal recognition in the company. Shaunak Mishra: And from the external side, when you publish papers and you file patents, you get a lot of external publicity as well. Both are rewarding for a scientist and that is what keeps us going and also helps us grow in our career. Rosalie Bartlett: Very cool. So let's talk about that external publicity. Last week I attended KDD with you and several members from the Yahoo Research team who were all presenting there. If you could reflect back on some of the themes or topics that you saw there that you found interesting or just personally very exciting. What comes to mind? Shaunak Mishra: Sure. So KDD is the best machine learning conference in the world in terms of applied machine learning and it's very competitive. For a company to have even four to five papers in that conference is a meaningful achievement because this year about 2,500 people from across the globe attended the conference. And a huge number of people submitted papers and the selection criteria was very strict. So it's a huge achievement for Yahoo Research to have such a strong presence in KDD. Shaunak Mishra: Having said that, the teams that I noticed in this year's conference, go along two directions. One is we are seeing a huge rise in the impact of AI and machine learning on domains outside the internet industry. For example, there were a lot of papers and keynote talks on using AI in healthcare and in transportation, including applications for ride-sharing companies. Shaunak Mishra: And in the long term this is going to have a huge impact on people. And the applications that are driving these use cases in healthcare and transportation have the roots in the internet industry, which is more mature in terms of using artificial intelligence. Shaunak Mishra: Having said that, in the broader context because I'm from the ad space, I was keenly looking at new ideas which are relevant for advertising. And I noticed that there is a rising trend in using deep learning models which process huge amounts of user data collected from online properties. And they process this data and come up with meaningful and insightful recommendations for users. Shaunak Mishra: These recommendations could be in the form of ads or videos or news articles, anything that helps the user engage more with an internet company. These were the two directions that I noticed and I think both of them are going to have a huge impact in the long run. Rosalie Bartlett: Love that. So you're seeing these impacts or seeing a lot of potential, not just in advertising, but you're seeing it applied to healthcare and transportation and probably just about every industry you can think about. Shaunak Mishra: Right. Rosalie Bartlett: Very exciting. You mentioned that your team presented quite a lot of papers at KDD and now I'd love to chat about the research that you personally shared. So you had two papers. The first one was “Learning from Multi-User Activity Trails for B2B Ad Targeting” and “Understanding Consumer Journey using Attention-based Recurrent Neural Networks”. As you look at these two papers, can you tell us about what inspired you to do this research and the problem that you're solving? Shaunak Mishra: Cool. That's a pretty deep question and at this point, I'm tempted to tell you a story that inspired us to go down this direction. Basically the relevance of ads is becoming a huge problem in the internet industry today. Users are complaining that they're not seeing relevant ads and they're scared to click on ads. And advertisers feel that they're wasting a lot of money on advertising. Shaunak Mishra: So at this juncture where advertising is really critical for a company's revenue and people are not satisfied with ads, it's really important to focus on making ads more relevant. And let me tell you a quick story to explain this better. Shaunak Mishra: The story is quite famous in the advertising research circles and it goes as follows. So there was a pizza shop owner and the owner hired three interns for the summer. And the job of the interns was to give coupons to potential customers and drive them towards the shop. Shaunak Mishra: So the deal was as follows, the owner told the interns, "Hey, I'll give you 100 coupons each and after a month I'll count how many coupons or how many customers used your coupon and came to my shop. And I'm going to reward you accordingly”. Shaunak Mishra: So after a month, the first intern managed to give out only 10 coupons. The second intern managed to give only 20 coupons and the third intern managed to sell or give all 100 coupons. So the owner was super impressed and gave the third intern a huge bonus. But the first two interns were all quite surprised. Like, "Hey," and they asked the third intern, "Hey, how did you manage to give away all 100 coupons? Whereas we just managed to give 20 or 10 coupons". And the third intern smiled and said, "Hey, I was just standing outside the door of the pizza shop." Shaunak Mishra: The story shows that coupons are like ads. They entice users to go and make a purchase. In this story, the purchase was that of buying a pizza from the pizza shop. And here you can see that the third intern simply gamed the system by selling off all the coupons and getting good credit. Whereas from an outsider's perspective, as you are hearing the story, you feel that, hey, he didn't make any difference. The people were anyway going to buy stuff from the shop. Shaunak Mishra: This is how the relevance of ads comes into the picture. So with this story in mind, we took some motivation from the marketing literature, where there exists many studies that say that there is a purchase funnel, which a user passes through as he or she makes a purchase decision. Let me explain this to you through some examples. Shaunak Mishra: Imagine you want to go to a theme park and let's say a theme park is advertising with an ad platform. And there might be users who might be aware of the theme park, they might know, “hey this theme park exists”. But may not be intent on going to the theme park anytime soon. But there might be users, by users I mean online users, who are just bent on visiting the theme park. They're searching for deals. Shaunak Mishra: You have two sets of users. One who is not intent on visiting the theme park and the other set is that they're really intent on visiting the theme park. And the idea was to customize ads based on these differences. For the one that was not intent, the ad could show or highlight new rides in the theme park and entice them to visit, just tempt them to visit them. Whereas the one who is intent on visiting the theme park, such users can be shown ads with deals, like hey, just take 20% off. Shaunak Mishra: This was the idea like, hey, you can separate or segregate the population of users into these different groups. One of them was aware or another shows intent. Or there could be an intermediate one who was just interested but still not sure about it. Shaunak Mishra: What we did was we used a deep learning model, namely an attention-based recurrent neural network, which processes the trail of activities that an online user has done. And automatically puts the user in one of these segments. And having done so we can show custom ads to these users and we did large scale online experiments. And what we learned was doing such ad customization based on funnel stages, has a huge increase in the click-through rate, which is a proxy for user engagement, as well as, advertiser satisfaction. Meaning people who clicked on ads really ended up going to the theme park in this example and they purchased more from the advertiser. Shaunak Mishra: This was the underlying theme of the paper called “Understanding Consumer Journey Using Attention-based Recurrent Neural Networks”. In the other paper which had the title, “Learning from Multi-User Activity Trails for B2B Ad Targeting”, we still use the idea of a purchase funnel but the main problem we focus on in that paper was in an organization, where a single person does not make a purchase decision. There might be multiple people in the organization. Shaunak Mishra: So for example, if the organization wants to buy, let's say a Wi-Fi service. Then maybe a market analyst or a junior employee in the organization does all the market research, which is the awareness or interest stage in the purchase funnel. And he or she influences the owner of the organization to make the purchase and the owner enters the intent stage where the intent is to buy the product. Shaunak Mishra: So here you can see that there is a purchase funnel, but multiple users in the organization are interacting with the purchase funnel. And we use this observation to learn from the activities done across users in an organization to help find relevant ads for that organization. So this was the theme of the second paper. Rosalie Bartlett: Such impressive work and it's also just a better user experience. Shaunak Mishra: Right. Rosalie Bartlett: It's relevant. It's what you're actually looking for versus something that you're getting and you're saying, "I don't really like these ads. They're not relevant to me". So for folks listening in, if they want to learn more about this research, where can they go? Shaunak Mishra: So since this is a community paper, that should be available on the ACM digital library for about a one year and after that the access is restricted to people who have the ACM subscription. But for the next year, if you are interested in this paper, please check out the ACM digital library with the name of the paper. Rosalie Bartlett: Well, those are all the questions from me. I know you're very busy so I won't take up any more of your time. I just want to say that it has been so great to chat with you today. Thank you so much. Shaunak Mishra: Thank you. It was great chatting with you. Gil Yehuda: If you enjoyed this episode of Dash Open, and if you want to learn more about the Open Source program at Verizon Media or any of the other things that we work on, visit our website at opensource.yahoo.com and you can find us on Twitter @YDN. And thank you for listening to the show.