Registration, welcome and breakfast

Opening

Combining Relevancy Scoring and Contextual Bandits for Personalized Ranking in Music Discovery

Since 2005, Pandora has built a powerful music recommendation platform, now attracting an audience of over 81 million monthly active listeners. While Pandora has accumulated significant expertise in audio-driven continuous recommendations, balancing popular and long-tail songs, a fully personalized Browse area is our first attempt in dramatically expanding user’s music discovery in a visually rich fashion.

Browse is aimed at providing users with a discovery experience. Different types of items, such as albums, curated music stations, and artists, are grouped into modules and presented to users on one scrollable page. In the first version of the product, modules were given a fixed order. However, shortly after launch, we set out to personalize the ranking of these modules. In this talk, we will introduce the challenges in ranking modules, namely, (1) finding a metric to compare relevance of different item types, (2) learning to rank from unbiased data, and (3) online learning of changing preferences. Next, we will discuss how we solved these problems by combining relevancy scores and contextual multi-arm bandits to greatly improve the MRR metric. Finally, we will share scalable implementation details. We hope this talk will provide other practitioners with a case-study that highlights the flexibility of contextual bandits in music discovery.

Dr. Tao Ye is a Principal Scientist at Pandora. She is a founding member of the Pandora science team, and has been working on personalized recommendation systems, measurements, and user modeling since 2010. Most recently, she has been leading the science and architecture of recommendation systems in Browse. She has two decades of experience in the software industry, holding research scientist and lead engineer positions in social media, networking and mobile systems. She holds 14 granted patents and has published 12 peer reviewed papers. She received her PhD from University of Melbourne in Electrical and Electronic Engineering, her MS from UC Berkeley in EECS and dual BS degrees from Stony Brook University in CS and Engineering Chemistry.

Mohit joined Personalization and Discovery team at Pandora as a Scientist. At Pandora, he has worked on various projects ranging from personalizing music for radio, artist recommendation and solving “ranking" problems in browse. Prior to Pandora, Mohit worked at Rdio as a data scientist where he lead music personalization based on listener context and features extracted from audio. He has over 5 years of industry experience specifically in large scale recommendation systems working in companies like Intel Research, American Express and One Kings Lane. He received his first Masters in Robotics from CMU and is currently pursuing his second Masters in CS from Georgia Tech.

Personalized Media Recommendations at Facebook

Facebook has over a billion pieces of content shared everyday but your News Feed surfaces only a small fraction of what might be interesting to you. To solve this we are building discovery surfaces serving personalized content recommendations.

The goal of this project is to deliver most interesting and important public content stories of a person’s day. In this talk we'll go over product considerations and then dive into candidate generation and ranking techniques in order to serve personalized recommendations for every user on Facebook.

Break

Scalable Collective Reasoning for Richly Structured Socio-Behavioral Data (slides)

Online behavior and social media provide richly structured
socio-behavioral data, which can be useful for personalization, recommendation and search. One of the challenges in making use of this data lies in being able to reason collectively about extremely large, heterogeneous, incomplete, and noisy interlinked data. Machine learning methods in these settings often need to infer unobserved
and/or latent attributes of individuals and their relationships. Performing these inferences correctly requires complex collective reasoning about dependencies. In this talk, I will describe some common inference patterns that are useful for socio-behavioral networks. I will then describe probabilistic soft logic (PSL), a highly scalable open-source probabilistic programming language being developed within my group to solve these challenges. Finally, I will describe a variety of applications of PSL, ranging from information integration to hybrid recommender systems.

Lise Getoor is a professor in the Computer Science Department at the University of California, Santa Cruz. Her research areas include machine learning, data integration and reasoning under uncertainty, with an emphasis on graph and network data. She is a Fellow of the Association for Artificial Intelligence, an elected board member of the International Machine Learning Society, serves on the board of the Computing Research Association (CRA), and was co-chair for ICML 2011. She is a recipient of an NSF Career Award and eleven best paper and best student paper awards. She received her PhD from Stanford University in 2001, her MS from UC Berkeley, and her BS from UC Santa Barbara, and was a professor in the Computer Science Department at the University of Maryland, College Park from 2001-2013.

Ethical considerations in online algorithm experimentation (slides)

Online experimentation (A/B testing) is a common practice for many of today's internet products that leverage personalization, search, and recommendation systems. By conducting online experiments with existing or potential customers, companies can identify algorithm improvements that are backed by statistical evidence. While this is beneficial for consumers from the perspective of accessing high-quality, ever-improving services, it also raises some ethical questions: Are consumers adequately aware of their participation in online experiments? How should other principles that have been formalized for human subjects research - such as beneficence and justice - apply to the online product world, where most experiments are trivial by comparison? This talk will explore how ethical considerations may (or may not) pertain to a variety of online algorithm experiments.

 

Caitlin Smallwood is the Vice President of Science and Algorithms at Netflix, where she and her team drive predictive decision models, algorithm / machine learning research, and experimentation science for all parts of the Netflix business. Caitlin is particularly passionate about personalization and other mechanisms of providing value to people through data.

Prior to joining Netflix in 2010, Caitlin worked at Intuit, Yahoo!, and several consulting firms (PwC, SRA), focusing on an array of analytic disciplines and products. She is experienced at leading strong teams and has built several data science groups from scratch. Caitlin holds an M.S. in Operations Research from Stanford University and a B.S. in Mathematics from The College of William and Mary.

Lunch

Interesting Finds on Amazon: Personalized, Diversified Ranking for Visual Browsing (paper)

Search queries are appropriate when customers have explicit intent, but they perform poorly when the intent is difficult to express or if the customer is simply looking to be inspired. Visual browsing systems allow e-commerce platforms to address these scenarios while offering the customer an engaging shopping experience. Here we explore extensions in the direction of adaptive personalization and item diversification within Interesting Finds, a new form of visual browsing and discovery by Amazon. Our system presents the customer with a diverse set of interesting items while adapting to customer interactions. When tested on live traffic, our algorithm shows a strong lift in session duration and next day revisits.

Personalized Content Blending in the Pinterest Homefeed (slides)

The Pinterest Homefeed is a personalized feed of content (or “Pins”) drawn from many sources, including followed users, followed topics, and recommendations, among other sources. Each types of content is ranked by its own specialized machine learning model, and then blended with a ratio-based round robin to create the final Homefeed.

This presentation dives into how the current system evolved, and describes in depth an approach for personalizing the content blending ratio. This method uses historical user action data and models the Pin action rates of each pin type as a Bernoulli distribution. Each content type’s overall utility is modeled as a sum of the Pin action rate distributions, weighted by action-specific reward constants. I will discuss different methods for assigning these blending ratios based on the utility distribution.

As we iterate on our blending systems, new questions have arisen as to how we measure success. Unlike traditional search ranking problems, Pinterest faces both short- and long-term optimization challenges as we balance immediate user-engagement metric movements and long term ecosystem health. This talk concludes with an overview of some of the different dimensions of success we currently monitor as we continue to work on blending.

Stephanie deWet is a software engineer at Pinterest on the homefeed team, where she builds personalized blending models and recommendations. She earned a MS in Computer Science from the University of Wisconsin in 2014. Prior to that, she spent 4 years developing realtime simulations at L-3 Communications.

Sequential Recommendation at YouTube

Feed-forward neural networks are increasingly used for industrial regression and classification tasks, but they often lack a rich understanding of the temporal sequence of users’ previous actions as well as their future trajectory. Recurrent Neural Networks (RNNs) promise to flexibly represent user history without laborious feature engineering. Reinforcement Learning (RL) provides a framework for modeling successive actions and progress towards a long-term goal. Leveraging simple RNNs and RL, we were able to improve one the largest-scale recommendation systems in existence. I’ll introduce basic concepts from these domains, share high-level insights from our experience and hint at future directions.

Paul Covington is a Staff Software Engineer at Google working on the YouTube Recommendations team. Prior to joining YouTube in 2014, he completed graduate work at Stanford in computational physics.

Closing, informal discussions and drinks

End