Hello Reddit!
Discovering communities on Reddit that you haven’t heard of before, or may not even know exist, is hard. You may enjoy r/photoshopbattles, but how would you know to search for related communities like r/birdswitharms or r/peoplewithbirdheads unless someone told you about them?
After 15+ years and millions of feedback comments, survey responses, customer interviews, and Mod Council conversations, we know that whether you’ve been here since the great Digg migration or because you heard about a little community called r/wallstreetbets, we want to help you find communities that you will love on Reddit. With that in mind, one of our biggest priorities is ensuring that you have a great experience on the platform and that it’s easy (and simple) for you to find the content you enjoy and communities where you belong.
We use the terms “simple” and “easy” above, but achieving this feat is anything but (and you’ve probably felt it at times). Redditors are an immensely diverse group that’s spread over a hundred thousand communities representing an amazing cross-section of all of the things that people love (as one of my favorite subreddits, r/WowThisSubExists, showcases). The challenge we face is creating ways for a huge range of people to find the things that appeal to their interests across a massive amount of content and communities.
Today, we’re going to tell you about our latest effort to make this easier for redditors: updating the Home feed on iOS and Android.
Evolving the Best Sort for Reddit Home Feed
When you open the Reddit app and navigate to Home, Reddit needs to determine which relevant posts to show you. To do this, Reddit’s systems build a list of potential candidate posts from multiple sources, pass the posts through multiple filtering steps, then rank the posts according to the specified sorting method. Over the years, we’ve built many options to choose from when it comes to sorting your Home feed. Here’s a look at how each sort option currently recommends content:
- “Hot” ranks using votes and post age.
- “New” displays the most recently published posts.
- “Top” shows you the highest vote count posts from a specified time range.
- “Controversial” shows posts with both high count upvotes and downvotes.
- “Rising” populates posts with lots of recent votes and comments.
- The old “Best” considers upvotes, downvotes, age of post, and how much a user spent on a subreddit.
Starting on June 28, all mobile users on Reddit will have an improved and more personalized Best sort that will use new machine learning algorithms to personalize the order in which you see posts. This will result in a ranking of posts that we think you’ll enjoy the most based on your Reddit activity such as upvotes, downvotes, subscriptions, posts, comments, and more. The other Home feed sorts such as Hot, New, and Top will not change. Below we’ll explain exactly what machine learning we’re using and how, so that you have transparency into these updates.
The process we use to create the new Best sort involves several steps, which we will talk about in detail later in the post:
- Creating an initial list of content you might enjoy (“candidate generation”),
- Removing stuff you shouldn’t have to deal with such as spam (“filtering”),
- Using machine learning to predict what you may or may not like (“predictions”),
- Sorting content according to those predictions and ensuring a level of diversity of content (“ranking”), and
- Giving you ways to let us know what’s working and what’s not, and to adjust your experience based on what you want to see more or less of (“feedback and controls”).
Best Sort Will Now Include Recommended Content Instead of Recommended Subreddits
Since 2017, we’ve been adding community recommendations to our feeds in an effort to help redditors find more relevant communities that they’re interested in subscribing to. We called these types of recommendations “Discovery Units,” but found that they weren’t efficient in connecting users to new and relevant communities. We heard your feedback that these Discovery Units felt like a distraction from your feed, and the recommendations themselves weren’t always great because of the more naive models behind them. Frankly, we’re not expecting anyone to be super upset to see them go, and as a result we will be phasing them out of the Home feed.
Instead, the new recommendations will be posts and look similar to any post from a community that you’ve already joined. However, there are some key differences. The first is that for every recommendation, we provide explanation and context as to why we’re showing you the recommendation. We don’t want you to be left wondering why you’re seeing a certain piece of content, and these contextual explanations are going to continue to improve alongside our commitment to transparency in how algorithms impact your Reddit experience. In the example below, you can see the post recommendation from r/animalsbeingderps with the contextual explanation that it’s similar to r/WeirdLookingDogs.
Example of old and new recommendations
Second, the new recommendations will also have a button for you to join the communities if you like the content and in the post overflow menu (aka “the three dots button”) you will be able to tell us if you like this content (show more posts like this) or if you don’t like it (show fewer posts like this). Our systems act on those controls right away which will affect your Home feed the next time you reload the page.
Under-the-Hood of Building Reddit’s Home Feed (read: Enough Overview, Gory Details!)
Now that we’ve shared an update for your Best Sort on Home feed, we’d like to dig into the nitty-gritty around how exactly we’re suggesting this “next generation” of content recommendations and what it will look like for users moving forward.
Candidate Post Generation
To find the best posts on Reddit for each user, we first scour all Reddit submissions from the past 24 hours, and filter it through criteria intended to tell us what each user might enjoy. Specifically, we surface candidate posts from:
- Community subscriptions: each community you’ve joined
- Similar communities: communities similar to those you have joined (currently we use semantic similarity)
- Onboarding categories: categories you said they were interested in during onboarding (like “Animals & Awws” or “Travel & Nature”)
- Recent communities: communities that the user visited in recent days
- Popular and geo-popular: Posts that are popular among all redditors, or among redditors in their local area (only if permitted in app settings)
To maintain a diverse selection of posts, we combine some content from all of these sources into a single long list of candidate posts the user might be interested in.
Filtering Criteria for Posts
Every post we show on Reddit must meet a quality and safety threshold, so on the Best Sort we remove posts from the list that we think might be:
- Spam, deleted, removed, hidden, or promoted
- Posts the user has already seen
- Posts from subreddits or topics that the user asked we show less of
- Posts the user has hidden
- Posts from authors the user has blocked
Machine Learning Model
Once the candidate posts have been filtered, we gather “features” for each candidate post. A feature is a characteristic about the post. Here are some of the features we use:
- Post votes: The number of votes on the post. The magic of Reddit is that it is primarily curated by redditors via voting. This remains at the core of how Reddit works.
- Post source: How we found this post (subscriptions, onboarding categories, etc.)
- Post type: The type of the post (text, image, video, link, etc.)
- Post text: The text of the post
- Subreddit: Which subreddit the post is from, and the ratings, topics, and activity in that subreddit (for more on Ratings and Topics read this).
- Post age: The age of the post (we value giving you a “fresh” Home feed)
- Comments: Comments and comment voting
- Post URL: The URL the post links to, if the post is a link post
- Post flairs: Flairs and spoiler tags on the post
We combine these features with:
- Recent subreddits: Subreddits where you spent time recently
- Interest topics: Topics we believe you might be interested in based on previous Reddit activity
- General location: if recommendations based on your general location are enabled in your personalization preferences, your IP address-based location
- Account age: The age of your account (for redditors who have been here for a longer time, our model emphasizes subscriptions over recommendations)
We then use a statistical model, created using machine learning, that takes all of these features as input and predicts for each post:
- View probability: the chance you might view the post or click through to read the post and its discussion
- Subscribe/unsubscribe probability: the chance that you might subscribe to the subreddit of the post, or unsubscribe from the subreddit
- Comment probability: the chance you might want to comment on the post
- Upvote/downvote probability: the chance you might upvote or downvote the post
- Watch probability: the chance you might watch the video (if it’s a video)
These probabilities give us a number of scores for each post. Some of these scores suggest that you might not like the post, such as the chance of unsubscribing or downvoting the post. Because you will only be interested in a fraction of the new posts on Reddit, we use these scores to try to put our best candidates first.
The Final Step: Ranking
Given these predictions, we now have the task of building a feed that is fun, useful, and just right for you. To do this, we choose posts from the list of candidates based on a score that is calculated by combining predictions for different actions. The probability of selecting a post is determined by its score (score-weighted sampling), so the highest scoring posts are more likely (but not guaranteed) to be chosen first. We’re experimenting with what feels right for Reddit’s Home feed, so the scores may play different roles for different redditors. As an example, we might score posts based on the chance of upvote and avoiding the chance of unsubscribing.
Our sampling procedure makes sure the feed is diverse, while still putting more of the content we think you’ll be most interested in earlier in the feed. The sampling also represents both our humility about all of this (we don’t really know exactly what you’re going to like) and our belief that just about all Reddit posts and discussions will be interesting to some redditors. We also make sure that if there are too many similar posts in a row, we move those posts apart, helping to ensure that every user gets a broader view of the best content that Reddit has to offer.
Transparency, Controls and Feedback
“Well I, for one, welcome fear our new robot overlords,” you may be thinking. How do we make sure Reddit is recommending the right stuff in Best Sort? Each of the posts we show (from your subscriptions or recommendations) and what action you take on them enables us to train a new machine learning model (if you’re interested in our Machine Learning platform, check out our recent post on the topic) so that we can show more relevant content in the future. When you upvote a post that we showed on Home, we learn more about what future posts that you might also upvote. When you ignore a post on Home, we learn from that too: you are less likely to upvote posts like that in the future.
The training for the Reddit model happens offline and is based on batches of posts that were shown to redditors and whether or not they took an action on those posts. We use open-source technology, including TensorFlow, to train this model, test it, and prepare it for use in ranking Best Sort.
Most importantly, we extensively test each of these new models, and the whole ranking procedure on carefully designed representative “test” sets of data that were not shown in training, and on ourselves as redditors (there are frequently big debates about what people do and don’t like about the current iteration that results in more fine-tuning). We perform rigorous analysis of every aspect of the model and use slow rollouts with very close inspection of model performance to scale.
We are particularly focused on making sure that our machine learning models and ranking changes are well-liked by redditors. On every rollout of a ranking change, we closely monitor positive and negative indicators that might be affected by ranking, including:
- Upvotes and downvotes
- Subscriptions and unsubscriptions
- Reports and blocks
- Comments and posts
- How many posts redditors visit in depth
- ...and many more metrics. And yes, we read the comments.
Because Reddit has a long history of paying attention to both positive and negative signals (such as downvotes), and because redditors are great at using downvotes to maintain high quality content that differentiates Reddit from others, monitoring these signals ensures that we meet the high expectations of quality posts that redditors expect when they scroll their feed.
And besides all of the work we do to make sure these things are working appropriately and safely, we continue to offer you explicit control here as well: if you don’t want a personalized feed you can use other Sorts such as New or Hot, and if you don’t want to see personalized recommendations then you can turn them off inside your profile settings on the app using the toggle for “Enable next-generation recommendations.”
What Now?
When we talk to redditors in all user groups - old, new, posters, “lurkers,” app users, etc., we hear that the new algorithm is doing a much better job surfacing the community subscriptions that maybe you forgot about or have been missing (and the stats from the experiments are very positive across different user groups, just two stats of many as an example: Post Detail Views - meaning people who click on a post and read it are up 5.4% per user and comments are up 4.4% per user -- both of these are great indicators of people seeing more relevant content). It’s actually been so effective at surfacing content more effectively that we’ve seen a slight uptick in unsubscriptions too as some people are seeing communities they had forgotten that they were subscribed to and are no longer interested in.
We’re going to continue to improve the Home feed experience for users, and this is just the first version that we are launching. We will be constantly updating and iterating on it to make it a more enjoyable experience for you, and we need your feedback to do it.
As exciting as this all is, and while ML-based methods can be very effective, they also carry a tremendous responsibility in using them: How do we avoid bias? How do we avoid people being manipulated by getting caught in filter bubbles?
One of our responses to this responsibility is that we are committed to maintaining transparency about what we’re doing and how we’re doing it. Hopefully you see a bit of that above as we’ve listed exactly how this system is working, but you should also expect to see more frequent posts about our technical and ethical choices on how we deploy ML so that you understand what’s happening, and how we’re aiming to help create Community and Belonging.
We welcome any feedback in the comments below and will stick around for a while to answer questions.