How do media companies build content recommendation systems?

Netflix's recommendation engine is responsible for over 80% of what its subscribers actually watch, according to a 2016 paper by Netflix researchers Gomez-Uribe and Hunt. That number has only grown since. For a company that spent roughly $17 billion on content in 2023, having software decide what gets surfaced is not a feature, it is the business model.

But the underlying machinery is not magic. It is a specific set of engineering choices, and those choices have costs and tradeoffs that every media company faces at every scale.

At its core, a recommendation system does one thing: it predicts which piece of content a specific user is most likely to engage with next. The prediction is based on two inputs, what you know about the user and what you know about the content.

The user side comes from behavioral signals. Every click, scroll, pause, skip, and replay tells the system something. A user who watches 90% of a documentary but skips the last 10 minutes sends a different signal than one who bails at the 30% mark. Spotify's research found that skip rates within the first 5 seconds are the strongest negative signal their system processes, outweighing explicit thumbs-down ratings in predictive power.

The content side comes from metadata: genre, format length, topic tags, language, publication date, and increasingly, AI-generated semantic tags that describe tone, complexity, and themes. A travel article tagged "budget travel, Southeast Asia, solo travel" matches against a user whose history shows exactly that cluster of interests, even if they have never read that specific author before.

Those two inputs feed into a model that generates a ranked list. The model is trained on historical data: what did users with this profile click on, and did they stay? Most systems combine two approaches. Collaborative filtering finds users with similar behavior and recommends what they liked. Content-based filtering matches content attributes to individual user preferences. Neither alone is good enough. Collaborative filtering fails for new users who have no history. Content-based filtering fails for unusual content that does not fit clean category boxes. Combining them is standard practice across teams at YouTube, Spotify, and Amazon.

At Netflix scale, this runs on hundreds of millions of user profiles updated in near-real-time. A McKinsey analysis from 2021 estimated that Netflix's combined recommendation and search capabilities generate over $1 billion in value annually by reducing subscriber churn. The system earns its budget.

Most publishers underinvest here, and it costs them. A recommendation model is only as good as the metadata it has to work with. Vague tags like "news" or "sports" are almost useless, they are too broad to drive meaningful personalization.

The metadata attributes with the highest predictive signal, based on published research from Spotify, YouTube, and academic teams at Cornell and Stanford, break into three groups.

Behavioral metadata is the most powerful. Completion rate (what share of users finish the content), return rate (do users come back to the same piece?), and share rate all outperform demographic signals like age or location as predictors. A 2022 paper from researchers at Google found that completion rate alone was responsible for 34% of the variance in their recommendation model's accuracy.

Semantic metadata covers what the content is actually about, not just its category label. Traditional taxonomy tags work, but AI-generated topic embeddings go further by capturing the conceptual relationship between pieces of content. An article about electric vehicle charging infrastructure and an article about urban planning share conceptual space even if their explicit tags do not overlap. A model trained on embeddings can surface that connection.

Contextual metadata accounts for when and where content gets consumed. Time-of-day patterns are consistent across large datasets: short-form content performs better in the morning and during commutes, long-form content in the evening. Platforms that ignore context end up recommending 45-minute documentaries to users who open the app at 7 AM on a weekday.

For a publisher building a recommendation system in 2024, the practical starting point is rigorous behavioral tagging and at least basic AI-generated semantic tags. AI tools that generate content embeddings cost a fraction of what they did three years ago, a publisher with a 100,000-article library can generate and store semantic tags for the full archive for under $2,000 using current API pricing.

This tension does not get discussed enough in technical literature, but every editorial team that has adopted recommendations eventually confronts it.

Pure personalization optimizes for engagement, and engagement is not the same as editorial value. A system trained entirely on clicks and watch time will consistently surface familiar content over challenging content, popular voices over emerging ones, and topics the user already knows over topics they might benefit from. Spotify's own research team published a 2018 paper acknowledging that pure collaborative filtering creates "filter bubbles" that reduce discovery over time and correlate with subscriber disengagement at 12-month horizons.

The solution most mature platforms use is a constraint layer on top of the personalization model. This layer enforces editorial rules that the pure ML model would otherwise violate. Diversity constraints ensure that no single topic, author, or format dominates a user's feed beyond a set threshold. Freshness rules weight recently published content upward for users who have shown novelty-seeking behavior. Promotional slots carve out fixed positions in recommendation feeds for content the editorial team wants surfaced regardless of predicted engagement.

The constraint layer is not anti-algorithm. It is what makes the algorithm work in service of a business that cares about more than raw engagement. The New York Times reported in 2023 that their recommendation team had introduced constraints specifically to surface reporting from underread international desks, which improved subscriber retention among users who engaged with that content by 18% over the following quarter.

For a media company building this in 2024, the practical structure is: let the model generate a ranked list of 30–50 candidates, then apply editorial constraints to select the final 5–10 shown to the user. The model handles personalization. The constraints handle editorial integrity. Neither replaces the other.

The cost structure breaks into three buckets: infrastructure, data storage and retrieval, and model training.

Infrastructure is the most variable. Running recommendations in real-time for millions of users requires compute that scales with traffic, not with your headcount. A publisher with 5 million monthly active users generating recommendations on every page load can expect to spend $8,000–$15,000 per month on compute alone, depending on model complexity and caching strategy. Publishers who pre-compute recommendations in batch (generating personalized feeds overnight rather than on-demand) cut this cost by 60–70%, with acceptable latency for most content formats. Live sports or breaking news platforms cannot pre-compute. Everyone else probably can.

Data storage scales with how much behavioral history you retain per user. Storing 90 days of click, scroll, and completion events for 5 million users runs about $1,500–$3,000 per month in cloud storage costs. Retention windows beyond 90 days offer diminishing returns for most content categories, because user preferences shift faster than models trained on 18-month-old data can track.

Model training is often the most misunderstood cost. Training a recommendation model from scratch on your data is expensive and slow. Most teams in 2024 are not doing this. They are fine-tuning pre-trained models, starting from a model that already understands user behavior patterns and adapting it to their specific content library. Fine-tuning a collaborative filtering model on a dataset of 10 million user interactions takes about 4–6 hours on standard cloud infrastructure and costs under $200 per run. Most teams retrain weekly.

Western data engineering agencies building a comparable system from scratch charge $200,000–$400,000 in development costs alone, before a single dollar of ongoing infrastructure. An AI-assisted development team builds the same recommendation pipeline for $40,000–$70,000, in 8–12 weeks rather than 6–9 months.

This is the question editorial teams at regional newspapers, B2B media companies, and niche content platforms ask most often. The short answer: yes, at 60–70% of the personalization lift, for a fraction of the cost.

Netflix's advantage is data volume. With 260 million subscribers, their models train on behavioral signals that smaller publishers simply cannot replicate. A publisher with 300,000 monthly active users does not have enough interaction data to train a deep collaborative filtering model with the same accuracy. But they do not need Netflix-level accuracy to generate meaningful engagement lift.

A 2023 study from the Reuters Institute found that publishers who implemented even basic recommendation systems saw a 23–35% increase in articles read per session and a 17% improvement in return visit rates within 90 days of launch. Those publishers were not running Netflix-scale infrastructure. They were using off-the-shelf recommendation APIs and a content metadata layer they built in-house.

The realistic path for a smaller publisher has three steps. Instrument the site to collect behavioral signals: what gets clicked, how long it is read, what gets shared. Tag the content library with semantic metadata, using AI-generated embeddings for any archive over 10,000 articles because manual tagging at that scale is not feasible. Then connect to a recommendation API like AWS Personalize or Google Recommendations AI, which are pre-trained at scale and priced on a consumption model, a publisher sending 10 million recommendation requests per month pays roughly $2,000–$4,000 depending on the service tier.

An AI-assisted development team can instrument, tag, and connect that full pipeline for $25,000–$40,000. A Western agency doing the same work quotes $120,000–$180,000. The output is the same recommendation system. The gap is the AI-native workflow, which compresses the repetitive instrumentation and integration work that inflates traditional agency invoices.

The one thing smaller publishers cannot shortcut is data quality. A recommendation system trained on a messy, inconsistently tagged content library will produce results that frustrate editors and confuse users. Cleaning and tagging the content library is usually 40% of the total project cost, and it is the 40% most publishers try to skip. Do not skip it.

Personalization is no longer a feature that requires Netflix's budget. For a publisher with a real content library and an audience worth keeping, the decision in 2024 is not whether to build recommendations. It is whether to build them with a team that charges for the old way of working, or one that has made the new way the default.

Book a free discovery call