X Algorithm Pipeline

The X recommendation algorithm processes posts through an 8-stage pipeline to generate the "For You" feed. Each stage transforms, filters, or scores the candidate posts.

Pipeline Overview

┌─────────────────────────────────────────────────────────────────────────────┐
│                         X RECOMMENDATION PIPELINE                           │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                             │
│   User Request                                                              │
│        │                                                                    │
│        ▼                                                                    │
│   ┌─────────────┐                                                           │
│   │ 1. Query    │  Hydrate user features, action history, socialgraph       │
│   │   Hydration │                                                           │
│   └──────┬──────┘                                                           │
│          ▼                                                                  │
│   ┌─────────────┐  Thunder (in-network) + Phoenix (out-of-network)          │
│   │ 2. Sources  │  In-network: Posts from followed accounts                 │
│   │             │  Out-of-network: ML retrieval from all posts              │
│   └──────┬──────┘                                                           │
│          ▼                                                                  │
│   ┌─────────────┐                                                           │
│   │ 3. Candidate│  Fetch tweet text, author data, visibility status         │
│   │   Hydration │                                                           │
│   └──────┬──────┘                                                           │
│          ▼                                                                  │
│   ┌─────────────┐                                                           │
│   │ 4. Pre-Score│  Age, duplicates, safety, blocked authors                 │
│   │   Filtering │                                                           │
│   └──────┬──────┘                                                           │
│          ▼                                                                  │
│   ┌─────────────┐  Phoenix ML → WeightedScorer → AuthorDiversity → OON      │
│   │ 5. Scoring  │  Each scorer adds/adjusts candidate.score                 │
│   │             │                                                           │
│   └──────┬──────┘                                                           │
│          ▼                                                                  │
│   ┌─────────────┐                                                           │
│   │ 6. Selection│  TopKScoreSelector: Keep top N by final score             │
│   │             │                                                           │
│   └──────┬──────┘                                                           │
│          ▼                                                                  │
│   ┌─────────────┐                                                           │
│   │ 7. Post-    │  Conversation dedup, previously seen, keywords            │
│   │   Filtering │                                                           │
│   └──────┬──────┘                                                           │
│          ▼                                                                  │
│   ┌─────────────┐                                                           │
│   │ 8. Side     │  Logging, analytics, impression tracking                  │
│   │   Effects   │                                                           │
│   └──────┬──────┘                                                           │
│          ▼                                                                  │
│      Feed Response                                                          │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Stage Details

1. Query Hydration

Enriches the request with user context:

User features (followed users, blocked users, muted users)
User action sequence (engagement history for ML)
Muted keywords
Subscription status
Bloom filters for seen posts

2. Sources

Two candidate sources provide posts:

Thunder Source (In-Network)

// home-mixer/sources/thunder_source.rs
// Posts from accounts the user follows
served_type: Some(pb::ServedType::ForYouInNetwork)

Queries Thunder service with user's following list
Returns recent posts from followed accounts
Includes conversation context (ancestors, reply chains)

Phoenix Source (Out-of-Network)

// home-mixer/sources/phoenix_source.rs
fn enable(&self, query: &ScoredPostsQuery) -> bool {
    !query.in_network_only  // Disabled for "Following" tab
}
served_type: Some(pb::ServedType::ForYouPhoenixRetrieval)

ML-based retrieval using user embedding
Finds relevant posts from the entire corpus
Enabled for "For You", disabled for "Following"

3. Candidate Hydration

Fetches full post data:

Tweet text content
Author information
Media metadata (video duration)
Visibility filtering results
Subscription requirements

4. Pre-Score Filtering

Removes ineligible candidates before expensive ML scoring:

AgeFilter - Too old
DropDuplicatesFilter - Duplicate IDs
VFFilter - Safety violations
AuthorSocialgraphFilter - Blocked/muted authors
CoreDataHydrationFilter - Missing data
IneligibleSubscriptionFilter - Subscription required

5. Scoring (4 Stages)

a) PhoenixScorer

// home-mixer/scorers/phoenix_scorer.rs
// Calls Phoenix ML to predict engagement probabilities

Produces phoenix_scores with 18 action probabilities.

b) WeightedScorer

// home-mixer/scorers/weighted_scorer.rs
// Combines probabilities into single score
weighted_score = Σ(weight × P(action))

Produces weighted_score from action predictions.

c) AuthorDiversityScorer

// home-mixer/scorers/author_diversity_scorer.rs
// Penalizes multiple posts from same author
multiplier = (1 - floor) × decay^position + floor

Adjusts scores to promote variety.

d) OONScorer

// home-mixer/scorers/oon_scorer.rs
// Adjusts out-of-network post scores
if !in_network: score *= OON_WEIGHT_FACTOR

Balances in-network vs out-of-network content.

6. Selection

// home-mixer/selectors/top_k_score_selector.rs
pub struct TopKScoreSelector;

impl Selector<ScoredPostsQuery, PostCandidate> for TopKScoreSelector {
    fn score(&self, candidate: &PostCandidate) -> f64 {
        candidate.score.unwrap_or(f64::NEG_INFINITY)
    }
    fn size(&self) -> Option<usize> {
        Some(params::TOP_K_CANDIDATES_TO_SELECT)
    }
}

Keeps top K posts by final score.

7. Post-Score Filtering

Fine-grained filtering after selection:

DedupConversationFilter - One post per conversation
RetweetDeduplicationFilter - One version per underlying post
PreviouslySeenPostsFilter - Remove seen posts
PreviouslyServedPostsFilter - Remove from current session
MutedKeywordFilter - User keyword mutes
SelfTweetFilter - Remove own posts

8. Side Effects

Non-blocking operations after response:

Impression logging
Analytics events
Cache updates

Data Flow Summary

Candidates start with:
├── tweet_id, author_id (from Sources)
├── tweet_text, metadata (from Hydration)
├── phoenix_scores (from PhoenixScorer)
├── weighted_score (from WeightedScorer)
├── score (from AuthorDiversity + OON)
└── Final ranking by score

PostCandidate Structure

pub struct PostCandidate {
    pub tweet_id: i64,
    pub author_id: u64,
    pub tweet_text: String,
    pub in_reply_to_tweet_id: Option<u64>,
    pub retweeted_tweet_id: Option<u64>,
    pub retweeted_user_id: Option<u64>,
    pub phoenix_scores: PhoenixScores,      // ML predictions
    pub weighted_score: Option<f64>,         // After WeightedScorer
    pub score: Option<f64>,                  // Final score
    pub served_type: Option<ServedType>,     // Source type
    pub in_network: Option<bool>,            // Following or not
    pub ancestors: Vec<u64>,                 // Conversation context
    pub video_duration_ms: Option<i32>,      // For VQV eligibility
    pub visibility_reason: Option<FilteredReason>,
    pub subscription_author_id: Option<u64>,
    // ...
}

Source Configuration

Tab	Thunder (In-Network)	Phoenix (Out-of-Network)
For You	Enabled	Enabled
Following	Enabled	Disabled

Related Skills

/x-algo-scoring - Detailed scoring formula
/x-algo-filters - All filter implementations
/x-algo-engagement - Action types and signals
/x-algo-ml - Phoenix ML model architecture

x-algo-pipeline