Introduction & problem statement
A content team maintaining hundreds of pages can't manually audit all of them. The natural question — "which pages should we look at first?" — usually gets answered by intuition: pages that feel old, or that someone remembers writing badly. This project replaces that intuition with an observed, measurable signal: pages whose click-through rate sits meaningfully below what similar pages, at a similar rank, in a similar content format, actually achieve.
The decision this supports is narrow and human-scoped on purpose: a reviewer with capacity for roughly 50 pages a cycle needs an ordered queue, with a reason attached to each entry, so they can decide — not have decided for them — whether a title or meta description rewrite is worth attempting. Getting the ranking wrong costs reviewer time on a page that was never the problem; it does not cost a client anything directly, since no action is taken automatically. That asymmetry is why this is framed as decision-support throughout, never as an automated fix.
Data
The project spans two scales using anonymized/pseudonymized FlyRank releases:
- Full warehouse release — a pseudonymized Hugging Face dataset (
FlyRank/internship-warehouse) of ~78.8M daily content-performance rows, ~520K content items, and 104 clients, used via DuckDB SQL for feature engineering, scaling limits, and data contract validation. - Starter release (Modeling Slice) — a focused 30,000-row, 44-column extracted slice covering search metadata, content characteristics, and 90-day GSC/GA4 performance. After filtering for visible pages (≥100 impressions), this resulted in a core modeling dataset of 22,006 pages.
Excluded on purpose: the final-month partition of the warehouse (June 2026) was treated as a sealed test month and never used for label design, since it's the natural outcome window of any past-to-future label. FlyRank's own internal product flags (health scores, priority scores) were not available in the starter slice and were never reconstructed. No client names, domains, URLs, or raw queries appear anywhere in this project.
Methodology
Task framing
Framed as binary classification producing a probability score, not regression — the real decision is a ranked queue, and classification lets the model be evaluated the same way the queue will actually be used: how many of the top K matter.
Label
A page is labeled a CTR anomaly (is_ctr_anomaly = 1) if its observed CTR falls below half the
median CTR of its peer group, defined as pages sharing the same position_tier and content_type.
This directly encodes "underperforming relative to comparable pages," not an arbitrary fixed threshold — and
deliberately avoids trend_direction/trend_pct, which an earlier leakage test
showed are themselves near-tautological with any trend-based label. Label rate
on the 22,006-row analytical slice: 31.6% positive.
Features
Five features, each knowable before the review decision is made: impressions_90d, avg_position,
content_age_days, word_count, engagement_rate. ctr and
clicks_90d were deliberately excluded because ctr is an ingredient in the target
label function, and clicks are highly correlated.
Baseline
A transparent hand-written rule, encoded before any model training: flag a page as
high_volume_low_ctr if it ranks in the top 10 with ≥500 impressions and CTR below 1.5, or as
striking_distance_opportunity if it ranks 11–20 with ≥250 impressions and CTR below 1.0. Score
scales with impressions × the CTR gap, so the highest-exposure misses surface first. (Note: Post-audit
analysis revealed the rule operates primarily as an impressions sorter, Spearman correlation 0.90,
rather than a strict CTR filter).
Validation design
Client-holdout split (GroupShuffleSplit, 75/25 on client_id) — no page from a
test-set client is ever seen during training. This matters more than it sounds: a naive random row split let
26 of 30 clients (87%) appear in both train and test, which is enough for a model to partly
memorize client-specific baseline CTR levels rather than learn a pattern that generalizes to a new client.
Section 4 shows exactly how much that inflates the apparent result.
trend_pct, the
exact percentage the trend label is derived from, pushed in-sample accuracy from an honest 0.6445 to a
hollow 0.9999 on the warehouse contract test. Separately, reintroducing ctr into the final
model's feature set pushed Precision@50 from 0.640 to 0.980. Both leaks were removed before any reported
result; the tests are kept here specifically to show the leakage harness catches a real leak when one is
deliberately planted. (Note: engagement_rate was found to be 63% zero-inflated during the
audit, requiring careful monitoring to ensure it doesn't act as a client fingerprint.)
Results: model vs. baseline, same split
All four rows below were scored on the identical held-out test set (4,610 rows, 8 clients never seen in training, positive rate 28.6%) with the identical metric, so the comparison is apples-to-apples.
| Method | Precision@20 | Precision@50 |
|---|---|---|
| Random chance (label's positive rate) | 0.286 | 0.286 |
| Week-4 hand-written rule | 0.350 | 0.260 |
| Logistic Regression | 0.550 | 0.480 |
| Random Forest (final) | 0.850 | 0.640 |
Note the baseline rule falls slightly below random chance at Precision@50 (0.260 vs. 0.286) — on unseen
clients, the
handwritten thresholds don't generalize much better than guessing. The learned model's advantage is real,
but so is its error pattern: of its top-20 highest-confidence picks, 3 were false positives, and all 3
clustered in the page_3_5 position tier at the low end of the impression floor (146–227
impressions) — exactly where a CTR estimate is noisiest (and often represents exactly zero clicks). Feature
importance was reassuringly spread rather
than dominated by one input (impressions_90d 37%, engagement_rate 25%, avg_position
24%, word_count 8%, content_age_days 6%), which argues against a single hidden
leak.
Why the split matters: before / after
The same Random Forest, same features, scored under two different splits on the same data:
| Split | Precision@20 | Precision@50 | Client overlap, train/test |
|---|---|---|---|
| Naive random row split | 1.000 | 0.940 | 26 / 30 clients (87%) |
| Client-grouped split (reported above) | 0.850 | 0.640 | 0 / 30 (by design) |
Limitations & honest framing
- Correlation vs. Causation: This model identifies observed CTR gaps. It does not causally prove that updating a page's metadata will recover the clicks.
- Single-Seed Variance: The 0.640 precision is an observed snapshot for a specific 8-client holdout (Seed 42). Across 20 random seeds, Precision@50 fluctuates (mean 0.74, std 0.15), meaning it generally outperforms chance, but the exact number is not a fixed guarantee.
- Opportunity Size Limit: The top-ranked pages often have exactly zero clicks and sit near the 250-impression floor. The expected absolute click gap is extremely small (< 1 click per 90 days), meaning this identifies relative statistical anomalies, not massive absolute traffic recoveries.
- Queue Sorting & Inference: The deployed queue is ranked by predicted model probability, not raw CTR gap. Furthermore, top-queue performance on strict holdout clients is closer to ~0.48, which is lower than the overall 0.640 evaluation metric.
- Zero-Click SERPs & Brand Queries: The data cannot distinguish between a page that has a bad title tag, versus a page that ranks for a "Zero-Click" AI Overview, or a query where the user specifically wanted a competitor's brand.
Ranked recommendations: the action playbook
The model score and the baseline rule are combined into six archetypes, each mapped to a confidence tier and
a concrete action. The queue is ranked top-down by confidence_tier and model_score
(predicted anomaly probability). Agreement between rule and model earns the highest confidence; a model-only
flag earns a
"look before you touch it" tier rather than an automatic top ranking.
Human review is required, not optional
Before acting on any flagged page, a reviewer should rule out:
- Expected Click Gap: Even high-confidence anomalies might only be missing a fraction of a click. Do not assume high probability equals high traffic reward.
- Brand/navigational queries: a page can rank well and still get few clicks because searchers want a competitor, not this result. Check top queries before rewriting anything.
- Zero-click SERPs: an AI Overview or featured snippet answering the query directly makes a metadata rewrite pointless. Confirm the SERP shape first.
- Thin-volume noise: flagged pages near the 250 impression floor can look anomalous from small-sample variance alone.
Items in quiet_risk_flag
specifically require a look, since the model is disagreeing with the transparent rule.
What should never be automated
- Auto-publishing a rewritten title or meta description from a model score alone.
- Treating a high confidence tier as proof a rewrite will recover clicks — this data is observational.
- Acting on any page below the 250-impression visibility floor.
- Bulk-refreshing content purely by age, given the mixed/possibly-confounded freshness evidence.
Monitoring & retrain triggers
Re-check Precision@50 against the 0.640 baseline whenever a new month's data becomes available — a drop below roughly 0.50 pauses the queue pending review. Track the label's positive rate (currently ~28.6% in the test split) for drift, and retrain on an updated client-grouped split whenever new clients are onboarded, rather than appending them to a frozen model.