Introduction & problem statement
A content team maintaining hundreds of pages can't manually audit all of them. The natural question — "which pages should we look at first?" — usually gets answered by intuition: pages that feel old, or that someone remembers writing badly. This project replaces that intuition with an observed, measurable signal: pages whose click-through rate sits meaningfully below what similar pages, at a similar rank, in a similar content format, actually achieve.
The decision this supports is narrow and human-scoped on purpose: a reviewer with capacity for roughly 50 pages a cycle needs an ordered queue, with a reason attached to each entry, so they can decide — not have decided for them — whether a title or meta description rewrite is worth attempting. Getting the ranking wrong costs reviewer time on a page that was never the problem; it does not cost a client anything directly, since no action is taken automatically. That asymmetry is why this is framed as decision-support throughout, never as an automated fix.
Data
Two data sources were used at different stages of the project, both anonymized/pseudonymized FlyRank releases:
- Starter release — a 30,000-row, 44-column anonymized CSV covering one content item per row: search metadata (volume, competition, CPC), content characteristics (word count, type, age), and 90-day and 30/prior-30-day GSC/GA4 performance (impressions, clicks, sessions, CTR, average position, trend direction). Used for framing, EDA, the baseline rule, the model, and the validation audit.
- Full warehouse release — a pseudonymized Hugging Face dataset (
FlyRank/internship-warehouse) of ~78.8M daily content-performance rows, ~520K content items, and 104 clients, used in the data contract stage to verify the grain, availability, and scale of the fuller panel behind the starter slice.
Time window: the starter slice's performance fields are 90-day trailing windows plus a
last-30 / prior-30 split, sourced from Google Search Console and GA4. The warehouse contract stage verified
March 2026 as a representative mid-panel month (9.84M rows, 3.61M with gsc_data_available IS
TRUE) and confirmed the grain directly: on 2026-03-15, total content rows and unique content_hash_id
counts matched exactly at 319,758 — one row per content item per day, as claimed.
Excluded on purpose: the final-month partition of the warehouse (June 2026) was treated as a sealed test month and never used for label design, since it's the natural outcome window of any past-to-future label. FlyRank's own internal product flags (health scores, priority scores) were not available in the starter slice and were never reconstructed. No client names, domains, URLs, or raw queries appear anywhere in this project.
Methodology
Task framing
Framed as binary classification producing a probability score, not regression — the real decision is a ranked queue, and classification lets the model be evaluated the same way the queue will actually be used: how many of the top K matter.
Label
A page is labeled a CTR anomaly (is_ctr_anomaly = 1) if its observed CTR falls below half the
median CTR of its peer group, defined as pages sharing the same position_tier and content_type.
This directly encodes "underperforming relative to comparable pages," not an arbitrary fixed threshold — and
deliberately avoids trend_direction/trend_pct, which an earlier leakage test
showed are themselves near-tautological with any trend-based label (see the leakage box below). Label rate
on the 22,006-row analytical slice: 31.6% positive.
Features
Five features, each knowable before the review decision is made: impressions_90d, avg_position,
content_age_days, word_count, engagement_rate. ctr and
clicks_90d were deliberately excluded — ctr is the numerator of the label itself,
so including it would reproduce the leakage trap demonstrated below in a new disguise.
Baseline
A transparent hand-written rule, encoded before any model training: flag a page as
high_volume_low_ctr if it ranks in the top 10 with ≥500 impressions and CTR below 1.5, or as
striking_distance_opportunity if it ranks 11–20 with ≥250 impressions and CTR below 1.0. Score
scales with impressions × the CTR gap, so the highest-exposure misses surface first.
Validation design
Client-holdout split (GroupShuffleSplit, 75/25 on client_id) — no page from a
test-set client is ever seen during training. This matters more than it sounds: a naive random row split let
26 of 30 clients (87%) appear in both train and test, which is enough for a model to partly
memorize client-specific baseline CTR levels rather than learn a pattern that generalizes to a new client.
Section 4 shows exactly how much that inflates the apparent result.
trend_pct — the
exact percentage the trend label is derived from — as a feature pushed in-sample accuracy from an honest
0.6445 to a hollow 0.9999 on the warehouse contract test, and pushed Precision@50 from 0.640 to 0.980 when
ctr was reintroduced into the final model's feature set. Both leaks were removed before any
reported result; the tests are kept here specifically to show the leakage harness catches a real leak when
one is deliberately planted.
Results: model vs. baseline, same split
All four rows below were scored on the identical held-out test set (4,610 rows, 8 clients never seen in training, positive rate 28.6%) with the identical metric, so the comparison is apples-to-apples.
| Method | Precision@20 | Precision@50 |
|---|---|---|
| Random chance (label's positive rate) | 0.286 | 0.286 |
| Week-4 hand-written rule | 0.350 | 0.260 |
| Logistic Regression | 0.550 | 0.480 |
| Random Forest (final) | 0.850 | 0.640 |
Note the baseline rule barely clears random chance at Precision@50 (0.260 vs. 0.286) — on unseen clients, the
hand-written thresholds don't generalize much better than guessing. The learned model's advantage is real,
but so is its error pattern: of its top-20 highest-confidence picks, 3 were false positives, and all 3
clustered in the page_3_5 position tier at the low end of the impression floor (146–227
impressions) — exactly where a CTR estimate is noisiest. Feature importance was reassuringly spread rather
than dominated by one input (impressions_90d 37%, engagement_rate 25%, avg_position
24%, word_count 8%, content_age_days 6%), which argues against hidden leakage.
Why the split matters: before / after
The same Random Forest, same features, scored under two different splits on the same data:
| Split | Precision@20 | Precision@50 | Client overlap, train/test |
|---|---|---|---|
| Naive random row split | 1.000 | 0.940 | 26 / 30 clients (87%) |
| Client-grouped split (reported above) | 0.850 | 0.640 | 0 / 30 (by design) |
Limitations & honest framing
- The label is a proxy, not ground truth. "CTR anomaly relative to peer median" is a reasonable, defensible construction — it is not a verified statement that a page's metadata is broken. Roughly 1 in 3 of the top-50 flagged pages, under this evaluation, is not a real anomaly.
- Correlation, not causation. This is observational data. Nothing here demonstrates that rewriting a title or meta description would recover clicks — that would require a controlled before/after experiment on actual edits, which this project does not run.
- Freshness is not a reliable lever, and is deliberately left out of the ranking. A
dedicated signal audit found
freshness_tieronly MIXED as a CTR predictor (fresher pages did not reliably outperform older ones in this slice). Separately, a comparable "freshness multiplier" finding elsewhere in FlyRank's own published research is self-flagged as statistically unstable at long windows and is plausibly confounded by selection bias — pages get refreshed because a team already believes they're worth saving. This project's queue ranks on the observed CTR gap itself, not on content age. - Thin-volume noise is a live risk. All three of the final model's top-20 false positives sat near the 100–250 impression floor. Below that volume, a CTR estimate is unstable enough to produce false alarms even when the model's overall precision is strong.
- Single dataset, single time slice. Trained and evaluated on one 22,006-row, 30-client anonymized slice. A Precision@50 of 0.640 describes this data and this label definition — it is not a guaranteed number for a different client mix, a different period, or FlyRank's actual internal flag logic.
Ranked recommendations: the action playbook
The model score and the baseline rule are combined into six archetypes, each mapped to a confidence tier and a concrete action. Agreement between rule and model earns the highest confidence; a model-only flag earns a "look before you touch it" tier rather than an automatic top ranking.
Human review is required, not optional
Before acting on any flagged page, a reviewer should rule out: brand or navigational intent
(a page can rank well and still get few clicks because searchers want a competitor, not this result),
zero-click SERPs (an AI Overview or featured snippet answering the query directly makes a
metadata rewrite pointless), and thin-volume noise (see Limitations). Items in quiet_risk_flag
specifically require a look, since the model is disagreeing with the transparent rule.
What should never be automated
- Auto-publishing a rewritten title or meta description from a model score alone.
- Treating a high confidence tier as proof a rewrite will recover clicks — this data is observational.
- Acting on any page below the 100-impression visibility floor.
- Bulk-refreshing content purely by age, given the mixed/possibly-confounded freshness evidence.
Monitoring & retrain triggers
Re-check Precision@50 against the 0.640 baseline whenever a new month's data becomes available — a drop below roughly 0.50 pauses the queue pending review. Track the label's positive rate (currently ~28.6%) for drift, and retrain on an updated client-grouped split whenever new clients are onboarded, rather than appending them to a frozen model.