The ad-organic-scraper captures organic competitor intelligence from YouTube — and where the platform is relevant, Instagram — without requiring an operator to trigger it. It runs on a schedule under Curator-Hermes. Every time it runs, it fetches up to 30 videos per channel, computes a VSR (view-to-baseline ratio — how many times more views this video earned than the channel's own typical content), assigns a signal tier in plain code, and routes only the high-signal videos to a language model for transcript analysis. Two tracks run in parallel: COMPETITOR mode maps the organic landscape for the expert's niche; OWNED mode monitors the expert's own channel for audience-validated topics and real comment-sourced pain language. All structured data is written to Supabase (the system of record for the Competitor Intelligence layer — Layer 3 of the seven data layers). One clean HTML review surface is generated from that data after the run (firm review-surface standard) — it is an export, not a source of truth. No Google Doc. Downstream skills, including ad-ideator-classify, query Supabase views only. This skill is wrapped by the /ad-research operator command, which runs both ad-data-scraper and ad-organic-scraper then reconciles curated memory with the fresh scan.
All calls go through OpenRouter and are logged with model identifier, provider, and cost to the agent_run audit trail. Counting, arithmetic, and data fetching are plain code — never a model.
| Task | Model | Why |
|---|---|---|
| Source identification — choosing which YouTube channels to track; judging whether a high-VSR video is genuinely transferable signal. First channel map for a new expert only. | claude-fable-5 (Fable 5) via OpenRouter — first channel map only | The first channel map for a new expert is irreversible: wrong choices corrupt Layer 3 for that expert's namespace permanently. Fable 5 (claude-fable-5) costs 2x Opus 4.8 and carries a mandatory 30-day data-retention obligation, so it is not the default for ongoing runs. Justified here — and only here — as a one-shot ceiling job. All later source-identification runs use Opus 4.8. |
| Source identification — ongoing runs (existing expert, channel list already established) | claude-opus-4-8 (Opus 4.8) via OpenRouter | Same high-consequence decision, but the channel map is already validated. Opus 4.8 is the correct model for this reasoning level without the Fable 5 cost and retention overhead. |
| Video metadata pull — view counts, subscriber counts, publish dates, video duration | Plain code — no language model | Deterministic fetch via the Apify grow_media/youtube-channel-video-scraper actor. A language model adds cost and hallucination risk with zero upside here. |
| Transcript extraction from video audio | Whisper (deterministic speech-to-text) — no language model | Transcription is a deterministic conversion task. Whisper produces verbatim text without inference. This runs before any model sees the content. |
| VSR calculation, channel median, and tier assignment | Plain code — no language model | VSR = video_views ÷ channel_median_views. Tier thresholds are fixed rules. Completeness is never a model's job. The model receives a pre-computed, tier-tagged list — it never decides tiers. |
| Comment handle redaction (OWNED mode — strips @handles and URLs before any model sees audience_voice_raw data) | Plain code — no language model | PII (personally identifiable information) removal must happen before any LLM (large language model) sees the data. A model cannot be trusted to reliably redact its own inputs. Non-optional pre-processing step. |
| Language-bank extraction, hook identification, and signal-category tagging from individual video transcripts | claude-sonnet-4-6 (Sonnet 4.6) via OpenRouter | Mid-complexity structured extraction against a controlled vocabulary. Cost-appropriate for high-volume per-video work. Does not require Opus-level synthesis. |
| Cross-video synthesis — what is working organically, ad creative transfer brief, concept seeds, voice extraction for OWNED mode | claude-opus-4-8 (Opus 4.8) via OpenRouter | The hardest reasoning job in this skill. Requires genuine cross-document pattern recognition and creative intelligence to produce actionable concept seeds and a credible transfer brief. |
What goes in, and what comes out. Curator-Hermes reads inputs from the expert config — no operator input is needed at run time.
| Input | What it is |
|---|---|
| Expert name and niche | The name of the expert (for example, Amelia Fenmore) and their niche (for example, interior-design business coaching). Pulled from the expert config at run time. |
| Mode | COMPETITOR (analyse other creators in the niche), OWNED (analyse the expert's own YouTube channel), or BOTH. Set in the expert config. Curator-Hermes runs both modes on schedule — an operator does not select this at run time. |
| Competitor list (COMPETITOR mode) | YouTube channel handles or URLs for known competitors. Optional — if absent, the agent discovers competitors via web search. Updated after each discovery run and stored back in the expert config. |
| Expert channel URL or handle (OWNED mode) | The exact YouTube URL or @handle for the expert's channel (for example, @theinterioracademy for IDA — Interior Design Academy). Must be in the expert config. The agent halts if it is missing and never guesses or searches for it, because expert channel names frequently differ from personal names. |
| Videos per channel | How many videos to pull per channel, sorted by view count (most-viewed first). Default is 30. 30 is enough to calculate a stable median and identify outliers reliably. |
| Comments per video (OWNED mode only) | How many comments to pull from each of the top five OWNED-mode videos. Default is 100 per video, sorted by relevance. |
| Previous run ID (from Supabase) | The agent_run identifier from the last completed run for this expert and mode. Pulled automatically from Supabase. Used to generate the "what changed since last run" diff. |
| Field | What it means |
|---|---|
| video_title | The title of the YouTube video as it appears on the channel. |
| channel_handle | The @handle of the YouTube channel that published the video. |
| channel_subscribers | Subscriber count of the channel at scrape time. Used as the VSR (view-to-baseline ratio) fallback denominator when fewer than 15 videos are available for the channel. |
| channel_median_views | The median view count across all videos pulled for this channel in this run. This is the primary baseline for VSR. Using the median rather than the mean prevents a single viral outlier from distorting the baseline. |
| video_views | View count on the video at scrape time. |
| vsr | View-to-baseline ratio — video_views divided by channel_median_views (or channel_subscribers as a fallback for channels with fewer than 15 videos). Calculated by plain code, never a model. A VSR of 3.0 means this video earned three times more views than the channel's typical video. |
| signal_tier | Assigned by plain-code arithmetic against fixed thresholds. TIER_1_OUTLIER (VSR ≥ 5x and ≥ 10,000 absolute views), TIER_1_HIGH (VSR ≥ 2x and ≥ 10,000 views), TIER_2_MODERATE (VSR ≥ 1.5x and ≥ 10,000 views), TIER_2_PROMISING (VSR ≥ 2x with 5,000–10,000 views on a small channel), TIER_3_BASELINE (skip). The model never decides tiers. |
| content_theme_tag | Controlled vocabulary tag assigned by Sonnet 4.6 after reading the transcript. One of seven values: ORIGIN_STORY, HOW_TO, MYTH_BUSTING, ASPIRATION, SOCIAL_PROOF, FEAR_BASED, COMPARISON. (TRANSFORMATION_NARRATIVE is an override flag on the signal_tier field, not a theme value.) |
| primary_pain_point | The single strongest pain point surfaced in the transcript, quoted verbatim from the video. |
| transformation_promise | The core before-and-after transformation the video promises, in the creator's own words. |
| ad_ready_hook | A rewrite of the video's opening as a three-second Meta (Facebook and Instagram advertising platform) ad opener. Written by Opus 4.8 during synthesis. |
| language_bank_json | JSON array of 10 or more verbatim phrases from the transcript containing strong pain, desire, identity, or proof language. These phrases are raw material for ad copy — the audience's own words. |
| transcript_status | complete, missing, truncated, or truncated_final. Videos with any status other than complete are not passed to any model. A video is flagged truncated when its character count is below the expected minimum for its duration (roughly 100 characters per minute). Re-fetch is attempted once; if still incomplete, it is stored as truncated_final and excluded from this run's analysis. |
| mode | COMPETITOR or OWNED. Indicates which track produced this row. |
| expert_namespace | The expert namespace this row belongs to (for example, 'ida', 'alba', 'elena'). Every query filters by namespace. A query for IDA reads IN ('ida','shared'). |
| created_by_run_id | The agent_run.run_id for the execution that wrote this row. Enables diffs, audits, and rollbacks. |
| scraped_at | UTC timestamp of when this video's data was pulled from YouTube. |
| Field | What it means |
|---|---|
| comment_text_redacted | Full comment text with the commenter's @handle and any embedded URLs stripped by plain code before this row was written. No model ever sees a handle. |
| source | Always 'youtube_organic' for rows produced by this skill. |
| source_video_id | YouTube video identifier the comment came from. |
| signal_category | One of six controlled-vocabulary values assigned by Sonnet 4.6 after reading the redacted comment text: PAIN, DESIRE, IDENTITY, PROOF, OBJECTION, QUESTION. |
| verbatim_flag | Boolean. True if this comment contains a phrase strong enough to use verbatim in ad copy, as judged by Sonnet 4.6. |
| pii_redacted | Always true. Set by plain code at handle-redaction time (Step 6 of the tool sequence). Never false. |
| redacted_at | UTC timestamp of when handle redaction was applied, before any model saw the text. |
| created_by_run_id | The agent_run.run_id that wrote this row. |
| Field | What it means |
|---|---|
| run_id | Unique identifier for this Curator-Hermes execution. Stamped on every row the run writes as created_by_run_id. |
| agent_name | Always 'ad-organic-scraper'. |
| triggered_by | 'cron' for scheduled runs; 'manual' for operator-triggered runs. |
| expert_namespace | Which expert namespace this run served (for example, 'ida'). |
| mode | COMPETITOR, OWNED, or BOTH. |
| started_at | UTC timestamp when the run opened. |
| completed_at | UTC timestamp when the run closed — on success or on halt. |
| status | 'success', 'partial' (some channels completed, others halted), or 'halted' (infrastructure failure — Supabase unreachable or Apify actor returned non-200 or zero results). A legitimate empty result (channel exists, no new qualifying videos) closes with 'success'. A connectivity failure closes with 'halted'. Empty-as-success is never allowed. |
| videos_scraped | Total videos pulled across all channels in this run. |
| tier1_count | Number of videos assigned Tier 1 signal in this run. |
| tier2_count | Number of videos assigned Tier 2 signal in this run. |
| halt_reason | If status is 'halted' or 'partial': the Apify actor name, input handle, HTTP status code, and timestamp of the failure. |
| diff_summary_json | Structured JSON listing: new high-signal videos not in the prior run, tier upgrades, videos removed (private or deleted), new competitors added. First section of the generated output document. |
| Section | What it contains |
|---|---|
| What changed since last run | First section — always. Plain-English summary of diff_summary_json: new high-signal videos, tier upgrades, removals, new competitors. A human can read only this section to stay current week to week. |
| Video Intelligence Log | Table of all Tier 1 and Tier 2 videos from this run: title, channel, view count, VSR, signal tier, content theme tag, primary pain point, and ad-ready hook. |
| Per-video deep analyses | Full Sonnet 4.6 extraction output per video: hook analysis, pain points, transformation promise, proof elements, objection handling, language bank, content theme. OWNED mode adds voice extraction, audience relationship notes, and ad-ready clippable moments. |
| Cross-video pattern summary (Opus 4.8) | Top five pain points by frequency, top three transformation promises, strongest hook patterns, combined language bank, white space — angles that have organic validation but no paid ad presence yet. |
| Comment Intelligence (OWNED mode only) | Synthesized from audience_voice_raw rows: top pain phrases, student transformation proof quoted from real comments, audience objections, desire language, identity language, and knowledge-gap questions. All sourced from handle-redacted text. |
| Ad Creative Transfer Brief | Actionable handoff for ad-ideator-classify: which organic hooks should be tested as paid ad hooks, which transformation narratives map to the seven-section ad formula, and three to five recommended concept seeds. OWNED-mode voice extraction feeds directly into ad-voice-profile for formal Voice Book construction. |
=== WHAT CHANGED SINCE LAST RUN === run_id: curator-hermes-20260611-ida-comp expert_namespace: ida mode: COMPETITOR previous run: 2026-05-26 | this run: 2026-06-11 NEW HIGH-SIGNAL VIDEOS (not in last run): [TIER_1_OUTLIER] @interiorincomeacademy Title: "I went from $0 to $12k/month interior design with zero customers — here's the method" Views: 187,000 | Channel median: 14,200 | VSR: 13.2x Theme: ORIGIN_STORY Pain: "I had the design eye but I had no idea how to turn it into actual income" Hook: "You already have the eye for design. The part nobody teaches you is how to turn that into a customer." [TIER_1_HIGH] @designyourbusiness_official Title: "Why your interior design portfolio isn't getting you customers (and what to fix)" Views: 41,000 | Channel median: 9,800 | VSR: 4.2x Theme: MYTH_BUSTING Pain: "I kept adding to my portfolio and nothing changed. Nobody was booking." Hook: "Your portfolio isn't the problem. Here's what's actually stopping customers from booking." TIER UPGRADES (crossed threshold since last run): @homedecoacademy — "How I got my first interior design customer with no experience" Was: Tier 2 Moderate (38k views / 1.9x VSR on 2026-05-26) Now: Tier 1 High Signal (52k views / 2.6x VSR) REMOVALS: none. NEW COMPETITORS: none. TRUNCATED TRANSCRIPTS: none. === CROSS-VIDEO PATTERN SUMMARY (Opus 4.8 — 8 Tier 1+2 videos analyzed) === Top pain points by frequency: 1. "I have the skills but I don't know how to get customers" (6 of 8 videos) 2. "I don't know what to charge — I'm scared I'll overprice and lose them" (5 of 8) 3. "I wasted years on a portfolio that didn't lead anywhere" (4 of 8) 4. "I didn't go to design school — I don't feel like a real designer" (3 of 8) 5. "Everyone says get on Instagram but I'm getting zero customers from it" (3 of 8) Strongest hook patterns: — Specific income number + short timeline + zero-experience framing — Myth-bust of conventional advice (portfolio, Instagram, formal credentials) — Identity permission: "you already have X, you're missing Y" WHITE SPACE (organic demand, no paid ad presence detected): — "No design degree" objection appears in 3 of 8 videos but zero scraped paid ads address it directly. — "Portfolio myth" angle (portfolio ≠ customers) is organically proven but absent from competitor paid creative. TOP 3 CONCEPT SEEDS for IDA (ad-ideator-classify handoff): SEED-A: 'No-degree permission' — "You don't need a design degree to charge $8k for a project. Here's what you actually need." SEED-B: 'Portfolio myth-bust' — "I spent 2 years perfecting my portfolio. It got me zero customers. This one thing changed everything." SEED-C: 'First $10k' — origin-story hook anchored to a specific income milestone, short timeline, no prior customers. === RUN CLOSE === status: success videos_scraped: 24 | tier1_count: 5 | tier2_count: 3 HTML review surface generated: IDA-organic-competitor-analysis.html (clean HTML, firm review-surface standard) Ready for: ad-ideator-classify (Supabase competitor_intelligence view)
Every abbreviation spelled out in full with contextual grounding.
| Term | In full / what it means |
|---|---|
| VSR | View-to-Baseline Ratio — The number that compares a single video's view count to the channel's typical view count. Calculated as: video_views divided by channel_median_views (the median of the top 30 videos pulled for that channel). A VSR of 3.0 means the video earned three times more views than the channel's average video. For channels with fewer than 15 videos, channel_subscribers is used as the fallback denominator. Always a plain-code calculation — never computed by a model. The model receives a pre-computed, tier-tagged list. |
| IDA | Interior Design Academy — Amelia Fenmore's signature offer: an online coaching programme that teaches people how to build an interior design business. Cold Meta ads drive traffic to a free training, which converts to booked sales calls. The expert namespace in all IDA Supabase rows is 'ida'. A query for IDA reads IN ('ida','shared'). |
| CPBC | Cost Per Booked Call — How much ad spend was required to get one person to book a sales call. The primary success metric for IDA cold traffic. Not computed by this skill — ad-organic-scraper operates in the organic research layer. Defined here because it appears in the broader system context. |
| CPL | Cost Per Lead — How much ad spend was required to get one person to register for the free training. An intermediate conversion proxy used before CPBC data is available. Not computed by this skill. |
| TOF | Top of Funnel — Cold traffic — people who have never seen Amelia Fenmore's content before. IDA cold Meta ads target TOF audiences and drive them to a free training before asking for a sales call booking. |
| MOF | Middle of Funnel — Warm traffic — people who have seen content or engaged with an ad but have not yet booked a call. Retargeting ads typically target MOF audiences. |
| BOF | Bottom of Funnel — Hot traffic — people who have seen the free training or visited the booking page but have not yet booked. High-intent retargeting. |
| UGC | User-Generated Content — An ad format that mimics organic, creator-style video — informal, handheld, less polished. Routed through the AI-UGC pathway in ad-production-brief. Not produced by this skill, but organic competitor content in UGC style is a valid signal source for COMPETITOR mode. |
| DTC | Direct to Camera — A video ad format where the presenter speaks directly into the camera lens, creating a one-on-one feel. One of the two main video formats in the IDA creative system (the other is Studio DTC with production lighting and a professional backdrop). |
| PII | Personally Identifiable Information — Any data that can identify a specific individual. In this skill's context: YouTube commenter @handles and profile links embedded in comments. PII must be stripped by plain code (Step 6 of the tool sequence) before any comment text is passed to a language model. |
| HITL | Human in the Loop — The operator approval gate. In V1 of the main creative pipeline, no autonomous execution proceeds without operator sign-off at each stage. Ad-organic-scraper itself runs autonomously (it is a background data-layer agent), but its outputs feed into ad-ideator-classify, which is an operator-gated step. |
| ICP | Ideal Customer Profile — The precise description of who the expert's programme is for: demographics, situation, pain, desire, and objections. IDA's ICP is a career-switcher or self-taught decorator who wants to build an interior design business without a formal design degree. |
| Meta | Meta Platforms (Facebook and Instagram advertising) — The paid advertising platform that runs ads on Facebook and Instagram. When this system refers to "Meta ads," it means paid social ads on those two platforms. Ad-organic-scraper feeds the research layer that eventually informs Meta ad creative — it does not run or manage Meta ads itself. |
| Voice Book | Expert Voice Book (Layer 2 of the seven data layers) — The structured document that captures the expert's speaking style, vocabulary, catchphrases, explanation patterns, and emotional register. Built by ad-voice-profile. OWNED-mode voice extraction from ad-organic-scraper feeds directly into ad-voice-profile as raw material for the Voice Book. The old name "Layer 3 calibration" is deprecated — do not use it. |
| Curator-Hermes | Curator-Hermes (always-on background agent) — The primary always-on agent that runs ad-organic-scraper on a cadence-driven schedule. Separate from the paid-ad Hermes instance (which handles ad swipe curation). Curator-Hermes covers organic YouTube signal only. A human can also trigger a manual run, but the primary operating model is automated maintenance of the persistent organic data layer. |
| Scout-Hermes | Scout-Hermes (daily discovery agent) — A separate always-on agent in the background fleet that runs daily competitor discovery. Distinct from Curator-Hermes. Mentioned here for clarity because the two agents are sometimes confused. |
| Supabase | Supabase (database platform) — The PostgreSQL-based database that hosts all seven data layers of the SKL (Stealth Kairos Labs) ad creative system — from raw ad captures through to launch learnings. Ad-organic-scraper writes to and reads from Supabase. Downstream skills query Supabase views only — never Google Drive or Fibery at runtime. |
| Fibery | Fibery (project management and creative entity CRM) — The system of record for all IDA creative entities (CREATIVE, HOOK, FILMING/SHOOT), their production status, and their Meta performance data. Fibery is a sync source for Supabase — not a runtime read for agents. This skill does not read or write Fibery directly. |
| OpenRouter | OpenRouter (multi-provider model routing API) — The API gateway through which all language model calls in this system are made. Every call is logged with model identifier, provider, and cost to the agent_run audit trail. |
| Apify | Apify (cloud web scraping and automation platform) — The platform hosting the YouTube scraping actors used in this skill. Specifically: grow_media/youtube-channel-video-scraper (for video metadata and transcripts) and streamers/youtube-comments-scraper (for comment data in OWNED mode). |
| Whisper | OpenAI Whisper (automatic speech recognition model) — A deterministic speech-to-text model that converts video audio into a verbatim transcript. No inference — it converts sound to text. Used here because transcription accuracy and determinism matter more than cost at this stage. |
| Sonnet 4.6 | claude-sonnet-4-6 (Anthropic language model, accessed via OpenRouter) — The mid-tier Claude model used for per-video structured extraction: classifying transcript phrases against a controlled vocabulary, tagging comment signal categories, and building the language bank. Cost-appropriate for high-volume extraction work. |
| Opus 4.8 | claude-opus-4-8 (Anthropic language model, accessed via OpenRouter) — The highest-capability Claude model in this skill's routing policy. Used for cross-video synthesis, the ad creative transfer brief, and concept seeds. Not used for extraction or tagging. |
| Fable 5 | claude-fable-5 (Anthropic language model, accessed via OpenRouter) — Reserved for the single ceiling-job in this skill: the first channel map for a new expert. Costs 2x Opus 4.8. Carries a mandatory 30-day data-retention obligation — never used on inputs containing customer PII (personally identifiable information) without explicit operator sign-off. Not the default for any ongoing run. |
| Tier 1 (signal tier) | Tier 1 — High Signal or Outlier — A video that earned at least 2x the channel's median view count AND at least 10,000 absolute views. The Outlier sub-tier is 5x or more. All Tier 1 videos are sent through extraction (Sonnet 4.6) and synthesis (Opus 4.8). Never skipped. |
| Tier 2 (signal tier) | Tier 2 — Moderate Signal or Promising — A video that earned 1.5x–2x the channel median with ≥ 10,000 views (Moderate), or 2x+ on a small channel with 5,000–10,000 absolute views (Promising). Sent through extraction if topically relevant. OWNED-mode threshold scaling applies for channels under 5,000 subscribers. |
| Tier 3 (signal tier) | Tier 3 — Baseline — A video that performed at or below the channel median, or has fewer than 5,000 absolute views. Skipped entirely. The only exception: a Transformation Narrative Override flag, which the agent sets on videos whose transcript structure mirrors the seven-section ad formula even if their VSR is below threshold. |
| competitor_intelligence | competitor_intelligence (Supabase Layer 3) — The Supabase table where video-level intelligence rows are written after each Curator-Hermes run. Contains VSR, tier, language bank, ad-ready hook, and all extraction outputs per analyzed video. source_type = 'youtube_video' for rows produced by this skill. |
| audience_voice_raw | audience_voice_raw (Supabase table) — The Supabase table where OWNED-mode YouTube comment data is stored after handle redaction. Every row contains one comment's text (handle stripped), the source video identifier, a signal category tag, and the run identifier that wrote it. |
| agent_run | agent_run (Supabase audit table) — The cross-cutting table that records every agent run. Ad-organic-scraper opens a row at Step 0 and closes it at Step 11. The row's run_id becomes created_by_run_id on every data row written in that run, creating a full trace from any data point back to the run that produced it. |
| expert_namespace | expert_namespace (data partitioning key) — The string that partitions all Supabase data by expert ('ida', 'alba', 'elena', 'shared'). Every Supabase query in this system filters by namespace. A query for IDA reads WHERE expert_namespace IN ('ida', 'shared'). |
| F1–F12 | Cold Traffic Core Hook Frameworks 1 through 12 — The 12 core cold-traffic hook frameworks in ad-hook-generation: Contrarian Claim, Unexpected Outcome, Hidden Truth, Specific Transformation, Identity Callout, Mistake Reveal, Paradox Hook, Question Interrupt, Time Collapse, What If Reframe, Social Proof Lead, Native Story Opening. Concept seeds from this skill's synthesis step reference these IDs when the organic pattern maps to a known framework. |
| HT1–HT12 | High-Ticket Cold Traffic Hook Frameworks 1 through 12 — The 12 high-ticket cold-traffic hook frameworks in ad-hook-generation (including Competence Test, Gatekeeper/Velvet Rope, Mechanism Reveal, Confession/Vulnerability Lead, and eight others). Frequently the correct framework mapping for IDA (a high-ticket coaching offer). |
| W1–W10 | Warm Traffic Retargeting Frameworks 1 through 10 — The 10 warm-traffic hook frameworks in ad-hook-generation. Only relevant for IDA retargeting ads targeting MOF (middle of funnel) and BOF (bottom of funnel) audiences. Cold TOF (top of funnel) ads use F1–F12 and HT1–HT12. |