Experimentation & A/B Testing

--- name: experimentation-and-ab-testing description: >- A/B testing and experimentation for social media content — evidence over opinion, at honest organic-scale rigor. Use when someone wants to "A/B test" or "split test" content, "test which version/hook/time/caption/thumbnail works," "set up an experiment," "what should I test," or to settle a content debate with evidence instead of opinion. Designs disciplined organic tests: one variable, controlled context, a decision rule set BEFORE publishing, enough duration and repetitions to separate signal from noise. Uses the TEST framework. Reads brand-profile + goals-and-kpis first. It DESIGNS the test and drafts variants; WoopSocial schedules them as controlled sequential posts (exception: YouTube's native Test & Compare); the result is read from native analytics via analytics-and-reporting. Organic can't reach true statistical significance; nothing is fabricated or p-hacked. Distinct from analytics-and-reporting (measures) and goals-and-kpis (sets targets). version: 1.0.0 --- # experimentation-and-ab-testing The **causation engine** — manipulate one variable under controlled conditions to learn what actually moves a KPI. This skill **designs** the test and drafts variants; **scheduling-and-queue → WoopSocial** publishes them; **analytics-and-reporting** reads the result. ## The POV: evidence, not vibes Most "testing" on social is vibes — post two things, eyeball the likes, declare a winner, learn nothing. Real experimentation turns guesses into evidence: change **one variable**, control everything else, set the **decision rule before you publish**, and run it **long and often enough** to separate signal from noise. Organic can't give clean statistical significance (small samples, an algorithm in the middle), so you compensate with tighter controls, a **~20%+ effect threshold**, **guardrail metrics**, and **3–5 repetitions** — and treat a single viral post as **noise, not a strategy.** ## Read these first 1. **brand-profile** — voice/format constraints for the variants. 2. **goals-and-kpis** — the **KPI/primary metric** the test must move. ## The framework: TEST (Depth: `references/the-test-framework.md`.) - **T — Target one variable:** a clear hypothesis; change ONE element (hook/first-frame/caption/CTA/time/ format), everything else identical; pick the highest-leverage one. - **E — Establish the decision rule first:** set the **primary metric + win threshold + guardrail** before publishing ("B wins if reach +15% and saves/reach not worse"); no post-hoc rationalizing. - **S — Set controls + sample:** same platform/format/topic/length/window; run ≥7 days (small accounts 2–4 weeks); judge on a **~20%+ consistent effect** (a tie = "test elsewhere"). - **T — Tally, repeat, scale:** **3–5 paired repetitions** before a "best practice"; log every test; scale winners into the playbook (`content-recycling`), retire the rest. ## What to test (highest leverage, in your control) Hook/first-frame (short video) → posting time (easy) → format → caption/CTA → thumbnail → hashtags — always tied to the KPI; test what's **in your control**, not algorithm-dependent factors. Run a **30-day sprint** with one test always running. Priority list, design template, sprint plan, testing log + worked examples: `references/what-to-test-and-recipes.md`. Full method + rules: `references/experimentation-2026-reality.md`. ## Honest scope (never violate) - **Organic isn't lab-grade** — results are **directional**; compensate with controls + effect-size + repetition, not p-value theater. - **WoopSocial has no A/B/audience-split surface** → organic testing = **controlled sequential posts**; the agent designs + drafts variants + schedules; the **primary metric is read from native analytics** (`analytics-and-reporting`). **One true native split exists: YouTube's Test & Compare** (YouTube Studio, long-form, not Shorts) — up to **3 titles, thumbnails, or title+thumbnail combos**; use it for YouTube title/thumbnail tests instead of sequential posts. (verify-quarterly) - **No p-hacking / HARKing / cherry-picking** — decision rule pre-set; a multi-variable change can't be pinned on one element; one post/one day is noise. **Never fabricate a result; a tie is valid.** (Scope, the loop role + connections: `references/scope-and-connections.md`.) ## Distinct from its siblings (route correctly) **experimentation (this)** = manipulate one variable to establish **causation** · **analytics-and-reporting** = observe/measure what happened · **goals-and-kpis** = set the target/primary metric · **content-recycling** = scale proven winners · **viral-reverse-engineering** = explain a *past* post (hindsight) vs testing forward. ## Where this connects Reads first: **brand-profile**, **goals-and-kpis**. Variants drafted via: **hook-writer**, **caption-writer**, **reels-script**/**tiktok-script**, **carousel-writer**, **image-prompt**/**ideogram**/**nano-banana**, **thumbnail-design**. Readout: **analytics-and-reporting** (native analytics). Scale/plan: **content-recycling**, **social-strategy**, **content-calendar**/**batch-content-plan**, every **\*-growth** skill. Publish variants: **scheduling-and-queue → WoopSocial** (controlled sequential posts). ## Definition of done A clear hypothesis testing ONE variable tied to a KPI; identical controlled context; a primary metric + win threshold + guardrail set **before** publishing; duration ≥7 days (2–4 weeks small accounts) and 3–5 paired repetitions; results read from native analytics and judged on a ~20%+ consistent effect (ties acknowledged); winners logged and scaled to content-recycling/strategy; organic limits stated, nothing fabricated or p-hacked, correctly distinguished from analytics-and-reporting and goals-and-kpis.

存档时间线

版本索引永久保留；内容副本只保留最近 2 版，更早版本仅留索引与哈希（存档时间线的证据链不会因此断裂）。

下载存档内容副本