Workflow

Phase 1: Group experiments by pattern

Cluster by change.type and content-of-change similarity. Example patterns:

• title_rewrite with "added number"
• title_rewrite with "added year"
• title_rewrite with "added bracket hook"
• desc_rewrite with "front-loaded value prop"
• schema_add of type FAQPage
• schema_add of type HowTo
• new_page of template alternatives
• internal_links_added between sibling pages
• content_refresh with new data points added
• markdown_version added to page

Phase 2: For each pattern, tabulate

Pattern: title_rewrite + add_number
  Experiments: 5
  Wins: 4 (avg lift: +34%, range +19% to +62%)
  Losses: 0
  Inconclusive: 1

Pattern: content_refresh + aggressive_rewrite (>50% content changed)
  Experiments: 3
  Wins: 0
  Losses: 2 (avg regression: -18%)
  Inconclusive: 1

Phase 3: Apply promotion rules

Promote to playbook when:

• ≥ 3 wins, AND
• 0 losses (or wins outnumber losses ≥ 3:1), AND
• Average lift is material (≥ 10% for CTR/traffic, ≥ 1 position for ranking, ≥ 5pp for citation rate)

• ≥ 2 losses in the same pattern, OR
• A single loss with Δ < -30% (severe)

• < 3 wins on the pattern
• Wins and losses roughly balanced

Phase 4: Write/update playbooks

For each pattern that meets promotion criteria, write or update {workspace}/reports/seo/memory/playbooks/{pattern-name}.md:

# Playbook: {pattern name}

When to apply
{Specific conditions under which this pattern wins. Be narrow —
a playbook that says "always do X" is useless.}

What to do
{Concrete steps. Include anti-patterns — what NOT to do within
this pattern.}

Why
{The mechanism. Why does this work? Cite the AEO/SEO principle
if relevant.}

When NOT to apply
{Edge cases where this pattern has failed or is expected to.}

Evidence

{exp-id}: {page} → {lift}
{exp-id}: {page} → {lift}
{exp-id}: {page} → {lift}

{link all experiment IDs for auditability}

Last updated: {YYYY-MM-DD}
Status: active | deprecated | under-review

If a playbook already exists:

• Append new evidence to the Evidence list
• Update average lift in "Why" if it's shifted meaningfully
• If average lift dropped below threshold → move to under-review status

Phase 5: Write lessons

For each pattern that failed ≥ 2 times OR had one severe failure, write lessons/lesson-{YYYY-MM-DD}-{short_id}.md:

# Lesson: {pattern name that fails}

What we tried
{The pattern in plain language.}

What happened
{The losses. Named experiments and numeric deltas.}

Why it failed
{Best-guess root cause. If unclear, say so — don't fabricate
a reason.}

Rule going forward
{Specific guardrail. Example: "Don't do aggressive rewrites
(>50% content changed) on pages with established rankings —
loses topical authority signals."}

Related

{exp-ids}
Playbook this updates: {link if applicable}

Phase 6: Update skill-level guardrails

If a lesson affects how another skill should behave, update that skill's SKILL.md to reference the lesson. Example: if we learn that "bundle changes into one experiment" consistently produces inconclusive results, update seo-experiment-log to warn against bundling.

This is how lessons become enforced defaults — the skill reads its own lessons before suggesting a change.

Phase 7: Write a learn summary

# Learn Summary — {month}

Playbooks promoted: {N}

add-numbers-to-title (new, 5 wins)
faq-schema-on-compare (updated, +2 wins)


Lessons written: {M}

aggressive-content-rewrites (2 losses)


Still-experimenting (hold): {K}

internal-linking-sprints: 2 wins, need 1 more to promote
markdown-version-added: 3 open experiments still in measurement window


Retired playbooks: {P}

bracket-hooks-in-titles — lift dropped below threshold in last 3 tests, moving to under-review

Quality bar

• [ ] Every measured experiment in the window is assigned to a pattern cluster
• [ ] No pattern is "promoted" with < 3 wins OR in the face of ≥ 2 losses
• [ ] Every new/updated playbook cites its experiment IDs (auditability)
• [ ] Every lesson has a concrete "rule going forward" — not vague reflection
• [ ] Learn summary exists at learn-log/learn-{date}.md

What I refuse

• To promote a pattern with only 1–2 wins. That's not a pattern; that's an anecdote.
• To keep a playbook active when recent evidence contradicts it. Playbooks are living documents; they get demoted if they stop working.
• To write lessons like "we learned to be more careful". Every lesson needs a testable rule.
• To cluster too aggressively. If two patterns behave differently, they're different patterns, even if they look similar on the surface.

Integration

Reads: