A/B Test Generator

Purpose

Generate data-driven A/B test configurations from Humblytics analytics and heatmap data. This skill creates complete test definitions including hypotheses, variant specifications with CSS/element selectors, success metrics, sample size requirements, and can launch tests directly through the Humblytics API.

When to Use

• Creating A/B tests from conversion data or heatmap insights
• Generating test hypotheses for a specific page or funnel step
• Launching no-code split tests via the Humblytics testing API
• Calculating required sample size and test duration
• Designing multivariate test matrices
• Reviewing and iterating on existing test results

Credentials

This skill reads a Humblytics API key from the environment. Never paste API keys directly into chat — they persist in transcripts and logs.

Setup (one time):

1. cp .env.example .env at the repo root and fill in HUMBLYTICS_API_KEY
2. source .env in your shell before running the agent (or use direnv, or add the exports to your shell profile)
3. Get the key from Humblytics Dashboard > Settings > API
4. The skill will ask for your Property ID (also in Dashboard > Settings > API)

• Base URL: https://app.humblytics.com/api/external/v1
• Docs: https://docs.humblytics.com/api

If HUMBLYTICS_API_KEY is not in the environment, stop and point the user at .env.example — do not accept the key in chat.

Before You Start

1. Confirm property and page — Which Humblytics property and which page URL to test
2. Pull current data — Retrieve page analytics, heatmap data, and current conversion rate
3. Check existing tests — Look for any running tests to avoid conflicts
4. Understand the goal — What is the primary conversion action on this page?
5. Verify traffic volume — Ensure enough traffic for statistical significance within a reasonable timeframe
6. Check for context — Look for product briefs, AGENTS.md, or existing CRO documents that inform test direction

Core Workflow

Step 1: Gather Page Intelligence

Pull data from Humblytics:

• Page analytics: Traffic volume, bounce rate, time on page, scroll depth
• Heatmap data: Click maps, scroll maps, attention maps
• Event data: CTA clicks, form interactions, video plays
• Device split: Mobile vs desktop behavior differences
• Source split: How different traffic sources behave on this page

API endpoints:

• GET /properties/{propertyId}/analytics/pages?url={pageUrl} — Page metrics
• GET /properties/{propertyId}/heatmaps?url={pageUrl} — Heatmap data
• GET /properties/{propertyId}/analytics/events?url={pageUrl} — Page events
• GET /properties/{propertyId}/tests — Existing tests

Step 2: Identify Test Opportunities

Analyze the data for signals:

High-value signals from heatmaps:

• Users clicking non-clickable elements (rage clicks) — make them clickable or remove confusion
• Low scroll depth — critical content is below the fold, move it up
• CTA getting few clicks despite visibility — copy, color, or placement issue
• Form field abandonment — simplify or reorder fields
• Dead zones — large page areas with no interaction

High-value signals from analytics:

• High bounce rate from specific sources — message mismatch
• Mobile conversion significantly lower than desktop — responsive layout issue
• High time-on-page but low conversion — users are interested but not persuaded
• Low time-on-page and low conversion — page fails to engage

Step 3: Formulate Hypotheses

For each opportunity, create a structured hypothesis:

Test Name: [descriptive-slug]
Page: [URL]
Hypothesis: IF we [specific change], THEN [primary metric] will [direction]
            BECAUSE [evidence from data]

Control: [current state description]
Variant: [proposed change description]

Primary Metric: [conversion event or goal]
Secondary Metrics: [engagement metrics to monitor]

Element Selector: [CSS selector for the element to modify]
Change Type: [text | style | visibility | layout | redirect]

Step 4: Calculate Sample Size and Duration

For each test, calculate statistical requirements:

Inputs needed:

• Baseline conversion rate (from current analytics)
• Minimum detectable effect (MDE) — typically 10-20% relative lift
• Statistical significance level — default 95%
• Statistical power — default 80%

Sample size formula (per variant):

n = (Z_alpha/2 + Z_beta)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2

Duration estimate:

Days = (n * number_of_variants) / daily_traffic_to_page

Present this clearly:

• Required sample per variant
• Total sample needed
• Estimated days to reach significance at current traffic
• Recommendation: proceed, increase traffic first, or test a larger change

Step 5: Define Test Configuration

Create the complete test spec for Humblytics:

{
  "name": "descriptive-test-name",
  "pageUrl": "/pricing",
  "hypothesis": "Changing CTA from 'Start Free Trial' to 'See Your Analytics' will increase clicks because heatmap shows users hesitate at commitment language",
  "variants": [
    {
      "name": "control",
      "weight": 50,
      "changes": []
    },
    {
      "name": "variant-a",
      "weight": 50,
      "changes": [
        {
          "selector": "#hero-cta",
          "attribute": "textContent",
          "value": "See Your Analytics"
        }
      ]
    }
  ],
  "primaryGoal": {
    "type": "event",
    "event": "signup_started"
  },
  "secondaryGoals": [
    { "type": "event", "event": "cta_clicked" },
    { "type": "metric", "metric": "bounce_rate" }
  ],
  "trafficAllocation": 100,
  "minimumSampleSize": 2400,
  "significanceLevel": 0.95
}

Step 6: Launch or Document

To launch via API:

• POST /properties/{propertyId}/tests — Create and start the test
• GET /properties/{propertyId}/tests/{testId} — Monitor test status
• GET /properties/{propertyId}/tests/{testId}/results — Pull results

To document for manual launch:

• Output the full test specification
• Include screenshot annotations if heatmap data informed the test
• Provide the hypothesis document for the team

Test Type Selection Guide

Scenario	Test Type	Notes
One element change	A/B test	Fastest to significance
Two element changes	A/B/C test	Test both independently
Multiple interacting elements	Multivariate	Needs 4x+ traffic
Completely different page	Split URL test	Redirect-based
Copy variations only	A/B test	Quick wins
Layout restructure	Split URL test	Build separate variant page

Common Test Categories

Headlines and Copy

• Value prop rewording
• Specificity (add numbers, timeframes, outcomes)
• Emotional vs rational framing
• Length (short punchy vs detailed)

CTAs

• Button text (action verbs, benefit language, urgency)
• Button color and size
• Button placement (above fold, after social proof, sticky)
• Single CTA vs multiple CTAs

Social Proof

• Testimonials vs logos vs metrics
• Placement (near CTA vs header vs throughout)
• Specificity (named customers vs anonymous)

Layout and Structure

• Long page vs short page
• Information hierarchy reordering
• Form length and field order
• Navigation presence vs removal

Pricing

• Price anchoring (show higher price first)
• Plan naming
• Feature comparison layout
• Free trial vs freemium vs demo

Avoiding Common Mistakes

1. Testing too small a change — If your MDE requires 50,000 visitors per variant, the change is too subtle. Test bolder.
2. Running too many tests on one page — Interaction effects corrupt results. One test per page at a time.
3. Stopping early on positive results — Peeking inflates false positives. Commit to the sample size.
4. Ignoring secondary metrics — A variant that increases signups but increases churn is not a win.
5. Not segmenting results — A test can be flat overall but show strong wins on mobile. Always segment.

Output Format

For each generated test, present:

1. Hypothesis — One sentence: IF/THEN/BECAUSE
2. Evidence — What data supports this test
3. Variants — Control and variant descriptions
4. Selectors — CSS selectors and exact changes
5. Metrics — Primary and secondary goals
6. Duration — Estimated days to significance
7. Expected Impact — Projected conversion lift range

Related Skills

• cro-optimizer — Identify which pages and funnel steps need testing
• page-cro — Deep page-level audit to inform test hypotheses
• copywriting — Generate high-quality copy variants for tests
• funnel-reporter — Track how test results affect downstream funnel metrics