Webhook Processor

Overview

This skill helps you build production-grade webhook ingestion endpoints that accept incoming HTTP callbacks, verify their authenticity, and process them reliably with exponential backoff retries and dead letter queues. It covers signature validation, idempotency keys, and graceful failure handling.

Instructions

1. Scaffold the webhook endpoint

Create an HTTP endpoint that accepts POST requests. Immediately return a 200 status before processing — webhook senders expect fast acknowledgment.

// webhook-receiver.ts
import express from "express";
import crypto from "crypto";
import { Queue } from "bullmq";

const app = express();
app.use(express.raw({ type: "application/json" }));

const webhookQueue = new Queue("webhooks", {
  connection: { host: "localhost", port: 6379 },
});

app.post("/webhooks/:source", async (req, res) => {
  const signature = req.headers["x-signature-256"] as string;
  const idempotencyKey =
    req.headers["x-idempotency-key"] ||
    crypto.createHash("sha256").update(req.body).digest("hex");

  await webhookQueue.add(
    "process",
    {
      source: req.params.source,
      payload: req.body.toString(),
      signature,
      idempotencyKey,
      receivedAt: new Date().toISOString(),
    },
    {
      jobId: String(idempotencyKey),
      attempts: 5,
      backoff: { type: "exponential", delay: 3000 },
    }
  );

  res.status(200).json({ received: true });
});

2. Verify webhook signatures

Always validate the signature before processing. Each provider uses different schemes:

function verifySignature(
  payload: string,
  signature: string,
  secret: string,
  scheme: "hmac-sha256" | "hmac-sha1"
): boolean {
  const algo = scheme === "hmac-sha256" ? "sha256" : "sha1";
  const expected = crypto
    .createHmac(algo, secret)
    .update(payload, "utf8")
    .digest("hex");
  const prefix = scheme === "hmac-sha256" ? "sha256=" : "sha1=";
  return crypto.timingSafeEqual(
    Buffer.from(prefix + expected),
    Buffer.from(signature)
  );
}

3. Implement the worker with retry logic

Process jobs from the queue. Failed jobs retry with exponential backoff. After all retries exhaust, move to a dead letter queue.

import { Worker } from "bullmq";

const worker = new Worker(
  "webhooks",
  async (job) => {
    const { source, payload, signature } = job.data;
    const secret = getSecretForSource(source);
    if (!verifySignature(payload, signature, secret, "hmac-sha256")) {
      throw new Error("Invalid webhook signature — will not retry");
    }
    const event = JSON.parse(payload);
    await routeEvent(source, event);
  },
  {
    connection: { host: "localhost", port: 6379 },
    limiter: { max: 50, duration: 1000 },
  }
);

worker.on("failed", (job, err) => {
  if (job && job.attemptsMade >= 5) {
    console.error(`Dead letter: job ${job.id} — ${err.message}`);
    // Move to dead letter queue for manual inspection
  }
});

4. Add idempotency tracking

Prevent double-processing with a deduplication store:

import Redis from "ioredis";
const redis = new Redis();

async function isProcessed(key: string): Promise<boolean> {
  const result = await redis.set(key, "1", "EX", 86400, "NX");
  return result === null; // null means key already existed
}

Examples

Example 1: Payment provider webhook

Prompt: "Set up a webhook endpoint to receive payment events. It should verify HMAC-SHA256 signatures, retry failed processing up to 5 times with exponential backoff, and log dead letter events."

Agent output:

Creates src/webhooks/payment-handler.ts with signature verification using the provider's signing secret
Creates src/workers/payment-worker.ts with BullMQ retry config (5 attempts, 3s/9s/27s/81s/243s backoff)
Creates src/utils/dead-letter.ts that stores failed events in a dead_letters database table
Adds integration test that simulates an invalid signature and verifies rejection

Example 2: Version control platform webhook

Prompt: "Build a webhook handler for repository push events that triggers CI builds. Include idempotency so duplicate deliveries don't start duplicate builds."