CIA Data Integration Skill

Purpose

This skill provides expertise in consuming JSON exports from the CIA (Citizen Intelligence Agency) platform, implementing robust caching strategies, and ensuring data quality through schema validation.

Core Principles

CIA is Source of Truth - Never modify CIA's pre-computed data
Validate Before Cache - Always validate against CIA-provided JSON schemas
Version Tracking - Track all CIA data updates with timestamps
Graceful Degradation - Fall back to cached data if CIA unavailable
Data Freshness - Monitor and alert on stale data (> 24 hours)
Audit Logging - Log all data operations for traceability

Enforces

CIA Export Consumption

Fetch 19 visualization products from CIA platform
Handle rate limiting and connection failures
Implement retry logic with exponential backoff
Circuit breaker pattern for API failures

JSON Schema Validation

// Validate CIA export against schema
import Ajv from 'ajv';

const ajv = new Ajv({ allErrors: true });
const schema = await fetchCIASchema(productName);
const validate = ajv.compile(schema);

if (!validate(data)) {
  throw new ValidationError(validate.errors);
}

Caching Strategy

data/cia-exports/
  current/          # Latest CIA exports (19 files)
  archive/          # Historical versions (date-stamped)
  metadata/         # Fetch timestamps, validation status

Data Freshness Monitoring

Track last fetch timestamp
Alert if data > 24 hours old
Display staleness warnings to users
Automatic fallback to cached data

When to Use

Implementing CIA export fetch workflows
Validating CIA JSON data
Designing caching strategies
Building data consumption pipelines
Monitoring data freshness
Handling API failures gracefully

Examples

CIA Export Client

class CIAExportClient {
  async fetchExport(productName) {
    const url = `https://www.hack23.com/cia/api/export/${productName}.json`;
    const data = await fetch(url, { timeout: 30000 });
    
    await this.validateSchema(productName, data);
    await this.cacheWithVersion(productName, data);
    
    return data;
  }
}

Automated Workflow

# .github/workflows/fetch-cia-exports.yml
name: Fetch CIA Exports
on:
  schedule:
    - cron: '0 2 * * *'  # 02:00 CET daily

jobs:
  fetch:
    steps:
      - name: Fetch CIA exports
        run: node scripts/fetch-cia-exports.js
      
      - name: Validate schemas
        run: node scripts/validate-cia-data.js
      
      - name: Commit if changed
        run: |
          if git diff --quiet data/cia-exports/; then
            echo "No changes"
          else
            git add data/cia-exports/
            git commit -m "Update CIA exports"
            git push
          fi

Remember

CIA maintains the digital twin - riksdagsmonitor consumes it
Validate all data before caching
Monitor freshness and alert on staleness
Provide graceful fallback to cached data
Never modify CIA's pre-computed intelligence
Log all operations for audit trail
Respect rate limits and implement backoff

References

Version: 1.0
Last Updated: 2026-02-06
Category: Data Integration
Maintained by: Hack23 AB

cia-data-integration