Skip to content

testing-strategy.md

Latest commit

 

History

History
249 lines (205 loc) · 7.94 KB

testing-strategy.md

File metadata and controls

249 lines (205 loc) · 7.94 KB
Raw

Testing Strategy

🧪 Comprehensive Testing Approach

The ChillFlow pipeline implements a robust testing strategy that ensures reliability, performance, and maintainability across all components.

📊 Testing Pyramid

graph TB
    %% Test Pyramid
    subgraph "Test Pyramid"
        UNIT["🧪 Unit Tests<br/>Fast, Isolated<br/>~100ms each"]
        INTEGRATION["🔗 Integration Tests<br/>Real Services<br/>~5-10s each"]
        E2E["🌐 End-to-End Tests<br/>Full Pipeline<br/>~30-60s each"]
    end

    %% Test Types
    subgraph "Test Categories"
        STREAM_TESTS["⚡ Stream Tests<br/>Kafka Integration (stream svc)"]
        BATCH_TESTS["📊 Batch Tests<br/>PostgreSQL Integration (batch svc)"]
        CONTRACT_TESTS["📋 Contract Tests<br/>Schema Validation"]
        SMOKE_TESTS["💨 Smoke Tests<br/>CLI Validation"]
        PERFORMANCE["⚡ Performance Tests<br/>Benchmarks"]
        INFRA_TESTS["🧱 Infrastructure Tests<br/>Platform/compose checks"]
    end

    %% Test Infrastructure
    subgraph "Test Infrastructure"
        TESTCONTAINERS["🐳 Testcontainers<br/>Real Services"]
        POSTGRES_TEST[("🗄️ PostgreSQL<br/>Test Database")]
        KAFKA_TEST[("⚡ Kafka<br/>Test Broker")]
        REDIS_TEST[("⚡ Redis<br/>Test Cache")]
    end

    %% Quality Gates
    subgraph "Quality Gates"
        PRECOMMIT["✅ pre-commit<br/>Code Quality"]
        LINTING["🔍 Linting<br/>flake8, isort"]
        FORMATTING["🎨 Formatting<br/>Black"]
        TYPE_CHECK["📝 Type Checking<br/>mypy"]
    end

    %% CI/CD Integration
    subgraph "CI/CD Integration"
        GITHUB_ACTIONS["🔄 GitHub Actions<br/>Automated Testing"]
        PARALLEL["⚡ Parallel Execution<br/>7 Test Jobs"]
        MATRIX["📊 Matrix Strategy<br/>Efficient Resource Usage"]
    end

    %% Test Flow
    UNIT --> INTEGRATION
    INTEGRATION --> E2E

    STREAM_TESTS --> TESTCONTAINERS
    BATCH_TESTS --> TESTCONTAINERS
    TESTCONTAINERS --> POSTGRES_TEST
    TESTCONTAINERS --> KAFKA_TEST
    TESTCONTAINERS --> REDIS_TEST

    PRECOMMIT --> LINTING
    PRECOMMIT --> FORMATTING
    PRECOMMIT --> TYPE_CHECK

    GITHUB_ACTIONS --> PARALLEL
    PARALLEL --> MATRIX
    MATRIX --> STREAM_TESTS
    MATRIX --> BATCH_TESTS
    MATRIX --> CONTRACT_TESTS
    MATRIX --> SMOKE_TESTS

    %% Styling
    classDef pyramid fill:#e8f5e8,stroke:#2e7d32,stroke-width:3px,color:#000
    classDef categories fill:#e3f2fd,stroke:#1565c0,stroke-width:2px,color:#000
    classDef infrastructure fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#000
    classDef quality fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px,color:#000
    classDef cicd fill:#e0f2f1,stroke:#00695c,stroke-width:2px,color:#000

    class UNIT,INTEGRATION,E2E pyramid
    class STREAM_TESTS,BATCH_TESTS,CONTRACT_TESTS,SMOKE_TESTS,PERFORMANCE,INFRA_TESTS categories
    class TESTCONTAINERS,POSTGRES_TEST,KAFKA_TEST,REDIS_TEST infrastructure
    class PRECOMMIT,LINTING,FORMATTING,TYPE_CHECK quality
    class GITHUB_ACTIONS,PARALLEL,MATRIX cicd
Loading

🏗️ Test Categories

1. Unit Tests

  • Purpose: Test individual components in isolation
  • Scope: Functions, classes, methods
  • Speed: ~100ms each
  • Coverage: >90% code coverage
  • Tools: pytest, unittest.mock

2. Integration Tests

  • Purpose: Test component interactions with real services
  • Scope: Database, Kafka, Redis integration
  • Speed: ~5-10s each
  • Infrastructure: Testcontainers
  • Tools: pytest, testcontainers

3. Contract Tests

  • Purpose: Validate data schemas and transformations
  • Scope: Pydantic models, API contracts
  • Speed: ~1-2s each
  • Tools: pytest, pydantic

4. Smoke Tests

  • Purpose: Verify basic functionality without dependencies
  • Scope: CLI commands, component initialization
  • Speed: ~500ms each
  • Tools: pytest, mocking

5. Performance Tests

  • Purpose: Benchmark processing speed and memory usage
  • Scope: Batch processing, serialization, memory
  • Speed: ~10-30s each
  • Tools: pytest, timeit, memory profiling

🔧 Test Infrastructure

Testcontainers Integration

  • PostgreSQL: Real database for integration tests
  • Kafka: Real message broker for streaming tests
  • Redis: Real cache for session tests
  • Benefits: Real service behavior, no mocking complexity

Test Data Management

  • Fixtures: Reusable test data generators
  • Factories: Dynamic test data creation
  • Cleanup: Automatic test data cleanup
  • Isolation: Each test runs in clean environment

🚀 CI/CD Integration

Parallel Test Execution

  • Matrix Strategy: parallelized suites
  • Resource Efficiency: Optimal resource utilization
  • Fast Feedback: Quick test results
  • Failure Isolation: Independent test failures

Quality Gates

  • pre-commit: Local code quality checks
  • Linting: flake8, isort, black
  • Type Checking: mypy for type safety
  • Formatting: Consistent code style

📊 Test Metrics

Coverage Targets

  • Unit Tests: >90% code coverage
  • Integration Tests: >80% component coverage
  • Contract Tests: 100% schema coverage
  • Smoke Tests: 100% CLI coverage

Performance Targets

  • Unit Tests: <100ms each
  • Integration Tests: <10s each
  • Smoke Tests: <1s each
  • Total Suite: <5 minutes

🔄 Test Execution & Layout

Repository layout (where tests live)

  • Root tests:
    • tests/unit/ — core unit tests
    • tests/contracts/ — schema/contract tests
    • tests/smoke/ — CLI smoke tests
    • tests/performance/ — basic benchmarks
    • tests/infrastructure/ — platform/compose checks
    • tests/e2e/ — end-to-end orchestrations (if present)
  • Service tests:
    • backend/chillflow-batch/tests/ — batch unit/integration
    • backend/chillflow-core/tests/ — core library tests
    • backend/chillflow-stream/tests/unit/ — streaming unit tests
    • backend/chillflow-stream/tests/integration/ — streaming integration

Local development commands

# All tests
pytest -q

# Root-level suites
pytest tests/unit -q
pytest tests/contracts -q
pytest tests/smoke -q
pytest tests/performance -q
pytest tests/infrastructure -q

# Service-specific
pytest backend/chillflow-batch/tests -q
pytest backend/chillflow-core/tests -q
pytest backend/chillflow-stream/tests/unit -q
pytest backend/chillflow-stream/tests/integration -q

CI/CD Pipeline (example split)

# Split by area (examples)
pytest tests/unit/ tests/contracts/ -q
pytest tests/smoke/ tests/infrastructure/ -q
pytest backend/chillflow-core/tests -q
pytest backend/chillflow-batch/tests -q
pytest backend/chillflow-stream/tests/unit -q
pytest backend/chillflow-stream/tests/integration -q

🛠️ Test Tools

Testing Framework

  • pytest: Primary testing framework
  • testcontainers: Real service integration
  • unittest.mock: Component mocking
  • pytest-cov: Coverage reporting

Quality Tools

  • pre-commit: Code quality hooks
  • black: Code formatting
  • flake8: Linting
  • isort: Import sorting
  • mypy: Type checking

📈 Continuous Improvement

Test Monitoring

  • Coverage Reports: Track test coverage trends
  • Performance Metrics: Monitor test execution time
  • Failure Analysis: Identify flaky tests
  • Quality Metrics: Track code quality trends

Test Maintenance

  • Regular Updates: Keep test dependencies current
  • Refactoring: Improve test maintainability
  • Documentation: Keep test documentation current
  • Best Practices: Follow testing best practices

🎯 Testing Best Practices

Test Design

  • Arrange-Act-Assert: Clear test structure
  • Single Responsibility: One test, one purpose
  • Descriptive Names: Clear test naming
  • Independent Tests: No test dependencies

Test Maintenance

  • Regular Updates: Keep tests current
  • Refactoring: Improve test quality
  • Documentation: Clear test documentation
  • Best Practices: Follow testing standards