Deployment Architecture

Overview

The Ibutsu backend uses a modern ASGI-based architecture with Gunicorn as the process manager and Uvicorn as the ASGI worker implementation. This document explains the deployment architecture, configuration, and rationale.

Architecture Stack

Layer 1: Gunicorn (Process Manager)

Role: Production-grade WSGI/ASGI application server

  • Manages multiple worker processes

  • Handles graceful restarts and zero-downtime deployments

  • Provides worker health monitoring and auto-restart

  • Load balances requests across workers

  • Integrates with reverse proxies (OpenShift routes, nginx, etc.)

Configuration: backend/config.py

workers = int(os.environ.get("GUNICORN_PROCESSES", "1"))
threads = int(os.environ.get("GUNICORN_THREADS", "1"))

Layer 2: Uvicorn Worker

Role: ASGI worker implementation for Gunicorn (uvicorn.workers.UvicornWorker)

  • Provides ASGI protocol support (required for Connexion 3.0)

  • Bridges Gunicorn’s process management with async I/O capabilities

  • Automatically enables high-performance components (uvloop, httptools)

Why UvicornWorker?

  • Connexion 3.0 is ASGI-only (not WSGI compatible)

  • Standard Gunicorn workers are synchronous (WSGI)

  • UvicornWorker allows Gunicorn to manage async ASGI applications

Layer 3: uvloop (Event Loop)

Role: High-performance asyncio event loop replacement

  • Written in Cython for near-native performance

  • 2-4x faster than standard Python asyncio event loop

  • Drop-in replacement, automatically used by Uvicorn

  • Powers all async/await operations in the application

Performance Benefits:

  • Fast I/O operations (database, Redis, HTTP)

  • Efficient handling of concurrent connections

  • Low latency for async tasks

Reference: https://github.com/MagicStack/uvloop

Layer 4: httptools (Protocol Parser)

Role: Fast HTTP request/response parser

  • Written in C for maximum performance

  • Handles HTTP/1.1 protocol parsing

  • Used by Uvicorn for request processing

Reference: https://github.com/MagicStack/httptools

Container Entrypoint

Production Dockerfile

File: backend/docker/Dockerfile.backend

CMD ["gunicorn", \
     "-k", "uvicorn.workers.UvicornWorker", \
     "-c", "config.py", \
     "--bind", "0.0.0.0:8080", \
     "--access-logfile", "-", \
     "--error-logfile", "-", \
     "ibutsu_server:connexion_app"]

Command Breakdown:

  • gunicorn - Main process manager

  • -k uvicorn.workers.UvicornWorker - Use Uvicorn as worker class

  • -c config.py - Load configuration from config.py

  • --bind 0.0.0.0:8080 - Listen on all interfaces, port 8080

  • --access-logfile - - Access logs to stdout

  • --error-logfile - - Error logs to stderr

  • ibutsu_server:connexion_app - ASGI application to serve

Development Entrypoints

For local development, simpler configurations are used:

Option 1: Direct Uvicorn (with hot reload)

uvicorn ibutsu_server:connexion_app --host 0.0.0.0 --port 8080 --reload

Option 2: Python module

python -m ibutsu_server --host 0.0.0.0

These bypass Gunicorn for easier debugging but lack production features.

Configuration

Environment Variables

Variable

Default

Description

GUNICORN_PROCESSES

1

Number of worker processes

GUNICORN_THREADS

1

Threads per worker

PORT

8080

HTTP server port

APP_CONFIG

config.py

Path to Gunicorn config file

Scaling Workers

Horizontal Scaling (Processes):

export GUNICORN_PROCESSES=4  # Multiple worker processes

Vertical Scaling (Threads):

export GUNICORN_THREADS=2    # Multiple threads per worker

Recommendation:

  • Start with GUNICORN_PROCESSES=1 for most workloads

  • Increase processes if CPU-bound operations dominate

  • ASGI async I/O typically doesn’t require multiple processes

  • Monitor memory usage when scaling (each process loads full application)

Reverse Proxy Configuration

The backend is designed to run behind reverse proxies (OpenShift routes, nginx, etc.):

# config.py
forwarded_allow_ips = "*"
secure_scheme_headers = {"X-Forwarded-Proto": "https"}

This ensures:

  • Proper client IP detection from X-Forwarded-For

  • HTTPS scheme detection for URL generation

  • Trust in proxy headers

Why This Architecture?

Problem Statement

  1. Connexion 3.0 requires ASGI: The modern Connexion framework (OpenAPI-first) is built on ASGI, not WSGI

  2. Need process management: Production deployments require worker lifecycle management, graceful restarts, health checks

  3. Want async performance: Ibutsu makes many I/O calls (database, Redis, Celery) that benefit from async

Solution: Gunicorn + Uvicorn

Component

Provides

Why?

Gunicorn

Process management

Battle-tested, production-ready, zero-downtime deploys

UvicornWorker

ASGI bridge

Allows Gunicorn to manage ASGI apps

uvloop

Fast async I/O

2-4x performance boost for async operations

httptools

Fast HTTP parsing

C-based parser for request handling

Alternatives Considered

Alternative

Why Not Used

Gunicorn alone

Doesn’t support ASGI (only WSGI)

Uvicorn alone

Lacks robust process management features

Hypercorn

Less mature than Uvicorn, smaller ecosystem

Daphne

Django-focused, heavier dependencies

Deployment Scenarios

OpenShift/Kubernetes

The container runs with the Dockerfile CMD:

containers:
  - name: ibutsu-backend
    image: quay.io/ibutsu/backend:main
    env:
      - name: GUNICORN_PROCESSES
        value: "1"
      - name: APP_CONFIG
        value: config.py
    ports:
      - containerPort: 8080

No command override needed - the Dockerfile CMD is production-ready.

Docker Compose

services:
  backend:
    image: ibutsu/backend
    environment:
      GUNICORN_PROCESSES: "1"
    ports:
      - "8080:8080"

Local Development (Podman)

For development with hot reload:

podman run -v ./backend:/app ibutsu/backend \
  uvicorn ibutsu_server:connexion_app --reload

Monitoring and Observability

Health Checks

The backend exposes health check endpoints:

  • GET / - Basic health check

  • GET /api/health - Detailed health status

Configure in Kubernetes:

livenessProbe:
  httpGet:
    path: /
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 30

readinessProbe:
  httpGet:
    path: /
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

Logs

Logs are written to stdout/stderr for container-friendly logging:

  • Access logs: HTTP requests (stdout)

  • Error logs: Application errors (stderr)

  • Application logs: Python logging to stdout

View with:

# Kubernetes/OpenShift
kubectl logs deployment/ibutsu-backend

# Podman
podman logs ibutsu-backend

Metrics

Gunicorn provides basic metrics through:

  • Process CPU/memory usage

  • Request count and latency

  • Worker status

For detailed metrics, integrate with:

  • Prometheus (via prometheus-flask-exporter)

  • OpenTelemetry

  • Application Performance Monitoring (APM) tools

Performance Tuning

Worker Configuration

Single Worker (Default):

  • Memory efficient

  • Sufficient for most workloads with async I/O

  • Recommended for containerized deployments

Multiple Workers:

GUNICORN_PROCESSES=4  # 2 x CPU cores is common starting point

When to scale workers:

  • ✅ CPU-bound operations (heavy computation)

  • ✅ High request volume (> 1000 req/min per worker)

  • ❌ Memory-bound (each worker duplicates memory)

  • ❌ Pure I/O-bound (async handles this efficiently)

Connection Pooling

Database and Redis connections are pooled:

  • SQLAlchemy connection pool for PostgreSQL

  • Redis connection pool for Celery/cache

  • Connections shared within a worker process

Database Query Optimization

Statement timeout set to prevent long-running queries:

# __init__.py
SQLALCHEMY_ENGINE_OPTIONS = {
    "connect_args": {"options": "-c statement_timeout=25000"}
}

Container Images

Image Registry

Images are built and pushed to Quay.io:

  • quay.io/ibutsu/backend:main - Latest from main branch

  • quay.io/ibutsu/backend:v2.5.8 - Versioned releases

Image Stream Tags

In OpenShift, ImageStreams track container images:

kind: ImageStream
metadata:
  name: backend
spec:
  tags:
    - name: main
      from:
        kind: DockerImage
        name: quay.io/ibutsu/backend:main
      importPolicy:
        scheduled: true  # Auto-update on new pushes

Documentation References

Official Documentation

  1. Uvicorn Deployment Guide

    https://www.uvicorn.org/#running-with-gunicorn

    • Comprehensive guide to Gunicorn + Uvicorn setup

    • Configuration examples and best practices

  2. Gunicorn Documentation

    https://docs.gunicorn.org/en/stable/

    • Configuration reference

    • Worker types and deployment patterns

  3. Connexion 3.0 Documentation

    https://connexion.readthedocs.io/en/latest/v3.html

    • ASGI application architecture

    • OpenAPI-first design patterns

  4. uvloop GitHub

    https://github.com/MagicStack/uvloop

    • Performance benchmarks

    • Implementation details

  5. httptools GitHub

    https://github.com/MagicStack/httptools

    • HTTP parser implementation

    • Performance characteristics

Troubleshooting

Container Won’t Start

Check logs:

kubectl logs deployment/ibutsu-backend --tail=100

Common issues:

  • Missing CELERY_BROKER_URL environment variable

  • Missing CELERY_RESULT_BACKEND environment variable

  • Database connection failure (wrong credentials/host)

  • Port 8080 already in use

Performance Issues

High CPU:

  • Check for slow database queries

  • Review async task implementations

  • Consider scaling workers (GUNICORN_PROCESSES)

High Memory:

  • Each worker duplicates memory

  • Reduce GUNICORN_PROCESSES if memory-constrained

  • Check for memory leaks in application code

Slow Responses:

  • Enable debug logging: --log-level debug

  • Check database query performance

  • Review Celery task execution time

  • Check Redis connection latency

Worker Crashes

Gunicorn automatically restarts crashed workers:

[2025-12-10 10:15:23 +0000] [1234] [WARNING] Worker with pid 5678 was terminated
[2025-12-10 10:15:23 +0000] [1234] [INFO] Booting worker with pid: 5679

Investigate crashes:

  • Check stderr logs for exceptions

  • Review memory usage (OOM kills)

  • Check for segfaults (native library issues)

Summary

The Ibutsu backend uses a layered architecture for production deployments:

  1. Gunicorn manages worker processes (lifecycle, health, scaling)

  2. UvicornWorker provides ASGI support for Connexion 3.0

  3. uvloop accelerates async I/O operations (2-4x faster)

  4. httptools provides fast HTTP protocol parsing

This architecture delivers:

  • ✅ Production-grade process management

  • ✅ High-performance async I/O

  • ✅ ASGI support for modern frameworks

  • ✅ Container-friendly logging and health checks

  • ✅ Horizontal and vertical scalability

For questions or issues, refer to the documentation links above or consult the development team.