74. Unified Logging System Design (Loki-based)¶

Overview¶

The Unified Logging System represents a critical observability component in professional quantitative trading systems, providing centralized log collection, storage, and analysis capabilities. This system transforms scattered container logs into a unified observability platform, enabling comprehensive system monitoring and troubleshooting.

🎯 Core Capabilities¶

System Architecture¶

Overall Architecture Flow¶

Microservice Standard Output → Promtail Collector → Loki Log Database → Grafana Log Dashboard

Key Design Principles: - ✅ Zero Code Changes: Microservices output to stdout without modification - ✅ Automatic Collection: Promtail automatically collects container logs - ✅ Efficient Storage: Loki provides compressed, high-performance log storage - ✅ Unified Viewing: Grafana enables multi-dimensional log search and analysis

Technology Stack Selection¶

Why Loki over ELK: - Lightweight: Lower resource consumption for microservices architecture - Cost-Effective: Reduced storage and processing requirements - Docker-Native: Seamless integration with containerized environments - Performance: Optimized for high-volume log ingestion

Microservice Logging Standardization¶

Logging Standards¶

Standardized Log Format: - Format: JSON structured logging - Levels: info, warning, error with proper categorization - Required Fields: timestamp, service_name, level, message - Optional Fields: strategy_id, account_id, order_id, error_code

Logging Guidelines: - Consistency: All services use identical log format - Completeness: Include all relevant context in log messages - Performance: Minimal logging overhead for high-frequency operations - Security: No sensitive data in logs (API keys, passwords)

Service-Specific Logging Patterns¶

Strategy Runner Logging:

{
  "timestamp": "2024-12-20T10:30:15.123Z",
  "service": "strategy-runner-001",
  "level": "info",
  "strategy_id": "momentum_btc_001",
  "account_id": "acc_12345",
  "message": "Strategy started successfully",
  "parameters": {"lookback_period": 20, "threshold": 0.02}
}

Risk Service Logging:

{
  "timestamp": "2024-12-20T10:30:16.456Z",
  "service": "risk-management-service",
  "level": "warning",
  "strategy_id": "momentum_btc_001",
  "account_id": "acc_12345",
  "message": "Position size exceeds 5% limit",
  "current_position": 0.06,
  "limit": 0.05
}

Portfolio Service Logging:

{
  "timestamp": "2024-12-20T10:30:17.789Z",
  "service": "portfolio-service",
  "level": "info",
  "account_id": "acc_12345",
  "message": "Portfolio updated",
  "total_value": 100000.50,
  "positions": {"BTC": 0.5, "ETH": 0.3}
}

Infrastructure Deployment¶

Docker Compose Configuration¶

Loki Service:

loki:
  image: grafana/loki:2.9.0
  ports:
    - "3100:3100"
  command: -config.file=/etc/loki/local-config.yaml
  volumes:
    - loki-data:/loki
  networks:
    - trading-network

Promtail Service:

promtail:
  image: grafana/promtail:2.9.0
  volumes:
    - /var/log:/var/log
    - /var/lib/docker/containers:/var/lib/docker/containers:ro
    - ./configs/promtail/promtail-config.yaml:/etc/promtail/promtail.yaml
  command: -config.file=/etc/promtail/promtail.yaml
  networks:
    - trading-network

Grafana Service:

grafana:
  image: grafana/grafana:latest
  ports:
    - "3001:3000"
  environment:
    - GF_SECURITY_ADMIN_USER=admin
    - GF_SECURITY_ADMIN_PASSWORD=admin
    - GF_INSTALL_PLUGINS=grafana-loki-datasource
  volumes:
    - grafana-data:/var/lib/grafana
  depends_on:
    - loki
  networks:
    - trading-network

Promtail Configuration¶

promtail-config.yaml:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker-containers
    static_configs:
      - targets:
          - localhost
        labels:
          job: docker-logs
          __path__: /var/lib/docker/containers/*/*.log
    pipeline_stages:
      - json:
          expressions:
            timestamp: timestamp
            service: service
            level: level
            message: message
            strategy_id: strategy_id
            account_id: account_id
      - labels:
          service:
          level:
          strategy_id:
          account_id:

Log Analysis and Visualization¶

Grafana Dashboard Configuration¶

Data Source Setup: - Loki Data Source: Configure connection to Loki service - URL: http://loki:3100 - Access: Server (default) mode - Authentication: None (internal network)

Dashboard Panels: - Real-time Log Stream: Live log viewing with filtering - Error Rate Monitoring: Error frequency by service and time - Strategy Performance Logs: Strategy-specific log analysis - System Health Overview: Overall system status from logs

Log Query Examples¶

Service-Specific Queries:

{service="strategy-runner-001"}                    # All logs from specific strategy runner
{service=~"strategy-runner.*"}                     # All strategy runner logs
{level="error"}                                    # All error logs
{strategy_id="momentum_btc_001"}                   # Specific strategy logs

Time-Based Queries:

{service="portfolio-service"} |= "error"           # Portfolio service errors
{service="risk-management-service"} |~ "warning"   # Risk service warnings
{account_id="acc_12345"}                          # Specific account logs

Complex Queries:

{service="strategy-runner-001"} |= "order" | json | line_format "{{.message}}"
{level="error"} | json | line_format "{{.service}}: {{.message}}"

Operational Benefits¶

Troubleshooting Capabilities¶

Alerting and Monitoring¶

Log-Based Alerts: - Error Rate Thresholds: Alert when error rates exceed limits - Service Health Monitoring: Detect service failures through logs - Strategy Performance Alerts: Monitor strategy execution issues - Security Event Detection: Identify suspicious activities

Alert Configuration:

groups:
  - name: trading-system-alerts
    rules:
      - alert: HighErrorRate
        expr: rate({level="error"}[5m]) > 0.1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High error rate detected"
          description: "Error rate is {{ $value }} errors per second"

Compliance and Audit¶

Regulatory Compliance: - Complete Audit Trail: All system activities logged and preserved - Data Retention: 90+ days of log retention for compliance - Secure Storage: Encrypted log storage with access controls - Audit Reporting: Automated compliance reporting capabilities

Performance Characteristics¶

Scalability Metrics¶

Resource Requirements¶

Integration with Existing System¶

Microservice Integration¶

Zero-Code Integration: - Standard Output: All services log to stdout/stderr - Automatic Collection: Promtail automatically discovers and collects logs - No Configuration: Services require no logging configuration changes - Immediate Visibility: Logs appear in Grafana immediately after deployment

Service Categories: - Strategy Services: Strategy runners, backtesting services - Core Services: Risk management, portfolio, execution services - Infrastructure Services: NATS, databases, monitoring services - Access Services: Market data gateways, trading gateways

Monitoring Integration¶

Prometheus + Grafana Integration: - Unified Dashboard: Combined metrics and logs in single interface - Correlation Analysis: Link metrics anomalies with log events - Comprehensive Observability: Complete system visibility - Alert Integration: Unified alerting across metrics and logs

Implementation Roadmap¶

Phase 1: Foundation (Week 1)¶

• Infrastructure Setup: Deploy Loki, Promtail, Grafana
• Basic Configuration: Configure log collection and storage
• Service Integration: Enable logging for core services
• Basic Dashboards: Create initial log viewing dashboards

Phase 2: Standardization (Week 2)¶

• Log Format Standardization: Implement JSON logging across all services
• Service-Specific Logging: Add structured logging to all microservices
• Log Validation: Ensure all services output proper log format
• Dashboard Enhancement: Create service-specific log dashboards

Phase 3: Advanced Features (Week 3)¶

• Alert Configuration: Set up log-based alerting rules
• Performance Optimization: Tune Loki and Promtail for high throughput
• Retention Policies: Configure long-term log retention
• Security Hardening: Implement log encryption and access controls

Phase 4: Production Ready (Week 4)¶

• High Availability: Deploy redundant logging infrastructure
• Backup and Recovery: Implement log backup and recovery procedures
• Compliance Features: Add regulatory compliance capabilities
• Performance Monitoring: Monitor logging system performance

Business Value¶

Operational Excellence¶

Competitive Advantages¶

Technical Implementation Details¶

Log Collection Architecture¶

Promtail Configuration Details: - Container Discovery: Automatic discovery of new containers - Log Parsing: JSON parsing with field extraction - Label Management: Dynamic labeling for filtering - Buffer Management: Efficient memory usage for high-volume logs

Loki Storage Configuration: - Chunk Storage: Efficient time-series log storage - Index Management: Fast query performance with minimal storage - Retention Policies: Configurable log retention periods - Compression: High compression ratios for cost efficiency

Query Performance Optimization¶

Indexing Strategy: - Label Indexing: Fast filtering by service, level, strategy_id - Time Indexing: Efficient time-range queries - Content Indexing: Full-text search capabilities - Query Caching: Frequently used query result caching

Performance Tuning: - Parallel Processing: Multi-threaded log ingestion - Memory Management: Optimized memory usage for large datasets - Network Optimization: Efficient data transfer protocols - Storage Optimization: SSD-based storage for high performance

Security and Compliance¶

Data Protection¶

Log Security Measures: - Encryption at Rest: All log data encrypted in storage - Encryption in Transit: Secure transmission of log data - Access Controls: Role-based access to log data - Audit Logging: Complete audit trail of log access

Compliance Features: - Data Retention: Configurable retention policies - Data Deletion: Secure deletion of expired logs - Access Monitoring: Track all log access and queries - Compliance Reporting: Automated compliance reports

Privacy Protection¶

Sensitive Data Handling: - PII Filtering: Automatic removal of personally identifiable information - API Key Masking: Secure handling of API credentials - Financial Data Protection: Secure handling of trading data - Access Logging: Complete audit trail of data access