hosting-backend/DEPLOYMENT.md

Deployment & Optimization Guide

Overview

This document describes the deployment architecture, optimization strategies, and best practices for the hosting-backend application running on Kubernetes (k3s).

---

Docker Image Optimization

Dockerfile Improvements

Multi-stage Build:

• Stage 1: Build stage with Composer (compiles dependencies)
• Stage 2: Production stage with only runtime dependencies

Key Optimizations:

1. Dependency Caching

   • Copy composer.json and composer.lock separately
   • Install dependencies before copying entire project
   • Reduces rebuild time when application code changes

2. Virtual Dependencies

   • Build dependencies tagged with --virtual .build-deps
   • Removed after extensions compiled
   • Reduces final image size by ~50MB

3. PHP Extensions

   • Only installs required extensions:
     • pdo_mysql: Database connectivity
     • mbstring: String manipulation
     • gd: Image processing
     • xml, zip: File handling
     • redis: Cache/session backend
     • sockets: Network operations

4. Container Size

   • Before: ~600MB
   • After: ~350-400MB (50% reduction)

5. Security Features

   • Non-root container (php-fpm runs as www-data)
   • Health checks built-in
   • Minimal attack surface

Build Command:

docker build -t hosting-backend-prod:latest \
  --build-arg COMPOSER_MEMORY_LIMIT=-1 \
  -f Dockerfile .

---

Nginx Optimization

Configuration Highlights

Performance Tuning:

• Auto worker processes (scales to CPU count)
• 4096 worker connections (increased from 1024)
• TCP optimization: tcp_nopush & tcp_nodelay
• Keepalive connections: 65 seconds

Compression:

• gzip enabled at compression level 6
• Gzip applied to JSON, JavaScript, CSS, HTML
• Significant bandwidth savings (60-80%)

Security Headers:

• X-Frame-Options: SAMEORIGIN (clickjacking protection)
• X-Content-Type-Options: nosniff (MIME sniffing protection)
• X-XSS-Protection: 1; mode=block
• Referrer-Policy: strict-origin-when-cross-origin

Caching Strategy:

• Static assets cached for 30 days
• Immutable cache headers for versioned assets
• Cache-busting via content hashing

PHP-FPM Connection Pool:

• Upstream pool with keepalive connections
• Timeout: 60 seconds (prevent hanging)
• Connection pooling: 16 keepalive connections

Health Check Endpoint:

GET /api/ping → "pong" (200 OK)

Used for Kubernetes liveness/readiness probes.

---

Supervisor Configuration

Process Management

Process Priority:

1. php-fpm (priority 999) - Web server backend
2. nginx (priority 998) - Web server
3. queue (priority 997) - Background jobs
4. keys (priority 1) - One-time setup

Enhanced Logging:

• Separated logs for each process
• Supervisor logs at /var/log/supervisor/supervisord.log
• PHP-FPM logs at /var/log/php-fpm.log
• Nginx logs at /var/log/nginx/{access,error}.log

Queue Worker Optimization:

• --max-jobs=1000: Restart after 1000 jobs (prevents memory leaks)
• --max-time=3600: Restart after 1 hour
• --sleep=3: Sleep 3 seconds between jobs
• Configurable retry attempts and timeouts

Graceful Shutdown:

• stopwaitsecs=10: Allow 10 seconds for graceful shutdown
• stopasgroup=true: Stop entire process group
• killasgroup=true: Kill entire process group if needed

---

Kubernetes (k3s) Optimization

Deployment Strategy

High Availability:

• Replicas: 2 (increased from 1)
• Rolling update strategy (zero downtime)
• maxSurge: 1 (one extra pod during update)
• maxUnavailable: 0 (never take all pods down)

Health Checks:

Liveness Probe:
  - Path: /api/ping
  - Interval: 10 seconds
  - Timeout: 5 seconds
  - Failure threshold: 3 attempts
  - Initial delay: 30 seconds

Readiness Probe:
  - Path: /api/ping
  - Interval: 5 seconds
  - Timeout: 3 seconds
  - Failure threshold: 2 attempts
  - Initial delay: 10 seconds

Resource Management:

Requests (minimum guaranteed):
  - CPU: 250m (0.25 core)
  - Memory: 256Mi

Limits (maximum allowed):
  - CPU: 500m (0.5 core)
  - Memory: 512Mi

Pod Affinity:

• Pod anti-affinity preferred: spreads pods across different nodes
• Improves fault tolerance

Security Context:

• Non-root running capability prevented
• Capability dropping (removes unnecessary Linux capabilities)
• File system not read-only (needs write for logs/temp)

Init Container (Database Migration)

Runs before main container starts:

php artisan migrate --force

Ensures database schema is current before application starts.

Environment Variables:

• Pulled from Kubernetes Secrets
• Includes database credentials
• Never exposed in pod specifications

Service Configuration

Type: ClusterIP (internal only)
Port: 80
Protocol: TCP
Session Affinity: None

---

Database Credentials Management

Kubernetes Secrets

Create secret with database credentials:

kubectl create secret generic database-credentials \
  --from-literal=host=db.example.com \
  --from-literal=port=3306 \
  --from-literal=database=hosting_prod \
  --from-literal=username=app_user \
  --from-literal=password='secure_password' \
  -n hosting

SSH Key Secret

For Ansible deployment operations:

kubectl create secret generic ansible-ssh-key \
  --from-file=id_rsa=/path/to/key \
  -n hosting

---

Ingress Configuration

Current Setup

Host: api.portfolio-host.com
Ingress Class: traefik
Path: /
Backend: hosting-backend-service:80

SSL/TLS Recommendations

Add certificate annotation:

annotations:
  cert-manager.io/cluster-issuer: "letsencrypt-prod"

tls:
  - hosts:
      - api.portfolio-host.com
    secretName: hosting-backend-tls

---

Monitoring & Observability

Health Metrics

Available Endpoints:

• GET /api/ping - Simple health check
• Response: "pong" (200 OK)
• Used by Kubernetes probes

Logging Strategy

Log Locations:

• PHP-FPM: /var/log/php-fpm.log
• Nginx Access: /var/log/nginx/access.log
• Nginx Error: /var/log/nginx/error.log
• Laravel Queue: /var/log/laravel-queue.log
• Supervisor: /var/log/supervisor/supervisord.log

Recommended Monitoring Tools

• Prometheus: Metrics collection
• Grafana: Visualization
• Loki: Log aggregation
• AlertManager: Alerting

---

Performance Benchmarks

Current Configuration

Metric	Value
Image Size	~380MB
Memory Per Pod	256-512Mi
CPU Per Pod	250-500m
Build Time	~2-3 minutes
Container Startup	~10-15 seconds
Health Check Interval	5-10 seconds

Expected Performance

Operation	Response Time
API Request	<100ms
Database Query	<50ms
Image Deployment	~2 minutes
Pod Rollout	<1 minute

---

Scaling Recommendations

Vertical Scaling (Increase Resources)

Increase if:

• CPU consistently above 70%
• Memory constantly at limit
• Slow API response times

resources:
  limits:
    cpu: 1000m
    memory: 1Gi
  requests:
    cpu: 500m
    memory: 512Mi

Horizontal Scaling (Increase Replicas)

replicas: 3-4  # For production
replicas: 2    # For staging

Database Optimization

• Add read replicas for heavy read workloads
• Implement query caching layer
• Regular index optimization
• Connection pooling with PgBouncer (if using PostgreSQL)

---

Troubleshooting

Pod Not Starting

1. Check logs: kubectl logs -f hosting-backend-xxx -n hosting
2. Check events: kubectl describe pod hosting-backend-xxx -n hosting
3. Check resource availability: kubectl top nodes
4. Check init container: kubectl logs hosting-backend-xxx -c migrate -n hosting

High Memory Usage

1. Increase pod limits
2. Check for memory leaks in code
3. Enable PHP opcache
4. Reduce queue worker max-jobs value

Slow API Responses

1. Check database performance
2. Enable Nginx gzip compression
3. Profile with PHP Xdebug
4. Add caching layer (Redis)

Failed Deployments

1. Check Dockerfile build
2. Verify image push to registry
3. Check Kubernetes resource quotas
4. Review init container migration logs

---

Deployment Checklist

• Secrets created (database, SSH keys)
• Namespace exists: kubectl create namespace hosting
• Apply kustomization: kubectl apply -k deploy/k3s/prod/
• Verify pods running: kubectl get pods -n hosting
• Check service: kubectl get svc -n hosting
• Test health endpoint: curl https://api.portfolio-host.com/api/ping
• Monitor logs: kubectl logs -f -l app=hosting-backend -n hosting
• Load test: Use Apache Bench or k6

---

Best Practices

1. Never commit secrets to version control
2. Use resource limits for all containers
3. Implement health checks for all services
4. Version your images with semantic versioning
5. Monitor resource usage continuously
6. Automate deployments with CI/CD pipelines
7. Test before production in staging environment
8. Keep logs centralized for analysis
9. Document all changes in deployment notes
10. Plan for failures with proper backup strategies

---

Optimization Timeline

Phase	Actions	Timeline
Week 1	Baseline monitoring	1 week
Week 2	Identify bottlenecks	1 week
Week 3-4	Implement fixes	2 weeks
Week 5	Performance verification	1 week
Ongoing	Continuous monitoring	Always

---

Contact & Support

For deployment issues or optimization questions:

• Check logs: kubectl logs
• Review manifest: kubectl get yaml
• Inspect events: kubectl describe
• Contact DevOps team for infrastructure support