# Production Deployment — Forest Department Recruitment System

This document captures everything required to run the system safely under live-exam load (4,000–10,000 concurrent candidates). Read it in full before any production deploy.

---

## 1. Required `.env` values for production

```dotenv
APP_ENV=production          # never "local" in prod — enables config caching + skips dev helpers
APP_DEBUG=false             # MUST be false — true leaks stack traces to users
APP_URL=https://<your-domain>

# Session — long enough for a 3-hour exam + buffer.
# Dropping below 240 risks mid-exam CSRF expiry for candidates who take
# the full time. 480 (8h) is the recommended minimum.
SESSION_DRIVER=database     # acceptable; use redis for 10k+ concurrent (see below)
SESSION_LIFETIME=480
SESSION_ENCRYPT=false       # leave false unless you have a specific compliance need

# Cache + queue — for 10k concurrent candidates, switch from database to redis.
# Database-backed cache/queue creates write contention on the `cache` / `jobs`
# tables that will serialise under load.
CACHE_STORE=redis
QUEUE_CONNECTION=redis

# Redis connection
REDIS_CLIENT=phpredis        # phpredis is ~2× faster than predis; install the PHP extension
REDIS_HOST=<your-redis-host>
REDIS_PORT=6379
REDIS_PASSWORD=<strong-password>
REDIS_DB=0
REDIS_CACHE_DB=1             # separate DB for cache vs queue to ease ops

# Logging — dedicated channels make the live-exam tail useful.
# See config/logging.php for the channel definitions.
LOG_CHANNEL=daily
LOG_LEVEL=info               # `debug` in prod fills disks fast
LOG_DAILY_DAYS=14
```

## 2. Pre-deploy build + cache steps

Run these in order on every deploy:

```bash
# Install PHP dependencies without dev packages; optimize autoloader.
composer install --no-dev --optimize-autoloader

# Build front-end assets (minified, fingerprinted).
npm ci
npm run build

# Laravel production caches. Re-run on every deploy — they read from .env at cache time.
php artisan config:cache
php artisan route:cache
php artisan view:cache
php artisan event:cache

# If using the queue for result emails:
php artisan queue:restart   # tells running workers to pick up new code

# Clear the opcode cache (fpm reload)
sudo systemctl reload php8.3-fpm
```

**Never run `php artisan config:cache` with `APP_DEBUG=true`**. The cached config will carry debug-mode into later requests even if you later change the `.env`. Always clear via `php artisan optimize:clear` first if you flip `.env` values.

## 3. Queue worker supervision

The `SendResultEmailJob` + PDF generation runs on the queue. In production, run a supervisor-managed worker:

```ini
; /etc/supervisor/conf.d/fdrs-queue.conf
[program:fdrs-queue]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/fdrs/artisan queue:work redis --queue=default --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/log/fdrs/queue.log
stopwaitsecs=3600
```

Run 4 workers per app server. Scale horizontally by adding more app servers behind the load balancer.

## 4. MySQL tuning (live-exam critical)

Default `max_connections=151` will exhaust at peak. Recommended minimum for 10k candidates:

```ini
# /etc/mysql/mysql.conf.d/fdrs.cnf
[mysqld]
max_connections           = 500
innodb_buffer_pool_size   = 4G       # tune to ~70% of RAM on a dedicated DB box
innodb_log_file_size      = 256M
innodb_flush_log_at_trx_commit = 2   # acceptable for exam-scale; faster writes
query_cache_type          = 0        # disabled (deprecated, harmful under contention)
thread_cache_size         = 64
table_open_cache          = 2000
wait_timeout              = 600
interactive_timeout       = 600
slow_query_log            = 1
slow_query_log_file       = /var/log/mysql/slow.log
long_query_time           = 1        # log queries over 1s
```

Restart MySQL after changes. Monitor with `SHOW PROCESSLIST` during the exam.

## 5. Nginx / web server hardening

```nginx
# /etc/nginx/sites-available/fdrs
server {
    listen 443 ssl http2;
    server_name recruitment.forest.gov.pk;

    root /var/www/fdrs/public;
    index index.php;

    # Long-cache fingerprinted assets from Vite build
    location ~* ^/build/assets/.*\.(?:css|js|woff2|woff|ttf|otf|eot|svg)$ {
        expires 1y;
        add_header Cache-Control "public, immutable, max-age=31536000";
        access_log off;
    }

    # Short-cache static vendor files
    location ~* ^/(vendor-.*\.js|favicon\.ico|robots\.txt)$ {
        expires 1d;
        add_header Cache-Control "public, max-age=86400";
    }

    # Compression — critical for slow connections.
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_types
        text/plain text/css text/javascript application/javascript
        application/json application/x-javascript application/xml+rss
        image/svg+xml application/xml application/atom+xml;
    gzip_comp_level 6;

    # If nginx-module-brotli is installed
    # brotli on;
    # brotli_comp_level 6;
    # brotli_types text/plain text/css text/javascript application/javascript application/json image/svg+xml;

    # Laravel
    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_pass unix:/run/php/php8.3-fpm.sock;
        fastcgi_index index.php;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        fastcgi_read_timeout 300;     # PDF download can take a moment
    }

    client_max_body_size 10M;          # accommodates profile doc uploads

    # Security
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
}
```

## 6. PHP-FPM pool sizing

For an 8 GB app server:

```ini
; /etc/php/8.3/fpm/pool.d/fdrs.conf
pm = dynamic
pm.max_children = 80
pm.start_servers = 20
pm.min_spare_servers = 10
pm.max_spare_servers = 30
pm.max_requests = 500              ; recycle workers periodically
request_terminate_timeout = 60s
```

Rule of thumb: `pm.max_children ≈ (available_ram_mb - os_reserved) / avg_process_memory_mb`.
On a 4-server fleet that's `4 × 80 = 320` concurrent PHP processes, which paired with MySQL `max_connections = 500` leaves headroom for queue workers + admin access.

## 7. Load test protocol (Tier 2 entry gate)

Before Tier 2 work or the live exam, verify:

```bash
# Simulated 500 concurrent candidates on staging for 30 minutes.
# Tool options: k6, Locust, Artillery. k6 example:
k6 run --vus 500 --duration 30m tests/load/exam-flow.js
```

Expected results:
- Median response time < 300ms for save-answer, heartbeat.
- 99th percentile < 2s.
- Zero 5xx errors.
- Zero dropped answers (exercise localStorage path: kill server mid-test).
- MySQL `Threads_running` stays below `max_connections × 0.5`.
- Redis memory stable.
- Queue backlog drains within 2 minutes of exam end.

## 8. Runbook — live exam day

**T-24h**
- Snapshot DB.
- Verify Redis is up + password is rotated.
- Deploy final code freeze.
- Smoke-test login, exam flow, result email.

**T-4h**
- `php artisan optimize` on every app server.
- Restart queue workers: `php artisan queue:restart`.
- Tail exam log channels in 4 terminals:
  ```bash
  tail -f storage/logs/exam-*.log          # exam lifecycle
  tail -f storage/logs/slow_query-*.log    # DB pain
  tail -f storage/logs/laravel.log         # generic errors
  sudo tail -f /var/log/mysql/slow.log
  ```

**T-1h**
- Pre-warm caches: hit `/health` and `/admin/dashboard` on each app server.
- Scale PHP-FPM workers up to target.

**T-0h — T+3h**
- Do not deploy. Do not run migrations.
- Watch MySQL `SHOW FULL PROCESSLIST` on contention alerts.
- Watch PHP-FPM pool status.
- If throttle 429s spike, check for runaway client in logs; likely a browser extension.

**T+3h**
- Verify queue backlog drains. `ResultEmailLog` should all go `pending → sent` within 10 minutes.
- Archive logs.

## 9. Rollback

- Code: `git revert` + redeploy. Never run destructive rollback during exam.
- Redis: flush cache keys for affected features only, never `FLUSHALL`.
- Schema: migrations in this project are additive-only; rollback would require a new migration.

## 10. Related documents

- `PERFORMANCE_AUDIT.md` — full audit findings, tier-ordered.
- `README.md` — developer onboarding.