Operations

This section covers operational aspects of the Keshless platform, including backup procedures, disaster recovery, and system maintenance.

Overview

Keshless implements a comprehensive backup and disaster recovery strategy to ensure data integrity and business continuity. The system uses automated Cloud Scheduler jobs to trigger backups at regular intervals.

Key Topics

Backup System

• Automated backup schedules
• PostgreSQL database backups
• GCP Cloud Storage for documents
• Secrets backup procedures
• Retention policies

Disaster Recovery

• Recovery procedures for PostgreSQL
• Document restoration from GCS
• Secrets recovery
• Full disaster recovery drills

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        GCP Cloud Scheduler                           │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐               │
│  │ Daily        │  │ Weekly       │  │ Weekly       │               │
│  │ PostgreSQL   │  │ Secrets 4AM  │  │ Cleanup 5AM  │               │
│  │ Backup 2AM   │  │              │  │              │               │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘               │
└─────────┼─────────────────┼─────────────────┼───────────────────────┘
          │                 │                 │
          ▼                 ▼                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    Keshless API (Cloud Run)                          │
│                        /backup-jobs/*                                │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐               │
│  │ postgres-    │  │ secrets-     │  │ cleanup-old- │               │
│  │ backup.ts    │  │ backup.ts    │  │ backups.ts   │               │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘               │
└─────────┼─────────────────┼─────────────────┼───────────────────────┘
          │                 │                 │
          ▼                 ▼                 ▼
┌─────────────────────────────────────────────────────────────────────┐
│                  GCP Cloud Storage (europe-west1)                    │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ gs://keshless-backups/                                         │ │
│  │   ├── postgresql/                                              │ │
│  │   │   ├── daily/YYYY-MM-DD/                                    │ │
│  │   │   └── monthly/YYYY-MM/                                     │ │
│  │   └── secrets/                                                 │ │
│  │       ├── weekly/YYYY-MM-DD.json.encrypted                     │ │
│  │       └── latest.json.encrypted                                │ │
│  └────────────────────────────────────────────────────────────────┘ │
│  ┌────────────────────────────────────────────────────────────────┐ │
│  │ gs://keshless-documents/                                       │ │
│  │   ├── kyc/                                                     │ │
│  │   ├── selfies/                                                 │ │
│  │   ├── vendor-kyc/                                              │ │
│  │   └── vendor-media/                                            │ │
│  └────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘

Environment Variables

The following environment variables are required for backup operations:

Variable	Description	Required
JOB_SECRET	Secret for Cloud Scheduler authentication	Yes
SECRETS_ENCRYPTION_KEY	32+ character key for secrets encryption	Yes
DATABASE_URL	PostgreSQL connection string	Yes
GCS_BACKUPS_BUCKET	GCS bucket for backups (keshless-backups)	Yes
GCS_DOCUMENTS_BUCKET	GCS bucket for documents (keshless-documents)	Yes

Security Considerations

• All backup endpoints are protected by X-Job-Secret header validation
• Secrets backups are encrypted using AES-256-GCM
• The encryption key must be stored securely offline (e.g., password manager)
• Cloud Scheduler jobs run in Google Cloud's trusted environment
• Document buckets have public access prevention enabled
• All document access uses time-limited signed URLs