Secret Management¶

Section: 5-security-architecture
Document: Secret Management & API Key Security
Status: 🚨 CRITICAL - SECURITY BLOCKER
Audience: DevOps, security teams, backend developers

🚨 CRITICAL SECURITY ISSUE¶

Status: ⚠️ HARDCODED SECRETS IN SOURCE CODE
Severity: HIGH
Priority: P0 - IMMEDIATE ACTION REQUIRED
Timeline: Must be fixed by Q1 2025

Problem Summary¶

11+ backend services contain hardcoded API keys, credentials, and secrets directly in source code. This poses a severe security risk:

Source code repository exposure = All secrets compromised
No secret rotation without code changes
No audit trail for secret access
Violates security best practices (OWASP, GDPR, DPDPA, HIPAA)
Blocks enterprise sales (security audit failure)

📋 Complete Inventory of Hardcoded Secrets¶

Services with Hardcoded Secrets¶

1. response-3d-chatbot-service ⚠️ 11 HARDCODED KEYS!¶

File: /response-3d-chatbot-service/src/main.py
Lines: Scattered throughout (2,000+ line file)

Hardcoded Secrets:

# Azure OpenAI Keys (11 different keys!)
openai_key_gpt_4_turbo = "9be2c9a5dfa84b3cb03e3af8caa94ccb"
openai_key_4_omini = "c5ce7da2c7024e6c8b8f7c3b2a5d68b3"
openai_key_gpt_4_omini_realtime = "c5ce7da2c7024e6c8b8f7c3b2a5d68b3"
openai_key_3_point_5_turbo = "07e62adba8a34e43969f5c21e2e797e9"
openai_key_gpt_4 = "9be2c9a5dfa84b3cb03e3af8caa94ccb"

# Together AI (Llama models)
together_api_key = "bd826d5f6dea7aba3b99b1c63e104c70e7a0f42a4f1c44e58d24f59cc5285feb"

# Groq API
groq_api_key = "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"

# DeepSeek API
deepseek_api_key = "sk-72fa0c0584dd43599098854e7e23bc0d"

# Qwen API
qwen_api_key = "sk-aba6b7ffdab84ac19c742eb7f5bc8eea"

# Azure TTS Keys (multiple instances)
tts_subscription_key = "2a87f12db4b7449aa1d20959c9b62fbe"
# Also: "ef2f59ac5f4d44e6a94e71c23fdb5c43" in voice service

# Azure Endpoints
azure_openai_endpoint_gpt_4 = "https://askgalore-france-central.openai.azure.com/"
# + 5 more endpoint variations

Risk Level: 🔴 CRITICAL
Impact: All LLM services compromised if repository accessed

2. response-text-chatbot-service ⚠️¶

File: Similar hardcoded Azure OpenAI keys

# Identical hardcoded keys as 3D service
openai_key = "9be2c9a5dfa84b3cb03e3af8caa94ccb"
together_api_key = "bd826d5f6dea7aba3b99b1c63e104c70e7a0f42a4f1c44e58d24f59cc5285feb"

Risk Level: 🔴 CRITICAL

3. response-voice-chatbot-service ⚠️¶

File: /response-voice-chatbot-service/src/main.py

# Azure TTS
tts_subscription_key = "ef2f59ac5f4d44e6a94e71c23fdb5c43"
tts_region = "centralindia"

# Azure OpenAI (shared keys)
openai_key = "9be2c9a5dfa84b3cb03e3af8caa94ccb"

Risk Level: 🔴 CRITICAL

4. llm-model-service ⚠️¶

File: /llm-model-service/src/main.py

# All LLM provider keys hardcoded
AZURE_OPENAI_KEY = "9be2c9a5dfa84b3cb03e3af8caa94ccb"
TOGETHER_API_KEY = "bd826d5f6dea7aba3b99b1c63e104c70e7a0f42a4f1c44e58d24f59cc5285feb"
GROQ_API_KEY = "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"

Risk Level: 🔴 CRITICAL

5. user-service ⚠️¶

File: /user-service/src/utils/email_utils.py

# Azure Communication Services
EMAIL_ENDPOINT = "https://mailing-sevice.india.communication.azure.com/"
EMAIL_ACCESS_KEY = "CgdEWi6fBCJv4c0EHOkq6ZUSML0VQSG49qXgEfrtlLmXY76HV7DeJQQJ99BBACULyCplbknJAAAAAZCSbc3T"
SENDER_EMAIL = "DoNotReply@machineagents.ai"

Risk Level: 🔴 CRITICAL (Email sending compromised)

6. auth-service ⚠️¶

File: /auth-service/src/main.py

# JWT Secret with weak default
JWT_SECRET = os.getenv("JWT_SECRET", "your_jwt_secret")  # ⚠️ Weak fallback!

Risk Level: 🟡 MEDIUM (Environment variable used, but weak default)

7-11. Additional Services ⚠️¶

Other services with similar issues:

chatbot-file-upload-service (MongoDB URI)
client-data-collection-service (Azure OpenAI keys)
data-crawling-service (MongoDB URI)
homepage-chatbot-service (Azure OpenAI keys)
gateway-service (reCAPTCHA secret potentially)

Complete Secret Inventory¶

Secret Name	Type	Services Affected	Current Storage	Risk Level
Azure OpenAI Keys (11 variants)	API Key	3D, Text, Voice, LLM	HARDCODED	🔴 CRITICAL
Together AI API Key (Llama)	API Key	3D, Text, LLM	HARDCODED	🔴 CRITICAL
Groq API Key	API Key	3D, LLM	HARDCODED	🔴 CRITICAL
DeepSeek API Key	API Key	3D	HARDCODED	🔴 CRITICAL
Qwen API Key	API Key	3D	HARDCODED	🔴 CRITICAL
Azure TTS Keys (2 variants)	API Key	3D, Voice	HARDCODED	🔴 CRITICAL
Azure Email Access Key	API Key	User Service	HARDCODED	🔴 CRITICAL
MongoDB Connection String	Connection String	Multiple	`.env` (some hardcoded)	🟡 MEDIUM
JWT Secret	Signing Key	Auth Service	`.env` (weak default)	🟡 MEDIUM
reCAPTCHA Secret	API Key	Gateway	`.env`	🟢 LOW
Razorpay API Key	API Key	Payment Service	`.env`	🟢 LOW
Razorpay API Secret	API Secret	Payment Service	`.env`	🟢 LOW

Total Secrets: 25+ unique secrets
Hardcoded: 18 secrets (🔴 CRITICAL)
In .env: 7 secrets (🟡 Better, but not ideal)

📈 Risk Assessment¶

Impact if Secrets Compromised¶

Azure OpenAI Keys:

Cost: Unlimited API usage → $$$$ monthly bills
Data: Training data poisoning
Rate Limits: Exhaust quotas, service disruption

Together AI / Groq / DeepSeek / Qwen:

Cost: API bills to customer's account
Abuse: Use for malicious purposes (spam, phishing content generation)

Azure TTS / Email:

Cost: Massive email sending bills
Reputation: Send spam emails from machineagents.ai domain → domain blacklisted
Privacy: Read email logs, user data

MongoDB Connection String:

Data Breach: Full database access
Compliance Violation: GDPR, DPDPA, HIPAA fines
Customer Trust: Complete loss

Attack Vectors¶

GitHub Repository Access (if ever made public)
Disgruntled Employee (has access to codebase)
Contractor/Third-Party (temporary access)
CI/CD Pipeline Compromise (secrets in build logs)
Docker Image Inspection (secrets baked into images)
Memory Dump (secrets in application memory)

🔧 Solution: Azure Key Vault Migration¶

Architecture¶

graph TB
    subgraph "Azure Key Vault"
        KV[(Secrets<br/>Encrypted Storage)]
    end

    subgraph "Managed Identity"
        MI[System-Assigned<br/>Managed Identity]
    end

    subgraph "Backend Services"
        S1[Response 3D Service]
        S2[Response Text Service]
        S3[Response Voice Service]
        S4[LLM Model Service]
        S5[User Service]
        S6[Auth Service]
        S7[Other Services...]
    end

    S1 --> MI
    S2 --> MI
    S3 --> MI
    S4 --> MI
    S5 --> MI
    S6 --> MI
    S7 --> MI

    MI -->|Authenticate| KV
    KV -->|Return Secret| MI
    MI -->|Provide to Service| S1
    MI -->|Provide to Service| S2

    style KV fill:#E3F2FD
    style MI fill:#FFF3E0
    style S1 fill:#FFE082

Benefits¶

✅ Centralized Management: All secrets in one place
✅ No Code Changes for Rotation: Update in Key Vault, restart services
✅ Audit Logs: Track every secret access
✅ RBAC: Control who can access which secrets
✅ Automatic Rotation: Schedule regular key rotation
✅ Compliance: Meets GDPR, DPDPA, HIPAA, SOC 2 requirements
✅ Encryption at Rest: FIPS 140-2 Level 2 validated
✅ Disaster Recovery: Geo-replicated, backed up

📝 Migration Plan¶

Phase 1: Setup Azure Key Vault (Week 1)¶

Step 1: Create Key Vault

# Azure CLI
az keyvault create \
  --name "machineavatars-kv" \
  --resource-group "machineavatars-rg" \
  --location "centralindia" \
  --enable-rbac-authorization

# Note Key Vault URL
# https://machineavatars-kv.vault.azure.net/

Step 2: Enable Managed Identity for Services

# For Azure App Service
az webapp identity assign \
  --name "response-3d-chatbot-service" \
  --resource-group "machineavatars-rg"

# Capture the Principal ID
PRINCIPAL_ID=$(az webapp identity show \
  --name "response-3d-chatbot-service" \
  --resource-group "machineavatars-rg" \
  --query principalId \
  --output tsv)

Step 3: Grant Permissions

# Grant "Key Vault Secrets User" role
az role assignment create \
  --role "Key Vault Secrets User" \
  --assignee $PRINCIPAL_ID \
  --scope "/subscriptions/{subscription-id}/resourceGroups/machineavatars-rg/providers/Microsoft.KeyVault/vaults/machineavatars-kv"

Phase 2: Migrate Secrets (Week 2)¶

Step 1: Add Secrets to Key Vault

# Azure OpenAI Keys
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "azure-openai-gpt4-turbo-key" \
  --value "9be2c9a5dfa84b3cb03e3af8caa94ccb"

az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "azure-openai-gpt35-turbo-key" \
  --value "07e62adba8a34e43969f5c21e2e797e9"

# Together AI
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "together-ai-api-key" \
  --value "bd826d5f6dea7aba3b99b1c63e104c70e7a0f42a4f1c44e58d24f59cc5285feb"

# Groq
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "groq-api-key" \
  --value "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"

# DeepSeek
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "deepseek-api-key" \
  --value "sk-72fa0c0584dd43599098854e7e23bc0d"

# Qwen
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "qwen-api-key" \
  --value "sk-aba6b7ffdab84ac19c742eb7f5bc8eea"

# Azure TTS
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "azure-tts-subscription-key" \
  --value "2a87f12db4b7449aa1d20959c9b62fbe"

# Azure Email
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "azure-email-access-key" \
  --value "CgdEWi6fBCJv4c0EHOkq6ZUSML0VQSG49qXgEfrtlLmXY76HV7DeJQQJ99BBACULyCplbknJAAAAAZCSbc3T"

# JWT Secret (generate new secure one!)
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "jwt-secret-key" \
  --value "$(openssl rand -base64 32)"

# MongoDB
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "mongodb-connection-string" \
  --value "mongodb+srv://..."

# Razorpay
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "razorpay-api-key" \
  --value "rzp_live_..."

az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "razorpay-api-secret" \
  --value "..."

Phase 3: Update Code (Week 3-4)¶

Step 1: Install Azure SDK

pip install azure-identity azure-keyvault-secrets

Step 2: Create Key Vault Client

# services/utils/secrets.py
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
import os
from functools import lru_cache

# Key Vault URL
KEY_VAULT_URL = os.getenv(
    "AZURE_KEY_VAULT_URL",
    "https://machineavatars-kv.vault.azure.net/"
)

# Initialize credential (uses Managed Identity in Azure)
credential = DefaultAzureCredential()

# Initialize Secret Client
secret_client = SecretClient(vault_url=KEY_VAULT_URL, credential=credential)

@lru_cache(maxsize=100)
def get_secret(secret_name: str) -> str:
    """
    Retrieve secret from Azure Key Vault.
    Results are cached to reduce API calls.

    Args:
        secret_name: Name of the secret in Key Vault

    Returns:
        Secret value as string

    Raises:
        Exception: If secret cannot be retrieved
    """
    try:
        secret = secret_client.get_secret(secret_name)
        return secret.value
    except Exception as e:
        logger.error(f"Failed to retrieve secret '{secret_name}': {e}")
        raise

# Example: Get Azure OpenAI key
def get_azure_openai_key(model: str = "gpt-4-turbo") -> str:
    """Get Azure OpenAI API key for specific model."""
    secret_name_map = {
        "gpt-4-turbo": "azure-openai-gpt4-turbo-key",
        "gpt-4": "azure-openai-gpt4-key",
        "gpt-3.5-turbo": "azure-openai-gpt35-turbo-key",
    }

    secret_name = secret_name_map.get(model, "azure-openai-gpt4-turbo-key")
    return get_secret(secret_name)

Step 3: Update Services

Before (response-3d-chatbot-service):

# ❌ HARDCODED
openai_key_gpt_4_turbo = "9be2c9a5dfa84b3cb03e3af8caa94ccb"
together_api_key = "bd826d5f6dea7aba3b99b1c63e104c70e7a0f42a4f1c44e58d24f59cc5285feb"

After:

# ✅ FROM KEY VAULT
from utils.secrets import get_secret, get_azure_openai_key

openai_key_gpt_4_turbo = get_azure_openai_key("gpt-4-turbo")
together_api_key = get_secret("together-ai-api-key")
groq_api_key = get_secret("groq-api-key")
deepseek_api_key = get_secret("deepseek-api-key")
qwen_api_key = get_secret("qwen-api-key")
tts_subscription_key = get_secret("azure-tts-subscription-key")

Update user-service (email):

Before:

# ❌ HARDCODED
EMAIL_ACCESS_KEY = "CgdEWi6fBCJv..."

After:

# ✅ FROM KEY VAULT
from utils.secrets import get_secret

EMAIL_ACCESS_KEY = get_secret("azure-email-access-key")

Update auth-service (JWT):

Before:

# ❌ WEAK DEFAULT
JWT_SECRET = os.getenv("JWT_SECRET", "your_jwt_secret")

After:

# ✅ FROM KEY VAULT (no fallback!)
from utils.secrets import get_secret

JWT_SECRET = get_secret("jwt-secret-key")

Phase 4: Testing (Week 5)¶

Test Checklist:

Testing Script:

# test_secrets.py
import pytest
from utils.secrets import get_secret

def test_azure_openai_key():
    key = get_secret("azure-openai-gpt4-turbo-key")
    assert key is not None
    assert len(key) == 32  # Expected key length

def test_together_ai_key():
    key = get_secret("together-ai-api-key")
    assert key is not None
    assert key.startswith("bd")  # Known prefix (temporary check)

def test_caching():
    """Verify caching works (no repeated Key Vault calls)."""
    import time
    start = time.time()
    key1 = get_secret("azure-openai-gpt4-turbo-key")
    first_call = time.time() - start

    start = time.time()
    key2 = get_secret("azure-openai-gpt4-turbo-key")
    second_call = time.time() - start

    assert key1 == key2
    assert second_call < first_call / 10  # Cached call much faster

Phase 5: Deployment (Week 6)¶

Deployment Steps:

Deploy to Staging:

# Deploy updated code to staging
git push staging main

# Verify all services start successfully
# Check logs for errors

Smoke Test in Staging:
Test all critical user flows
Verify LLM responses
Check TTS audio generation
Test OTP emails
Production Deployment:

# Deploy to production (blue-green deployment)
git push production main

# Monitor error rates
# Check performance metrics

Rollback Plan:

# If issues detected, rollback immediately
git revert HEAD
git push production main

Phase 6: Cleanup (Week 7)¶

Remove Hardcoded Secrets:

# Create cleanup branch
git checkout -b cleanup/remove-hardcoded-secrets

# Find all hardcoded secrets
grep -r "9be2c9a5dfa84b3cb03e3af8caa94ccb" .
grep -r "bd826d5f6dea7aba" .
# ... etc for all secrets

# Replace with Key Vault calls
# Commit changes
git commit -m "security: Remove all hardcoded API keys"

# Merge to main
git checkout main
git merge cleanup/remove-hardcoded-secrets

Rotate All Secrets:

Since secrets were hardcoded and potentially exposed:

# Regenerate ALL API keys
# Azure OpenAI - create new keys in Azure portal
# Together AI - regenerate in Together dashboard
# Groq - regenerate in Groq dashboard
# DeepSeek - regenerate
# Qwen - regenerate
# Azure TTS - regenerate
# Azure Email - regenerate

# Update Key Vault with NEW secrets
az keyvault secret set \
  --vault-name "machineavatars-kv" \
  --name "azure-openai-gpt4-turbo-key" \
  --value "{NEW_KEY}"

# Restart all services to pick up new secrets

🔄 Secret Rotation Policy¶

Automatic Rotation:

# Set rotation policy (Azure CLI)
az keyvault secret set-attributes \
  --vault-name "machineavatars-kv" \
  --name "jwt-secret-key" \
  --rotation-policy-file rotation-policy.json

rotation-policy.json:

{
  "lifetimeActions": [
    {
      "trigger": {
        "timeBeforeExpiry": "P30D"
      },
      "action": {
        "type": "Notify"
      }
    }
  ],
  "attributes": {
    "expiryTime": "P365D"
  }
}

Rotation Schedule:

Secret Type	Rotation Frequency	Auto-rotation
JWT Secret	90 days	✅ Planned
API Keys (Azure, LLM providers)	180 days	Manual
Database Credentials	365 days	Manual
Payment API Keys	On compromise	Manual

🔐 Best Practices¶

Secret Naming Convention¶

Format: {service}-{provider}-{secret-type}-{variant}

Examples:

azure-openai-gpt4-turbo-key
together-ai-api-key
azure-tts-subscription-key
mongodb-connection-string
razorpay-api-secret

Secret Versioning¶

Key Vault automatically versions secrets:

# Get latest version (default)
secret = secret_client.get_secret("azure-openai-gpt4-turbo-key")

# Get specific version
secret = secret_client.get_secret(
    "azure-openai-gpt4-turbo-key",
    version="abc123..."
)

Emergency Access¶

Break-glass Procedure:

Service account with Key Vault admin access (stored securely offline)
Multi-person authorization required
Audit log review after emergency access

📊 Monitoring & Alerts¶

Azure Monitor Alerts:

# Alert on secret access
az monitor metrics alert create \
  --name "KeyVault-HighAccessRate" \
  --resource-group "machineavatars-rg" \
  --scopes "/subscriptions/{...}/resourceGroups/machineavatars-rg/providers/Microsoft.KeyVault/vaults/machineavatars-kv" \
  --condition "count ServiceApiResult > 1000" \
  --window-size 5m \
  --evaluation-frequency 1m

Logging:

import logging

logger.info("Retrieved secret from Key Vault", extra={
    "secret_name": "azure-openai-gpt4-turbo-key",
    "user_id": user_id,
    "service": "response-3d-chatbot",
    "timestamp": datetime.now().isoformat()
})

✅ Migration Checklist¶

Pre-Migration:

Azure Key Vault created
Managed Identities assigned to all services
Permissions granted (Key Vault Secrets User role)
All secrets added to Key Vault
Backup of current secrets (encrypted, offline)

Code Changes:

Azure SDK installed in all services
secrets.py utility module created
All hardcoded secrets replaced with get_secret() calls
Error handling for secret retrieval failures
Caching implemented (@lru_cache)

Testing:

Unit tests for secret retrieval
Integration tests (all services)
Staging deployment successful
Smoke tests passed
Performance tests (no degradation)

Deployment:

Production deployment completed
All services healthy
Error rates normal
Performance metrics normal

Cleanup:

Hardcoded secrets removed from code
All secrets rotated (new keys generated)
Git history cleaned (BFG Repo-Cleaner)
Old secrets revoked in provider dashboards

Security:

Encryption - Key Vault encryption details
Authentication & Authorization - JWT secret usage
API Security - API key authentication

Backend:

Response 3D Service - Hardcoded keys identified
User Service - Email key hardcoded

"A secret once committed is a secret forever exposed." 🔐⚠️

TIMELINE: Q1 2025 - NO EXCEPTIONS

Secret Management¶

🚨 CRITICAL SECURITY ISSUE¶

Problem Summary¶

📋 Complete Inventory of Hardcoded Secrets¶

Services with Hardcoded Secrets¶

1. response-3d-chatbot-service ⚠️ 11 HARDCODED KEYS!¶

2. response-text-chatbot-service ⚠️¶

3. response-voice-chatbot-service ⚠️¶

4. llm-model-service ⚠️¶

5. user-service ⚠️¶

6. auth-service ⚠️¶

7-11. Additional Services ⚠️¶

Complete Secret Inventory¶

📈 Risk Assessment¶

Impact if Secrets Compromised¶

Attack Vectors¶

🔧 Solution: Azure Key Vault Migration¶

Architecture¶

Benefits¶

📝 Migration Plan¶

Phase 1: Setup Azure Key Vault (Week 1)¶

Phase 2: Migrate Secrets (Week 2)¶

Phase 3: Update Code (Week 3-4)¶

Phase 4: Testing (Week 5)¶

Phase 5: Deployment (Week 6)¶

Phase 6: Cleanup (Week 7)¶

🔄 Secret Rotation Policy¶

🔐 Best Practices¶

Secret Naming Convention¶

Secret Versioning¶

Emergency Access¶

📊 Monitoring & Alerts¶

✅ Migration Checklist¶

🔗 Related Documentation¶