Notification Webhooks (Internal)¶

Complete guide to internal webhook endpoints for the notification scheduling system. These endpoints are called by Cloud Scheduler and Cloud Tasks to execute scheduled notifications.

Overview¶

The notification webhook system provides internal endpoints for:

Promotion Trigger - Cloud Scheduler → promote notifications from MongoDB to Cloud Tasks
Notification Trigger - Cloud Tasks → execute scheduled notification at scheduled time
Promotion Statistics - Monitor pending/promoted notification counts
Manual Retry - Retry failed notifications (admin/support use)
Health Check - System health monitoring

Key Concepts:

Two-Tier Scheduling = MongoDB (>30 days) + Cloud Tasks (≤30 days)
Promotion = Moving notifications from MongoDB tier to Cloud Tasks
Version Tracking = Prevents stale execution of rescheduled notifications
Idempotency = Prevents duplicate notification delivery

Internal Endpoints

These endpoints are NOT included in Swagger/ReDoc (include_in_schema=False). They are system-only endpoints called by Cloud Scheduler and Cloud Tasks, not intended for direct API consumption.

Two-Tier Scheduling Architecture¶

The notification system uses a two-tier approach to handle scheduling beyond Google Cloud Tasks' 30-day limit:

graph TB
    subgraph Tier 1: Cloud Tasks
        CT[Cloud Tasks Queue]
        CT --> |At scheduled_at time| Trigger["/webhooks/notifications/trigger"]
        Trigger --> Send[NotificationService.send]
    end

    subgraph Tier 2: MongoDB
        MDB[(MongoDB<br/>scheduled_notifications)]
        MDB --> |T-25 days| Promote["/webhooks/notifications/promote"]
        Promote --> CT
    end

    subgraph Cloud Scheduler
        CS[Cloud Scheduler<br/>Every 5 minutes]
        CS --> Promote
    end

    New[New Scheduled Notification] --> Decision{Days until<br/>scheduled_at?}
    Decision --> |≤30 days| CT
    Decision --> |>30 days| MDB

Flow Summary:

New Notification Scheduled
If ≤30 days away → Immediately create Cloud Task (status: PROMOTED)
If >30 days away → Store in MongoDB (status: PENDING, tier: MONGODB)
Promotion Process (Cloud Scheduler)
Runs every 5 minutes via Cloud Scheduler
Calls /webhooks/notifications/promote
Promotes notifications within 25-day window to Cloud Tasks
Updates status: PENDING → PROMOTED
Execution (Cloud Tasks)
Cloud Task fires at scheduled_at time
Calls /webhooks/notifications/trigger with payload
NotificationService sends actual notification
Updates status: PROMOTED → COMPLETED or FAILED

Subscription Plan Limits¶

Notification channel access and quotas affect which channels can be used when notifications are executed:

Channel	FREE	PRO	ENTERPRISE
Push (FCM)	Unlimited	Unlimited	Unlimited
Email (SES)	100/month	1,000/month	Unlimited
WhatsApp (Fonnte)	50/month	500/month	5,000/month

Quota Enforcement

Quotas are checked at execution time, not scheduling time. If a tenant exceeds their quota when the notification triggers, that channel will be blocked but other channels will still be attempted.

Promotion Endpoint¶

Trigger promotion of scheduled notifications from MongoDB to Cloud Tasks.

Endpoint¶

POST /api/v1/webhooks/notifications/promote

Authentication: None (system-only)

Access: Cloud Scheduler / Cloud Function only

Purpose¶

Called by Cloud Scheduler (via Cloud Function) to promote notifications that are scheduled within the promotion window (default: 25 days).

Process Flow¶

sequenceDiagram
    participant CS as Cloud Scheduler
    participant CF as Cloud Function
    participant API as /webhooks/notifications/promote
    participant PS as PromoterService
    participant DB as MongoDB
    participant CT as Cloud Tasks

    CS->>CF: Trigger (every 5 min)
    CF->>API: POST /promote
    API->>PS: promote_batch()
    PS->>DB: Query PENDING notifications<br/>(scheduled_at ≤ now + 25 days)
    DB-->>PS: Batch of notifications
    loop For each notification
        PS->>CT: Create Cloud Task
        PS->>DB: Update status → PROMOTED
    end
    PS-->>API: Promotion stats
    API-->>CF: 200 OK + stats

Response¶

{
  "status": "success",
  "stats": {
    "total_found": 10,
    "promoted": 8,
    "failed": 1,
    "skipped": 1,
    "success_rate": 80.0,
    "duration_ms": 1250
  },
  "timestamp": "2025-01-15T10:30:00.000Z"
}

Stats Fields:

Field	Description
`total_found`	Number of notifications eligible for promotion
`promoted`	Successfully promoted to Cloud Tasks
`failed`	Failed to create Cloud Task
`skipped`	Already promoted or invalid
`success_rate`	Percentage of successful promotions
`duration_ms`	Batch processing time in milliseconds

Error Response¶

{
  "status": "error",
  "error": "Database connection failed",
  "timestamp": "2025-01-15T10:30:00.000Z"
}

Always Returns 200

This endpoint always returns HTTP 200 even on partial failures. This prevents Cloud Function retries and allows monitoring systems to track stats. Check the status field for actual success/error.

Business Rules¶

Only promotes notifications with status=PENDING and tier=MONGODB
Only promotes notifications within promotion window (default: 25 days from scheduled_at)
Batch size configurable via PROMOTER_BATCH_SIZE (default: 100)
Idempotent: Already-promoted notifications are skipped

Notification Trigger Endpoint¶

Execute a scheduled notification triggered by Cloud Tasks.

Endpoint¶

POST /api/v1/webhooks/notifications/trigger

Authentication: None (system-only)

Access: Cloud Tasks only

Purpose¶

Called by Cloud Tasks when a scheduled notification is due. This endpoint receives the notification payload and sends the actual notification through configured channels.

Request Body¶

{
  "scheduled_notification_id": "507f1f77bcf86cd799439011",
  "tenant_id": "507f1f77bcf86cd799439012",
  "version": 1
}

Parameters:

Field	Type	Required	Description
`scheduled_notification_id`	string	Yes	ID of scheduled notification to execute
`tenant_id`	string	No	Tenant ID for validation
`version`	integer	No	Version for stale detection (default: 1)

Process Flow¶

sequenceDiagram
    participant CT as Cloud Tasks
    participant API as /webhooks/notifications/trigger
    participant DB as MongoDB
    participant NS as NotificationService
    participant Ch as Channels (FCM/SES/WA)

    CT->>API: POST /trigger (at scheduled_at)
    API->>DB: Find scheduled notification
    DB-->>API: Notification data

    alt Version mismatch
        API-->>CT: 200 {status: "skipped", reason: "Version mismatch"}
    else Status not PROMOTED
        API-->>CT: 200 {status: "skipped", reason: "Not PROMOTED"}
    else Valid notification
        API->>DB: Update status → PROCESSING
        API->>NS: send(notification_data)
        NS->>Ch: Deliver via channels
        Ch-->>NS: Delivery result
        NS-->>API: SendResult
        alt Success
            API->>DB: Update status → COMPLETED
            API-->>CT: 200 {status: "success"}
        else Failure
            API->>DB: Update status → FAILED
            API-->>CT: 200 {status: "error"}
        end
    end

Success Response¶

{
  "status": "success",
  "notification_id": "507f1f77bcf86cd799439011",
  "sent_via": ["push", "email"],
  "message": "Notification sent successfully"
}

Skipped Response (Version Mismatch)¶

{
  "status": "skipped",
  "reason": "Version mismatch (notification was rescheduled)",
  "notification_id": "507f1f77bcf86cd799439011"
}

Skipped Response (Status Not PROMOTED)¶

{
  "status": "skipped",
  "reason": "Status is CANCELLED, not PROMOTED",
  "notification_id": "507f1f77bcf86cd799439011"
}

Error Response¶

{
  "status": "error",
  "notification_id": "507f1f77bcf86cd799439011",
  "error": "Email delivery failed: Invalid recipient"
}

Always Returns 200

This endpoint always returns HTTP 200 to prevent Cloud Tasks automatic retries. Errors are handled internally with retry logic. Check the status field for actual success/error.

Version Check (Stale Prevention)¶

When a notification is rescheduled, its version is incremented. The old Cloud Task (with old version) will still fire but will be rejected:

# Rescheduling flow
original_version = 1  # Cloud Task created with version 1
# ... notification rescheduled ...
new_version = 2       # New Cloud Task created with version 2

# Old Cloud Task fires with version 1
# API checks: expected=1, current=2 → MISMATCH → Skip

Status Transitions¶

From	To	Condition
`PROMOTED`	`PROCESSING`	Trigger received, starting execution
`PROCESSING`	`COMPLETED`	All channels sent successfully
`PROCESSING`	`FAILED`	All channels failed
`PROMOTED`	(skip)	Version mismatch or wrong status

Promotion Statistics Endpoint¶

Get statistics about pending and promotable notifications.

Endpoint¶

GET /api/v1/webhooks/notifications/promotion-stats

Authentication: None (system-only)

Access: Monitoring systems / health checks

Purpose¶

Monitoring endpoint for Cloud Scheduler health checks and dashboards. Returns counts of notifications in various states.

Response¶

{
  "status": "success",
  "pending_mongodb": 150,
  "ready_for_promotion": 25,
  "promoted": 1000,
  "failed": 3,
  "promotion_window_days": 25,
  "promoter_batch_size": 100,
  "timestamp": "2025-01-15T10:30:00.000Z"
}

Fields:

Field	Description
`pending_mongodb`	Total notifications waiting in MongoDB tier
`ready_for_promotion`	Notifications within promotion window
`promoted`	Notifications currently in Cloud Tasks
`failed`	Notifications that failed to send
`promotion_window_days`	Current promotion window configuration
`promoter_batch_size`	Current batch size configuration

Error Response¶

{
  "status": "error",
  "error": "Failed to query database",
  "timestamp": "2025-01-15T10:30:00.000Z"
}

Manual Retry Endpoint¶

Manually retry a failed scheduled notification.

Endpoint¶

POST /api/v1/webhooks/notifications/retry/{notification_id}

Authentication: None (system-only, but should be admin-protected in production)

Access: Admin tools / support dashboards

Purpose¶

Admin/support endpoint for retrying notifications that failed due to transient errors (network issues, temporary service unavailability).

Path Parameters¶

Parameter	Type	Required	Description
`notification_id`	string	Yes	Scheduled notification ID to retry

Request Body¶

{
  "force": false
}

Parameters:

Field	Type	Required	Description
`force`	boolean	No	Force retry even if max retries (3) exceeded

Success Response¶

{
  "status": "success",
  "notification_id": "507f1f77bcf86cd799439011",
  "retry_count": 2,
  "sent_via": ["push", "email"],
  "message": "Notification sent successfully on retry"
}

Retry Failed Response¶

{
  "status": "error",
  "notification_id": "507f1f77bcf86cd799439011",
  "retry_count": 3,
  "error": "Email delivery failed: Quota exceeded",
  "message": "Retry failed, notification marked as FAILED again"
}

Error Responses¶

Notification Not Found (404):

{
  "detail": "Scheduled notification not found"
}

Invalid Status (400):

{
  "detail": "Cannot retry notification with status 'COMPLETED'. Only FAILED notifications can be retried."
}

Max Retries Exceeded (400):

{
  "detail": "Max retries (3) exceeded. Use force=true to override."
}

Business Rules¶

Only FAILED notifications can be retried
Maximum 3 retry attempts (configurable)
Use force=true to override max retry limit
Retry count incremented on each attempt
Creates audit log entry for tracking

Retry Flow¶

sequenceDiagram
    participant Admin as Admin/Support
    participant API as /webhooks/notifications/retry
    participant DB as MongoDB
    participant NS as NotificationService
    participant Ch as Channels

    Admin->>API: POST /retry/{id} {force: false}
    API->>DB: Find notification
    DB-->>API: Notification (status: FAILED)

    alt retry_count >= 3 and force=false
        API-->>Admin: 400 {error: "Max retries exceeded"}
    else Valid retry
        API->>DB: Update status → PROMOTED, retry_count++
        API->>NS: send(notification_data)
        NS->>Ch: Deliver via channels
        Ch-->>NS: Delivery result
        alt Success
            API->>DB: Update status → COMPLETED
            API-->>Admin: 200 {status: "success"}
        else Failure
            API->>DB: Update status → FAILED
            API-->>Admin: 200 {status: "error"}
        end
    end

Health Check Endpoint¶

Health check endpoint for notification system monitoring.

Endpoint¶

GET /api/v1/webhooks/notifications/health

Authentication: None (system-only)

Access: Monitoring systems / load balancers

Purpose¶

Used by monitoring systems and load balancers to verify the notification system is operational.

Checks Performed¶

Database Connectivity - MongoDB ping command
Cloud Tasks Client - Verify client is available
Collection Access - Query scheduled_notifications counts

Healthy Response¶

{
  "status": "healthy",
  "database": "connected",
  "cloud_tasks": "available",
  "timestamp": "2025-01-15T10:30:00.000Z",
  "details": {
    "pending_count": 150,
    "promoted_count": 25,
    "failed_count": 3
  }
}

Degraded Response¶

{
  "status": "degraded",
  "database": "connected",
  "cloud_tasks": "disabled",
  "timestamp": "2025-01-15T10:30:00.000Z",
  "details": {
    "pending_count": 150,
    "promoted_count": 0,
    "failed_count": 3
  }
}

Unhealthy Response¶

{
  "status": "unhealthy",
  "database": "error: Connection refused",
  "cloud_tasks": "unknown",
  "error": "Failed to ping database",
  "timestamp": "2025-01-15T10:30:00.000Z"
}

Status Values¶

Status	Description
`healthy`	All systems operational
`degraded`	Some systems unavailable (e.g., Cloud Tasks disabled)
`unhealthy`	Critical systems failed (e.g., database down)

Scheduled Notification Statuses¶

Status	Description	Next State
`PENDING`	Stored in MongoDB, waiting for promotion	`PROMOTED` (on promotion)
`PROMOTED`	Cloud Task created, waiting for scheduled time	`PROCESSING` (on trigger)
`PROCESSING`	Currently executing notification send	`COMPLETED` or `FAILED`
`COMPLETED`	Successfully sent via all channels	(terminal)
`FAILED`	Failed to send (can be retried)	`PROMOTED` (on retry)
`CANCELLED`	Manually cancelled	(terminal)
`EXPIRED`	TTL expired before execution	(terminal)

stateDiagram-v2
    [*] --> PENDING: Schedule >30 days
    [*] --> PROMOTED: Schedule ≤30 days

    PENDING --> PROMOTED: Promotion
    PROMOTED --> PROCESSING: Cloud Task fires
    PROCESSING --> COMPLETED: Send success
    PROCESSING --> FAILED: Send failure
    FAILED --> PROMOTED: Manual retry

    PENDING --> CANCELLED: User cancellation
    PROMOTED --> CANCELLED: User cancellation

    PENDING --> EXPIRED: TTL expired

Configuration Parameters¶

Parameter	Default	Description
`PROMOTION_WINDOW_DAYS`	25	Days before scheduled_at to promote
`CLOUD_TASKS_MAX_DAYS`	30	Cloud Tasks hard limit
`PROMOTER_BATCH_SIZE`	100	Max notifications per promotion run
`NOTIFICATION_MAX_RETRIES`	3	Maximum retry attempts

Troubleshooting¶

Notifications Not Being Promoted¶

Symptoms: Notifications stuck in PENDING status

Checks:

Verify Cloud Scheduler is running (GCP Console)
Check Cloud Function logs for errors
Verify /promote endpoint returns success
Check promotion-stats for ready_for_promotion count

Fix:

Manually trigger Cloud Function
Check MongoDB indexes on scheduled_notifications
Verify scheduled_at dates are within promotion window

Cloud Tasks Not Firing¶

Symptoms: Notifications stuck in PROMOTED status

Checks:

Verify Cloud Tasks queue is active (GCP Console)
Check task creation timestamps
Verify webhook URL is correct in Cloud Tasks
Check task payloads contain correct scheduled_notification_id

Fix:

Check Cloud Tasks error logs
Verify network connectivity to API
Re-promote affected notifications

Version Mismatch Errors¶

Symptoms: "Version mismatch" in trigger logs

Cause: Notification was rescheduled but old Cloud Task still fired

Expected Behavior: Old task should be skipped

Action Required: None - this is expected behavior for rescheduled notifications

Failed Notifications Not Retrying¶

Symptoms: Notifications stuck in FAILED status

Checks:

Check retry_count - may have exceeded max (3)
Check error_message for root cause
Verify channels are not quota-blocked

Fix:

Use manual retry endpoint with force=true
Resolve underlying error (quota, invalid recipient, etc.)
Check subscription plan limits

Security Considerations¶

Endpoint Protection¶

These endpoints are excluded from Swagger (include_in_schema=False) but should have additional protection in production:

Recommended Measures:

IP Whitelisting - Only allow requests from GCP Cloud Tasks/Scheduler IPs
Internal Network - Deploy behind VPC/internal load balancer
Service Account Auth - Add IAM-based authentication for GCP services
Rate Limiting - Prevent abuse of retry endpoint

Validation Layers¶

ObjectId Validation - All notification IDs validated before database lookup
Status Checks - Only appropriate status transitions allowed
Version Verification - Stale notifications rejected
Tenant Isolation - Tenant ID validated if provided

Document	Description
Notification Management	Send/schedule notifications via API
Notification Settings	Configure templates and reminder timings
Device Management	Register FCM tokens for push notifications
Push Notification Integration	FCM setup and integration guide
Webhook Integration	Paper.id payment webhooks

API Reference Summary¶

Endpoint	Method	Purpose	Access
`/webhooks/notifications/promote`	POST	Promote MongoDB → Cloud Tasks	Cloud Scheduler
`/webhooks/notifications/trigger`	POST	Execute scheduled notification	Cloud Tasks
`/webhooks/notifications/promotion-stats`	GET	Get promotion statistics	Monitoring
`/webhooks/notifications/retry/{id}`	POST	Retry failed notification	Admin/Support
`/webhooks/notifications/health`	GET	System health check	Load Balancer

Next Steps:

Configure Cloud Scheduler to call /promote every 5 minutes
Set up Cloud Tasks queue with correct webhook URL
Monitor promotion stats via /promotion-stats
Configure alerting for unhealthy health status
Review Notification Management for scheduling API

Notification Webhooks (Internal)¶

Overview¶

Two-Tier Scheduling Architecture¶

Subscription Plan Limits¶

Promotion Endpoint¶

Endpoint¶

Purpose¶

Process Flow¶

Response¶

Error Response¶

Business Rules¶

Notification Trigger Endpoint¶

Endpoint¶

Purpose¶

Request Body¶

Process Flow¶

Success Response¶

Skipped Response (Version Mismatch)¶

Skipped Response (Status Not PROMOTED)¶

Error Response¶

Version Check (Stale Prevention)¶

Status Transitions¶

Promotion Statistics Endpoint¶

Endpoint¶

Purpose¶

Response¶

Error Response¶

Manual Retry Endpoint¶

Endpoint¶

Purpose¶

Path Parameters¶

Request Body¶

Success Response¶

Retry Failed Response¶

Error Responses¶

Business Rules¶

Retry Flow¶

Health Check Endpoint¶

Endpoint¶

Purpose¶

Checks Performed¶

Healthy Response¶

Degraded Response¶

Unhealthy Response¶

Status Values¶

Scheduled Notification Statuses¶

Configuration Parameters¶

Troubleshooting¶

Notifications Not Being Promoted¶

Cloud Tasks Not Firing¶

Version Mismatch Errors¶

Failed Notifications Not Retrying¶

Security Considerations¶

Endpoint Protection¶

Validation Layers¶

Related Documentation¶

API Reference Summary¶