Article summary
Plan large-scale activity checking with arrival rate, throughput, backlog, recovery objectives and expiry windows instead of treating a peak demo as sustainable capacity.
Capacity for large-scale activity checking is not the fastest minute in a platform demo. It is whether the complete system can prepare, process, accept, reconcile and review exceptions within a required time. If downstream staff can accept only 20,000 rows per day, an upstream return of 100,000 per hour merely moves the bottleneck into an Excel backlog.
Define four capacity variables
arrival_rate is the new unique phones entering daily; service_rate is the observations fully accepted; backlog is unfinished inventory; freshness_window is how long a result can serve the decision. A stable system sustains service_rate above arrival_rate and absorbs peaks.
Map the pipeline as a capacity table
| Stage | Capacity unit | Main bottleneck |
|---|---|---|
| Normalization | Unique phones/hour | Country evidence and exceptions |
| Task processing | Batches/hour | Queue and external return |
| Excel acceptance | Rows/hour | Schema and reconciliation |
| Collision review | Cases/person-day | Mapping and identity |
| Downstream write | Observations/minute | Idempotency and locks |
Why activity has a time constraint
WhatsApp, Telegram, Zalo and Viber activity results can change. If total job duration approaches freshness_window, the first shard and the last no longer share a useful observation period. Give each cohort a completion SLA and expiry before work begins.
Shard to isolate capacity risk
Split immutable shards by platform, country, source and task. A high-exception feed gets smaller shards; a stable feed may use larger ones. Each shard stores input count, hash, attempt, state and owner. A failure reruns one fragment instead of restarting a million-phone job.
Set recovery objectives
| Objective | Question | Design response |
|---|---|---|
| RTO | How quickly must processing resume? | Persistent queue and reentrant worker |
| RPO | How much progress may be lost? | Row or batch checkpoint |
| Retry budget | How many attempts are allowed? | Error classes and backoff |
| Reconciliation SLO | How quickly must totals balance? | Automated difference report |
Price a closed loop, not a submission
The unit-cost denominator is not submitted phones; it is timely, reconciled observations that remain appropriate for the purpose. The numerator includes cleaning, task, storage, review, retry, expiry deletion and incident work. More unknowns or collisions rapidly increase cost per usable observation.
A capacity scenario
At 80,000 daily arrivals and 100,000 complete daily service, spare capacity is only 20,000. A failure that creates a 200,000 backlog takes ten days to clear while arrivals continue. The team needs temporary scaling and prioritization or must allow some cohorts to expire and stop.
Priority is not simply customer value
The queue balances business deadline, result lifetime, processing cost, user impact and permission. Remove opt-outs, ended purposes and records without evidence before processing. A high-value label cannot bypass authorization or permanently starve lower-risk service customers.
A load test includes failure
Test slow returns, missing-column Excel, duplicates, network interruption, worker restart, mapping collision and database lock—not only peak throughput. Observe duplicate writes, checkpoint recovery and the time required to drain backlog.
The essential capacity dashboard
Show arrival and service rate, queue-age distribution, oldest shard, stage utilization, error class, retry budget, reconciliation and volume nearing expiry. An average hides the tail; include P50, P95 and count beyond SLA.
When to stop scaling
If more results do not change decisions, human review becomes a permanent bottleneck or output expires before use, reduce the cohort and task instead of adding machines. Good capacity planning completes necessary observations within their useful life; it does not maximize phone volume.
