Article summary
An engineering playbook for sharding, manifests, resumable runs, TXT generation, Excel reconciliation and targeted retries in large WA/WS checking jobs.
The hard part of bulk WhatsApp number checking is not placing hundreds of thousands of phones in one TXT. It is making the job divisible, pausable, repeatable and reconcilable. When one huge file fails, the team cannot tell which rows completed. Uploading the entire database again creates duplicate observations and cost. Treat the phone set as multiple immutable shards governed by a manifest.
Freeze the scope of the job
Create job_id for one business request and record the source snapshot, task, owner, purpose and deadline. New phones enter another job rather than being appended during processing. A frozen scope is the basis of reconciliation and the boundary for rollback.
Preserve three phone representations
Keep phone_raw, phone_candidate and phone_normalized. Cleaning handles approved whitespace, separators, country code and explicit rules. Missing country evidence, suspect length or multiple phones in one value becomes an exception. Never discard the source or guess a country to increase a pass rate.
Deduplicate without losing business relationships
The processing layer creates a dedupe_key for each normalized phone and checks the unique value once. A relationship table still preserves all contact_id, order or lead references. After Excel returns, distribute the observation through those relationships rather than sending the same phone repeatedly.
How to choose shard size
| Factor | Reason for smaller shard | Reason for larger shard |
|---|---|---|
| Source complexity | Many exceptions, easy isolation | Highly uniform format |
| Recovery objective | Reduces retry scope | Whole batch can wait |
| Acceptance capacity | Review while processing | Central automated acceptance |
| Country / task | Definitions must stay separate | Same rule and schema |
The manifest is the control plane
For every shard, store job_id, batch_id, task, country, unique input count, TXT name, SHA-256, creation time and state. Useful states are prepared, submitted, returned, accepted, failed and superseded. Do not encode workflow state only in a filename.
Generate a minimal TXT
AIPUSH accepts TXT with one phone per line. Generate rows in manifest order, with no header and no name, segment or permission data. Keep a controlled crosswalk with row_key, dedupe_key and internal contact references. Recount lines, validate encoding and calculate the hash after generation.
The task defines the Excel contract
| Task | Expected core columns | Assumption to reject |
|---|---|---|
| WS Registration | Phone, registration result | Activity, personhood or permission |
| WS Activity | Phone, activity time, active days, mapped phone | Purchase intent |
| WS Full Format | Defined activity, profile, avatar and business fields | Every column is populated |
Six acceptance steps when Excel arrives
- Associate the input through batch identity and hash;
- confirm columns against the task version;
- parse types and quarantine bad values;
- reconcile unique inputs, returns, unknowns and errors;
- join through the crosswalk rather than row order;
- land in staging and produce acceptance_report.
If a step fails, the batch stays returned or failed and does not enter production observation tables.
Resumable does not mean blind retry
A system error reruns only the failed shard and assigns a new attempt_id linked to the original batch. Format exceptions return to cleaning; missing fields require schema investigation; unknowns remain defined business outcomes. Retry never converts unknown to no and never erases the first response.
Guarantee idempotency
Use job_id, batch_id, row_key and task_version as a write key so reimporting the same Excel does not add another observation. A genuine recheck at a new time receives a new batch_id and checked_at. Network recovery or an accidental click then cannot corrupt totals.
The bulk-operations dashboard
Show shard count by state, queue age, exception rate, unknown rate, reconciliation, duplicate prevention and retry count. Break these down by country and source to locate a bad feed quickly. Processed volume alone is not enough; throughput says nothing about usability.
Balance three ledgers before closing
The quantity ledger explains how unique inputs became returns, exceptions and unknowns. The version ledger ties every result to the correct TXT and schema. The business ledger states which observations entered an approved workflow and when they expire. Close the job and clean temporary files only when all three reconcile.
