Article summary
Calculate real number-deduplication savings from source rows, normalized successes, unique phones, exceptions, unit price, and rework instead of promising 15%.
“Number deduplication saves 15%” is true only when the actual duplicate rate supports it. Real savings depend on source rows, unique phones after normalization, exceptions, billing units, whether one phone maps to several CRM objects, and cleaning and rework time.
This article provides a reusable model without using customer or AppShai billing data. Every number in the illustration teaches the formula; it is not an average duplicate rate, product price, or savings promise.
Core formula: Avoided processing units = qualified rows that would have been submitted − unique qualified phones after normalization. Direct task savings = avoided units × actual unit price. Net benefit then subtracts added cleaning, review, and tooling cost.
Separate four kinds of duplicate
| Duplicate type | Example | Merge? | Caution |
|---|---|---|---|
| Exact | Same text twice | Usually one task row | Keep both source IDs |
| Format | Country-code, domestic, punctuation variants | Merge after normalization | Verify country rule first |
| Shared phone | Several contacts use one phone | Task row may dedupe | Do not auto-merge contacts |
| Suspected | Similar length or suffix | Do not auto-merge | Require complete normalized evidence |
Raw-text dedupe misses format duplicates. Similarity matching can collapse different phones. Preserve raw_phone, build normalized_phone, and assign duplicate_group from the normalized value.
Variables for a deduplication cost model
| Variable | Definition | Source |
|---|---|---|
| N | Nonblank source phone rows | Frozen source file |
| V | Successfully normalized, submission-eligible rows | Cleaning output |
| U | Unique eligible phones after normalization | Deduplication output |
| X | Exception or uncertain rows | Exception file |
| P | Actual per-processing-unit price | Current quote/bill |
| H | Cleaning and review hours | Time record |
| R | Rerun units caused by errors | Task log |
N should reconcile into normalized, exception, blank, and other excluded categories. If counts do not close, do not calculate a savings percentage yet.
Three steps to direct task savings
- Avoided units: A = V − U. Count only eligible phones that otherwise would be submitted repeatedly.
- Direct fee savings: S_direct = A × P. Use the real billing unit and current price.
- Net benefit: S_net = S_direct + avoided downstream operating cost − added cleaning and review cost.
| Illustrative input—not business data | Value | Calculation |
|---|---|---|
| Nonblank source N | 10,000 | Formula demonstration only |
| Normalized eligible V | 9,400 | 600 exceptions/exclusions |
| Unique U | 8,200 | After normalization |
| Avoided A | 1,200 | 9,400−8,200 |
| Direct savings | 1,200×P | Insert your real unit price |
The example leaves P blank because task, plan, and time can change pricing. Insert the actual quote rather than inventing one universal unit price.
Which denominator belongs in duplicate rate?
| Measure | Formula | Answers |
|---|---|---|
| Source compression | (N−U)÷N | Overall file shrinkage, including exception effects |
| Eligible duplicate rate | (V−U)÷V | Duplication among submit-ready rows |
| Exception rate | X÷N | Records requiring repair/exclusion |
| Task input rate | U÷N | Unique task rows as share of source |
| Import expansion | CRM records to update ÷ U | How many source objects receive each result |
Use eligible duplicate rate for task-fee savings and show exception rate for data quality. Do not mix denominators in one trend.
Deduplicating task rows is not deleting customers
A task needs one normalized phone once, while CRM may contain several orders, branch contacts, or household members sharing it. Keep a one-to-many mapping, then expand the result back without merging permission or lifecycle.
| Object | Deduplication treatment | Preserve |
|---|---|---|
| Task TXT | One row per normalized phone | Normalized phone |
| Mapping table | One phone to several CRM IDs | Raw phone, source, duplicate_group |
| CRM contact | Do not auto-merge | Relationship and permission |
| Check result | Expand through mapping | task_id, checked_at |
| Suppression | Apply phone and subject rules | Opt-out reason and time |
Hidden costs in both paths
| Cost | Without dedupe | Dedupe can also create |
|---|---|---|
| Repeated task fee | Same phone processed repeatedly | Rule maintenance |
| Repeated send/support | One person contacted several times | Shared-phone review |
| CRM pollution | Format variants create duplicates | False-merge risk |
| Rework | Result rows do not reconcile | Exception handling |
| Audit | Source/version cannot be explained | Mapping retention |
Complex automation may not pay for a one-time list with few duplicates. In a recurring workflow, fixed rule and mapping cost spreads across more batches. Calculate your own break-even point.
Application to number-checking tasks
AppShai platform tasks accept TXT with one phone per line and export Excel after completion. Cleaning and deduplication happen before upload; output fields depend on the selected service. Review a relevant product page such as the WhatsApp Number Checker.
| Stage | Count | Check |
|---|---|---|
| Source freeze | N and blanks | Matches CRM export |
| Normalization | V and X | Exception reasons explainable |
| Deduplication | U and duplicate_group | Every source ID traceable |
| TXT upload | Nonblank rows=U | One phone per line |
| Excel export | Results plus exceptions reconcile | Do not default missing to no |
| CRM import | Expanded records | Do not overwrite permission |
Shared finance and data checklist
- Name period, source, and task.
- Report N, V, U, X, and R together.
- Take P from a dated quote or bill.
- Separate direct savings from net benefit.
- Label illustrative and measured data.
- Preserve normalization rules and duplicate mapping.
- Do not merge customers that share a phone.
- Do not double-count downstream and task savings.
Frequently asked questions
Does deduplication always save 15%?
No. A 2% eligible duplicate rate cannot produce 15% direct task savings without another factor; a higher rate may save more. Use your own V, U, and P.
Are invalid phones duplicates?
No. Unknown country, damaged format, short code, or extension belongs to exceptions. It may reduce submission volume because it was excluded or held, not because it was deduplicated.
What if one phone belongs to three CRM contacts?
Process one TXT row but retain three CRM IDs in the mapping. Expand the result while preserving each record’s permission, orders, and relationship.
A credible savings report does not need an attractive fixed percentage. It needs every count, price, and assumption to trace to evidence.
