Handling Low Model Accuracy: A Practical Guide for Agencies
When an MMM model fails to reach the accuracy threshold (R² ≥ 0.75), you face one of the most commercially and reputationally sensitive moments in any engagement. This guide tells you how to diagnose what happened, what to do next, and — critically — how to handle the refund question.
Step 1 — Diagnose Before You Decide Anything
Low accuracy has two very different causes. Your response depends entirely on which one you are dealing with.
Category A: Data Problems (Potentially Fixable)
The data the client provided does not meet the quality standard needed to support a reliable model. This is the most common root cause for SME clients.
Signs:
- The dependent variable is GA4 revenue with visible tracking gaps
- Weekly revenue data has unexplained spikes or zeroes that don’t match any known event
- Spend data is missing for certain channels or periods
- The promotional calendar was not provided, or is incomplete
- The dataset is at monthly granularity when weekly is required
What this means: The failure is in the data, not the methodology. More analysis runs will not fix this. The client needs to provide better data before meaningful modeling is possible.
Category B: Structural Factors (Not Fixable With This Client’s Setup)
The business model, market, or channel mix makes MMM inherently difficult — regardless of how complete the data is.
Common structural factors:
| Factor | Why It Causes Low Accuracy |
|---|---|
| Long B2B sales cycles (>3 months) | The lag between marketing exposure and closed revenue is too long and unpredictable for the model to isolate |
| Very low transaction volume | Weekly revenue is so noisy that no statistical pattern can be reliably extracted |
| One dominant channel (>85% of spend) | Insufficient channel variation for the model to isolate relative effects |
| Very high organic baseline (>90% of revenue) | Advertising contribution is statistically invisible against organic noise |
| Mid-period structural break | A full rebrand, product pivot, supply chain crisis, or market exit creates a discontinuity the model cannot bridge |
| Very short trading history | Less than 12 months doesn’t give the model enough variation across seasons and campaign cycles |
What this means: The engagement was not suitable for MMM analysis from the start. This should have been identified at the qualification stage (Phase 0 of the SOP). If it wasn’t, the agency bears some responsibility.
Step 2 — What to Deliver Based on R² Level
Do not treat all low-accuracy outcomes the same. The response should be calibrated to how far the model fell short.
| R² Result | Confidence Level | Recommended Action |
|---|---|---|
| ≥ 0.75 | High | Deliver in full. Standard process applies. |
| 0.60 – 0.74 | Moderate | Deliver with explicit confidence caveats. Label all budget scenarios as directional only. Disclose the R² score in the report. |
| 0.45 – 0.59 | Low | Do not deliver budget recommendations. Deliver a Diagnostic Report instead. Discuss root cause and a data improvement roadmap with the client. |
| < 0.45 | Too low to be meaningful | Stop. Do not deliver any analytical conclusions. Deliver a transparent explanation, a root cause summary, and a proposed path forward. |
Rule: Never deliver budget reallocation recommendations based on a model with R² below 0.60. The risk of the client acting on unreliable findings — and attributing poor outcomes to your advice — is too high.
Step 3 — The Refund Question
This is the question every agency dreads. The answer is rarely a simple yes or no, and it depends heavily on how the engagement was structured and at what point the failure occurred.
The Core Principle
The agency is entitled to compensation for work genuinely done. The client is entitled to receive what was promised. If the deliverable cannot be provided as contracted, some form of remedy is appropriate. What that remedy looks like depends on the specifics.
Decision Framework: When Did the Failure Occur?
If the failure was caught at QG-1 (Data Readiness Gate — before the full analysis run)
This is the best-case scenario. The SOP is working correctly. The Data Audit Report has flagged data problems, and you have not yet committed significant modeling resources.
The right move:
- Charge only for the work completed to date: discovery, kickoff, and the Data Quality Audit.
- Do not charge the full project fee.
- Propose a path forward: the client improves their data, and the engagement resumes (with or without a reduced restart fee).
If the engagement was scoped with a milestone-based payment schedule (e.g., 50% upfront and 50% on delivery), this outcome is clean: the upfront payment covers the audit work, and the second payment is held until conditions are met.
Key lesson: This outcome is an argument for milestone-based pricing rather than full payment upfront.
If the failure occurred at QG-2 (Model Validity Gate — after the full analysis run)
The full engagement has been run: data collected, model executed, output reviewed. The model did not reach the accuracy threshold. Significant work has been done.
Whether a refund is appropriate depends on the root cause:
| Root Cause | Was It Foreseeable? | Recommended Position |
|---|---|---|
| Data problems that weren’t visible during the audit | Partially | Partial refund or credit is reasonable. The audit should have caught this, but some data issues only emerge during modeling. |
| Structural factors that should have been caught at qualification | Yes — agency error | A meaningful refund is warranted. The client should not pay full price for an engagement that was not suitable from the start. |
| Structural factors that were not foreseeable from the qualification data | No — neither party’s fault | A reduced fee for work completed (data, modeling, diagnostic) is reasonable. A credit toward a future re-run is a fair gesture of goodwill. |
| Client provided data that did not meet the agreed minimum threshold despite clear documentation | Client responsibility | No full refund required. Charging for work completed is defensible. Offer a re-run credit when data is improved. |
Alternatives to a Cash Refund
A full cash refund is almost never the right first offer. It immediately frames the engagement as a failure with no value delivered, and it does not help the client solve their actual problem. Consider these alternatives first:
1. Diagnostic Report as the Deliverable
Reframe the outcome as a different product: a detailed analysis of why the model struggled, what the data shows by way of directions (even if the model fit is low), and a concrete roadmap for what data improvements would enable a reliable analysis. Charge a reduced rate for this — not zero. Effort was expended.
2. Credit Toward a Future Engagement
Offer the client a credit (not necessarily the full fee) toward a future re-run once their data is in better shape. This keeps the relationship active, gives the client an incentive to solve the data problem, and gives your agency a path to recover the fee through future work.
3. Reduced Fee for a Confidence-Caveated Delivery
If R² is in the 0.60–0.74 range, offer to deliver the analysis at a reduced price — with explicit documentation of the model’s limitations, confidence levels on all outputs, and no budget recommendations beyond directional. Some clients prefer this to walking away with nothing.
4. Co-investment on a Re-Run
If the root cause is a specific fixable data problem, offer to run the model again once the issue is corrected — splitting the credit cost with the agency absorbing a portion as a goodwill gesture. This is appropriate when the agency bears partial responsibility for not catching the issue earlier.
5. Full Refund (Last Resort)
Appropriate when:
- The engagement should never have been sold — the qualification process failed to catch a clear disqualifying factor.
- The agency made demonstrable errors in the data preparation or model configuration.
- The client relationship is at serious risk and the long-term cost of the dispute outweighs the fee.
Even in this case, consider whether a partial refund (for work genuinely done) is more defensible than a full one.
Step 4 — How to Have the Conversation
The way you communicate a low-accuracy outcome matters as much as what you decide commercially.
Don’t: Present the finding as a surprise at the delivery meeting. The client should never be blindsided.
Do: Flag the issue as soon as it’s known — before the Strategy Session. A brief written update is far better than silence followed by bad news.
The conversation structure:
- Acknowledge what happened: be direct, non-defensive, and factual.
- Explain the root cause in plain language. Avoid statistical terminology.
- Present the options — what can be delivered now, and what would need to change to deliver more.
- Make a recommendation — don’t leave the client to choose from a menu of bad options without guidance.
- Agree on a path forward in writing before the meeting ends.
Step 5 — Prevent This From Happening Again
Most low-accuracy outcomes in the SME segment are preventable. The most effective mitigations:
At the proposal stage:
- Make the minimum data requirements non-negotiable and in writing — not just a checklist, but a signed acknowledgement that the engagement proceeds on the condition that the data meets the threshold.
- Apply the disqualification criteria from the SOP seriously. When in doubt about eligibility, do not propose — offer a paid data diagnostic first.
At kickoff:
- Assign a named data owner on the client side. Clients without a clear internal owner consistently provide incomplete data late.
- Build a data buffer. Ask for 24 months of history even if you only need 12. You are far more likely to get 18 months in practice.
During data collection:
- Run the Quick Data Check in MMM Pilot before committing to a full run. This is a low-cost validation that catches format errors, gaps, and structural issues before modeling begins.
- Invest in the Data Audit Report at QG-1. A thorough audit is the cheapest intervention in the process.
A Word on Repeat Offenders
Some industries and business types will consistently produce low-accuracy results. If your agency encounters more than one failed engagement in a specific segment, consider building an exclusion list — or a specialist qualification protocol — for that category.
This is not a failure. It is good business practice. Agencies that are honest about which clients they cannot serve well build stronger long-term reputations than those who take every project and deliver inconsistently.