CMS secret shopper calls follow a predictable script — and most plans fail for the same four completely preventable reasons.
The Call Your Directory Doesn't Know About
CMS conducts secret shopper surveys as part of its Medicare Advantage oversight program. The surveys are not announced, not scheduled, and not limited to new plans or plans already under scrutiny. Any MA plan, in any market, can receive a secret shopper call in any contract year.
Most network teams know this. What they typically don't know is exactly what the caller is testing — and what constitutes a failure. The gap between "we know secret shoppers exist" and "we understand what they're looking for" is where plans get into trouble.
Here is what actually happens on a CMS secret shopper call, and what each type of failure triggers.
What the Secret Shopper Actually Does
The caller uses your published provider directory — the same one your members use — to select a provider. They then call that provider's listed phone number and attempt to book an appointment as a new patient enrolled in your plan. They are testing four specific things:
- Reachability: Does the phone number connect to a live person or functioning scheduling system? A disconnected number, perpetually full voicemail, or phone that rings without answer is an immediate failure.
- Location accuracy: Is the provider still practicing at the address listed in your directory? If the provider has moved, closed, or the address is otherwise wrong, that is a directory accuracy failure.
- Plan participation: Does the provider actually accept your plan? "We don't take [plan name]" is a critical failure — it means your directory lists a provider who has no active contract or whose contract has lapsed.
- Appointment availability: Is the provider accepting new patients, and can they offer an appointment within the CMS-required timeframe? For primary care, that is typically 30 days for a routine appointment. For specialists, it varies by specialty but generally ranges from 15 to 60 days.
The secret shopper isn't trying to catch you. They're testing exactly what a confused 68-year-old enrollee would experience trying to use your plan. That's the standard.
The Four Most Common Failure Modes
Disconnected or non-functional numbers are the most common single failure. A provider who moved practices left the old number behind. A solo practitioner retired and the number was reassigned. A group practice changed its scheduling system and the published number is now a fax line. Each of these appears in your directory as a valid listing until someone actually dials the number. CMS counts each disconnected number call as a failed survey response, which feeds directly into your directory accuracy score.
Wrong address failures follow closely behind. Providers move more frequently than most plans realize — especially in competitive urban markets. Without proactive re-attestation, your directory drifts from reality at a rate of 15–25% annually in high-turnover markets. A wrong address doesn't just fail the secret shopper — it also fails your members, which is ultimately the point CMS is making with the survey.
"We don't take that plan" is the failure that causes the most regulatory concern, because it indicates a systemic data problem. Either the provider's contract has terminated and your directory hasn't been updated, the provider was listed erroneously in the first place, or a credentialing issue has caused a gap between the directory and the actual contracted network. Any of these triggers an inquiry into your directory accuracy processes.
Appointment timeframe failures are tricky because they reflect real capacity, not just data quality. A provider who is technically accepting new patients but can't offer an appointment within the required window is still a survey failure. In underserved specialties and rural markets, this is often the hardest problem to solve — and it's the one that most directly connects to whether your network actually serves your members.
What a Failed Call Actually Triggers
CMS uses secret shopper results to calculate your directory accuracy rate — the percentage of listed providers who pass all four tests. Plans that fall below CMS thresholds receive a compliance warning. Plans with persistent or widespread failures receive deficiency notices and, ultimately, the heightened oversight and enrollment action consequences that come with them.
A single failed call doesn't sink a plan. A pattern of failures in a county or specialty area — even with an overall directory accuracy rate that looks acceptable in aggregate — can trigger targeted enforcement. CMS's analysis has become more geographically and specialty-specific over time. Aggregate scores are less protective than they used to be.
How to Prepare Without Waiting for CMS to Call First
The playbook for secret shopper readiness is the same as the playbook for running a high-quality network: continuous re-attestation, proactive outreach to providers with data anomalies, and systematic appointment availability monitoring in your highest-risk counties and specialties.
- Conduct your own internal secret shopper calls on a quarterly basis, sampling at minimum 5% of your directory across each service area.
- Prioritize re-attestation for providers who have not responded to a directory verification in more than 90 days — these are your highest-risk listings.
- Track appointment availability data, not just network participation. A provider who passes the first three tests but can't see a new patient within the required timeframe is still a problem.
- Build triggers for automatic directory flags when contract termination notices are received — the window between termination and directory update is where "we don't take that plan" failures originate.
Secret shoppers don't find surprises. They find the gaps that already exist in your directory. The plans that pass consistently are the ones running ongoing monitoring programs — not the ones who spend the two weeks before their assumed survey window doing emergency outreach.