Enterprise Search Renewal: Measure Trusted Answers, Not Indexed Documents

Indexing more documents is easy to sell. It is also a weak reason to renew an enterprise search or retrieval-augmented generation platform.

The platform may connect to SharePoint, Google Drive, Confluence, Slack, a ticketing system, and a dozen other repositories. That proves the connectors ran. It does not prove an employee received the right answer, saw the right source, respected the right permissions, or avoided an outdated policy.

Before renewal, stop counting indexed documents as the main outcome. Test whether the platform produces answers your people can use and your company can defend.

The decision should be whether to renew, resize, repair, bridge, or compare the platform based on trusted answer performance and full operating cost.

Start with decisions, not repositories

Build a test set from work people perform. Include the questions that affect revenue, customer commitments, employee actions, security, finance, and compliance.

A useful test record includes:

Field	What to capture
Business question	The question an employee would ask in normal language
Intended user	Role, department, geography, and access level
Approved answer	What a qualified owner says the answer should contain
Authoritative sources	The documents or systems that should support the answer
Freshness requirement	How current the source must be for this decision
Expected action	What the employee should do with the answer
Failure risk	The effect of a wrong, incomplete, stale, or overexposed answer
Result	Pass, fail, partial, or correct refusal
Evidence	Answer, citations, retrieved sources, time, cost, and defect record

Do not let the vendor build the whole test set. The vendor knows the demo. Your business owners know the work.

Include easy questions, conflicting sources, outdated documents, acronyms, incomplete prompts, role-specific answers, and questions the system should refuse to answer. An enterprise search product earns trust partly by knowing when it does not have enough approved evidence.

NIST’s AI Risk Management Framework Playbook supports this approach. Its Measure guidance calls for documented test sets, metrics, and tools. It also says performance should be demonstrated under conditions similar to the deployment setting and monitored in production. A generic vendor benchmark is not your deployment setting.

Separate retrieval failure from answer failure

When an answer is wrong, find where it broke.

The system may have retrieved the wrong documents. It may have found the right document but ranked an older version first. The answer model may have ignored a critical sentence. The source may be correct but inaccessible to the employee who owns the decision. The source itself may be wrong.

Those are different problems with different owners.

For each failed test, preserve the query, retrieved sources, source versions, generated answer, citations, user identity, configuration, and time. Then classify the failure:

No relevant source was retrieved
A relevant source was retrieved but ranked too low
A stale or superseded source won
The answer was not supported by the retrieved material
The answer omitted a required condition or exception
The citation did not support the sentence attached to it
The system answered when it should have declined
The result exposed content the test user should not see

Microsoft’s current Foundry observability guidance lists RAG-specific evaluation measures such as groundedness and relevance, along with production signals including latency, error rates, token consumption, and quality scores. Those measures are useful, but your renewal scorecard still needs business tests. A technically grounded answer can be complete nonsense for the process if it cites an obsolete policy.

Test permissions with identities, not slides

Permission-aware search is a buying requirement for most internal enterprise deployments. Treat it as something to prove.

Create controlled identities that represent common access cases: a standard employee, a manager, a restricted department, a contractor, and a recently transferred or terminated user. Give the identities known access to test content. Ask the same questions from each account and compare the retrieved sources, answer, citation preview, follow-up behavior, export, and conversation history.

Then change access at the source. Remove a user from a group. Restrict a document. Move a person into another business unit. Measure how long it takes for the search result and generated answer to reflect the change.

The timing matters. An identity provider update, connector sync, index refresh, cache, and existing conversation can each behave differently. “It uses your existing permissions” is not enough detail for a security review.

Google’s current Agent Search documentation gives buyers a concrete example of why architecture details belong in the evaluation. Google says data-source access control uses the organization’s identity provider to determine which documents an end user can receive. It also lists configuration limits, including that an access-controlled data store must be set that way when it is created and cannot be switched on or off later.

Your provider may work differently. Ask for the exact identity flow, group limits, sync behavior, cache behavior, failure handling, audit records, and rebuild requirements. Then test them.

Pair this review with the enterprise AI data-retention questions if the platform stores prompts, retrieved passages, answer history, feedback, or administrator logs. Search permissions and retained conversation data can create different exposure paths.

Prove freshness and deletion

A useful answer from last year’s policy can still create this year’s problem.

Choose documents with known owners and expiration dates. Update one. Supersede one. Delete one. Revoke access to one. Add a conflicting draft that should not become authoritative. Record when each change appears in search results and generated answers.

Check more than the user interface. Test direct search, generated answers, follow-up questions, source previews, saved conversations, APIs, exports, analytics, and caches where they exist.

Azure AI Search documentation shows why this cannot be assumed. Microsoft states that change detection for Azure Storage content occurs automatically through timestamps, but deletion detection does not. Its indexer guidance requires a deletion strategy to avoid orphaned search documents. It also warns that if the deletion policy was not in place before the first indexer run, previously deleted documents can remain in the index and may require a new index and indexer.

That is a product-specific example of a broad buyer concern: deleted at the source does not always mean deleted from every retrieval path.

Require a written freshness and deletion design for each connector. It should name the sync interval, change signal, deletion signal, failed-sync alert, document owner, stale-content treatment, and evidence retained after a correction.

Count correction work as part of the price

Enterprise search does not maintain knowledge on its own. Someone has to resolve duplicate policies, assign content owners, review failed answers, tune retrieval, fix permissions, investigate connector errors, and retire bad sources.

Measure that work for a normal month. Capture:

Administrator hours for connector and identity work
Business-owner hours for source review and answer approval
Security and compliance review time
Support cases and escalation time
Failed-answer review and correction volume
Repeated questions that still end in manual research
Model, query, indexing, storage, connector, and API consumption
Professional services and custom integration costs

Avoid fake precision. If the team has not tracked the work, start a short measurement period before renewal rather than inventing an annual savings number.

The same rule applies to adoption. Monthly active users and query counts show activity. They do not prove value. Link usage to a defined workflow outcome such as reduced policy-research time, fewer escalations, faster case preparation, higher first-answer acceptance, or fewer corrections after an answer is used.

Our enterprise AI productivity metrics guide explains how to connect AI usage to a business outcome. For search, add answer acceptance, source quality, correction effort, and access-control failures to the operating view.

Force the commercial model onto the evidence

Reconcile every charge with actual usage and a required business outcome.

Include user licenses, premium search or AI tiers, indexed volume, connector fees, query consumption, model tokens, storage, data transfer, API access, sandbox environments, implementation work, support, and internal administration. Identify minimum commitments and overage rates. Separate recurring fees from remediation work you need only because the original deployment never reached an acceptable state.

Then ask the vendor to price three scenarios:

The current scope with corrected quantities and consumption assumptions
A narrower scope limited to proven workflows and authoritative sources
A growth case with written volume assumptions and rate protections

If a bundled software renewal includes the search feature, do not let a discounted bundle end the analysis. Use the AI software add-on renewal checklist to separate the AI decision from the base software decision.

Make one of five renewal decisions

Renew as proposed when the test set shows useful, supported, permission-correct, current answers and the full cost fits the value.

Renew with corrections when the platform fits but licenses, data sources, controls, service levels, pricing, or contract terms need work.

Resize when a smaller set of users, repositories, or workflows produces most of the proven value.

Use a short bridge when defects can be fixed, but the evidence will not be ready before the deadline. Put remediation milestones and exit rights in writing.

Compare alternatives when the provider cannot prove permission behavior, source traceability, freshness, correction ownership, export, or a credible cost model.

Your renewal packet should include the business test set, answer results, permission tests, freshness and deletion tests, open defects, operating labor, consumption data, pricing scenarios, contract changes, and named decision owners.

Indexed documents belong in that packet. They just do not get the deciding vote.

If your enterprise search or RAG agreement is approaching renewal, request a Contract and Spend Risk Review. Bring the agreement, proposal, connector inventory, usage export, test results, access model, support history, and current cost data. Catch Advisors will help you decide what to renew, repair, resize, or compare before another term turns deployment activity into assumed value.