How to Evaluate a Business Records Sample for AI Training

How do you decide whether a sample is useful?

Evaluate it against one intended task, with acceptance criteria set before the review. Check where the records came from, what portion of the source they represent, whether cases can be followed from input to outcome, and what remains after filtering. Keep rights and privacy review separate from technical quality. Record unknowns instead of turning a promising sample into a claim about the full corpus.

Start with provenance and permitted use

Document the source systems, date ranges, selection method and any transformations made to the sample. Confirm that the proposed reviewer, purpose, access method and retention are within the written evaluation permission. A technical assessment cannot supply missing rights. If the permission is unresolved, hold sample access until it is settled.

Measure usable coverage

Count records received, records excluded and records that remain usable after format checks, deduplication and missing-field review. Report the reasons for exclusions. A large raw archive can yield few complete cases. A sample chosen by the owner is not a random draw; treat it as non-representative unless a selection method was agreed in advance. Compare the sample's period and systems with the inventory before extrapolating.

Inspect complete workflows

For a few cases, ask whether you can see the initial request, relevant context, actions, revisions and outcome. Check whether links between email, tickets, documents and structured records survive preparation. Note where an expert would need to infer a missing step. If the buyer needs task-shaped examples, test whether a prompt and a defensible answer or grader can be derived from the evidence.

A case that can be followed this way is what researchers call a trajectory. For long-horizon work, meaning tasks that take a person hours or days, count how many cases in the sample run the full length you need. METR's research measures models by the length of task they can complete, and a sample with many short exchanges and few complete jobs will not support long tasks. If the records are meant for a reinforcement learning (RL) environment, check that they cover the systems the environment will reproduce.

Review privacy and sensitivity

Check for direct identifiers and contextual details that could still identify a person or reveal restricted information. Document exclusions and residual risk. Removing names alone is not a complete privacy assessment; the risk depends on what the recipient can combine with the data. If direct identifiers or restricted material appear, stop the review, notify the provider and handle the affected records only as the written permission allows. A separate qualified review may be needed before a buyer sees or uses a transformed sample. For US health records, HHS guidance describes the two recognized de-identification methods, Expert Determination and Safe Harbor.

Run the buyer's acceptance test

Use a held-out task or benchmark chosen for the buyer's intended use. Training or fine-tuning on a sample needs to be expressly allowed by the written evaluation permission; that permission should address derived outputs, embeddings and model changes when access ends. If training is allowed, compare performance and cost against a baseline and have someone independent of sample preparation assess the result. If it is not allowed, use only the agreed non-training checks. Report uncertainty and failure cases. If no model test was run, say “not measured.” Structural checks, usable-token counts and expert impressions can guide a next step, but they are not evidence of model improvement.

End with a short decision record: what was measured, what passed, what failed, what remains unknown and what the buyer would need for a larger evaluation. The Measure section of NIST's AI RMF Playbook is a useful reference for documenting measurement methods and evaluation in the intended context. To prepare the request, read How to Write a Business Data Request for AI Training; for handling requirements, see Security and data handling.

Frequently asked questions

Does a good sample prove the full corpus is good?

No. A sample review is limited to the permission, selection and measurements actually completed. A sample chosen by the owner is not a random draw, so treat it as non-representative unless a selection method was agreed in advance.

What makes a sample usable for long-horizon tasks?

Complete cases. You should be able to follow a case from the initial request through the context, actions and revisions to the outcome, and the links between email, tickets, documents and structured records should survive preparation.

Can a buyer train or fine-tune a model on a sample?

Only if the written evaluation permission expressly allows it. That permission should also address derived outputs, embeddings and model changes when access ends. If training is not allowed, use only the agreed non-training checks.

What should a reviewer do if identifiers appear in a sample?

Stop the review, notify the provider and handle the affected records only as the written permission allows. Removing names alone is not a complete privacy assessment.

Is a token count or an expert's impression evidence of model improvement?

No. Structural checks, usable-token counts and expert impressions can guide a next step, but they are not evidence of model improvement. If no model test was run, report the result as not measured.