Colour theme
Book a consultation

Testing AI productivity on a real task

Test AI on one real task, measure review time and errors, limit access and decide whether the result is useful enough to keep in your workflow.

A black alarm clock on a white desk between a computer monitor and a keyboard.

An AI tool can produce a draft quickly and still leave your team with more work. Someone has to check the facts, correct the wording and deal with mistakes. A useful pilot measures the effort needed to finish an acceptable piece of work.

Choose one recurring task with a clear reviewer and a result you can check. Drafting an internal procedure from source material you’ve already checked is a better starting point than allowing a tool to send client advice or approve payments on its own.

Define the task and an acceptable result

Write down the input, expected output and the person who signs off the result. For a procedure draft, the checks might include correct steps, complete warnings, accurate references and wording the team can follow.

Identify the errors that would make the result unusable. A missing safety step matters more than a heading that needs editing. If no one on the team can judge the output, resolve that gap before starting the pilot.

Record how the task is done today, including preparation, review and corrections. Use several representative examples so that one unusually easy case doesn’t set your expectations.

Set the information and access boundaries

Check that the tool and account you use are allowed to handle the information involved. Start with public or synthetic examples where possible. Don’t paste client records, staff information, passwords or confidential contracts into a trial account just to make the test realistic.

The Cyber Centre’s generative-AI guidance recommends keeping personal and sensitive business information out of prompts, checking how the provider collects and stores data, and verifying generated content. A business subscription won’t make every use safe.

Keep the pilot’s permissions narrow. A drafting test may need only a selected document, with no ability to search every shared folder or change records. If the tool reads emails, websites or uploaded files, treat instructions inside that material as untrusted. Review connected access separately using the AI-agent guide.

The AI policy guide covers who decides which tools and uses are allowed. The public-AI data guide covers input handling. Use those rules throughout the test.

Count checking and rework as part of the task

Run the same kinds of examples through the existing process and the AI-assisted process. Keep enough detail to compare:

  • Time spent preparing inputs and generating the draft.
  • Time spent checking sources, correcting errors and formatting the result.
  • Material omissions or incorrect statements caught before sign-off.
  • Cases abandoned because the output couldn’t be used.
  • Subscription and integration costs needed to repeat the task.

Have the reviewer check against the original records, without relying on another generated summary. For calculations, reconcile the figures independently. For code, use your normal code review and tests. For factual writing, open the cited source and confirm that it supports the statement.

Keep useful evaluation records without retaining unnecessary personal or confidential material. Counting prompts or logins won’t tell you whether the work got faster.

Decide whether to continue, change or stop

Agree on success criteria before interpreting the results. You might require acceptable quality across the sample, less total effort and no use of information outside the scope you set. Set the thresholds for this task; there isn’t one percentage that fits every business.

If the test fails, look at the cause. Missing source information may need a documentation fix. Heavy correction work may mean the task is unsuitable. A permissions problem needs to be resolved before expansion.

If the pilot succeeds, document the input pattern, reviewer, limitations and fallback process. Train a small group, check their results and revisit the test after significant tool or workflow changes. Keep a way to pause the AI-assisted step and complete the work manually.

Talk to us if you need help choosing a pilot and checking its effect on the work your team has to finish.


Featured Image Credit