General Manager
Before starting a pilot, I want the organisation and the provider to agree what decision the pilot is meant to support. “See whether people like the simulation” is a possible question. It is not the same question as whether the programme improves a particular workplace behaviour.
Practice-Native Learning is the methodology we are developing at Altaius around realistic decisions, consequences, evidence and feedback. A pilot should examine how well a specific implementation serves its intended audience, rather than treating the name of the methodology as evidence of effectiveness.
An illustrative pilot might ask whether a practice programme helps managers check constraints before making commitments, and whether the organisation can provide the support needed to run it. The final decision might be to continue, redesign the programme or stop.
Agree what evidence would inform each option before results arrive. Do not invent a success threshold afterwards because one available metric happened to improve. If the organisation cannot yet specify a meaningful outcome, the first phase may need to clarify the need rather than claim impact.
This seven-part brief is an Altaius working proposal, not a standardised validated instrument. Adapt it to the programme and the significance of the decision.
A small pilot can show whether people can access the experience, whether the scenario is understandable, whether the feedback is usable and whether the support model is manageable. Those are worthwhile findings.
It may not have the sample size, comparison or observation period needed to establish an effect on workplace performance. Determine the evaluation design and required sample with suitable expertise and the actual question in mind. A convenient number of available participants is not automatically an adequate effectiveness study.
Training-transfer research gives a reason to examine the work environment, while assessment standards give a reason to specify the intended interpretation of a result. Neither source establishes that an Altaius pilot will succeed. [1, 2]
Where practical, use a comparable group, a staged introduction or another appropriate design. Keep track of differences in support, task difficulty and prior exposure. If the evaluation relies only on before-and-after observations, describe the alternative explanations that remain.
Use a new case when the question concerns applying a principle beyond the rehearsed answer. Record whether assistance was available. Keep participation, scenario performance, workplace application and organisational results separate in the report.
Include results that challenge the preferred conclusion. If learners like the programme but cannot use the feedback, that is valuable information. If the scenario reveals an unclear organisational policy, the response may need to extend beyond learning design.
I would want the final report to contain the original question, what actually ran, the evidence gathered, important deviations, limitations and the reason for the next decision. Costs should include the people and evaluation work required, not just software access.
The term Practice-Native Learning gives us a shared design position. The pilot gives us a way to test an implementation. Keeping those two jobs separate is how I want Altaius to build a body of evidence that other people can inspect and, where justified, challenge.
Read the Altaius definition of Practice-Native Learning
Previous in this series: Governing AI in Practice-Native Learning
