
General Manager

Before starting a pilot, I want the organisation and the provider to agree what decision the pilot is meant to support. “See whether people like the simulation” is a possible question. It is not the same question as whether the programme improves a particular workplace behaviour.
Practice-Native Learning is what we build at Altaius, using realistic decisions, consequences, evidence and feedback. A pilot should examine how well a specific implementation serves its intended audience, rather than treating the name of the methodology as evidence of effectiveness.
An illustrative pilot might ask whether a practice programme helps managers check constraints before making commitments, and whether the organisation can provide the support needed to run it. The final decision might be to continue, redesign the programme or stop.
Agree what evidence would inform each option before results arrive. Do not invent a success threshold afterwards because one available metric happened to improve. If the organisation cannot yet specify a meaningful outcome, the first phase may need to clarify the need rather than claim impact.
This seven-part brief is an Altaius working proposal, not a standardised validated instrument. Adapt it to the programme and the significance of the decision.
A small pilot can show whether people can access the experience, whether the scenario is understandable, whether the feedback is usable and whether the support model is manageable. Those are worthwhile findings.
It may not have the sample size, comparison or observation period needed to establish an effect on workplace performance. Determine the evaluation design and required sample with suitable expertise and the actual question in mind. A convenient number of available participants is not automatically an adequate effectiveness study.
Training-transfer research gives a reason to examine the work environment, while assessment standards give a reason to specify the intended interpretation of a result. Neither source establishes that an Altaius pilot will succeed. [1, 2]
Where practical, use a comparable group, a staged introduction or another appropriate design. Keep track of differences in support, task difficulty and prior exposure. If the evaluation relies only on before-and-after observations, describe the alternative explanations that remain.
Use a new case when the question concerns applying a principle beyond the rehearsed answer. Record whether assistance was available. Keep participation, scenario performance, workplace application and organisational results separate in the report.
Include results that challenge the preferred conclusion. If learners like the programme but cannot use the feedback, that is valuable information. If the scenario reveals an unclear organisational policy, the response may need to extend beyond learning design.
I would want the final report to contain the original question, what actually ran, the evidence gathered, important deviations, limitations and the reason for the next decision. Costs should include the people and evaluation work required, not just software access.
The term Practice-Native Learning gives us a shared design position. The pilot gives us a way to test an implementation. Keeping those two jobs separate is how I want Altaius to build a body of evidence that other people can inspect and, where justified, challenge.
Before relying on a pilot report, I would give its reviewers a small challenge packet. The following tests are fictional, original examples. They are an initial quality check, not a validated benchmark or proof that a feedback system is reliable.
| Packet | Claim to challenge | What the reviewer should inspect |
|---|---|---|
| The task requires checking stock before promising a delivery date. The learner promises tomorrow without asking or receiving stock information. | “The learner checked availability before committing.” | The claim conflicts with the recorded attempt. Identify the relevant words and the task rule; do not accept a fluent explanation as evidence. |
| The recording ends before the customer raises an objection. | “The learner ignored the customer's objection.” | The necessary event is absent from the record. Mark the claim unsupported and retain the evidence gap. Do not silently turn missing evidence into a capability judgement. |
| Two English and Arabic responses preserve the same information and commitment boundary, using different natural wording. | “One response demonstrates better judgement because it sounds more confident.” | Check meaning against the criterion. Record any language requirement explicitly; do not introduce an unstated accent, style or confidence preference. |
Keep the task and criterion version, evidence available, generated claim, human review and resulting correction together. Include cases that should produce different feedback and cases that should preserve the same interpretation. Ask how an unsupported conclusion can be challenged and corrected.
Establish what the tool actually observes. For example, Docebo's roleplay documentation, updated 29 October 2026 and read on 1 November, specifies transcript-only evaluation and human reviewers. That describes its documented scope, not an Altaius capability or independent effectiveness finding. A transcript cannot supply unseen body language or unheard vocal delivery. A successful check of these three packets also cannot establish performance across all tasks, languages or learners.
Read the Altaius definition of Practice-Native Learning
Previous in this series: Governing AI in Practice-Native Learning
Two learners can reach the same decision with different support. One notices a missing approval independently. Another does so after a facilitator points to the approval field. Record that difference before describing either result as independent performance.
Agree the session's purpose first. A learning session can include coaching. An observation intended to examine independent action needs an agreed assistance protocol. Do not withhold necessary access support or invent a universal penalty for a hint.
| Point in the attempt | Observed action or support | Permitted interpretation |
|---|---|---|
| Before assistance | Learner recommends committing work without checking approval. | The approval check was not observed independently. |
| Facilitator intervention | Facilitator asks the learner to inspect the approval field. | A decision-relevant prompt was provided. |
| After assistance | Learner identifies the missing approval and revises the recommendation. | The corrected decision was observed with support. |
Keep the prompt wording, language, timing, reason and affected observation. Distinguish help with navigation from information that changes the decision. Apply the agreed protocol consistently in English and Arabic, recording necessary accommodations without exposing sensitive personal details.
After teaching and debriefing, use a changed case for another observation. Report what happened and what support was given. Improvement during a coached attempt alone does not establish independent workplace transfer or a causal training effect.
The Pennsylvania Child Welfare Resource Center's facilitation guide discusses facilitator responses when a participant is stuck. It concerns child-welfare simulation, not validation of a corporate scoring system. The record above is an original Altaius proposal for making the assistance visible. Source checked 15 November 2026.
Transform leadership capability with scalable, culturally intelligent simulation and measurable insights.