Demo: reading a pilot's results that look better than they are
On This Lesson 3 topics
This lesson is planned. The outline below lists the topics it will cover. The full lesson has not been published yet.
In this lesson
critique pilot data that appears positive but hides problems; prepares SRB-022-LAB1.
Draft outline details
Step by step
- Read the headline: pilot teams shipped 25% more pull requests.
- Check the baseline and control: the control team also rose, as a release deadline passed.
- Check quality: defect rate rose slightly, and review time per pull request grew.
- Planned mistake: report the headline to executives. Fix: present throughput, quality, review load, and survey together, with caveats.
- Identify biases: self-selected volunteers, short window, novelty effect.
- Propose the next step: extend the pilot, fix review capacity, retest.
Check your understanding
What did the control team's rise reveal?
Show answer
Part of the improvement was not due to the tool.
Practice
Apply this lesson with the module's lab, exercises and quiz.
Marking a lesson complete is optional and you can undo it any time. Your progress stays in this browser.