Skip to content
Ameen Abozaid
Englishالعربية
SRB-013

Technology evaluation

  • 8 Lessons
  • 3.5 h practice (estimate)
Start module
On this page 9 topics

Overview

Topics

  • Proofs of concept
  • evaluation matrices
  • decision criteria

Learn first

3.5 h of practice (estimate)

Learning outcomes

  1. Frame an evaluation around the problem and weighted criteria before looking at products.
  2. Design a time-boxed proof of concept that tests the riskiest assumptions.
  3. Compare options with an evaluation matrix, including total cost and exit cost.
  4. Avoid common biases (novelty, sunk cost, vendor demos).

Lessons

  1. 01Start with the problem
  2. 02Criteria and weights
  3. 03Proofs of concept that answer questions
  4. 04Evaluation matrices
  5. 05Exit costs and lock-in
  6. 06Bias in technology choices
  7. 07Demo: evaluating two workflow engines in a two-day PoC
  8. 08Demo: scoring options and checking the matrix for bias

Lab and exercises

Lab

SRB-013-LAB1 · Workflow engine choice

Outcomes: 1–4 · Time: ~3.5 h

  1. Write the evaluation charter and weighted criteria for FulfillCo's returns process before naming products.
  2. Run a time-boxed PoC of one workflow engine against the hand-built saga baseline, testing failure paths.
  3. Build the evaluation matrix with a three-year cost and exit cost, a sensitivity check, and a bias checklist.
  4. Write the recommendation with an exit plan.

Acceptance: recommendation includes exit plan and cost over three years.

Exercises (assessment: evaluation report review)

IDOutcomeTask
SRB-013-EX011Rewrite the request "we should adopt a graph database" as a problem statement with constraints, options, and decision date.
SRB-013-EX021Define five criteria, mark must-haves, and assign weights for a message broker choice.
SRB-013-EX032Case: a PoC plan only shows a demo flow. List five riskier questions and the tests for them.
SRB-013-EX043Given a filled matrix, run a sensitivity check and decide whether the result is robust.
SRB-013-EX053Estimate three-year total cost for a managed workflow service versus self-hosted, listing every cost line.
SRB-013-EX063Write an exit plan for a vendor-hosted workflow service: triggers, data export, rewrite effort, test of the exit.
SRB-013-EX074Case: a lead picked a favorite and then wrote the criteria. Identify the biases and propose corrections.
Show solutions
  • EX01 Problem: relationship queries are slow and complex in current model; constraints: skills, operations, budget; options include optimized relational, graph database, do nothing; decision owner and date named.
  • EX02 Example criteria: delivery guarantees (must), operability, throughput headroom, team skill, cost; weights sum to 100 and are written before scoring.
  • EX03 Worker death mid-run, retries and duplicates, timeout handling, upgrade with running instances, observability of stuck instances. Each has a pass criterion.
  • EX04 Vary the top weights; if the winner flips with a small change, the choice is not robust and the contested criterion needs stakeholder decision.
  • EX05 Licence or hosting, infrastructure, engineering build time, operations and on-call, training, upgrades, support; compare over 36 months with stated assumptions.
  • EX06 Trigger examples: price rise, outage record, roadmap change. Export workflow definitions and state in an open format, estimate rewrite weeks, rehearse export and re-import of a sample.
  • EX07 Pre-chosen favorite and criteria fitted to it. Correct by rewriting criteria from the problem, having a second reviewer, scoring the do-nothing option, and re-running the matrix.
  • Outcome 2 is also covered by the lab PoC and Q03; outcome 4 by Q07-Q08.

Workplace evidence: a technology evaluation you ran and the decision it produced.

Quiz

Outcomes: O1 framing and criteria · O2 proofs of concept · O3 matrices, total cost, exit cost · O4 bias.

A self-check for you: nothing is graded, saved or sent anywhere. Choose an answer, then check it. You can try again as often as you like.

  1. 1The first step of an evaluation is:O1
  2. 2Criteria weights should be agreed:O1
  3. 3A good PoC primarily tests:O2
  4. 4The PoC code should be:O2
  5. 5In a scenario where the top option wins only if one weight stays high, you should:O3
  6. 6Total cost over three years should include:O3
  7. 7A vendor demo is risky evidence because it:O4
  8. 8"We already spent six months on this tool" is an example of:O4
  9. 9An exit plan should include:O3
  10. 10Why score the "do nothing" option?O1

Practical part

Submit the evaluation report for the workflow engine choice (SRB-013 assessment) and complete SRB-013-LAB1. Pass: at least 8/10 on the quiz and a report with criteria written before scoring, failure-path PoC evidence, three-year cost, and an exit plan.

Common mistakes

PoCs that only test the happy path; criteria chosen after the favorite.

Assessment

Evaluation report review.

Workplace evidence

A technology evaluation you ran and the decision it produced.

Career relevance

Technology assessment (ARC-008) and vendor evaluation (ARC-040).

Type to search levels, phases, modules and lessons.