01 learn & understand
→
02 make & experiment
→
03 profile
→
04 get practical solution & apply
01 · learn & understand
your problem
Mira is a backend engineer at a mid-size fintech. Her on-call rotation had become a nightmare — two or three 3am pages a week, most of them noise, a few of them real. Her team was burning out.
"what do I actually need to know to build a system that handles 80% of incidents without paging a human?"
layers surfaced:
harness
skills
tools / mcp
multi-agent
context
oss vs frontier
02 · make & experiment
three stacks running
A · Claude-native
canonical agentic loop · frontier-only
harnessClaude Agent SDK
routingSonnet 4.5
workersSonnet 4.5
mcpdatadog · pagerduty · linear
17/18 ✓
·$0.42/inc
·p50 4.1s
B · OpenAI-native
plan-first · parallel tool calls
harnessOpenAI Agent SDK
routingGPT-5
workersGPT-5
mcpdatadog · pagerduty · linear
16/18 ✓
·$0.51/inc
·p50 3.8s
C · hybrid OSS
orchestrator + workers · fanout/fanin
harnessOpenAI Agent SDK
routingSonnet 4.5
workersQwen3-235B
mcpdatadog · pagerduty · linear
14/18 ✓
·$0.07/inc
·p50 5.2s
03 · profile
three shapes — same eval, different strengths
A · Claude-native
strong & rounded · expensive
strong everywhere, but pays for it — shape leans heavy on the right; OSS-readiness near zero.
B · OpenAI-native
fast · weaker on tool-loops
fastest of the three — but tool-loop dents on the right; drops calls when MCP chains get deep.
C · hybrid OSS
surprising · sharp asymmetry
strong on routing & tool-stab, weak on long context — leans hard left; cheap.
trace · case replay
why each stack reached its score
case #02 · 03:14 AM · datadog: payments-api 5xx burst · 412 errors / 90s
classify(alert)
A · Claude
→ "p1 · upstream db saturation" · routes to diagnose-5xx
✓
B · OpenAI
→ "p1 · 5xx · likely db" · plans 4 parallel checks
✓
C · hybrid
→ router (Sonnet) classifies, dispatches to diagnose-5xx (Qwen)
✓
diagnose(query datadog + pagerduty)
A · Claude
3 tool calls in sequence · finds db connection-pool exhaustion
✓
B · OpenAI
parallel calls · synthesizes from 4 sources — invents "redis-cache" service that doesn't exist
✗
C · hybrid
Qwen worker · narrow-scope CoT · returns "db pool · likely cause" · cited datadog rows
✓
act(page or escalate)
A · Claude
draft postmortem · page on-call · auto-resolved at 03:21
✓
B · OpenAI
cascading hallucinated services · pages on-call with wrong runbook
✗
C · hybrid
router (Sonnet) escalate-senior · sends summary · resolved 03:19
✓
This is why C wins on tool-stab in the radar — Qwen workers stay tightly scoped, so they don't synthesize fake context. B's tool-loop dent isn't a model problem — it's a stack-shape problem. The trace shows it at the diagnose step, not the verdict.
04 · get practical solution & apply
Mira didn't pick a winner. She composed a hybrid.
Frontier where the agent reasons, OSS where it executes — exactly what each profile said each was strong at. Sonnet 4.5 routes, Qwen3-235B workers handle diagnosis & postmortem drafting.
SLA · this month
99.4%
5 weeks · 1,247 incidents
handled without paging human
handled without paging human
Cost / incident
$0.13
−69% vs stack A
(oss workers carry the load)
(oss workers carry the load)
P50 latency
4.4s
stable across hour-of-day
no thundering-herd dents
no thundering-herd dents
01 learn & understand
→
02 make & experiment
→
03 profile
→
04 get practical solution & apply
01 · learn & understand
your problem
Liu runs ops at a wealth management firm. Every month, her team lost two weeks to manual close work — reconciliation, compliance checks, client report generation, all done by hand across three different systems.
"what do I actually need to know to build agents that automate our monthly financial close without touching production data wrong?"
layers surfaced:
harness
skills
tools / mcp
multi-agent
audit / compliance
context
02 · make & experiment
three stacks running
A · Claude-native
single-agent · long-context · audit-trail first
harnessClaude Agent SDK
routingSonnet 4.5
workersSonnet 4.5
mcperp · gl · compliance
12/12 ✓
·$2.10/close
·p50 22s
B · OpenAI-native
plan-first · structured outputs
harnessOpenAI Agent SDK
routingGPT-5
workersGPT-5
mcperp · gl · compliance
11/12 ✓
·$1.80/close
·p50 18s
C · hybrid OSS
orchestrator + workers · audit-locked
harnessOpenAI Agent SDK
routingSonnet 4.5
workersQwen3-235B
mcperp · gl · compliance
10/12 ✓
·$0.40/close
·p50 28s
03 · profile
three shapes — same eval, different strengths
A · Claude-native
deep context · expensive
complete & defensible — but $2.10 per close at this volume is real money.
B · OpenAI-native
fast · brittle on long PDFs
fastest — drops one case on long-form policy PDF; latency wins, audit-trail middling.
C · hybrid OSS
cheap · uneven on edge cases
5× cheaper — but 2 misses on cross-system reconciliation; audit-trail soft.
trace · case replay
why audit-trail is load-bearing here
case #07 · proration on a mid-cycle plan downgrade · client owes $174.32 · audit needs each step cited
retrieve(policy + ledger)
A · Claude
long-context recall · cites policy §4.2 · pulls correct ledger row
✓
B · OpenAI
structured retrieval · finds policy · misses one historical ledger entry
✓
C · hybrid
router → reconcile worker (Qwen) · cites policy correctly
✓
compute(proration)
A · Claude
$174.32 · shows formula · all 3 systems agree
✓
B · OpenAI
$172.10 · off by historical entry (above)
✗ −$2.22
C · hybrid
$174.32 · matches A · cheaper compute
✓
audit(write step-by-step)
A · Claude
complete trail · every retrieval cited · compliance review < 1 min
✓
B · OpenAI
summary trail · misses intermediate calc · compliance asks 2 follow-ups
⚠
C · hybrid
Qwen trail thin on context · auditor flags 1 step
⚠
Audit is non-negotiable here — Liu's compliance team has to defend every number. Long-context recall and explicit citations are the load-bearing axes, not raw accuracy.
04 · get practical solution & apply
Liu shipped A for the audited core, C for the bulk reconciliation.
Sonnet 4.5 owns proration + audit writes. Qwen workers handle high-volume reconciliation on rows where the audit trail comes from the source data itself, not the model. Two stacks, one canvas.
Close time
2 days
was 14 days · −86%
same audit standard
same audit standard
Cost / close
$0.78
−63% vs A-only
(C handles 80% of rows)
(C handles 80% of rows)
Compliance · 3 mo
0
follow-up requests
from audit since launch
from audit since launch
01 learn & understand
→
02 make & experiment
→
03 profile
→
04 get practical solution & apply
01 · learn & understand
your problem
Kai is an ML engineer at a computer vision startup. Every model retrain was a 3-day manual slog — data prep, training runs, eval, registry push, canary deploy, all orchestrated by him personally. He was the bottleneck.
"what do I actually need to know to build an agent system that runs the full CV pipeline automatically — retrain, eval, and deploy — without me?"
layers surfaced:
harness
multi-agent
orchestration
tools / mcp
eval-loop
canary
02 · make & experiment
three stacks running
A · Claude-native
canonical agentic loop · single brain
harnessClaude Agent SDK
routingSonnet 4.5
workersSonnet 4.5
mcps3 · w&b · sagemaker
8/8 ✓
·$8.40/run
·p50 47m
B · OpenAI-native
plan-first · graph-of-tasks
harnessOpenAI Agent SDK
routingGPT-5
workersGPT-5
mcps3 · w&b · sagemaker
7/8 ✓
·$9.10/run
·p50 41m
C · hybrid OSS
orchestrator + 6 specialist workers
harnessOpenAI Agent SDK
routingSonnet 4.5
workersQwen3-235B
mcps3 · w&b · sagemaker
8/8 ✓
·$1.20/run
·p50 52m
03 · profile
three shapes — same pipeline, different strengths
A · Claude-native
capable but expensive
does it all — but $8.40 × N retrains/month is the bill that ends the project.
B · OpenAI-native
parallel-friendly · brittle on graph
parallel wins on parallelism — but stalls on the dependency chain (eval blocks deploy).
C · hybrid OSS
workhorse shape · cheap parallel
strongest on orchestration + cost — Sonnet plans, Qwen workers do the work cheaply.
trace · pipeline replay
why orchestration is the load-bearing axis
retrain run · 2026-04-22 · new label batch from labelers · 8 stages · 47-min target
prep(data) · stage 1-2
A · Claude
single brain plans + does · clean
✓
B · OpenAI
parallel chunks well · 4× speed-up
✓
C · hybrid
orchestrator dispatches 4 prep workers · in parallel
✓
eval(canary) · stage 5
A · Claude
runs eval · catches mAP regression on subset 3
✓
B · OpenAI
tries to parallelize eval + deploy · races · skips canary check
✗ deploy blocked
C · hybrid
orchestrator gates · eval worker reports · holds deploy
✓
deploy(canary → prod)
A · Claude
canary 5% · monitors 20m · promotes
✓
B · OpenAI
manual intervention required · graph stalled
⚠
C · hybrid
deploy worker promotes after canary signal · clean
✓
This is why C wins on orchestration — explicit gates between stages means a bad eval blocks a bad deploy. B's parallel-everywhere plan loses precisely the dependency it most needs.
04 · get practical solution & apply
Kai shipped C — and stopped being the bottleneck.
Sonnet orchestrates, Qwen workers handle prep, train, eval, registry push, canary monitoring. 6 specialist workers, one router. He runs retrains by approving a Slack message instead of a 3-day slog.
Retrain time
52 min
was 3 days · −98%
fully unattended
fully unattended
Cost / retrain
$1.20
−86% vs A-only
retrain rate 4×/week feasible
retrain rate 4×/week feasible
Kai's hours
~0
approves canary in Slack
back to model research
back to model research