Developer tools
Pilot
Developer productivity: DevQuest, Eng Hub and Quarterly Planning
I reframed DevQuest from gamified speed to review confidence, and the stakeholder goal grew from completion time alone to completion time plus PR quality.
My part
Designer: journey maps, role model, direction, AI-built prototypes
Duration
Jul 2025 – Jul 2026
Crew
PM, UX research, engineering squad
Ships on
VS Code · Azure DevOps (pilot)
Context & role
Azure engineering systems teams build the tools Microsoft engineers use to plan, write, review and ship code. From July 2025 to July 2026 I designed across three of them: DevQuest, a VS Code extension for pull request work aimed at Azure Core developers; a redesign of Engineering Hub, the documentation platform engineers rely on; and a quarterly planning concept for Azure DevOps. On DevQuest I was the designer in a squad with a PM, a UX researcher and engineering. I owned the journey maps, the author and reviewer role model, the design direction and the AI-built prototypes; the PM ran the concept interviews with research guidance, and engineering built the pilot.
Stakes
Long pull request reviews cost Azure Core developers days, and the work was split across VS Code, Azure DevOps, GitHub, Teams and browser tabs. The existing review bot gave general documentation feedback rather than analysis of the change, so preventable issues surfaced late, when they were most expensive to fix. The brief measured success by completion speed, which can't tell a fast, careless review from a fast, well-evidenced one.
Reframe
Asked: make PRs faster with gamification. Reframed: make PRs review-ready.
An early iteration built the brief as given: a leaderboard for participation and completion speed. Building it exposed the hidden theory that rewarding activity produces better behaviour, and teams reviewed so differently that one speed target fit neither culture. I redefined the problem as review readiness: developers move faster when a PR arrives with the evidence a reviewer needs, so quality is the cause and speed is the byproduct. With stakeholders I used airport pre-check as the shorthand: help people make better decisions, not do more.
Illustration: a leaderboard rewards activity; the review-ready view names the one missing piece of evidence.
Evidence
4 of 5 developers asked for quality, not speed
Concept interviews with five inner-loop developers (December 2025 and February 2026, PM-led with research guidance) set the direction. Four of five raised PR quality as the theme that mattered, and they valued clearer signals, inline feedback and less tab switching. Participants rated the concept 3.75 out of 5. With five people this is directional, and we said so; final UI, feasibility and impact were listed as follow-up work.
01
Speed was a proxy. A completion-time metric can't tell a fast, careless review from a fast, well-evidenced one.
02
Quality was the need. Four of five developers raised PR quality, not speed, as the theme that mattered to them.
03
Docs fail on trust, not findability. Readers found pages but couldn't tell which one was authoritative, so they judged by breadcrumbs and authors.
Key decision
Drop the leaderboard. Hold PRs at "Not Review-Ready".
We explored three directions: keep gamification and optimize completion time as briefed, consolidate PR work inside VS Code, or add an evidence layer that marks a PR "Not Review-Ready" until build, test, risk, security and customer-impact evidence is in place. The team kept consolidation, which shipped in the pilot, and I moved the design focus from the leaderboard to evidence. I chose review-readiness evidence over a speed leaderboard because the research showed quality was the need and a leaderboard rewards the wrong behaviour, accepting a slower path to a headline speed number and more friction for authors before review.
Second call
Show who owns a doc, without sending questions to the wrong person
Engineering Hub had a smaller trade-off of the same kind. Readers wanted to see who owned a page, but one warned that "last edited by" sends questions to the wrong person, and another warned against unmoderated Q&A. I kept author visibility optional and tied feedback to existing workflows, trading some transparency for fewer misdirected questions. The platform team built its homepage requirements from these reader findings: search kept prominent, an Ask Copilot entry point added, favorites easy to reach, and static links cut to the top 30 percent by page views.
Illustration: three directions explored, one kept and one chosen.
Craft
Designing for what the AI can't prove
The confidence prototype treats AI output as evidence to be checked, not an answer to trust. Each PR shows a score ("Confidence: 1/8 signals passed"), and each signal states pass or fail with one line of reason. The testing detail separates what existing tests prove from what they don't, lists the questions a reviewer should still ask, and recommends a specific fix, such as a unit test for a null fallback. For agent workflows I named six risks to design against, from false confidence to weakened human judgement, and proposed a bounded pilot: replay past PRs against real human review comments, measure detection and false positives, and turn misses into better guidelines. I prototyped with AI coding agents from low fidelity to working VS Code builds, so the squad could test ideas in days rather than wait on specs.
Illustration: every signal state, including when the evidence can't be computed.
System
Evidence before approval, for code, docs and plans
The model has three layers: context (what changed and why), quality signals (what passed and what didn't) and evidence (what a reviewer can verify). It carried into two related efforts: in the Engineering Hub docs redesign, freshness, ownership and breadcrumbs became trust signals, and in a quarterly planning concept, AI checks work items for completeness before anyone commits to them. The principle across all three: a person or an agent should only be asked to act on what the system can show evidence for. The PR Insights team plugged its AI code-pattern recommendations into the pilot.
Illustration: one evidence model applied to pull requests, docs and plans.
What shipped
What shipped in the pilot
The DevQuest pilot shipped as a VS Code extension with a Developer Hub: tiles for PRs in review, authored PRs and work items across Azure DevOps and GitHub, and a PR review panel with Details, Files, Comments, CI Status and Quality tabs. The confidence prototype carried the evidence layer: a review-readiness score, pass-or-fail signals with one line of reason, and a suggested fix for each gap. The PR Insights team plugged its AI code-pattern recommendations into the pilot, and invitations went to the developers from our concept interviews. Alongside it, the Engineering Hub findings became the platform team's homepage requirements, and the quarterly planning concept was presented at an internal hackathon.
Outcome
What changed
The stakeholder goal grew from completion time alone to completion time and PR quality, and DevQuest went to an early-access pilot in April 2026. Nothing measured has moved yet: the pilot launched without a baseline, so it shows direction and reach, not impact. When the org shifted its investment to other review systems in July 2026, I carried the evidence model into a talk, "Designing for Confidence: Looking Beyond PR Completion Time".
Reflection
Check whether the metric is the outcome or a proxy
I would run research before building for a metric: the gamification iteration came before the interviews that showed quality was the need. I would also agree a baseline for review readiness and rework with PM and engineering before the pilot, so it could show change and not only sentiment. The principle I took away: when you're handed a metric, check whether it is the outcome or a proxy before you design for it. Faster PRs were a byproduct of confidence, and confidence comes from evidence people can check.
