Part 3: Agile Methodologies — Scrum, Kanban & Lean Startup

Introduction — The Crisis That Created Agile

By the late 1990s, the software industry had a dirty secret: most projects failed. The Standish Group's CHAOS Report (1994) found that only 16% of software projects were delivered on time, on budget, and with the required features. A staggering 31% were cancelled before completion. The remaining 53% were "challenged" — late, over-budget, or missing critical functionality.

The dominant approach was heavyweight process: elaborate specifications, extensive upfront design, rigid change control boards, and "Big Bang" integrations at the end. These processes assumed that software development was like building a bridge — that you could fully specify the outcome before construction began. They were wrong.

The problem wasn't that developers were incompetent. The problem was that heavyweight processes were designed for predictable, well-understood work. Software development is largely unpredictable, creative work where learning happens during construction, not before it.

The 2001 Snowbird Meeting

In February 2001, seventeen software developers met at the Snowbird ski resort in Utah. They represented diverse lightweight methodologies: Extreme Programming, Scrum, DSDM, Adaptive Software Development, Crystal, Feature-Driven Development, and Pragmatic Programming. Despite their differences, they agreed on fundamental values.

The result was the Manifesto for Agile Software Development — a one-page document that would reshape the entire industry. The signatories included Kent Beck, Martin Fowler, Robert C. Martin (Uncle Bob), Ken Schwaber, Jeff Sutherland, and Alistair Cockburn.

The Agile Manifesto

The manifesto is deceptively simple — four value statements and twelve principles. Its genius lies in what it doesn't say. It doesn't prescribe tools, ceremonies, or team structures. It establishes a philosophical foundation from which any number of concrete practices can grow.

The Four Values

Each value statement follows the pattern: "We value X over Y. That is, while there is value in the items on the right, we value the items on the left more."

1. Individuals and Interactions over Processes and Tools

A brilliant team with mediocre tools will outperform a mediocre team with brilliant tools every time. Processes exist to serve people, not the other way around.

Concrete Example: Your team uses Jira, but a critical design decision needs to happen now. The agile choice: walk over to the whiteboard and have a 15-minute conversation rather than spending 2 hours writing a Jira ticket, assigning it, waiting for acknowledgement, and scheduling a meeting.

2. Working Software over Comprehensive Documentation

A 200-page specification that nobody reads is waste. A working prototype that stakeholders can click through generates real feedback. Documentation is not bad — but it's only valuable if someone reads it, and even then it's less valuable than working code that proves the concept.

Concrete Example: Instead of spending 3 weeks writing a technical design document for a new feature, build a working prototype in 3 days. Demo it to stakeholders. The prototype reveals assumptions the document would have hidden. Then write the documentation you actually need — likely 2 pages instead of 200.

3. Customer Collaboration over Contract Negotiation

Fixed-scope, fixed-price contracts assume you can predict the future. You can't. Instead, work with the customer continuously. Show them working software regularly. Let them steer. The result is a product they actually want, delivered incrementally.

Concrete Example: A client asks for a "reporting dashboard." A contract-first approach writes a 50-page requirements document, gets it signed, and builds exactly that — only to find six months later that the client actually needed real-time alerts, not historical reports. A collaborative approach shows them a working dashboard every two weeks and adjusts based on their feedback.

4. Responding to Change over Following a Plan

Plans are useful, but they become dangerous when they prevent adaptation. The world changes. Competitors launch. Users surprise you. Regulations shift. An agile team embraces change as a competitive advantage — the faster you can pivot, the more valuable you become.

Concrete Example: Your team planned a six-month feature roadmap. Three months in, a competitor launches a feature that makes half your plan irrelevant. The agile response: acknowledge reality, re-prioritise the backlog, and redirect effort to where it creates the most value now. The waterfall response: continue building features nobody needs because "it's in the plan."

The 12 Principles Behind the Manifesto

The principles provide more concrete guidance than the values. Here they are with modern interpretations:

Key Agile Concepts

Before diving into specific methodologies, let's establish the vocabulary that cuts across all agile approaches.

Timeboxing

A timebox is a fixed period of time during which work is performed. The key principle: when the time expires, the activity ends — regardless of whether all planned work is complete. This creates natural checkpoints for inspection and adaptation.

Think of it like a cooking timer. When it rings, you take the dish out — whether it looks perfect or not. You assess, adjust, and put it back if needed. You never stand there indefinitely waiting for perfection.

Iterative and Incremental Development

These terms are often confused but mean different things:

• Iterative: Revisiting and refining. You build Version 1, get feedback, improve it to Version 2. Like sculpting — you start with rough form and refine progressively.
• Incremental: Adding new pieces. Each increment adds new functionality. Like building with LEGO — each piece adds to the whole.

Most agile approaches use both: you build incrementally (adding features) while iterating on existing features (refining based on feedback).

User Stories

A user story is a short, informal description of a feature from the end user's perspective:

# User Story Format
As a [type of user],
I want [some goal],
So that [some reason].

# Example
As a frequent shopper,
I want to save my payment details,
So that I can check out faster next time.

User stories are deliberately incomplete. They're placeholders for conversations, not comprehensive specifications. The details emerge through discussion between the development team and the product owner.

Velocity

Velocity measures how much work a team completes per iteration (usually measured in story points). It's a planning tool, not a performance tool. It helps teams predict how much they can commit to in the next sprint.

Definition of Done (DoD)

The Definition of Done is a shared checklist that defines when a work item is truly "done" — not just coded, but completely finished. A typical DoD includes:

• Code written and peer-reviewed
• Unit tests pass (≥80% coverage for new code)
• Integration tests pass
• Documentation updated
• Deployed to staging environment
• Acceptance criteria verified by Product Owner
• No known defects

Definition of Ready (DoR)

The Definition of Ready answers: "When is a backlog item ready to be pulled into a sprint?" A typical DoR includes:

• User story written with clear acceptance criteria
• Dependencies identified and resolved (or planned)
• Estimated by the team
• Small enough to complete in one sprint
• Design/UX mockups available (if applicable)

Scrum Overview

Scrum is the most widely adopted agile framework. It provides a structured approach to iterative development with defined roles, events (ceremonies), and artifacts. We'll cover Scrum in much greater depth in Part 4 — here's the essential overview.

The Three Roles

The Five Events (Ceremonies)

• Sprint: A timebox (1-4 weeks) during which a "Done" increment is created
• Sprint Planning: Team selects backlog items and plans how to deliver them
• Daily Scrum: 15-minute daily sync — what did I do, what will I do, what's blocking me
• Sprint Review: Demo of the increment to stakeholders; gather feedback
• Sprint Retrospective: Team reflects on the process and identifies improvements

The Three Artifacts

• Product Backlog: Ordered list of everything the product might need
• Sprint Backlog: Subset of Product Backlog selected for the current sprint + plan for delivery
• Increment: The sum of all completed backlog items — must be in a usable condition

flowchart LR
    PB[Product Backlog] --> SP[Sprint Planning]
    SP --> SB[Sprint Backlog]
    SB --> DEV[Development Work]
    DEV --> DS[Daily Scrum]
    DS --> DEV
    DEV --> INC[Potentially Shippable Increment]
    INC --> SR[Sprint Review]
    SR --> PB
    INC --> RET[Retrospective]
    RET --> SP
                            

Kanban

Kanban (看板, literally "visual signal" or "card" in Japanese) originated at Toyota in the late 1940s as a scheduling system for just-in-time manufacturing. David J. Anderson adapted it for software development in 2004, and it has since become the primary alternative to Scrum for agile teams.

Core Principles

Kanban has six core practices:

1. Visualise the workflow — Make work visible on a board
2. Limit Work in Progress (WIP) — Cap the number of items in each stage
3. Manage flow — Monitor and optimise the movement of work
4. Make policies explicit — Write down the rules (DoD, WIP limits, priorities)
5. Implement feedback loops — Regular cadences for review and improvement
6. Improve collaboratively, evolve experimentally — Small changes based on data

The Kanban Board

A Kanban board visualises work flowing through stages. Each column represents a state in your workflow. Cards (work items) move left to right as they progress.

flowchart LR
    A["Backlog
(∞)"] --> B["To Do
(WIP: 5)"]
    B --> C["In Progress
(WIP: 3)"]
    C --> D["Code Review
(WIP: 2)"]
    D --> E["Testing
(WIP: 2)"]
    E --> F["Done
(∞)"]
                            

WIP Limits — The Secret Weapon

WIP (Work in Progress) limits are the single most powerful concept in Kanban. They cap how many items can be in a column at any time. When a column hits its WIP limit, no new work can enter until something exits.

Why this works: Imagine a motorway. When traffic is light, cars flow freely. As more cars enter, speed drops. At peak congestion, everyone crawls. WIP limits prevent the "motorway at rush hour" problem — they keep the system below congestion.

The Pull System

In Kanban, work is pulled, not pushed. A developer finishes a code review → the "Code Review" column now has capacity → the developer pulls the next highest-priority item from "In Progress" into review. This contrasts with push systems where managers assign work regardless of capacity.

Flow Metrics

Kanban vs Scrum

Extreme Programming (XP)

Extreme Programming, created by Kent Beck in the late 1990s, takes good software engineering practices and pushes them to their logical extreme. If code review is good, review all code all the time (pair programming). If testing is good, test everything continuously (TDD). If integration is good, integrate many times per day (continuous integration).

Core XP Practices

The TDD Cycle

# The Red-Green-Refactor cycle:
# 1. RED    — Write a failing test
# 2. GREEN  — Write the minimum code to make it pass
# 3. REFACTOR — Clean up the code while keeping tests green

# Example in pseudocode:
# RED: test("adds two numbers") { assert add(2, 3) == 5 }  → FAILS (no add function)
# GREEN: function add(a, b) { return a + b }               → PASSES
# REFACTOR: (already clean — move on)

When XP Works Best

• Small, co-located teams (2-12 developers) — pair programming requires proximity (physical or virtual)
• Frequently changing requirements — TDD + refactoring makes code safe to change
• High-quality requirements — when bugs are expensive (financial systems, healthcare)
• Experienced developers — TDD and pairing require skill and discipline
• Customer available daily — the planning game requires real-time business input

Lean Startup

The Lean Startup methodology, popularised by Eric Ries in 2011, applies agile principles beyond software delivery to the entire product/market fit question: "Should we build this at all?"

While Scrum and Kanban help you build the thing right, Lean Startup helps you build the right thing. It's the methodology of validated learning — using experiments to test business hypotheses before investing heavily in development.

The Build-Measure-Learn Loop

flowchart TD
    IDEAS[Ideas / Hypotheses] --> BUILD[Build MVP]
    BUILD --> PRODUCT[Product / Feature]
    PRODUCT --> MEASURE[Measure Data]
    MEASURE --> DATA[Metrics / Evidence]
    DATA --> LEARN[Learn / Validate]
    LEARN --> DECISION{Pivot or Persevere?}
    DECISION -->|Persevere| IDEAS
    DECISION -->|Pivot| NEWIDEA[New Hypothesis]
    NEWIDEA --> BUILD
                            

Core Concepts

Minimum Viable Product (MVP)

An MVP is the smallest version of your product that lets you start the learning loop. It's not a prototype, not a demo, not a half-finished product. It's the minimum amount of effort needed to test a specific hypothesis.

Examples of MVPs:

• Dropbox: A 3-minute video showing the product concept → validated demand before writing a single line of sync code
• Zappos: Founder photographed shoes at local stores, posted online, bought and shipped manually → validated that people would buy shoes online
• Buffer: A landing page with pricing tiers and a "Sign Up" button (no product behind it) → validated willingness to pay before building

Validated Learning

Validated learning is progress measured by running experiments. Instead of "we think users want feature X," you state: "We hypothesise that adding feature X will increase conversion by 15%." You build the minimum, measure the result, and learn whether your hypothesis was correct.

Pivot vs Persevere

At each iteration of the loop, you face a critical decision:

• Persevere: The data supports your hypothesis. Double down. Scale up.
• Pivot: The data contradicts your hypothesis. Change strategy while keeping what you've learned.

Types of pivots:

• Zoom-in pivot: A single feature becomes the whole product (Flickr started as a game)
• Zoom-out pivot: The whole product becomes a feature (Instagram started with check-ins + photos; pivoted to photos only)
• Customer segment pivot: Same product, different audience (Slack was built for a game studio; pivoted to general teams)
• Technology pivot: Same solution, different tech (Netflix: DVDs → streaming)

SAFe, LeSS, and Scaling Agile

Agile was designed for small, co-located teams (5-9 people). But what happens when you have 500 developers across 50 teams building one product? That's where scaling frameworks enter.

The Major Scaling Frameworks

Common Scaling Pitfalls

Warning signs your scaled agile is just "waterfall with post-its":

• Teams can't deploy independently — they wait for quarterly "release trains"
• Product decisions are made by committees, not empowered Product Owners
• Architects dictate solutions to teams rather than teams owning their design
• The "scaled Scrum of Scrums" meeting takes longer than the actual work
• Every team has the same velocity target set by management

Choosing the Right Methodology

There is no universally "best" methodology. The right choice depends on your context: team size, product maturity, regulatory requirements, customer access, and organisational culture.

Decision Framework

Practical Decision Rules

• Start with Scrum if you need structure, have a dedicated product team, and are new to agile
• Choose Kanban if your work is interrupt-driven, you have mixed priorities, or you need continuous flow (DevOps, support teams)
• Adopt XP practices (TDD, CI, pair programming) regardless of framework — they're complementary engineering practices, not a competing methodology
• Use Lean Startup when you don't yet know if you're building the right thing — validate before you invest
• Combine freely: "Scrumban" (Scrum cadence + Kanban WIP limits) is extremely popular and often the pragmatic best choice

Common Agile Anti-Patterns

After two decades of agile adoption, certain failure modes repeat across organisations. Recognising these patterns saves teams from the most common traps.

1. "Agile Waterfall" (Mini-Waterfalls in Sprints)

The team runs two-week sprints, but within each sprint they do one week of design, then one week of coding, then scramble to test on the last day. The sprint boundary provides an illusion of agility, but the work within the sprint is sequential waterfall.

Fix: Break stories small enough that a developer can complete code + tests in 1-2 days. Swarm on stories (multiple people work on one story) rather than spreading the team across all stories simultaneously.

2. "Cargo Cult Agile" (Ceremonies Without Values)

The team holds daily stand-ups, sprint reviews, and retrospectives — but nothing changes. The stand-up is a status report to the manager. The retro produces action items that nobody follows up on. The sprint review is a PowerPoint presentation, not a live demo.

Fix: Ask "why" for every ceremony. If you can't articulate how it creates value, stop doing it. Measure: are retrospective action items actually implemented? If not, the retro is theater.

3. "Sprint Zero Forever"

The team spends sprint after sprint on "infrastructure setup," "architecture design," and "foundation work" without delivering user-facing value. Sprint Zero is legitimate for initial setup (1-2 sprints max), but it becomes an anti-pattern when it extends indefinitely.

Fix: Every sprint must produce something a user could hypothetically use. "Hello World deployed to production with CI/CD" is a valid Sprint 1 deliverable — it proves the infrastructure works AND delivers user value (albeit minimal).

4. Ignoring Technical Debt

The Product Owner fills every sprint with features. There's never time for refactoring, upgrading dependencies, or improving test coverage. The codebase slowly rots until development slows to a crawl.

Fix: Allocate 15-20% of sprint capacity to technical health. This is non-negotiable. Teams that invest in continuous refactoring move faster over time; teams that don't eventually grind to a halt.

5. No Retrospective Action Items

The team identifies problems in the retrospective but never assigns owners, never creates tickets, and never follows up. Three months later, the same issues appear in every retro.

Fix: Limit retro outputs to 1-2 concrete actions per sprint. Assign an owner. Put them in the sprint backlog. Review them at the start of the next retro before generating new actions.

6. Using Story Points as Performance Metrics

Management asks: "Why did Team A complete 50 points this sprint and Team B only 30?" This incentivises point inflation, undermines trust, and destroys the psychological safety needed for honest estimation.

Fix: Story points are a planning tool for the team's internal use. If management needs metrics, give them outcomes (features shipped, customer satisfaction, lead time) not outputs (story points).

Exercises

Conclusion & Next Steps

Agile is not a methodology — it's a philosophy. Scrum, Kanban, XP, and Lean Startup are different expressions of the same core values: embrace uncertainty, deliver incrementally, learn continuously, and improve relentlessly.

The key takeaways from this article:

• The Agile Manifesto provides values and principles, not prescriptions — it's a compass, not a map
• Scrum provides structure through sprints, roles, and ceremonies — ideal for product development teams
• Kanban provides flow management through WIP limits and pull systems — ideal for operations and unpredictable workloads
• XP provides engineering excellence through TDD, pairing, and continuous integration — practices any team can adopt
• Lean Startup provides validation through Build-Measure-Learn — essential when requirements are uncertain
• Scaling frameworks exist but carry high risk of "agile theater" — organisational change matters more than process change
• Anti-patterns are predictable — recognise them early and fix them before they calcify