Any company with a software product these days is perpetually on the lookout for ways to trim overhead from the development process and accelerate the delivery of value to customers. From Agile to DevOps to a half-dozen other buzzwords, there’s no shortage of philosophies and toolkits billed as the next Holy Grail. But as we wrote previously, a philosophy doesn’t go very far without a broader cultural shift.

To rapidly and safely deliver changes to our end users, Walmart Technology has been focusing on the principles and practices of Continuous Delivery, or CD. Like Agile and DevOps, CD, to the uninitiated, may seem like a matter of buying the right set of tools and handing them over to development teams. But new tooling without new behavior simply makes it easier to break things.

As we began our path towards CD, we needed to overcome the classic “artisanal” development process where:

  • Business defines requirements and hands them off to developers.
  • Developers code a solution based on their best understanding and hand it off to QA.
  • QA tests the application based on their understanding of the requirements and hands defects back to developers.
  • Defect resolution tasks are injected into the next or even current sprint.
  • Delivery is delayed.

Ironically, the shortcomings of this process are compounded by multiple issues arising from the practice of Agile development, which is often less than agile:

  • Developers cannot execute Continuous Integration correctly. Without a clear understanding of what the functional tests should be, they cannot commit tested code daily to mainline. Instead, they tend to hold changes back for days or even weeks while they seek understanding of the problem and explore potential answers. This leads to integration delays at the end of a sprint and additional “hardening” sprints before release.
  • Time is wasted in meetings discussing the validity of defects resulting in an adversarial relationship between Dev, QA, and Business.
  • When QA testing finds defects at the end of a sprint, defect resolution cost is multiplied 51 times.
Chart created by Bryan Finster, adapted from learnings from “Software Assessments, Benchmarks and Best Practices”- Capers Jones.

Our method for addressing these issues is Behavior Driven Development or BDD.

BDD is a process that takes user stories and decomposes them into testable outcomes. The “Three Amigos” — business, developers, and QA — work together to refine the testable outcomes of each story, and each brings to the table their unique perspective on the problem. Each stakeholder becomes an owner in the resulting solution.

We’ve seen several improvements since adopting BDD at Walmart:

  • “Done” can now be quantified. The importance of this can’t be overstated. Since all of the testable outcomes for a story are agreed upon, we can mark a story done once the application passes those tests. Stories and tasks no longer drag on with added work after being accepted into the sprint. Consequently, velocity becomes predictable. Any missed requirements are simply added as acceptance tests in a following sprint.
  • Developers can write effective functional tests for their code before committing their changes to version control. This is a prerequisite for effective Continuous Integration.
  • The previously adversarial relationship between Business, Dev and QA evolves into a partnership as the three roles collaborate together, which enables us to drive out uncertainty more rapidly. Rather than pointing fingers, we work as a team.
  • Instead of testing after the fact, QA can focus on discovering how something might fail during story refining, injecting that knowledge into the acceptance tests. This “drives tests left” and has saved countless hours of wasted development.
  • Because the entire delivery team contributed to creating the test definitions, a developer can focus on how to implement the behaviors instead of deciding what behaviors need to be implemented. This results in more rapid development with a higher degree of confidence and faster value delivery.

To help formalize the BDD thought process, we use the Gherkin language to express our tests. It is a business readable, Domain Specific Language that lets you describe behavior without detailing how that behavior is implemented. It is used both as documentation and to enable automation.


  • Given some pre-condition
  • When some action occurs
  • Then an expected outcome happens.

Good Gherkin is simple, declarative, and written in active voice. Using it allows developers to perform Acceptance Test-Driven Development leveraging their current testing frameworks (JUnit, Jasmine, RSpec, etc).

Here’s a simple example:

  • Scenario: Remove all containers from a delivery
  • Given containers exist for delivery “123
  • When I request that the containers are removed
  • Then no containers will exist for delivery “123

Using BDD frameworks such as Cucumber accelerates development even further and treats business requirements as code. Because we commit the acceptance tests to version control and Cucumber is validating the unit tests against them, the acceptance tests become both the driver of new features and the living documentation of the current application.

Cucumber parses the Gherkin tests and outputs the following JUnit stubs:

@Given(“^containers exist for delivery “(\\d+)”$”)public void containers_exist_for_delivery(int arg1) throws Throwable {
// Write code here that turns the phrase above into concrete actions throw new PendingException();
@When(“^I request that the containers are removed$”)public void i_request_that_the_containers_are_removed() throws Throwable {
// Write code here that turns the phrase above into concrete actions throw new PendingException();
@Then(“^no containers will exist for delivery “(\\d+)”$”)public void no_containers_will_exist_for_delivery(int arg1) throws Throwable {
// Write code here that turns the phrase above into concrete actions throw new PendingException();

We now have three failing tests that are reporting “pending implementation needed”. The developer then simply adds the concrete actions to test and resolves the breaking tests for the feature.

@Given(“^containers exist for delivery “(\\d+)”$”)public void containers_exist_for_delivery(int arg1) throws Throwable {
Assert.assertNotEquals(0,container. getContainersForDelivery(deliveryId).count());
@When(“^I request that the containers are removed$”)public void i_request_that_the_containers_are_removed() throws Throwable {
@Then(“^no containers will exist for delivery “(\\d+)”$”)public void no_containers_will_exist_for_delivery(int arg1) throws Throwable {
Assert.assertEquals(0, container.getContainersForDelivery(deliveryId).count());

While test automation is powerful and one of the first things that attracts developers to BDD, the tight collaboration between Business, QA, and Dev and the increased certainty of value delivery is where the real power lies. Discovering potential bugs during requirements gathering dramatically reduces costs. In some cases, a five minute conversation about a test behavior has resulted in the decision not to develop something that originally sounded like a good idea. This has saved countless hours of effort and support.

Decomposing stories into tests encourages more modular, testable code. That, in turn, enables proper Continuous Integration. And proper Continuous Integration enables the rapid value delivery required to be a sustainable enterprise in today’s market.

A good starting point for more information can be found at Liz Keogh’s website. She has been working with BDD starting with Thought Works in 2004 and has many blog posts and videos on how to apply it and anti-patterns to avoid.


  1. Walmart will never succeed with continuous delivery until they actually check on deliveries from their warehouses to the stores instead of taking the shipment listing as gospel. If you start with a flawed inventory it will do nothing but perpetuate the problem

  2. Whattt?? This company has one of the largest and well funded lobbiest organizations in the US. That’s what brings these results.

Comments are closed.

Register for the Latest News