Developer Day Notes 3: Natural Language Acceptance Testing

by Patrick Reagan

Small shop, QA is a part-time job of developer and project manager.
But quality is important and testing communicates the intent of code…
and maybe communicate user expectations of the code.

Goals: developer & PM pair, tests first, drive a web browser.

Start with a story: User can register for an account (success state).
Requires email, pwd, pwd confirm, accept TOS (the form). Acceptance
includes no dupe emails, pwd at least 6 chars (error conditions).

Behavior is defined by locations (URL), page state (existence of
elements), and message content (eg flash response).

Should define state, steps to execute, and expectation of results.

Manual testing is straightforward but time-consuming, error-prone, and
most importantly isn’t a regression suite so you don’t know if you’re
improving or breaking things.

Selenium IDE is a macro recorder for your browser. Easy to use, but the
HTML table output is sort of weird. And it’s easy to have problems with
browser state, like if you run login a second time it’ll fail because
you’re already logged in and because a user by that name is registered.
It’s hard to maintain, reuse, or understand.

Selenium Remote Control lets you write in a familiar programming
language. Requires a proxy Java server to drive traffic. Mostly you can
do black-box testing, but you’ll probably need a URL to create a clean
state. But it still requires a developer and is not readable by

Can the test statement be executable?

  * Given no users exist in the system


  1. Visit the registration page
  2. And I enter valid registration information
  3. And I submit the form


  1. Then I should be redirected to the home page
  2. And I should see a success message displayed

Yes. Cucumber is a Ruby testing tool that speaks to customers/domain
experts first and code second.

Create a Feature that has a readable English statement of what the steps
and expectations. When you first run Cucumber, it’ll create the skeleton
so you can define the verbs and nouns.

Given /no users exist in the system/ { open ‘clean_slate_url’ }

Problems: Still depends on a developer and no user control over input data.

But you can make this dynamic, change the steps to say things like “When
I enter ‘’ to the ‘user_email’ field” Cucumber will create
a skeleton that picks up those values and your tests can use them.

What’s important is that this makes it extensible — a method to enter
text into a text form field can be reused on later tests.

Then you can move higher up, make a template that says “And I enter
‘ for ‘user_email'” and create a table of email/pwd combos and
it’ll run the test for each row of data.

(Demo of the example test in the slides)

In the future, you can refactor tests to remove duplication and data
like “Given a user exists with the email ‘‘” or ‘And I enter
valid registration data’.

One problem with this testing is that it’s fairly slow, on the order of
10s for a single test. But you can run specific named tests or tag tests
to be able to run a subset easily.

Selenium RC is easy to run cross-platform, you can make it run a browser
in a VM so you can test browser/OS combos with continuous integration.


How do you deal with scale? Tests could take 40m. A: Uh, no really good
answer to this, unfortunately.

What’s your experience with it? A: It can cut down on back-and-forth
between customers and devs. But I haven’t used it on a production