Developer Day Notes 3: Natural Language Acceptance Testing

by Patrick Reagan

Small shop, QA is a part-time job of developer and project manager. But quality is important and testing communicates the intent of code… and maybe communicate user expectations of the code.

Goals: developer & PM pair, tests first, drive a web browser.

Start with a story: User can register for an account (success state). Requires email, pwd, pwd confirm, accept TOS (the form). Acceptance includes no dupe emails, pwd at least 6 chars (error conditions).

Behavior is defined by locations (URL), page state (existence of elements), and message content (eg flash response).

Should define state, steps to execute, and expectation of results.

Manual testing is straightforward but time-consuming, error-prone, and most importantly isn’t a regression suite so you don’t know if you’re improving or breaking things.

Selenium IDE is a macro recorder for your browser. Easy to use, but the HTML table output is sort of weird. And it’s easy to have problems with browser state, like if you run login a second time it’ll fail because you’re already logged in and because a user by that name is registered. It’s hard to maintain, reuse, or understand.

Selenium Remote Control lets you write in a familiar programming language. Requires a proxy Java server to drive traffic. Mostly you can do black-box testing, but you’ll probably need a URL to create a clean state. But it still requires a developer and is not readable by non-developers.

Can the test statement be executable?

State:
  * Given no users exist in the system

Steps:

  1. Visit the registration page
  2. And I enter valid registration information
  3. And I submit the form

Expectation:

  1. Then I should be redirected to the home page
  2. And I should see a success message displayed

Yes. Cucumber is a Ruby testing tool that speaks to customers/domain experts first and code second.

Create a Feature that has a readable English statement of what the steps and expectations. When you first run Cucumber, it’ll create the skeleton so you can define the verbs and nouns.

Given /no users exist in the system/ { open ‘clean_slate_url’ }

Problems: Still depends on a developer and no user control over input data.

But you can make this dynamic, change the steps to say things like “When I enter ‘user@host.com’ to the ‘user_email’ field” Cucumber will create a skeleton that picks up those values and your tests can use them.

What’s important is that this makes it extensible — a method to enter text into a text form field can be reused on later tests.

Then you can move higher up, make a template that says “And I enter ‘‘ for ‘user_email’” and create a table of email/pwd combos and it’ll run the test for each row of data.

(Demo of the example test in the slides)

In the future, you can refactor tests to remove duplication and data like “Given a user exists with the email ‘‘” or ‘And I enter valid registration data’.

One problem with this testing is that it’s fairly slow, on the order of 10s for a single test. But you can run specific named tests or tag tests to be able to run a subset easily.

Selenium RC is easy to run cross-platform, you can make it run a browser in a VM so you can test browser/OS combos with continuous integration.

Q&A:

How do you deal with scale? Tests could take 40m. A: Uh, no really good answer to this, unfortunately.

What’s your experience with it? A: It can cut down on back-and-forth between customers and devs. But I haven’t used it on a production project.