Wednesday, January 03, 2007

Why we don't have testers

Hello, answering Pawel's comment to my last entry I'll elaborate a bit on testers in my present context.

"How can you actually live without them?" well we just don't, when I came here there weren't any and I haven't asked to hire any so, no there aren't any dedicated testers. But of course there are numerous people that take the tester's role. For example when it comes to specifying acceptance criteria this is a job of analysts (or people in the analyst's role at the moment). When it comes to punching all the application buttons to see if it breaks then usually we have some people from helpdesk to do the job, or other non-programmers taking the tester's role. But yes we have to dedicated and trained testers. For the record, I wish we did but for now I have more serious problems to deal with introducing basic agile practices.

I disagree that customers shouldn't play a role in testing, I agree that things have to be tested internally before deployed on a customer accessible machine, but I would strive to let the customer see the developed features as soon as possible under all cost. Rapid feedback is what tells us we are right, without it we may be just waisting our time and his money.

"Unit testing is a good concept to check the code itself. Still, there are other aspects of software that are quite difficult to test with unit testing. Examples: user interface, localization, static information stored in databases and files. It's nearly impossible to check these without testers."

Well for starters I think the interface testing problems are very much resolved by acceptance test frameworks that execute functional tests on the system through the interface itself (Selenium, Canoo WebTest for the browser, other frameworks I can't recall now for rich client GUI's).

That said I should emphasize the need to have the UI as thin as possible, through the usage of FIT (or FitNesse) - Framework for Integrated Tests. Testing the top level UI is sometimes necessary but it's usually better to test just 5mm below - that's why you need the UI to be dumb and thin. That's because the UI usually changes, a lot. But it's possible to mix and match those approaches (and that's what I'm recently planning to do) and e.g. call Selenium from FitNesse.

Localization? I can think of at least one way to check for that, while validating the GUI

Static info in DB and files. Well that can easily be done by all testing frameworks, best shot is probably xUnit or FIT (depending on the test nature). During writing the test just write additional code that will check the contents of a file or execute a statement against a database.

Having said all that I agree that are some things that can't be tested automatically, at least not for a reasonable price. That include compliance to a formal standard, having a nice look and feel, etc. My point is - we try to bring the number of those things down by magnitudes, so that the costs (and time) of manual tests are far lower. Then manually checking for those few things isn't such a horrible thing to do.

"Moreover, it's good to have real people checking what you've done as oppose to cold machines which will never really understand requirements. Software testing software...? I think it's Sci-fi"
not really, think of regression testing, once people who understand the requirements have set up acceptance criteria and a proper environment machines may take the role of running the system against those criteria. This is a common problem we also have when we don't have automated acceptance tests. At the end of a cycle there is very little time to check for all those things that were OK 2 months ago if they are still working properly. Very often we just do a little fix, which in turn breaks some functions but we just don't have time to check the whole system for this. Having acceptance tests would have minimized this problem greatly. I didn't say the problem is totally eliminated, I said it is minimized. I may be wrong but I would guess that it would catch over 90% of cases.

1 comment:

Anonymous said...

Hi again Ilfrin,

I think I can't stop myself commenting your posts and provoke you to write more. I hope you don't mind.

Basically, there are three things I'd like to explain more:
- how do you find working with tools that automate black-box web applications testing? How do they cope with JavaScript? What web browsers do they imitate? Don't they limit your web interface options while requiring really thin web user interface? What if I want more sophisticated web interface?
- I have no clue whatsoever how you see automatic testing of software localization. The question for localization testing software is -
Does following Japanese text have any sense on a welcome page: "こ습んに欢ちは、 JOE BLOGGSさん"?
- Tell me more details about database information automatic testing. I find it quite hard, actually. What approach do you use here? Are you trying to cover all information within database?

Can't wait to read your thoughts on these topics.