Computerworld: Testing times for slack software development

I've been quoted in a recent Computerworld article entitled Testing times for slack software development. Answering the questions for that article gave me time to reflect on the improvements to testing that I've implemented at Squiz and what I still have left to do. I thought I'd post the questions I was asked and my responses with a bit more of an explanation for each.

Is it important to retain software testing in-house? Why?

For the testing we do, I believe it is. We know our product best, and we know how our clients use it and what they are most concerned about. We tailor our tests and testing frameworks based on that knowledge. MySource Matrix is highly complicated, especially when you consider the interactions between different parts of the system, so testing only the surface functionality is never going to be adequate. We need testers who know all the weird and wonderful ways our software can be used.

I'm sure outsourcing testing is a viable solution for plenty of software products out there, but it's not for a product like MySource Matrix. Some software products are designed with a clearly defined set of functions they must perform. It is very easy to write test cases for those products based on the original specification. A product like MySource Matrix is designed to allow complex interactions between its modules. Essentially, the product is designed to be used in ways that we have have not dreamed of yet. For this kind of product, implementation experience is essential for designing test cases, and that is something that a 3rd party testing company doesn't have.

The survey claimed the automation of software testing is low - do you agree? Why/why not?

Automated software testing tools are available for almost every programming language. Open source products are notorious for being badly tested, but unit testing (at least) has really grown in popularity over the last few years. So I'd say this problem is getting better.

From my experience working with other organisations, automated testing is rare. You often here about testing teams developing "test scripts" but the name is sometimes misleading. Ideally, a suite of test scripts is something that can be run over a product without user intervention, allowing for automated testing before rolling out each new product update. Often, test scripts are simply a set of steps that need to be manually performed by a tester, and the results recorded. This leaves the testing process open to human error. If your testing is not automated, you cannot guarantee that your test results are accurate.

A survey claimed 85 percent of defects in software projects originate in the requirements phase - do you agree?

For Squiz, I don't think this is correct. In my experience, software defects creep into the product during the system architecture design and development phases. Maybe we do the specification phase differently?

This figure of 85% is useless without knowing how the researcher defined the word "defect". If a defect is classified as a failure of the product to conform to the requirements, which is how I would roughly define it, then the only defects produced in the requirements phase and going to be conflicting requirements. While I've seen my share of conflicting requirements, a figure of 85% is certainly too high.

I see most defects during the development of new functionality and the maintenance of existing functionality once a product has been released. The first version of a product should have the least amount of defects, increasing over time based on the amount of additional development performed on the product. The first version of a product has the least number of features and those features tend to be very well tested by their creators. Of course, corporate reality sometimes kicks in and forces a product release before it is ready, so that statement doesn't always hold true.

Fixing errors in the post-implementation phase is 100 times higher than correcting bugs during development - do you agree?

When you consider the time it takes to upgrade and test a client's system after a bug fix, it's easy to see that post-implementation bug fixing is significantly more costly than bug fixes made before the product is released to users. Is it 100 times more costly? That depends on how many client systems you have out in the wild with the bug and how those systems are patched.

I'm not sure where the figure of 100 came from, but I certainly can't dismiss it. It's very hard to qualify just how much more expensive bug fixes are post-implementation but it should be obvious to any software vendor that it's worth putting in the time to detect and fix as many bugs as possible before a release. Not only is it far cheaper, but it also increases client satisfaction, and that in itself saves money.

How can the quality of software testing be upheld?

The quality of software testing is like everything else; you need to measure the effectiveness and constantly improve your processes. Each organisation needs to find some metrics that suit their product (e.g., code coverage, DB performance) and then measure them constantly as the product changes during the development phase. A process needs to be in place to identify undesirable changes and rectify the product. I believe that automation is the best way to ensure consistent quality of testing.

This really sums up what I have been trying to do at Squiz. Both unit testing frameworks I developed at Squiz had metrics built in from the beginning. Simply knowing that your tests are passing is not enough. You need to be constantly increasing the coverage of those tests and ensuring that new code is not adversely affecting the performance on the product.

The best example I have of this was a few years ago when I introduced an assertion class into MySource Matrix. Developers were able to assert that function parameters were valid before using them, allowing us to use a design by contract methodology. The problem was when to stop validating input. One core function required a DB query to validate its input, which was later found to be incredibly slow. We found this problem through the performance metrics built into our testing framework. When all your tests start reporting increased running times, you know you've made a mistake somewhere.

So where to next?

For me, it's interface testing. Testing interfaces is tricky, but there are now tools to make the job easier. For PHP, the integration between PHPUnit and the Selenium RC server (using the Testing_Selenium PEAR package) can't be beat. Selenium is easy to write test scripts for (you can record them in Firefox using an extension if you want) and the PHPUnit integration makes incorporating these scripts into your existing framework easy.

I'm looking forward to implementing a new testing framework that will allow Squiz to move beyond traditional unit testing and into an area more like functional and user acceptance testing, which has always been tough for browser-based applications.