Computerworld: Testing times for slack software development

I've been quoted in a recent Computerworld article entitled Testing times for slack software development. Answering the questions for that article gave me time to reflect on the improvements to testing that I've implemented at Squiz and what I still have left to do. I thought I'd post the questions I was asked and my responses with a bit more of an explanation for each.

Is it important to retain software testing in-house? Why?

For the testing we do, I believe it is. We know our product best, and we know how our clients use it and what they are most concerned about. We tailor our tests and testing frameworks based on that knowledge. MySource Matrix is highly complicated, especially when you consider the interactions between different parts of the system, so testing only the surface functionality is never going to be adequate. We need testers who know all the weird and wonderful ways our software can be used.

I'm sure outsourcing testing is a viable solution for plenty of software products out there, but it's not for a product like MySource Matrix. Some software products are designed with a clearly defined set of functions they must perform. It is very easy to write test cases for those products based on the original specification. A product like MySource Matrix is designed to allow complex interactions between its modules. Essentially, the product is designed to be used in ways that we have have not dreamed of yet. For this kind of product, implementation experience is essential for designing test cases, and that is something that a 3rd party testing company doesn't have.

The survey claimed the automation of software testing is low - do you agree? Why/why not?

Automated software testing tools are available for almost every programming language. Open source products are notorious for being badly tested, but unit testing (at least) has really grown in popularity over the last few years. So I'd say this problem is getting better.

From my experience working with other organisations, automated testing is rare. You often here about testing teams developing "test scripts" but the name is sometimes misleading. Ideally, a suite of test scripts is something that can be run over a product without user intervention, allowing for automated testing before rolling out each new product update. Often, test scripts are simply a set of steps that need to be manually performed by a tester, and the results recorded. This leaves the testing process open to human error. If your testing is not automated, you cannot guarantee that your test results are accurate.

A survey claimed 85 percent of defects in software projects originate in the requirements phase - do you agree?

For Squiz, I don't think this is correct. In my experience, software defects creep into the product during the system architecture design and development phases. Maybe we do the specification phase differently?

This figure of 85% is useless without knowing how the researcher defined the word "defect". If a defect is classified as a failure of the product to conform to the requirements, which is how I would roughly define it, then the only defects produced in the requirements phase and going to be conflicting requirements. While I've seen my share of conflicting requirements, a figure of 85% is certainly too high.

I see most defects during the development of new functionality and the maintenance of existing functionality once a product has been released. The first version of a product should have the least amount of defects, increasing over time based on the amount of additional development performed on the product. The first version of a product has the least number of features and those features tend to be very well tested by their creators. Of course, corporate reality sometimes kicks in and forces a product release before it is ready, so that statement doesn't always hold true.

Fixing errors in the post-implementation phase is 100 times higher than correcting bugs during development - do you agree?

When you consider the time it takes to upgrade and test a client's system after a bug fix, it's easy to see that post-implementation bug fixing is significantly more costly than bug fixes made before the product is released to users. Is it 100 times more costly? That depends on how many client systems you have out in the wild with the bug and how those systems are patched.

I'm not sure where the figure of 100 came from, but I certainly can't dismiss it. It's very hard to qualify just how much more expensive bug fixes are post-implementation but it should be obvious to any software vendor that it's worth putting in the time to detect and fix as many bugs as possible before a release. Not only is it far cheaper, but it also increases client satisfaction, and that in itself saves money.

How can the quality of software testing be upheld?

The quality of software testing is like everything else; you need to measure the effectiveness and constantly improve your processes. Each organisation needs to find some metrics that suit their product (e.g., code coverage, DB performance) and then measure them constantly as the product changes during the development phase. A process needs to be in place to identify undesirable changes and rectify the product. I believe that automation is the best way to ensure consistent quality of testing.

This really sums up what I have been trying to do at Squiz. Both unit testing frameworks I developed at Squiz had metrics built in from the beginning. Simply knowing that your tests are passing is not enough. You need to be constantly increasing the coverage of those tests and ensuring that new code is not adversely affecting the performance on the product.

The best example I have of this was a few years ago when I introduced an assertion class into MySource Matrix. Developers were able to assert that function parameters were valid before using them, allowing us to use a design by contract methodology. The problem was when to stop validating input. One core function required a DB query to validate its input, which was later found to be incredibly slow. We found this problem through the performance metrics built into our testing framework. When all your tests start reporting increased running times, you know you've made a mistake somewhere.

So where to next?

For me, it's interface testing. Testing interfaces is tricky, but there are now tools to make the job easier. For PHP, the integration between PHPUnit and the Selenium RC server (using the Testing_Selenium PEAR package) can't be beat. Selenium is easy to write test scripts for (you can record them in Firefox using an extension if you want) and the PHPUnit integration makes incorporating these scripts into your existing framework easy.

I'm looking forward to implementing a new testing framework that will allow Squiz to move beyond traditional unit testing and into an area more like functional and user acceptance testing, which has always been tough for browser-based applications.

Ignore files and folders with PHP_CodeSniffer

A feature request came through about a month ago to add a new command line argument to PHP_CodeSniffer that allows certain files and folders to be skipped. I added this feature in a couple of weeks ago and I've just finished updating the end user docs to show how it is used.

Sometimes you want PHP_CodeSniffer to run over a very large number of files, but you want some files and folders to be skipped. The --ignore command line argument can be used to tell PHP_CodeSniffer to skip files and folders that match one or more patterns.

In the following example, PHP_CodeSniffer will skip all files inside the package's tests and data directories. This is useful if you are checking a PEAR package but don't want your test or data files to conform to your coding standard.

$ phpcs --ignore=*/tests/*,*/data/* /path/to/code

As it turns out, this feature comes in handy for the new automated PHP_CodeSniffer report being run over PEAR CVS thanks to Brett Bieber.

PHP_CodeSniffer contains a lot of test files that violate the PEAR coding standards for the purpose of checking that violations are detected. Those files were causing thousands of errors to be reported against the PHP_CodeSniffer package, which was not an accurate reflection of its adherence to the coding standards.

Brett has now modified his script to use the new command line argument to ignore all files inside test and data directories, which matches the newly modified PhpDocumentor ignore pattern being used by PEAR.

Testing PHP_CodeSniffer without installing

PHP_CodeSniffer has always contained extensive unit tests for all code sniffs, but the testing structure required the package to be installed so that tests could be placed into an expected location. The tests also use PHP_CodeSniffer itself, and it requires the package to be installed so that the require statements that rely on the PEAR include path would work correctly. So basically, neither PHP_CodeSniffer or the unit tests could be run without installing the package first.

PHP_CodeSniffer is not a library, it is a development tool, so this shouldn't matter. However, PEAR has recently pushed to get repository-wide code coverage stats for unit testing. All unit tests must now run correctly when the package is checked-out from CVS and without being installed.

To solve the problem, I turned to the newly proposed PEAR2 standards, which are being hotly debated at the moment on pear-dev.

To be honest, I wasn't sure about those standards when I first read them. I could see the merit for some libraries, but it was hard to see how those changes would make PHP_CodeSniffer better. I still don't think they are going to make PHP_CodeSniffer easier to use, but they have certainly made it easier to develop and test for.

The change that I made was to remove all my require_once() statements and replace them with class_exists() and interface_exists() calls with the optional second argument for autoloading. The PHP_CodeSniffer class now has an autoload method that includes the class files from different locations depending on if the package is installed or not.

The benefit of doing this is that the AllTests.php file that starts the unit testing off only needs to find the CodeSniffer.php file and its own test files. Once CodeSniffer.php is included, all other files are included automatically. Couple this change with a few other minor tweaks to remove some hard-coded assumptions about file paths and lengths, and you've got a unit test suite that runs from CVS. As an added benefit, modifying a single include inside the phpcs script allows you to run PHP_CodeSniffer itself without installing the package.

Index: scripts/phpcs
===============================================
RCS file: /repository/pear/PHP_CodeSniffer/scripts/phpcs,v
retrieving revision 1.16
diff -r1.16 phpcs
20c20
< require_once 'PHP/CodeSniffer.php';
---
> require_once '../CodeSniffer.php';

While these changes were made primarily for the PEAR unit testing system, they have also meant that I can stop installing the package to test every change I make. That is making development easier for me, and that's a good thing.

What is inline content editing?

Update: Squiz decided to tackle this problem and we've created our own truly inline WYSIWYG editor called Viper. You can find out more, and watch a video of it in action, on the MySource Mini inline editing feature page.

A lot of products claim to have inline content editing, but it's obvious that everyone has their own interpretation of that phrase. Inline content editing seems to come in six main flavors, although I've yet to see a true inline content editing interface.

Stand-alone WYSIWYG editing

This style of content editing displays a stand-alone WYSIWYG editor embedded into an editing interface. The editing interface and the site frontend have no common elements. The WYSIWYG editor takes the styles from the site's style sheets and displays them inside the WYSIWYG editor. This allows the page author to see their content as it would be displayed on the page, but they cannot see how the content will look when the other page elements are placed around it. Generally, a preview is provided to allow the author to switch between editing and previewing the content.

The MySource Matrix administration interface uses this style of inline editing. The content editing screen and the preview screen are accessed separately, but the site design's styles are shown while editing in the WYSIWYG editor.

The advantage of this style of editing is that the author has a large area in which to edit content. The disadvantage is that the author cannot see how their content will flow inside the page without saving their work and switching to a preview.

Embedded WYSIWYG editing

This style of content editing is similar to the stand-alone WYSIWYG editor, but this style places the content items around the WYSIWYG editor so the author has an idea of how the content will fit with the overall style of the page. This inline preview is limited though. For complex pages with multiple content areas, the WYSIWYG editor itself changes the page layout and makes the inline preview less accurate. The author still requires a preview mode to confirm that the page looks the same as it does during editing.

The MySource Matrix simple edit interface uses this style of inline editing. The editor is the same as that in the administration interface, but the size is now the same as the content area and the site's design is shown around the editor. IP.Board also uses embedded WYSIWYG editing in their new quick edit forum post editing interface.

The advantage of this style of editing is that the author can see quickly how the content of the page will look within the site's design. The disadvantage is that the editing area is often smaller, resized to fit within the body area of the page. Some editors provide a full-screen mode to get around this size limitation, but that transforms the editing interface into a stand-alone editor, with all the disadvantages that brings.

Popup WYSIWYG editing

This style of content editing was made famous by RedDot. Content areas within a page preview are marked as editable using an icon that floats over the content or some other obvious style. Clicking that icon displays a stand-alone WYSIWYG editing interface. Once the content is saved, the new content is written back to the page preview.

For a good demonstration of this style of content editing, take a look at this inline editing demo video for Community Server. The first part of the video shows true inline content editing with text boxes only, not an uncommon thing to see, but the second part shows inline editing of complex page elements like HTML areas.

The advantage of this style of editing is that the preview is always accurate. The disadvantage is that content is still edited using a stand-alone editing interface, not allowing the author to see a true preview of the content until they have completed editing.

It is worth noting that some editing interfaces take this style a bit further and write back to the page preview in real-time. This allows you to switch to the preview without saving or to have the page preview and editing interface side-by-side. Certainly, this is an improvement, but it does require multiple monitors to really make use of this feature and is does require the author to switch focus frequently while editing.

Whole page WYSIWYG editing

To get around the problem of having a preview mode and an edit mode, this style of editing uses a WYSIWYG editor to edit the entire contents of the page, including the site design and all page elements. The author gets inline editing and a very accurate preview.

The advantages are obvious, but the disadvantages outweigh them in my opinion. When the entire page is editable, there is no way to lock parts of the page and stop the author from editing them. For anything but the most simple of sites, this is not an acceptable situation. Sure, you can build components into the WYSIWYG editor to stop the author from editing some content in WYSIWYG mode, but switching to the common "HTML source" mode allows the author to change all parts of the page.

Some editors get around the disadvantages by removing the source mode, or providing their own source mode that locks part of the page from being edited. Most editors that do this are written in either Flash or Java so the developers have full control over the editing interface.

If you ask me, a full screen custom WYSIWYG editor is just an application running inside the browser and not the inline content editing interface of a content management system. As new features need to be added to the CMS, they need to be built into the full-screen WYSIWYG editor. So what you end up with is just a CMS written in Flash or a Java applet.

Inline WYSIWYG editing

This style of content editing is similar to embedded WYSIWYG editing except that the editor is made to be almost invisible. By removing the borders of the editor and resizing it dynamically as the content within it changes, the author is given an almost perfect editing and preview mix. The editing feels inline and the preview looks accurate. Better yet, the author does not have to shift focus between editing and preview modes.

So is this perfect? It is very close, but not quite what I would call true inline content editing. The content generated by the WYSIWYG editor still needs to be saved back to the content repository and cleaned to remove browser-specific code that is added by the browser's editable region. Using a non-JavaScript WYSIWYG editor, such as Flash or Java, can get around this but then adds the requirement of a browser plugin.

True inline WYSIWYG editing

True inline WYSIWYG editing combines the advanced preview mode of popup WYSIWYG editing interfaces while maintaining the clean lines of inline WYSIWYG editing interfaces. For me, true inline content editing needs to be done with a JavaScript WYSIWYG editor, but not one that simply uses the browser's own editable region. A true inline WYSIWYG editor needs to alter the source of the page as the author makes changes to the content. By editing the source directly, and not just inside an editable iFrame, the author gets a truly accurate preview while editing content without having to switch to a preview mode.

This is really the holy grail of content editing interfaces, bringing inline HTML editing programs like Dreamweaver into the browser. So what's the catch? You'll be very hard pressed to find a JavaScript WYSIWYG editor that can edit the source of a page directly without requiring some sort of custom browser plugin or having strict browser requirements, like IE or Firefox support only. Mozile, the Mozilla Inline Editor, is about as close as you can get at the moment, but that editor is far from complete and doesn't work in IE.

PHP_CodeSniffer 0.7.0 (beta) released

Version 0.7.0 of PHP_CodeSniffer was released today. It wasn't intentional, but it has been almost two months since the last release. Consequently, it has the largest changelog of any release so far, including a couple of backwards compatibility breaks for those writing their own coding standards. There are also ten new sniffs for you to use in your own standards, or as examples for writing your own sniffs.

The most important feature added was the auto-detection of line endings, previous discussed here. This feature allows for PHP_CodeSniffer to be used correctly for projects that don't use Unix newline characters.

This release also has also seen the unit test suite grow to 100 tests, a nice milestone.

You can download the new release, and view the changelog, on the download page.

MySource 4.0 internal demo

Today, I gave the first internal demonstration of MySource 4.0 to Squiz. At lot of people have been waiting a long time to get a look at our progress and it was great to finally be able to show them what we've been working on.

The first part of the demonstration was MySource 4.0 itself. I showed the installation process and created a simple five page website with an imported design. The inline content editing interfaces were shown with raw HTML text boxes only. Finally, I demonstrated MySource 4.0's WebDAV integration by editing a Word Document that was uploaded through the content editing interface.

The second part was a batch processing test. I added and deleted a web path and role to 135,000 pages that we had previously imported into a test system. Adding took around 26 seconds. Deleting took around 6 seconds. To compare that to MySource Matrix, we would be acting upon somewhere between 540,000 and 800,000 assets and both adding and deleting would take somewhere between 10 and 24 hours, depending on hardware and load.

The last part of the demonstration was a preview of Viper, the MySource 4.0 WYSIWYG editor. This was really exciting and drove home our goal of inline content editing. Unlike other WYWISYG editors, Viper is written in JavaScript but does not use the browser's built-in editable region. The result is an editor where we have complete control over how it works, how it looks and the HTML it produces.

We are now busy preparing features for the first public demonstration at the MySource Matrix International User Conference 2007 in September.

MySource 4.0 public demo in September

The first MySource Matrix International User Conference will be held in September this year. It's a chance for MySource Matrix users to get together and discuss the product, their implementations of it and how it can improve the organisation and delivery of content. It's also a chance for MySource Matrix users to see a preview MySource 4.0.

MySource 4.0 is still a fair way from being feature complete, but the demonstration will still showcase some exciting improvements to our content creation and editing interfaces. It will also be a chance for us to describe where we are going with this product and the kinds of development that we will be able to do in the future.

The details of the conference are:

MySource Matrix International User Conference 2007

The conference is designed to bring our user community together on an annual basis to provide best practice advice in all areas of content management and web technology.

Squiz will coordinate a cutomised program delivering functional, practical solutions that maximise the benefits of your investment with MySource Matrix.

The agenda offers a broad range of presentations and case studies lead by recongnised industry leaders and current users. Held at the Radisson Hotel on the Gold Coast, the venue will be conducive to collaboration, innovation and learning.

Your registration also includes a seat at the Squiz Innovation Awards Dinner on Thursday night, 13th September. Prizes will be awarded for users, who demonstrate excellence in MySource Matrix implementation & website innovation.

Why you should attend:

• To meet and liaise with industry leaders, topic experts and the user community
• To broaden your skill set and knowledge base - delivering more relevance to the organisation
• To gain efficiencies and best practice techniques in using MySource Matrix
• To ensure you are up to date with the latest software developments
• Be one of the first to see MySource version 4 to be released in 2008

Key Agenda Topics:

• Designing usable Web 2.0 applications
• Techniques to ensure maximum accessibility, usability and functionality
• First public preview of the much anticipated MySource version 4
• User oriented information architecture for effective web content management
• What's happening with de-centralised publishing? Pros and Cons.
• MySource Matrix Tips, Tricks, Shortcuts and more (Manual included)
• User workshops

To register for the conference visit:
http://www.hotelnetwork.com.au/conferences/conferences#squiz7
For Further details please call 02 8507 9000