It has been five years since I first used NUnit and have used it with great success on almost every project since then. This post briefly reflects the pros and cons of Automated Developer Testing I experienced.
. Zero Defects can be the norm
Given a serious amount of control on projects I insist on ~100% coverage for the most complex pieces. Every time this happened the complex code was delivered with zero (or very close) defects. Admittedly before NUnit I received a lot of praise of stable code and attention to detail, but with with NUnit we can even redesign working code to be more maintainable and still release with almost 100% stable code. Redesigning for maintainability is not practical without a suite of automated tests – it normally requires a complete rewrite.
. 100% Coverage is for Fools
Sometimes creating Automated Tests make prefect sense, and sometime it costs more time than it saves. I aim for 100% coverage on reusable libraries, critical architecture and anytime I think the business rules are far too complex for QA to completely understand let alone regression test on a regular basis. 100% Coverage on simple data manipulation web page will be a frustrating waste of your time.
. TDD – Good luck with that!
Unless your company has several years experience writing Automated Tests you will not get buy in for TDD. Learn to walk before you try to run.
. Continuous Integration is an easier win
I can install a Cruise Control in a day and educate all devs in one 30 minute presentation. Before looking at Automated Developer Tests I highly recommend rolling Continuous Integration our first. It is a massive win for very little effort. Managers and developers alike always see the value, this should build up your credibility enough to start an NUnit pilot.
. Screw Mocks, use a real database, ldap etc
Yes Mocks can work, but in my experience I have seen developers waste time coding Mocks, Object Mothers etc. A far simpler approach is to commit to a reasonably large amount of upfront effort so the Automated Tests setup and tear down a real environment. This framework should be created one expert developer and will result in some complex, hard to maintain code, but it then becomes very simple for even junior developers to test large sections of their code, not just wimpy Unit Tests. This is the foremost reason I have had so much success with NUnit. I have my own framework for SQL Server/ Oracle which took about three re-writes to become as simple and stable as it is today – you will need great database skills and good C# but once written it just works. Via Cruise Control we automatically spot problems in build scripts instantly, if anyone changes sproc parameters without updating the DAL we spot that too! I even have a suite of test on one project that does an end-to-end publish mimicking data coming from several internal feeds, being data scrubbed by our system, transformed into records in our system and push out the other end to table ready for user consumption! Probably 100klocs of C# and PL/SQL are executed by a single test – if anything breaks we know about it very quickly.
. Hundreds of small tests or a few Large Tests?
A classic mistake I see is developers taking the Unit word too literally and set about writing literally hundreds of test for one module. Although sound in theory, it is madness in practice. These developers run out of time and generally I see a stack of pretty useless tests. You should aim for the most coverage you can per test, of course when such tests break it is not immediately obvious what broke but you will be aletred that something broke, and Cruise Control will show you what code changed since the last build – making it an easy fix almost every time. So do you want to spend weeks writing hundreds of tiny tests or a few days writing all-encompassing tests? In reality no team I have worked on has every has been permitted the time to write hundreds of test. Also most developers don’t have that kind of patience which is why TDD has a low adoption rate. To clarify this point in general go black box rather than white box – not I said in general, use your common sense here :)
. NUnit can really stress developers out
This is a serious problem when developers are under the gun to meet deadlines and an NUnit test breaks the Cruise Control build. Of course everyone knows all work should stop until the build is fixed, but 90% of developers will just mark the test with [Ignore] and work on what their manager is shouting for.
. Some Managers think that is what (cheaper) QA staff is for
No explanation is needed I am sure. Some managers get the SDLC and some do not, and I can offer no solution to this problem. Some managers can only focus on very short-terms goals.
Paul Lockwood Development (General)
