Every company develops their own methods for testing. A lot of these patterns naturally form around development cycles. Some are pretty informal while others grow into complex structures with various levels of validation. With that said, there are some definite patterns for testing that have been established as good practice as well as those that should be avoided.
Martin Fowler has a great writeup on the “testing pyramid” – a favorite in visualizing where testing efforts should be. This idea lays out a simple-to-understand setup for testing: UI tests are at the top of the pyramid, Service tests are in the middle, and Unit tests make up the bottom or foundation of the pyramid.
There are various opinions on how each level of testing should be distributed. Some argue that this layout suggests the level of effort should be focused on unit tests while UI tests should make up a much smaller subset. Though this pyramid may not necessarily show the needed quantity of tests per level, it does accurately reflect the level of reliance of UI tests.
Alister Scott, a core contributor to the Watir community, wrote an article about the dangers of inverting that pyramid and relying heavily on UI and manual testing alone. In his article, he mentions how easy it is to channel all of a team’s focus for testing to just manual and automated UI tests. He coins this inverted visual “the software testing ice-cream cone.” The ice cream cone is considered an anti-pattern as it is an inefficient model for testing. When there are large suites of automated UI tests then the majority of a team’s time is held up in maintenance or troubleshooting problems.
The level of confidence in those tests shrinks as run times and the number of false failures increase. A “false failure” could be anything that doesn’t reveal a legitimate bug like bad data or a server going offline. All of this undermines the core purpose of automation – save time and help ensure a healthy product.
Creating automated tests at any level requires striking the right balance. UI tests can help expose gaps where a unit test may be the better option. If there is a heavy reliance on manual and UI tests, it may be time to weed through them and determine if they would be better at the service or unit test level. Have you found a good balance for your UI, Service, and Unit level tests? Let us know!
