Automation plays an increasingly important role in the global economy and in daily experience. Engineers strive to combine automated devices with mathematical and organizational tools to create complex systems for a rapidly expanding range of applications and human activities. Many roles for humans in industrial processes presently lie beyond the scope of automation. The problems related to automation system failure are discussed in this paper. Different features for building good automation systems are also highlighted in this paper.
Automation is the new mandate in today’s fast paced world of software development. Companies are hard-pressed to get their products out as quickly as possible. Traditionally testing has long been seen as the bottleneck to getting software releases out the door and software automation as the solution to removing that bottleneck. However, software automation more often than not fails to live up to expectations and in fact seldom returns the investment put forth. Although manual tests may find many defects in a software application, it is a laborious and time consuming process. In addition it may not be effective in finding certain classes of defects. Test automation is a process of writing a computer program to do testing that would otherwise need to be done manually. Once tests have been automated, they can be run quickly. This is often the most cost effective method for software products that have a long maintenance life, because even minor patches over the lifetime of the application can cause features to break which were working at an earlier point in time.
2. The Problems
A majority of automation projects fail. In fact, most if not all automation projects fail to live up to peoples’ expectations and many automation projects are flat out abandoned. This fable illustrates several problems that plague test automation projects:
2.1 Spare time test automation. People are allowed to work on test automation on their own time or as a back burner project when the test schedule allows. This keeps it from getting the time and focuses it needs.
2.2 Lack of clear goals. There are many good reasons for doing test automation. It can save time, make testing easier and improve the testing coverage. It can also help keep testers motivated. But it’s not likely to do all these things at the same time. Different parties typically have different hopes. These need to be stated, or else disappointment is likely.
2.3 Lack of experience. Junior programmers trying to test their limits often tackle test automation projects.
The results are often difficult to maintain.
2.4 High turnover. Test automation can take a while to learn. But when the turnover is high, you lose this experience.
2.5 Reaction to desperation. Problems are usually lurking in the software long before testing begins. But testing brings them to light. Testing is difficult enough in itself. When testing is followed by testing and retesting of the repaired software, people can get worn down. Will the testing ever end? This desperation can become particularly acute when the schedule has dictated that the software should be ready now. If only it weren’t for all the testing! In this environment, test automation may be a ready answer, but it may not be the best. It can be more of a wish than a realistic proposal.
2.6 Reluctance to think about testing. Many find automating a product more interesting than testing it. Some automation projects provide convenient cover stories for why their contributors aren’t more involved in the testing. Rarely does the outcome contribute much to the test effort.
2.7 Technology focus. How the software can be automated is a technologically interesting problem. But this can lose sight of whether the result meets the testing need.
3. Deciding what to automate
Organizations sometimes jump into automation without carefully considering what to automate. For example, many decide to automate the most complicated items, such as a difficult test case, thinking that automation will free them of this headache. But soon they find themselves spending more time automating than testing.
When deciding what to automate, organizations should consider the following questions:
• How often do these items get used?
• How many people use them or rely on them?
• Are these components part of a test that everyone has to use?
• Are they difficult to maintain?
• Which items do my customers add (or not add) value to?
When building an automation strategy, teams must decide what to automate based on what will yield the greatest benefit overall, not simply aim for the most technically ambitious projects. As illustrated in Figure 1, organizations should put aside the desire to do the most difficult task first and focus on the tests that deliver immediate business value.
4. Evaluating technology against vendor claims
In the past, vendors have asserted that their automation software allows organizations to create robust and maintainable test assets. As many organizations have discovered after making their purchases, the technology behind these products often does not stand up to the claims. Worse yet, without future-proofing technology, organizations are realizing only half of automation’s potential value. When evaluating automation software, organizations should add maintainability to their list of requirements.
5. Rethinking processes from a team perspective
To realize the potential of automation and maximize its efficiencies, testers and developers need to think of themselves as part of an assembly line. If test engineers build test cases that they know will need to be automated, they can save the automation team a step by building in abstraction. Though it might require a little extra effort up front, this action can save the entire organization time throughout the testing process.
With the right technology, testers can start to build and run tests in a way that makes it easier for the rest of the team to move those tests through automation, as well as maintain tests for future releases. More importantly, the test organization can build a scalable framework for communication and asset sharing. Organizations need to start taking an assembly-line approach to testing and development, with individuals focused on creating efficiencies that benefit the entire team.
6. Automation is akin to any Complex Software System
Building Automation solutions is akin to building any (complex) Software system and requires the same effort (usually high effort) to implement effectively. You will be integrating a system with many parts as well as software (i.e. scripts and glue software). Therefore the same lessons learned in building systems and software should be applied to your automation project. Some items to consider are:
6.1 What are the high level goals of the automation system: performance testing, unit testing, feature testing?
6.2 What are the features you need in the automation system? Some examples are listed below.
a. Logging capabilities
b. Graphical automation scenario creator
c. API for a remote procedure calls
e. Resource Management of test/lab resources
f. Concurrent scripts
g. Capability to preparing or “clean up” the test environment
h. Versioning of test scripts
i.Documentation for administration/test script creation.
6.3 Design and document an automation architecture. An automation project needs a framework and the framework needs to be implemented and documented. The framework should lay the foundation and rules that everyone lives under. For example a shared software library for automation scripts should be implemented and used by all automation script writers. Some other items to consider are listed below:
a.Automation Architecture must be extensible. Automation requirements will grow in step with the requirements of the software under test and the automation system must be able to cope with that growth.
b.Must have instrumentation for debugging purposes. Running scripts is the easy part. Debugging scripts is what takes the most time. Be sure you have a system in place that quickly summarizes all the problems from a test execution run.
c.Your scripts should be modularized.
d.Keeping statistics on your executions. Identify test cases with a high false positive rate and either fix the root cause or remove from your executions. Conversely, identify tests that find real software defects.
e.Can the automation scripts be re-used later for different purposes. For example you might develop scripts for feature testing and later re-use them for integration testing or performance testing. Will other groups repurpose your scripts?
6.4 Tool Selection (Open source vs. Commercial vs. In- House). Regardless of which direction you go (i.e. Open Source, Commercial, Home Grown) be sure you realize the risks/rewards you will have whichever path you choose. Ask yourself if the tool fits your specific requirements right now and in the future. Some other points for consideration:
a. Is the tool well supported?
b. What is the average time for a feature request or bug fix to be implemented?
c. If using Open Source, is the said community active in developing features and fixing
defects? Perhaps this is not an issue if you have people on staff that can maintain the software.
d. If the software is home grown do you have developers inhouse to support the product? If so how likely is it these developers or development group will be available three to five years from now?
7. Consider the Long Term Implications of your Choices (i.e. don’t bite off more than you can chew) Every choice in building an automation platform has consequences. Some consequences seem fine in the short term but can be project killers in the long term. Be aware that you have scarce resources: time, computing, money, and people. Some issues to be aware of are as follows:
a. Focusing on the low hanging fruit allows the automation architect to “prototype” the new system and easily make changes to his/her approach. This also allows the automation team to firmly establish and enforce the correct processes for running an effective automation program.
b. Some scenarios are too much work to automate. For example if a scenario takes 1 hour to run manually but 40 hours to automate you don’t want to automate the scenario if it is run only once a year. That means it would take 40 years for a return on your labor investment. The average life-span of software is a mere 5 years.
c. Some scenarios introduce a much greater level of complexity to the automation system. This complexity can affect the stability and the scalability of the automation system as a whole. There should be a very strong business case to justify the risk in automating in these situations.
d. Don’t over-engineer your automation test cases. Just because it seems reasonable to perform certain “extra” steps when running test cases manually does not mean implementing the same checks in automation scripts is a good thing. Adding extra functionality in scripts unless absolutely needed adds a greater chance of test cases failing due to false-positives and obfuscates the real intent of the test cases. If you really need to check something not specifically listed in a test case then create a new test case for that specific scenario.
8. Full-Time Personnel must be qualified and fully committed to build and maintain an effective automation program
As mentioned earlier an effective automation platform demands a disciplined focus. Some issues to consider when staffing up for your automation endeavor should include
a. Carefully vetting your team from a list of job requirements including but not limited to past work experience, industry certifications, and personal recommendations related to the job position. When staffing up for an automation project create a list of job requirements that will be required or desirable for the new team members. Be sure your team reflects the appropriate level of talent.
b. If automation staff focus is divided between multiple complicated projects this will reduce the chance of success for your automation program.
c. Staff for the project is transient. Staff for the automation project is cycled through without regard to the deleterious effects of the automation system.
d. Don’t bother automating more test scenarios if you don’t have the staff to maintain the increased quantity.
9. The Test Tool Product you Buy is a “Product” not “Your Automation Solution”
Don’t expect to purchase an automation system in a box. You will be required to do much of the work yourself when creating an automation platform. Purchasing/Obtaining a tool is just one of many steps along the way.
10. Defining a Successful Automation System
Create a list of criteria for determining success/failure and revisit these criteria at specific intervals over the life-span of the automation project. Some possible criteria is listed below:
a. Low rate of False Positives
b. High rate of defects found
c. Reducing or maintaining staff growth costs
d. Reducing Manual testing time
Successful automation involves careful planning, disciplined execution, meticulous maintenance procedures, and a dedicated staff as well as a management with a willingness and patience to fully support (with money, time, and labor) an automation project. The first and most important step in building an automation system is realizing the complexities involved and the positive consequences of careful planning and execution of your automation system.