Conflair Inc. - IT Solutioning Company

Test Automation

If you know the theory and just want to see what can Conflair do for you, click here...

What is Test Automation?

As with many great things like sliced bread or organic food, the idea behind functional test automation is simple: let’s have the computer drive the application-under-test and compare its outputs to the expected results. After all, doing something over and over again quickly and in exactly same way is a human weakness yet a computer’s strength.
In a sense, Test Automation tool mimics a human tester: it clicks buttons, enters data, invokes web services, runs SQL queries and then compares the output to the expected results. If the actual results match the expected, it reports Pass; otherwise it reports Fail.
Test Automation is supported by a number of tools. There are numerous experts who have deep knowledge and extensive experience in using these tools. However, as always with the IT, it’s not the tool, and even not the test automation that you want – you are looking for a solution to certain problems: shortening the test cycle time, improving the reliability of testing, allowing tests that are based on the data generated on-the-fly. Conflair, in the spirit of IT Solutioning, creates Test Automation solutions to fit your needs and to answer your objectives.

Test Automation Benefits

There are two major benefits from functional Test Automation: increased reliability and reduced timeline.

1. Increased reliability
Automated scripts are more accurate in test execution than manual testers in terms of specifying inputs and comparing actual and expected results. Consider the following tasks in order to compare between manual tester and an automated script:
  • Entering 2,345 records into a database through a web form and comparing the records in the database with the expected results
  • Comparing two output files, actual and expected, 7,987 rows each, with 57 fields in each row
  • Running an on-line request through web services and comparing the response to the output of the system-under-test within 3 seconds.
2. Reduced timeline
Automated scripts, when developed and executed properly, can be executed much faster than manual tests. Going through 173 fields in 7 tabs would be quite a time-consuming task for a manual tester. For an automated script it would take almost same time as going through 5 fields, provided that the system response time does not slow it down significantly.

Both of these benefits, of course, become real if and only if the automation is done properly. Having a good tool is important, yet not enough: expertise in test automation is just as important.

Test Automation ROI

There are a number of sources for Test Automation return on investment:

- Testing time and resource savings
Since the scripts can do it faster, you can run the same number of scripts using fewer resources. Even taking into account the initial investment to create the scripts, the savings are quite significant. Consider, for example, the following scenario:

Manual Test Execution:
5 Test Engineers running a 5-days test cycle every week, hourly cost - $65. The labor cost for one Test Cycle will be $13,000, given 8 hours working day.

Automated Testing – Initial Scripting:
3 Test Automation Experts working 50 days to establish a Test Automation Infrastructure and automate 80% of the manual test cases, hourly cost - $100. The cost of the setup will be $120,000. 80% of the Test Cases that take 95% of the testing time will be automated, leaving manual the ones that require analysis rather than repeated and tedious execution.

Automated Test Execution:
In order to execute the automated scripts and the remaining manual test cases, two Test Engineer days will be required. Thus, the cost of one test cycle will be $1,040, given $65/hour.

In this scenario, after 28 weeks Test Automation will start bringing benefits from the pure time and resource saving perspective, saving close to $300,000 by the end of the first year.

Contact Conflair to perform Test Automation ROI analysis for your company

- Support costs savings due to higher quality
Properly developed and maintained, automated scripts will catch more bugs than the same manual tests prior to going live. There are two reasons for that. First, automated script is more accurate than human testers in comparing files, messages, options and other actual outputs to the expected ones. Second, human testers will have more time to concentrate on really challenging tests, and will have a better chance to find problems there. This means less defects making it into production, and less support calls.

- Company image improvement and increase in sales due to higher quality
Same reasons that lead to the reduction of support costs will drive the user satisfaction upwards – so the sales might go up as well.

Test Automation Challenges

Test Automation has its share of challenges. Among them:

- Coping with change
Automated scripts are sensitive to changes in the system-under-test. If not designed properly, automated script will fail not because of a defect in the system under test but due to login button changing its caption from [OK] to [Submit]...

- Selecting what to automate and what not to automate
It is really sad to spend time and money on automating a functionality that will be phased out in three months... Or to invest 80 hours in automating something that takes 3 minutes to execute. As with many issues in IT, aligning Test Automation with project needs and development goals is crucial yet not easy.

- Making test execution process smooth
A Test Engineer executing manual script employs (hopefully...) her or his best judgment as for when and how to run it. Many times a number of setup elements, so obvious for a human tester that they never make it into test case's pre-conditions, are an insurmountable roadblock for an automated script. It is extremely frustrating to start 5,000 test scripts test suite at 5:30PM to discover next morning that it has stopped because test case #13 could not continue since test case #12 had logged out... Conflair has a number of solutions for these problems. Some of them are universal; others are company-specific. Most of them are in-between.
Want to know more? Drop us a line at

Common misconceptions

“I can automate it all” – no, you cannot. And you probably do not want to. In most cases, it’s worth automating 20% of the test cases that take 80% of the time of your testing team yet only 20% of their execution requires some analytical thinking. The rest of the test cases, ones that require more thorough analysis, you probably would like to leave to the human mind.

“I can do record-and-replay type of automation” – no, you cannot. It usually does not work even the first time, not to mention after you make changes to the application.

“Each change in the application will require re-writing of all automated scripts” – this is true only if you do not do it right. If you automate your testing according to a sound methodology, maintenance will be manageable.

“Automation is for stable products only. I'm implementing a customized software package, and I have to be done in three months. Automation cannot help me” - in fact, it can help you a lot. The approach should be different, though. Conflair has a special methodology for these cases – the R&R.

What kind of Test Automation you need?

There are two different types of Test Automation – for Development Projects and for Implementation projects.

Click on one of the links below that describes your project better:

I have a Development project
I’m developing a software application that has multiple versions, should be tested repeatedly for regression and requires tedious and accurate testing.

I have an Implementation project
I’m implementing a generic software application. The implementation requires quite a bit of customization that might affect the core functionality. On the other hand, I want to test all the business processes the application supports. I do not have much time, so thorough manual testing is not really possible.

Comprehensive Test Automation for Application Development

The goal of Comprehensive Test Automation is to fulfill the automation promise by providing a solid basis for automating company's testing. In order to do that, we follow the top-down approach. The principles of this approach are described below.

Test Automation Architecture
Before the scripting begins, it's important to define the overall framework – just like in software development. The first element is Test Automation Architecture – the components of the automation suite and the ways they integrate with each other.
Figure 1 provides and example of a typical Conflair's Test Automation Architecture:

Figure 1: Typical Conflair's Test Automation Architecture

Having Test Automation Architecture that fits the functionality and technological environment of the system under test, aligned with the business goals of the organization and flexible enough to allow further expansion is a necessary pre-condition for a successful Test Automation.

Automation Infrastructure
Automation Infrastructure consists of two components: procedural and technological.
The procedural component, or Test Automation Guidelines, defines the way to design, code and maintain the automated scripts. It describes coding rules, naming conventions, templates, automated script life cycle. Having solid Guidelines ensures higher maintainability of the scripts as well as shorter learning curve for new Test Automation Engineers.
The technological component consists of:
  • Reusable functions: for accessing interfaces, pulling data, reporting results, etc.
  • Data structures
  • Focused aids, e.g., a tool for tracking Requirements traceability

Creation of Automated Scripts
Well, after we have the architecture and the guidelines in place, that should be easy :).
The scripting stage is the one where Test Automation Engineer's expertise plays the most important role. From the management standpoint, it is important to ensure that the automation project is on time and within budget. In order to do that, we track the scripting progress and remove all obstacles on the way to meeting this stage's goal. At times, this requires re-prioritization of script development: if certain system module is not ready for scripting, it makes sense to delay creation of automated scripts for this module and deal with another piece for meanwhile. Another consideration is incremental delivery of automated scripts, so they can gradually replace segments of manual testing and thus start bringing benefits prior to the completion of the automation project.

Quality Control
As any other software deliverable, automated scripts should be subject to Quality Control. Usually we conduct the following Quality Control activities throughout the life cycle of an automated script:
  • Script design review
  • Code review
  • Test run
Conducted by a Senior Test Automation Engineer, these activities are aimed at ensuring the quality of the automated script.

Comprehensive Test Automation is an excellent way to automate test suites for multi-version software applications. However, software implementation projects require a different approach – Rapid and Robust Test Automation.

Rapid and Robust Test Automation for Software Implementation

Contrary to a common misconception, Test Automation can bring a lot of value for a software implementation project. Two major benefits of Test Automation – shortened test execution time and high reliability – are of crucial importance to these projects, where short timelines coincide with high quality requirements. However, in order to be useful for implementation projects, Test Automation should be approached differently.

The goal of Test Automation in implementation project is shortening testing timelines while maintaining high accuracy level. In order to achieve this goal, Test Automation should be rapid and test scripts – robust. To get there, we follow a number of principles:

- Automate processes, not test cases
In implementation projects, we prefer to automate a small part of the test case quickly and start getting the benefit of the saved time, rather than automating the whole test case and endangering the timeline. For example: Adding New User test case consists of filling out a form with 72 fields and of 15 validation points. For an implementation project we will automated filling out the fields based on external test data – the most laborious part. The validation will remain manual. If the manual test case took 3 hours to run, 2 of which were entering the data, automating the data entry piece might save almost two hours for each test run. If creation of the script took 8 hours, it will start saving time after running this test case four times.

- Avoid GUI whenever possible
While GUI is a natural way to run many applications, its unstable nature poses some challenges to Test Automation. If not necessary, automation for implementation projects seeks other ways to drive the application under test and obtain actual results. For example, sometimes it is quicker and more reliable to create records through a web services interface and to verify expected results by running database queries, and to verify the GUI manually.

- Focus on end-to-end business processes
Thorough test cases usually specify dozens of verification points. In implementation, however, there is rarely a need for thorough validation of different controls; the end result is of higher importance. Test Automation for these projects should focus on the same goal.

- Think Agile
As with Agile software development, Test Automation for implementation projects assumes the possibility and the benefit of future refactoring (changes to the code that do not alter the functionality) of the automated script. This is an opposite to stressing the infrastructure in Comprehensive Test Automation. The infrastructure is very beneficial when you are expecting multiple versions of the software in the future and significant amount of regression testing. In implementation, however, you expect the project to end soon – well, hopefully...

Making it happen
In the spirit of timeliness, the process here is simple:
  • Identify processes where Test Automation might be useful
  • Create scripts to automate these processes
  • Perform Quality Control activities
  • Re-use scripts for other tests whenever possible

Test Automation is an expense that can pay off and bring significant benefits, or turn to be a lost investment. It all depends on having the right expertise, the right approach and the right people. Conflair can help you in that. Please drop us a line at
 Get more info
 I'm interested in: 

 Sign Up for our Newsletter 

 © 2006-2012 Conflair Inc.  |   Contact us  |   Terms of Use  |  Privacy Policy