In this era of everchanging technology and product functionality it is very important to identify the right test automation tool that can enable you to accelerate high quality software delivery by considering both the current and future roadmap of your product. Selecting the optimal automation tool is not an easy task as we think, unless we follow a clear process that can lead to right decision.
Key Characteristics
of a SMART Automation Tool:
S – Simple: A smart tool should be simple to learn and use and
this is the key characteristic for scaling the tool to wider teams and
projects.
M – Measurable: A smart tool should have inbuilt capabilities to measure
automation progress and report of on various automation metrics including the
benefits/ROI.
A – Achievable: A smart tool should be able to achieve seamless across
different technology landscapes and achieve our desired automation targets.
R –
Robust: A smart tool should be robust to
handle the technology upgrades and DOM changes more gracefully and have minimal/no
false failures
T – Timely: A smart tool should be more efficient/productive in developing/maintaining automation scripts in less time. It should be able to do faster execution/runs generating more effort savings.
Selecting a best-fit SMART automation tool is always challenging and complex due to the availability of wide range of tools with varied features and their big, boasted claims. We need to understand no single tool in the market is the perfect automation tool. All we need to understand and identify the one that will best fit our need to automate our complex technology landscape and help us to overcome the current limitations in our automation. Over and above the tool that we select should enable us to reach our quality/automation targets in less-time and effort. Context is very key for tool selection.
For this reason, we need to invest considerable time and effort to make this right decision. We can’t decide on a tool just by looking into its features and its claims. Once you decide on a tool to implement, be clear that it is a one-way journey, and it will hurt a lot to reverse your decision in the middle of the journey. So, I will recommend having a well structure assessment and selection process for making the right decision without any bias.
Key recommended evaluation stages:
Stage 1: Static Evaluation –
At this stage, you will do static evaluation of the top market leading tools without leveraging any trial/product licenses. You will shortlist 4 to 5 top market/industry leading tools and gather details on the supported technology, features, limitations, license cost and client reviews/feedback. Based on the details gathered from multiple sources (check different forums to get unbiased details), do rank the tools based on which is close to meet your needs. From this static study/evaluation, shortlist 2 to 3 tools for next stage of evaluation.
Stage 2: POC Evaluation -
At this stage, you will engage with the vendors that you have shortlisted in previous stage and plan to conduct a detailed Proof of Concept (POC) using the trial version on your organization environment. You will choose few key businesses critical and complex to automate applications for the POC evaluation. This evaluation needs to be done in a detailed manner as it is a critical decision-making stage. I will recommend doing this phase in partnership with the vendor, as we don’t want to waste lot of time in understanding the tool. We need to capture all the evaluation outcomes for various parameters in a well structure manner, which we will discuss at the later stage of this article. At the end of this evaluation, you will shortlist 1 tool which best fits your need and proceed for limited purchase.
Note: I will not recommend doing POC at external application/environment as that will not expose the challenges/limitations and will not give complete confidence for buying. If you are doing evaluation at your organisation environment, then do note that it may take long time for completing all the process/approvals required for enablement.
Stage 3: Pilot Evaluation –
At this stage, based on the outcome of POC stage I
would recommend buying limited number of licenses to do a pilot on your
key project/scrum teams, before you go for major adoption/ implementation. This
Pilot stage is very critical to make the decision for long-term adoption. At end
of this stage, if your pilot is successful then you will proceed to buy
additional licenses for implementing this tool on long-term basis across all
your projects/teams and if not successful then you can decide to
limit the use of the tool for short-term on limited project or decide to
terminate the decision to move-forward.
The success of this stage depends on various factors including scalability of this tool across different technologies within your organization context/roadmap, integration capabilities, E2E automation capabilities, ease of learning/simplicity to adapt by new users and productivity/automation efficiency…etc.,
Key Requirements
for Tool Selection:
It is very important that the tool can automate seamlessly on various application technology landscape used in your organization. It is recommended not just to focus on the current technology landscape, but also on the future technology/product roadmap. Various technologies to consider for automation feasibility study are listed below.
Web application | Mobile Apps | Desktop application
| Mainframe | API | Documents | DB/ETL | FTP | Email | PDF | Images | Business domains
(SAP/Pega/SF/Oracle)
รจ Check-out the below list of detailed key parameters to be measured during evaluation.
Current Automation Limitations:
Once you have asserted that the new tool has good capabilities to automate on different technologies in your organization landscape, it is very important to establish that the tool has enhanced capabilities to overcome the limitations/challenges in your current automation.
I have listed few major limitations that we face in
our test automation journey.
- Need for technical skilled testers to build automation.
- Automation development/maintenance takes more time due to complex coding.
- Automation not able to cope up with the application change
frequency.
- Unable to achieve higher or 100% automation coverage.
- Need for multiple automation solutions/frameworks to automate
different technologies.
- Unable to automate E2E scenarios that involves multiple
interfaces/technologies.
- Automation is restricted only for regression test scope and unable
to scale into In-sprint test scope.
- Complexity in object identification and performing desired actions.
- Current automation scripts are not robust and result in false
failures over time.
- Difficulties in integrating with other test management, Agile and
DevOps tools.
- Unable to generate various automation metrics related to coverage
and execution.
- Need for dedicated Automation SMEs to enhance/maintain automation
frameworks.
- Inability in sharing the automation assets across test projects.
- High complexity in scaling automation frameworks to cope up with
technology upgrades.
- Challenges in integrating custom solutions with existing automation
frameworks.
- Challenges in storing and accessing automation test results in
centralized platforms.
Adaptability skills:
One of the key characteristics of any good
automation tool is its adaptability. It should be easy to learn and implement
in quick time and shouldn’t require expertise in coding skills. This will
accelerate you automation journey and help to achieve your targets on timely
manner. Code-less automation tools will be a great choice, considering
the smaller learning curve, faster to adapt and implement. Take my advice, not
every single code-less tools available in the market are simple and easy to
adapt. So, making the better choice based on your context of resource skill
set available in your team is very key for the success of your automation
journey.
Another key aspect of adaptability is flexibility/customization offered by the tool and vendor's support system that helps you to overcome challenges in adoption. For any tool, during its early stages of adoption, you will encounter lots of issues and it is very important to have vendor support team that is responsive and ready to provide solutions. This is key factor for successful adoption of the identified tool.
Cost/Effort constraints:
Last but not the least, the key requirement/factors
like the availability of budget or cost constraints and resource
capacity play a key role in selecting the optimum best fit automation tool
for your team. I would recommend not to discard any leading licensed tool
because of only cost constraints. As long as if you can substantiate your tool proposal
with data that it can generate good ROI and it will help to propel
your automation roadmap to meet your business targets, it would be a worthwhile
investment to make considering the future of the business.
List of detailed parameters to be measured during evaluation:
Below list is just an example, feel free to add based on your organization context.
|
Key Evaluation Parameters at the Tool Level |
|
Object Recognition and Ease of
Maintenance |
|
Ease
of using Record & Play feature and leveraging its outcome to script
development |
|
object
repository management & ease of maintenance |
|
Ability
of the smart scan engines to capture objects accurately with less effort |
|
Availability
of inbuilt actions and commands to perform desired operations on the UI |
|
Ease
of script maintenance for application technology upgrade |
|
Level
of self-healing ability to fix scripts based on UI changes |
|
Level
of Al/ML feature enabling automation of complex UIs & technologies |
|
Test Execution Controls & Speed |
|
Ability
to execute the scripts with greater speed and accuracy |
|
Test
data control for various test suites (Smoke, Regression, In-Sprint) |
|
Ability
to recover from failures and progress |
|
Re-run
of failed test cases to desired reattempts |
|
Parallel
Execution capability for different technologies (Web/Desktop/Mainframe/Mobile Apps)
|
|
Reporting Capability |
|
Ability
to capture the consolidated & detailed test results at the test case
& test suite level |
|
Ability
to filter & drill down the test execution results |
|
Ability
to create custom reports based on the need |
|
Report
publishing capability through email with summary & attachments |
|
Additional Features |
|
Shared
workspace for collaboration & storing project artifact's |
|
Support
for BDD test framework (Given/Precondition, When/Action, Then/Outcome) |
|
Level
of support for adding custom coded functions |
|
Ability
to integrate existing Selenium/UFT scripts with Code-less scripts execution |
|
Integration Capability |
|
Level
of integration with CICD Orchestration tools (Jenkins/Azure etc.,) |
|
Level
of integration with Agile management tool (IRA/Azure etc.,) |
|
Level
of integration with Test Management tools (Zephyr/QTest/Azure/ALM etc.,) |
|
Level
of Integration with cross platform/browser testing tools (BrowserStack/SauceLabs
etc.,) |
|
Pricing & Support |
|
POC
engagement & support |
|
Tool
Technical Support |
|
Tool
Upskilling Support |
|
Cost
of Investment |
|
Value
for Money |
Note: The example given below is a focused on a particular
technology (web applications), please do extend for other technologies appropriately.
|
Key Evaluation Parameters at the Application
Technology Level |
|
Technology -
Web application evaluation |
|
Ease
of recognizing UI objects and able to perform desired actions |
|
Ease
of developing scripts using inbuilt methods & actions |
|
Ease
of using reusable modules in script development |
|
Ease
of adding various assertions/checkpoints to validate results |
|
Ease
of script maintenance for UI & functional changes |
|
Ease
of validating UI results with data in db |
|
Ease
of running the script in multiple platforms/browsers |
|
Ease
of running the script for multiple data sets |
|
Ease
of managing test data for the scripts |
|
Ease
of doing failure analysis |
|
Stability
& consistency of the script results during multiple runs |
|
Ease
of navigating from once technology flow to another technology |
|
Productivity
Assessment: |
|
Script
development time (mins) - consider test complexity |
|
Script
maintenance time (mins) - consider test complexity |
|
Script
execution time - Single run (mins) |
|
Script
execution time - Parallel 10 thread (mins) |
Scoring mechanism for evaluation:
You can use rating-based score system based on the level of support provided by the tool for a given evaluation parameter.
|
Support
level |
Rating Score |
|
Very Good |
10 - 8 |
|
Good |
7 - 5 |
|
Moderate |
4 - 1 |
|
No Support |
0 |
Final Summary:
§ Identifying the right automation tool needs to be
done through various stages of evaluation and clear-structured process
that can lead to right decision.
§ Context is very
key when determining the best fit tool for your organization. So, define
the best-fit criteria for your team and measure the tool against the defined criteria’s.
§ In this age of cutting-edge technologies and AI
trends, it is better to identify and invest on a right smart automation tool
(code-less + AI enabled) to achieve your automation roadmap and quality
targets.
In my
recent past automation tool evaluations, I have recommended “ACCELQ” as
one the best AI based code-less automation tool, which has exceeded the score
when compared against other similar code-less tools in the market. Again, I
stress that when you say a tool as best-fit it is based on your context and
expectations from the tool for a given organization.
If you are
looking for my expert consultations on the test automation strategy/best-fit tool
identification, please do reach out to me (pg0rakesh@gmail.com). Thanks, and all the very best!