Cloud environment Automated Regression testing for automated regression testing created from scratch.
How Etteplan became the test automation partner thanks to our extensive expertise and also our knowledge of both QA and DevOps.
Background
Around 2018 one of our customers was focusing on ensuring the high quality of their application, which was continuously developed to deliver the best possible software for the end-users. This web application is used by professionals who build and maintain communication networks enabled by the customer. The frequent new releases of course required testing, posing challenges from cost and test volume points of view when handled manually. Therefore, the customer evaluated Quality Assurance (QA) partners who would be able to support them with Test Automation needs and requirements, as the only option was to move towards regression test automation. Etteplan became the test automation partner thanks to our extensive expertise and also our knowledge of both QA and DevOps.
Business needs
The main goal was to build a Continuous Integration Environment for Test Automation, covering the key functionalities of the web application. We were able to design and create the regression test environment from scratch (using existing manual testing scenarios as the starting point). The customer trusted our expertise, however, from their point of view it was crucial to have a reliable, maintainable, and inexpensive Environment in place for Automated Regression testing execution which would cover the key functionalities of the web application.
Solution
Evaluation
We had the unique opportunity to start with a blank page, assess the situation, and have a dialog with the customer while going through our recommendations. The first instance of the Continuous Integration Test Environment was built and hosted on Etteplan premises with a dedicated server with Jenkins managing the whole regression. Each new application build triggered the regression execution. In parallel, we addressed topics such as versioning and storage of test cases, official test report, and regression execution process. By mid-2019, we had reached a satisfactory coverage level and moved to a stabilization phase.
The natural move was then to consider migrating the regression environment to the Cloud. Even though the environment setup was not very complex, it still required maintenance and availability and security expectations also generated needs for resources. To sum up, these were the aspects we wanted to optimize.
Three top cloud providers were taken into account. The comparison criteria were: offered services by each cloud, references, user-friendly portal layout, and price competitiveness. In our case, we had quite precisely defined expectations in terms of needed services (based on the existing on-premise solution), even though we didn’t replicate it 1 to 1 to the cloud, but introduced few improvements to the environment architecture.
After a thorough evaluation, Azure was chosen as the preferred provider. The reason behind this was the highest score in our comparison. Apart from that, the additional factor which convinced us was the fact that the customer already had some Azure services in other departments of the company.
Cloud migration
At the beginning of 2020, we launched the migration project and designed a cost-effective solution. We planned to migrate the existing, up and running on-premise environment to the cloud. The migration included steps such as system elements configuration — Jenkins server configuration, Jenkins jobs migration, creation of virtual test machine image with Win 10, Chrome installation with update freeze, Python3.7/Selenium/RF, etc., which will be used to execute the regression run, network configuration for synced devices
The high-level architecture of a newly built solution is as follow:
All performed tasks can be grouped into few areas (Azure, Jenkins, Setup…):
- Azure — under one Resource Group we have:
- Two Virtual Machines created and configured:
— Linux – host virtual machine for Jenkins server
— Win10 – Test machine with Jenkins agent configured, started, and used only when test regression is running (pay-as-you-go)
- Network interfaces configured — inbound and outbound rules allowing the only customer and Etteplan networks
- Storage account and disk to store the latest build files
- Other services like Public IP address, Network Security Group, Routing Table, Virtual Network
2. Jenkins
- Configuration
— Cloud storage mounted to store latest application builds
— Jobs imported from Etteplan Jenkins from an original on-premise solution
— Created admin and tester users with adequate roles
— Generated tokens to trigger builds remotely (using POST method)
— Configured access log
- Jobs refactoring
— Application builds are copied to Jenkins from cloud storage instead of on-premise storage
— Virtual machine (Win 10) is now started just before the regression test are executed and deallocated after the regression run is finished to reduce costs (pay-as-you-go)
3. Regression setup
- The regression execution stays as is, which means Short regression immediately after new application build, full regression every night on the latest application build
- E-mail notification send after each regression execution
- Jenkins accesses granted for customer’s technicians to allow verification of regression execution details
After a smooth handover between the two environments, we continue to maintain the regression test automation cloud environment and suggest enhancements to the test scenarios.
Experience from daily usage of Azure
Nearly one year of daily usage proofs that it was a good idea to move from an on-premise solution to the cloud even though the environment architecture is relatively simple. The customer’s business needs are fulfilled. The key benefits are:
- 24/7 stable environment availability without engaging any internal resources (human and hardware)
- Easy access to all resources via Azure Portal
- The predictable and low monthly cost
- Cost-effectiveness with use of pay-as-you-go pricing :
— Win 10 VM which is running and charging only during regression execution
— Switching off the environment during periods when application development and regression running is on hold (e.g. monthly summer break)
As a result, the customer now has a state of the art regression test automation environment. They are able to make rapid and reliable sanity checks of the latest application releases and provide quality software for the end-users.
