What ISVs Should Know - Part 7 - QA teams
So, after meeting with many companies during their move to the cloud, I find myself in two types of conversations:
- QA teams that don’t think anything should change as their organization is moving to the cloud.
- QA teams that are not sure what their roles should be now that their organization is moving to the cloud. I received a question from a load testing team leader asking, “If the cloud is elastic with infinite capacity, what am I supposed to be testing?”
The fact is, QA teams should adjust their processes and tools to improve productivity, efficiency, and visibility.
Let’s start with the obvious – cost optimization. In data centers, QA needs to buy hardware for testing. Since they also need to run load testing, they often buy hardware that is similar or identical to the hardware in production, and they run all their tests on that hardware.
In the cloud, you pay for what you use. So running all the tests on production equivalent instances is inefficient. Note the following:
-
Separate the testing into application testing and load testing. For application testing, there is no need to run the tests on full-scale hardware.
-
Load testing can be on full-scale hardware, but it should be limited in time – once the tests are done, the instances can be stopped.
-
Running automated tests on one instance for 10 hours costs the same as running those tests on 10 instances for 1 hour. If you can, use that to shorten your testing and time to market.
-
Manual testing requires instances to run during office hours only. Make sure to shut down the environments during off-hours and monitor it closely. If needed, you can schedule to launch the environments at a specific hour to save time for the teams.
-
Test for cost:
- How many requests of each type can an instance/pod/task process?
- How much would a tenant cost?
- How much would the additional functionality impact performance?
- How much would the heavy query cost?
- What is the impact of different features on Storage, Networking, Compute?
I’ll elaborate more in a post dedicated to cost-per-tenant.
In short, there are adjustments to be made to the processes and procedures in QA, but when done well, they impact the organization’s operational spending, time to market, and pricing strategy.