No Time To Test and No Time To Automate

This series of stories is based on my presentation “No Team, No Tests, No Time or How To Survive and Thrive when Resources are Limited” that I gave at Test Leadership Summer 2020. Test Leadership is one of the best testing conferences in North America, and only one conference that specifically designed for QA Managers, Team Leads, Directors, and everybody else who aims to become or already is a QA leader.

If you ever joined the startup as a first QA or worked on a project with strict deadlines because the company needs to hit that marked opportunity, or worked on a project with a lot of bugs in production you know that there are no limits to the limited resources. You might have an understaffed QA team, no automation or not enough automation, no processes, no budget to buy a tool, no time to build your own.

If at least one of the above resonates with you, I bet you might feel very stressed; if two or more you might want to pull your hair out. We also now live in unprecedented times and have a COVID-19 on top of everything else.

For every issue, there is a solution. This solution might be temporary, you might cut corners, and lower the standards. This solution might not satisfy everybody including you and will require a lot of compromises. Just remember that during the crisis your main goal is to survive, keep your mental health intact and ship out the best possible version of the software in the current situation.

To solve any problem you need to have a plan. It’s ok if, in the beginning, your plan looks like this: run in circles and cry out loud.

I should say it's not a bad plan. If you feel frustrated, stressed, overwhelmed go outside, walk around a block. If it did not help —repeat until the level of endorphins is increased enough so you can start to think clearly again. This simple plan saved me in a lot of difficult situations, helped me not to overreact in many cases, and, as a free bonus, to stay fit (probably not that fit, to compensate for sitting in front of computer days and nights, I need way more walking and way less working)

Before we begin I must add the disclaimer: the tools and methods described below are “save the day” tools that can help you to come out of the crisis and help you during crunch time. They can be used when “war is over” but only as a supplement to the robust automation test suite, good test strategy, and clear processes.

There is no time to test

The most limited resource that money cannot buy is time. We, as QAs, are often asked to put something in production by “yesterday”. Many of us regularly hear that “testing takes too long”(like fixing all bugs we found takes no time at all) and sometimes we just need to deliver something to a specific date. So what to do if there is a need to test something really fast?

First of all, you can use the old good “test party”. Pickup a time, order some pizza, invite products, developers, QAs, support, and anybody else who is interested, put them in the room, and start testing. The test party allows us to find a lot of bugs quickly, questions are answered very fast and issues are fixed on the fly. For the best results make sure that the software you want to test is pretty stable. If nobody can log in, if the program is crashing all the time, if there are blocking issues there is no sense to have such a party, it would be a waste of everybody’s time. It also works best if you have a rough plan and some tests documented, even if documentation is very high level.

The next one — make developers test their own code. I know, every testing book says that if you wrote it, you should not test it. I do agree with this point, but we are not talking about peace and prosperity time here. If there is nobody else and there is no time — let developers do it. Make sure that they know that there will be no other verifications after their testing, that there are no safety nets. I do believe, although this is unpopular thought in the testing community, that developers can test and can do it well. Nobody wants to fix the production issue at 11 PM on Saturday night so they are as interested in the quality as we are (maybe even more).

Dogfooding. Which is actually means that you are using your own software. It’s not possible for all types of software. For example, I lead the testing of a platform that automates negotiation between TV advertisement buyers and sellers, so I probably do not need this software in my daily life. But if you are working on some dating or banking app, blogging platform, online store, or any other b2c type of software — make sure that employees of your company use it. To benefit most from such testing make sure that:

• it is easy to install the version of the software that needs to be tested
• it is easy to report issues
• you can distinguish “test users” and the real ones
• it is fun (people should gain something for testing it, not only a free account but some other benefits as well).

If you need to test something in a very short time — test the “money flows” and security. Money flows are flows that bring your company money, the flows that are the focal points in your sales pitch. It will be the ability to find someone they like and exchange contact information for dating sites. It will be adding and removing items to the list for the shopping list app. It will be searching for the dress they like, adding it to the shopping cart, and checking out for the bridal gowns store.

If you don’t have time — don’t waste time testing secondary functionality (again, we are talking about the crisis here, do test it when it's a new feature or you have a reasonable amount of time).

Don’t skip security testing although. We are all aware of some scandals that involved big companies. They did survive but they are big. Such a scandal can be the last nail to the coffin for a small startup.

Use your customers as testers (if you have any. Sorry it’s probably not a good joke, but I can’t resist). You can use all customers or a selected group. Make sure that issues can be reported easily and increase the logging level for these specific customers. Do not make them test completely unstable versions though, they are still your customers and need to enjoy using the application.

Feature Toggling. Feature toggling is a tool that allows you to enable/disable features. If you don’t know what is it — here is my favorite article on this topic. If possible (and it is not always the case) implement dynamic toggles, so you can turn features off and on on the fly, without rebuilding, redeploying, or restarting the application.

Just ship it. This is the least popular option among QA engineers because all our life we were told that we should not do it. It is sometimes the only option, unfortunately. It should not be “just ship it”, you need to have a plan. The plan can be as simple as the readiness of the team to work around the clock for the next 3 weeks fixing bugs. The plan can be to rollback the new build if something goes extremely wrong, or release it only to the preselected groups of customers. It does not matter what the plan is, you just need to have one and rehearse it if possible (talking from experience — never, never, ever do the rollback for the first time in the production)

To busy to Automate

How often do you hear your pears complaining about a huge automation backlog and no time to attack? Probably pretty often. We move forward with such speed that frequently we leave a bunch of functionality without automation coverage.

According to the different studies, 50–70% of all testing done in the world is still done manually. I personally do not see anything wrong with any percentage of automated vs manual as long as it fits team needs. If you are able to move as fast as you want, the quality of the product is high and everybody is happy — why add more automation? So the first question that you need to ask before automating one or another thing is the question “Do I really need to automate it”. Don’t get me wrong — I am an automation preacher, and if something can be automated I am the first one to say — let’s do it. Although we are talking here about the situations when you have limited resources, so you need to think twice before investing these resources into any type of activity. If you decided that you need to automate some tests think again, especially considering what will happen if this remains not automated.

If you have a buggy feature in production that causes a lot of problems, or you plan a big redesign in some specific area in the near future, the answer is kinda obvious — you need to add automation. If you have a feature that is pretty stable, there are no improvements planned or it is rarely used by customers — leave it alone. Invest in places where you must have automation, in high-risk areas, in some additional exploratory testing, in self-education after all. Do not automate something just for a sake of automation, or to reach some automation % that somebody said you must have. You must not, especially if resources are scarce. The only thing that you must — is to deliver the software that your customers enjoy using and be profitable as a company.

When you decided that you really need to automate something — automate it on the lowest possible level. Do not add end-to-end tests where the unit test is enough.

Do you need to automate tests for bugs found in production? It is a good practice when a) you have enough resources; b) the feature is buggy and you already have a plan to add more automation to prevent bugs from escaping to prod. If you don't have enough resources and you need to move fast — let it be; it's a very rare case when exactly the same issue reappears in production again and again, and automation will not help you to catch anything else except exactly the same issue.

Use modern tools to help you automate. Nowadays the vast majority of automation is still done by good old coding. There is nothing wrong with it, but coding is slow, requires to have a team with programming skills, and continuous maintenance. There are many tools on the market that provide the ability to just record tests. Recording allows you to create an initial set of tests quickly and involve in this process a product team, support team, and users. It helps to decrease maintenance time as well(it is much faster to rerecord a new tab/screen/page than write a code for it). If we go further with modern tools consider using tools that help you to run your tests on different devices and browsers. It is usually cheaper (if you count all expenses) to use the tool than build that device farm, maintain it and orchestrate the execution tests on all of them. The same goes for different browsers and operating systems. The people who build such tools know what devices and browsers are used most and how to run these tests in parallel to speed up test execution.

Finally, if you are talking about tooling, there are plenty of tools that can help you to ease testing in many other ways, like collecting users' data to know exactly who your users are and what functionality they use.

Another thing to try if you really do not have time to automate is to find what is the slowest, the most time-consuming, the most error-prone part of the testing process and automate it. For example, you can automate test setup, environmental cleanup, data comparison, etc. Later you can use these scripts when automating tests.

What about coverage? Coverage is so overloaded world in testing and it is so noninformative metric (from my perspective) that I recommend ignoring the coverage numbers. You don't need to have 70, 80, or 100% coverage. Repeating myself: you need quality products that your customers enjoy using, not a number. If everybody is happy and there is only 25% of the functionality covered by tests, maybe it is enough. If a problem comes up — do not mindlessly try to increase coverage — try to solve the problem itself. Finally, ten stable, reliable tests that the team trust are much better than a hundred flaky tests that nobody trust so they as useful as no tests at all.

Read about tips and tricks that might be useful if you need to select a testing (or any other tool) in a very short period of time, maximizing usage of limited testing budget, and more in my next story. Stay tuned!