Regression Testing for Software – Your FREE Complete Guide to becoming a better tester

Whether you’re new to software testing or a seasoned professional, there’s one thing you’ll come across time and time again. Regression testing!

This article will be your mini regression testing guide where I will attempt to firstly define what regression testing is, go through some scenarios and try to answer some common questions.

Some of these points will be about how to create a regression pack, how to select test cases and how to run them.

I also hope to go through some of the benefits of regression testing as well as some of the disadvantages.

This is not a technical article so if you’re here to learn about creating a Java regression suite, this is probably not for you.

I hope to relay the concept of regression testing and how it can be implemented regardless of which testing level you are or technologies you use.

Hopefully this article will act as an evergreen resource so you can take actionable steps to make your regression testing efforts more efficient.

What is regression testing in Software Testing?

The official definition of regression testing as per the ISTQB is;

“Testing of a previously tested component or system following modification to ensure that defects have not been introduced or have been uncovered in unchanged areas of the software, as a result of the changes made.”

Why is Regression Testing Required

Regression testing is required to ensure that no new defects have been found in unchanged areas as a result of a change.

The purpose of regression testing and what does this means in laymen’s terms can be illustrated in the example below.

Let’s assume that you’ve already completed tested an application which consists of 100 test cases. We will call this “Test Cycle 1”

All test cases have passed and no defects were found. Good job, high fives all around!!!

Now, the business decides they want to introduce new functionality to the application you just tested.

It has been determined that the new functionality WILL NOT impact any part of the application that was tested as part of Test Cycle 1.

The new change gets deployed.

To ensure that nothing is broken or no new issues have been found, you re-run all 100 test cases. We will call this ‘Test Cycle 2’.

5 test cases fail and you uncover 10 defects.

Your expected result is that all 100 test cases should have passed with no defects.

However, your actual result is a 5% failure of the test cases run.

This means that from the last code deployment;

• New issues were found
• No code changes were made to the existing application that were run as part of ‘Test Cycle 1’
• Test execution expectation were no test case failures and no defects should have been introduced.

The regression test has successfully served it’s purpose by uncovering new defects from “unchanged” software.

On the assumption that your test pack will be run on parts of the application that have NOT changed, there are a number of potential outcomes;

• Outcome of Test Cycle 2 should be the same as Test Cycle 1 (baseline).
• The ideal scenario is that all test cases pass with no defects.
• You identify defects and failed test cases when you were expecting NO failures and defects.

I’m sure you can see how valuable Regression testing can be.

An illustrated regression testing example

Different Types of Regression Testing

Don’t get this confused with testing levels.

There are soo many ways to test. I like to keep things simple. There’s either a full or a partial/targeted regression. Everything else in my opinion are just sub categories of these two regression types.

Targeted regression

A targeted regression is when you run a partial set of test cases from your test suite.

This is quite common in a lot of cases where you may need to run more than a sanity check but less than a full run of the regression pack.

For example if there are any code changes and you want to test a particular functionality or any integration points, a targeted regression may be a good strategy.

Full Regression

A full regression is where you would run every single test case in the test suite to test whether or not any new defects have been introduced as a result of a change.

Difference between Regression vs

There are a number of different types of software testing terms that are often used interchangeably with Regression.

I’ll try and clarify these differences as simply as I can.

What is the Difference between Smoke testing vs Regression Testing?

Smoke testing or “Sanity” testing as it’s otherwise known is a method of testing your software to ensure the system “stands up”.

An example of a smoke test would be when you have a new release and want to ensure the basic functionality of the application is working. This kind of testing is generally very high level to ensure the application is ready to be tested fully.

Checking navigation and that screens are being rendered correctly in a web application could be regarded as a ‘smoke test’.

What we don’t want is an application that doesn’t work when start our test execution.

I usually include this as part of my deployment entry criteria so I am confident the application has been smoke tested beforehand by the teams that deploy the application.

Regression testing on the other hand consists of more detailed and granular testing of specific functionality. It really goes in depth and to a much lower level.

So, in summary, ‘Smoke Testing; is high level and Regression Testing is more lower level.

What is the difference between integration testing and regression testing

Integration testing allows you to test various components of the application together to make sure it works. Regression testing ensures that the system hasn’t broken as a result of any new change.

What is the difference between progression and regression testing

According to the ISTQB the term ‘Progression testing’ doesn’t exist.

However, my understanding leads me to believe this is a form of functional Integration testing where you would test new functionality for the first time.

This is probably no different to what you are doing already.

“Progression testing” tests new functionality to find new errors in an incremental way and regression testing tests to make sure no new defects are found.

What do you mean by ‘incremental way’?

Let’s say you have component A; you test this for the first time. Then you test component B. Then both components together.

What is the difference between retesting and regression testing

Retesting is when you re-run a failed test case after a fix or change has been deployed.

It might be worth familiarising yourself with the different testing types.

What is a Regression Test Suite / Test Pack?

A regression “test suite” or a “regression pack” is essentially a collection of all test scripts/cases you have for the application under test.

The aim of a regression pack is to build a complete asset library of tests that covers all requirements. This could be business/ functional/non-functional requirements, GUI tests, manual, automated and the list goes on.

The aim is to be a comprehensive library to test every single aspect of the system.

Regression suites are agnostic of the testing level and or whether they are automated or manual.

How do you create a Regression Test Pack

As mentioned above, your regression test pack consists of ALL your existing signed off test cases.

So let’s say for example that we have a brand new software application that needs to be tested.

In total we create 500 test cases with 100% coverage of all the business and functional requirements.

You can use these test cases to form part of your initial test pack (test suite).

If any changes are made to the application then new test cases will be added, existing test cases updated and in some rare occasions deleted.

This will ultimately be down to how your organisation works but I have found that in most place I have worked they are sorted by testing level > function/component > requirement type.

So for example System Testing > Login > User attempts more than 3 failed login attempts.

How we map requirements to test cases is outside the scope if this article.

Is there any criteria when writing regression test cases?

I’m unaware of any formal best practice on specifically writing regression test cases. However, if I was about to execute a test pack, I’d want to make sure that I have at least the following things as an absolute minimum;

• Full Coverage – Ensure that all requirements from specification documentation has been covered in your test cases. You might want to consider creating a Requirement Traceability Matrix (RTM) if you don’t incorporate this already.
• Low level of detail and granularity – I absolutely love attention to detail. I want to therefore make sure that we cover as much low level of detail that is possible an which is not exhaustive.

I don’t write test cases specifically for a regression pack. Just use whichever best practice you adopt for writing test cases. That should be sufficient.

By ensuring the above, you will be confident in your test coverage as well as quality and so will stakeholders.

How do you select test cases from a regression suite?

Now that we have a test pack, how do you choose the regression test cases?

Selection can depend on a number of factors.

Firstly, regression testing should be identified in your Test Strategy or Test Plan documentation, so your stakeholders are aware of your approach.

Asking yourself the following questions may help to understand your test scope.

• What is being changed to the existing system?
• Which parts of the application/system are directly impacted as a result of the change?
• Do you have test cases that cover the areas impacted?
• Is it REALLY necessary to run the entire regression suite?

The first thing to do is to understand the change(s) that’s are going to be deployed.

The second thing to do is to carry out an impact analysis.

You’ll need to gauge whether any areas of the existing system are impacted directly as a result of the change.

This will possibly involve speaking to other teams such as Business Analysts of Software Developer teams so they can advise.

Once you have all this information, you can go through the test pack and pick out the test cases that cover the areas impacted.

My suggestion would be to always have a set of “stand up” sanity test cases that you should run first.

I always test using a risk-based approach and found that prioritising my test cases this way means we can either prove or find errors in key functional areas quickly.

You can also use the concept of ‘defect clustering’ as part of your test case selection. This means, looking at previous testing activities to identify where most defects occurred.

When should regression testing be performed?

Ideally, you can perform this during the Unit, System or System integration Testing level. This is not something you would expect to be done during UAT as most defects should have been identified by this point.

When you have tight deadlines (who doesn’t) and constant code drops, your Delivery Managers do NOT like the word regression testing.

Regression testing can be performed during different times during your testing life cycle as there is no “one size fits all” rule.

However, I would suggest that it be used sparingly as opposed to the first type of testing you do after a release.

The reason is because it can be VERY time consuming. That means a longer timeline and costs!

How to carry out regression testing

So how do you perform regression testing?

“It depends”! 🙂

My career has been predominantly focussed on Manual System, System Integration and UAT so I’ll try and answer from that perspective.

I try and avoid running a comprehensive regression where possible as often times it isn’t required.

My approach is to run a “targeted regression” as opposed to going full on! I generally;

• Pick out the “Stand up” tests so that a smoke test can be run.
• Design the test cases for the new change
• Identify test cases that address the touch-points/integrations where the change will impact the application. This will then be your targeted regression test cases.

If we are highly confident that this is sufficient coverage to carry out test execution then this would be good enough.

If a targeted regression is not enough, then replace this with a complete regression cycle at the end of your functional testing.

We do have to be pragmatic here as there can so many ways you can test.

Some people prefer to carry out a baseline and carry out regression at the beginning of the cycle, again it depends on your application, complexity, risk and other features.

Maintenance and Updating your Regression Test Pack

Having a regression pack is incredibly handy and if you have software that needs regular updates or changes.

A word of caution, the pack is only as good as it’s updates.

It is absolutely imperative that you periodically review your regression pack to ensure it is still up to date.

The best way I’ve found to do this is to review and make changes when you have an application specific project.

My usual approach is to ask my colleagues, “does a regression pack exist?”. There usually is, but you sometimes find that it hasn’t been updated in a while. Not good if your testing on an outdated GUI design or if the business logic has changed (even worse)!

If I have selected specific test cases, I usually update them during the design phase of my project.

If you want to review the whole test pack then this might be something you do on an annual basis. This depends on how often the application is likely to be tested.

The key point is that whatever you decide, your Regression suite needs to stay current and up to date.

It becomes more important when you outsource the work offshore, have a new starter that may not be a Subject Matter Expert, or have an urgent project that you need to deliver.

Your stakeholders don’t care whether the person is new or not, they just want you to deliver and expect the same level of service as they are used to from a seasoned Tester.

A situation I have too often found myself in.

Regression packs REALLY do help with the learning curve and save a great deal of time, so maintenance is essential.

Pros of Regression Testing

So what are some of the advantages of Regression Testing and having a test suite?

• Having a regression pack can mean that Test effort and planning can be estimated more accurately based on historical data. For example, you will be able to give more accurate timelines to your project managers since test cases have already been run before.
• Test design work can be significantly reduced if test cases already exist.
• Test execution can take place quicker if test cases have already been designed.
• Cost saving – with assets that already exist, there is a long-term cost saving.
• A central repository of test cases means that all the team can access them. As a result, duplication and redundancy can be reduced.
• Once a regression pack has been created, small maintenance changes can be quick.
• A regression pack is a great way to transfer knowledge to new starters or someone not familiar with the application. This is immensely helpful if you have an offshore team.
• Regression testing allows you to ensure the system under test can be tested thoroughly and uncover any new defects. This reduces risk and exposure of defects in a PRODUCTION environment.
• A regression pack that has comprehensive coverage gives confidence to the business/end user that testing is being conducted at a high standard.
• Quality of testing can get better and better when test packs are updated.
• A regression pack allows you to execute tests at short notice.

Cons of Regression Testing

So what are the Disadvantages of Regression Testing?

• Setting up from scratch requires a lot of effort.
• Executing a test pack can be very time consuming, especially if they are manual test cases.
• Maintenance and updating test cases can seem a chore when you have other projects on the go. As a result, someone will need to own the process of maintaining the suite of tests.
• Project Managers are usually not willing to pay to update regression packs from their budget.

Entry and Exit Criteria for Regression Testing

Your entry criteria for regression testing should be no different to any entry phase of testing. My general entry criteria for System Testing is the following;

• Ensure the previous testing phase includes no Severity 1 (Showstoppers) or Severity 2 (Critical) defects.
• Severity 3 and 4 defects should have a mitigation plan or workaround in place.
• Ensure the test cycle has been completed (all test cases executed) with none in a ‘not started’ status.

Best Practices for Regression Testing

There are a number of points that have been mentioned in this article. When thinking about regression testing, below are some best practices you may want to think about;

• Identify the areas that are being changed or modified in your application.
• Identify the impact of this change to the rest of the application
• Prioritise your test cases using a risk-based approach. For example, are there any high-risk areas of the application that may be impacted? If so, these should be tested first.
• Mark each of these test cases as High, Medium or Low priority.
• Engage and get a ‘sign off’ from your stakeholders on the test cases you are going to run. This ensures you are all aligned and prevents any issues later such as finger pointing.
• If you don’t need to use ALL your test cases, then don’t’

Best Regression Testing Tools

What are the best automation tools for regression testing?

Automation can be a great way to make your testing activities more efficient and quicker.

You can adopt a number of tool types such as;

• UI Testing tools
• Browser based regression tools
• Regression tools for web and mobile applications

Unit and component testing where code is concerned is usually automated. Using techniques such as Test Driven Development means that automated regression tests can be created as part of the development process.

I won’t cover too much on regression tools here since that is a separate article in itself.

However, to give you an idea, some of the best automation tools for regression testing on the market are listed below;

• AdventNet QEngine
• Apache JMeter
• Cucumber
• Functionize
• HP UFT (Unified Functional Testing) or QTP
• Httest
• IBM Rational Functional Tester (RFT)
• Katalon Studio
• LambdaTest
• LEAPWORK
• mabl
• Percy
• Postman
• QA Wizard
• Ranorex Studio
• Regression Tester
• SahiPro
• Screenster
• Selenium
• Serenity
• Silktest
• SoapUI
• Squish
• Telerik Test Studio
• TestArchitect
• TestCafe Studio
• TestComplete
• TestCraft
• TestDrive
• Testim
• TestingWhiz
• TimeShiftX
• Tricentis Tosca
• Watir
• WebKing
• WorkSoft Certify
• Zephyr

There are so many tools available for regression testing that this would be such a long list.

When it comes to System or System Integration testing, automation isn’t always possible so don’t be obsessed with automation.

In some cases, test case automation IS possible, however it might not make financial sense.

To summarise, a regression pack is a great way to streamline and make your testing more efficient.

If an application has been fully tested once then it’s always a good idea to use the tests as part of a future regression pack, regardless of the size of the application under test.

Automation is a great buzzword in the world of software testing but remember that it’s not always a good idea to adopt automation testing if its not required. Manual testing may be a much better option so don’t get too worked up about automation.

Regression testing is an integral part of testing any software and has a great deal of benefits, however you should always ask yourself if is required all the time or if there is a way that you can reduce your test effort whilst still maintaining the highest possible quality.

I’m interested to know your approach to regression testing.

What methods do you use?

How do you select your test cases?

How do you maintain any test packs?

What are your favourite tools?

I would love to hear from you.

Please feel free to leave your comments below.