How to Test Software - Part 2: Non-Functional Testing

In part 1 of this article, we dealt with tests that evaluate the base functionality of a piece of software. Does a button do what it’s supposed to? Is the database updating correctly? Are the business needs met? These are the types of questions that functional testing answers.

However, software has many important aspects beyond its base functionality. The software needs to be responsive, deal with realistic user loads, and be easy to use. Non-functional tests deal with all these various aspects of the system’s performance.

Load Testing 

A real system will have multitudes of users on it simultaneously. A load test evaluates how the system handles a normal or peak amount of users. This is important because if a system cannot withstand the load placed upon it, it will at best slow down, and at worst crash altogether.

As load testing requires you to simulate thousands of users, it is almost necessary to automate it. There are dedicated tools designed for this, such as WebLOAD, LoadView, and Loadrunner. They will allow you to create test cases that target specific areas of your system.

These tests should be realistic and linked to the software's business requirements. From there, you specify the number of users you want to simulate and run the rest.

You should run the test in a dedicated environment. Otherwise, it will cause performance issues for other users. 

Example 

Our online store is having a promotion and is likely to see an uptick in traffic. A load test can simulate 1000 users adding products to their basket and going through checkout at the same time. 

Stress and Endurance Testing 

These are both subcategories of load testing. They use the same methods but with different values. 

Stress Testing evaluates what happens when a system experiences a load higher than its specified capacity. In this scenario, a plan should remain robust and be able to handle any errors. The goal is to avoid a complete system crash. So, if we design our e-commerce application to handle a peak of 1500 users shopping at the same time, a stress test would simulate 2000 users.

 A perfect example of why this type of testing is important came courtesy of Coronavirus. As the illness forced more people to work from home, these people utilised tools such as Microsoft Teams. This sudden upsurge in traffic proved too much for Microsoft’s software, and it experienced significant downtime.

Endurance Testing assesses how a system responds to its normal or peak operation capacity over an extended period of time. So, taking our designed peak of 1500, we could test 1000 users using the system over 5 hours. This test reveals if there are more minor errors that start to accumulate into very real problems.

Usability Testing 

So far, we’ve focused on how the system functions and performs - but another side of the equation exists. Usability testing evaluates how user-friendly a system is. A system should be easy to use; otherwise, it could result in unnecessary frustration for users. If the user experience is bad, people will not use the system.

This type of testing is best performed by people who have not had much exposure to the system before. It can be easy to overlook obvious usability issues if you’ve gotten used to them - especially if you’re the one who built the system. Fresh eyes ensure that you get a non-biased assessment.

Usability testing is often carried out through workshops. The testing team will invite users to test out the system, give them a goal, and observe how they complete it. The user’s screen is often recorded so that testers can analyse it at a later time. In general, this sort of testing is looking to reveal things such as:

• How easy the system is to learn. 

• Whether the UI allows users to achieve their goals intuitively and efficiently. 

• Whether the design is accessible to users who have disabilities (e.g., vision impairment). 

• How many errors do users make, and how do they recover from them? 

• Whether the user is satisfied at the end of their session. 

It’s common to do this both in-person and remotely.

Penetration and Security Testing 

It should go without saying that a system needs to be secure, even more so if it handles sensitive or identifying data. A data breach can damage a business, so once you’re sure the system works, this should be a top priority.

Penetration testing is one of the most common forms of security testing. It simulates a calculated cyber-attack on a system. Testers will use tools that brute-force the system or attempt to inject and run outside code. They may even use social engineering techniques to manipulate users into giving away private information. All this works to expose vulnerabilities and make sure hackers can't tamper with the software.

It’s common to run security testing internally on automated cycles throughout the development phase, but like usability testing, it’s wise to run at least one full test with a third party. They will return a full report on how your system stood up their attacks. If they find any vulnerabilities, they will say how important they are to fix them. Also, remember to inform the server team that you’ve planned an authorised attack. Otherwise, they’ll get quite a startle!

Example 

Our system has text boxes in which users can type things. A penetration tester will attempt to enter malicious code into these inputs with the aim of getting it to run on the server. This is called Cross-Site Scripting (XSS) and is one of the most common forms of cyber-attacks. If an attacker succeeds, they can easily steal other users’ data.

In Closing 

Non-functional testing is just as necessary as functional testing. All of the best functionality in the world is useless if it falls over once it’s live. If the usability is poor, users won’t be able to utilise the software to its potential, not to mention the trouble you’ll have if your system security is breached.

Hopefully, this 2-part article has shown you the true breadth of software testing. PayReview software is tested with all the methods discussed and is always benefiting from methodology improvements. If you’d like to talk to us about any of our products or the way we make them, please don’t hesitate to get in touch.