User Acceptance Testing (UAT) is a big part of most software realization and implementation projects. The goal of this test can be diverse and ranging from the wide “finding everything that is wrong with this software” to the narrow “have a look and see it is fine”. These extremes are both ineffective in reaching the true goal of the UAT, which is to have the business end users test and ideally accept the implementation as it is planned to move to production.
In theory, the core purpose of UAT should be to validate the business processes against the software implementation using business data, business processes and all roles and responsibilities set as intended. Often when I am hired to organize the UAT for a software implementation project the purpose turns out to be either “get them to accept it” when dealing with IT centered management or “we need to find everything that is wrong with the software” when the vendor is distrusted. While both have their merits, they have pitfalls that can impact project results when not addressed early in the planning stage.
Before doing anything else, I sit down with my employer and talk about the added value and limitations of User Acceptance Testing. The added value is clear: the end users have a hands-on first experience with the implementation and can signal where changes need to be made for a smooth transition. But the limitations are just as important and very easily overlooked. Here I will list a few and discuss how to mitigate the risks.
Users are not testers, users are users.
The UAT is done with the actual users of the product, they are taken from their day to day work and are now looking at unfamiliar screens, colors and display fields. Some discomfort is to be expected and when this is the first time the business has a look, your users can be quite skeptic. Some may have their own agenda, some will be clueless why they are there, some are happy to be away from the daily grind or worried what will be waiting for them when they come back and others are motivated to enter all exceptions they can think of to prove the system wrong. The main thing to remember is that your end-users will most likely not look at the designs and stay within the designed path but are more prone to divert from their tracks, move back and forth and enter very unlikely data. Nothing of this is planned much and most is ad hoc so reproduction of defects will be a nightmare so you are to make sure there is a robust system in place that is of high maturity and almost ready for go-live.
To many defects: out with confidence.
As seen above, a low product maturity will cause many defects when entering the UAT. Test early and often, absolutely, but use test professionals for this. They have a thick skin and are used to working with unfinished products. When asking your day to day business people to test a product that is not ready for UAT, they will wonder why they are interrupting their daily activities for something that is not ready. But worse, the word will spread your product is horrible and will ruin all efficiency that people worked so hard to implement. For a smooth transition and business changes, people need to be willing to work with the new product. Starting UAT to early will undermine the willingness to cooperate.
A big overhaul will cause massive delays
The biggest limitation on UAT is that no big changes are possible anymore if the deadline is to be made. Small display changes, report fine tuning and role and responsibilities can be adjusted easily but bigger changes to internal logic, backend communication or data processing cannot be made without going through a full design, build and test cycle. Planning your UAT four weeks before go-live, which is not uncommon, prevents any full cycles to be completed especially if big data conversions or interfaces are involved.
Testing on production data can be expensive or impossible
While the test professionals can create their own data based on production samples, the users will want to see the actual data as it will be after go-live. Every project I have been involved in and where IT was not able to deliver production data in the UAT environment ran into defects that were complaining about the testdata. It is understandable to the project manager and the realization team that the test environment is not suited for the huge dataset that will be used in production but always try to include at least a subset of data and communicate clearly to your users what this subset is. Sometimes laws and regulations or the fact that there is no production data at all prevent the project from using actual data. Then strive to have the data completed as much as possible, nicely rounded off cases and as real as possible. Rather use your company’s phone registry to have a bunch of names than use “testcustomer1” and “testcustomer2”, this will engage your users more.
When conducting the User Acceptance Test it all comes down to managing expectations. Manage the project’s expectation with regard to what the UAT will deliver, manage the users’ expectation with the new product can and cannot do and manage the expectation what can be changed and what cannot be changed. Keep in mind that the UAT is one of the first opportunities for the end-users to see the product in action, make sure it is an impression that counts.