This article describes between subjects design in the context of multi-user usability testing.
Setting up experiments to test the usability of multiple user interfaces and conducting user surveys requires some planning. One consideration is whether to choose interval research or use an interdisciplinary research approach.
It also discusses the significant benefits that can help you use this method. To test the usability of multiple user interfaces (including the same test-taker) in a single survey, you need to find a way to match test-takers.
What is Between Subjects Design?
And let’s start with the essence of this method. The between subjects design, as opposed to a within subjects design, is based on a comparison of one user interface in a study across a group of subjects or a test subject. In a between subjects design, each condition is checked by a different person or group, but each test participant has access to only one user interface at a time.
The subjects are placed in identical test conditions, where they will need to rebuild in terms of use with each new step. Stages are called the change of subjects (in our case, these are users) when tasks are performed in one condition, making participation in another state impossible.
The fundamental problem with research in the between subjects design is to create groups with all inputs but the independent variable equal.
In theory and practice
Let’s look at the method with a more specific example. Each of the stages of the independent variable affects the experience of the subjects. In this scenario, the independent variable may be subjective, for instance, in a study of two separate groups, where people gain experience in the experiment and cannot participate in the investigations under different conditions of the individual variable.
How the between subjects design works can be shown with a simple example.
The company would like to test which of its two new sites will be more effective in attracting more customers. And as part of the experiment, three test groups were created. The first test group receives the current website of the company. The remaining two receive one new version of the site per group. Each group interacts with only one of the site options, and the researchers observe which of the options the subjects liked the most and use this data for further development.
Why do you need to use Between Subjects Design?
In usability testing, the principal goal of the between subject design vs within subjects design research can be comparison:
- Different versions of the same user interface.
- Own site and site of a competitor.
- How users with different levels of knowledge and skills interact with the interface.
We give each experimental group an assignment and material that the designer believes will affect the results, while we assign the control groups a task with an independent variable.
We formed groups with random participants to ensure comparability of the initial characteristics of the participants across the groups. This way helps us to get more objective results.
During testing, the designer compares the effectiveness of the dependent variable between groups to see whether the independent variable is influential in the control groups. If the groups differ significantly, it is possible to conclude that autonomous manipulation of the variables is likely to have caused the differences.This significantly reduces testing time than if we subject each of the groups to sequential testing of each of the samples.
It is also worth taking care that the participants could not know which group they belong to, the experimental or the control group. This is done in order to prevent participants from unintentionally or deliberately changing the test results. Such actions of the subjects will lead to the absolute bias of the test. And this also significantly affects objectivity. Users have no previous experience and pass the test immersing themselves as much as possible in the task.
Besides usability comparison, with the help of between subjects design, you can compare groups that differ in key characteristics. Yes, this way, you can test not only the design or functions but also your audience. In this case, you can select the target audience. You can take into account age, knowledge of the topic, skills, or any other characteristics.
This characteristic will be your independent variable, with different stages that differentiate the groups from each other. And in this case, there will be no control or experimental groups because all participants go through the same procedures.
In addition, you can link certain group characteristics to certain design features. This approach will help you test the functionality as accurately as possible. It will also help you save a lot of time and break down more complex tasks.
Advantages and Disadvantages of Between Subjects Design
Like any other method, the between subject design has its pros and cons.
Advantages of Between Subjects Design
In certain circumstances, between subject designs vs within subject design is highly beneficial and offers researchers the chance to carry out an experiment that is little affected by external variables.
No experience transfer
Since, within the framework of the between subjects design, the subject participates in only one type of condition, then the transfer of experience from one test to another does not occur.
Easy to set up
The same can be said for customization. Since the test takers have only one condition, the planning focuses on one test condition.
Takes up less time
A good feature of the between subjects design is that it takes little time to test one condition within one experiment. As a result, the designer gets data analysis more quickly, which allows more experiments to be done at the same time.
Disadvantages of Between Subjects Design
The apparent disadvantage of the intersubjective design is the possibility of selection bias. Since we placed different people in different conditions, there is always the possibility that any differences found reflect not the actual effect of the experimental manipulation but the initial individual differences between the subjects.
In the between subjects design, in which we examined each subject under each condition, this probability is absent. Note, this advantage of the between subjects design correlates with the previously described advantage of the longitudinal method over the between subjects method.
There are two ways to rule out the possibility of selection bias in a between subjects design. The first is to equalize the potentially significant variables; below we will look at the pros and cons of equalization. Another approach is to assign subjects to different groups randomly.
If the sample size is large enough, if the assignment is indeed random, the initial differences between the subjects will be controlled, and confusion of subject variables and conditions will be avoided. Thus, the logic of the random approach is flawless; the problem is to provide a positive answer to both its.
How to use Between Subjects Design? [Short Instruction]
And since we began to consider methods in the context of between subject vs within subject, then we must take into account two key criteria:
- The number of independent variables which we will control;
- The order of forming groups taking into account different testing conditions.
And let’s start right away from the first point. In the between subjects design, we have only one independent variable, and in the case of the user interface, this can be a classic design, which you can read about in our blog. It can also be a one-factor design. However, also, a plan can have two or more independent variables. This is called a factorial design.
In our case, with usability testing, it is best when the number of independent variables does not exceed three. This is because the variables determine the number of conditions and, therefore, the number of test participants required. And it would be best if you remembered that along with the number of conditions, the number of participants grows.
On the other hand, the complexity of design is directly related to the complexity of statistical analysis. For example, There is Design A X B, where A and B represent the number of levels in each controlled variable. To find out the number of experimental conditions, we just need to replace A and B with the number of levels of each of these variables.
In the case of any user interface, if we can change the color theme (green or blue) and interactive features like an order form or something else. In this case, we get a 2 X 2 between subjects design, that is, four experimental conditions. And then, we subject each group to a test using only one condition per group.
For Designers
And we think the question is the rationale of what is the use of this for the designer. And the answer lies on the surface. Between subjects methods, help designers observe the user in a natural context of use.
And this is the key to subsequent productive changes for any project where user interaction with the interface is essential, such as an e-commerce store. At this stage, it becomes possible to consider all the wishes of real users and make the product as comfortable as possible.
Along with this, there is also the identification of problems that the user faces. Next comes the objective identification according to the level of difficulties that the user faces. And what is essential is their subsequent elimination.
And one more opportunity that comes with introducing between subjects design is carrying out all possible measurements during testing.
For Business Owner
And here are some arguments to help you understand if the between subjects design is necessary:
- If you’re short on time, you can use the between subjects design.
- If you’re on a tight budget, you might move to the between subjects design and hire fewer test-takers to keep costs down.
- If you want to minimize knowledge transfer across different testing conditions, the between subjects design is the right option.
As a business owner, you should understand the importance of usability testing research with the between subjects design or even within subject design, depending on your goals and resources.
Why is Between Subjects Design Good?
Since we figured out how the between subjects design works, now we can consider what this method brings to your testing.
And the first thing I want to focus on is an actual experience. The between subjects design is carried out in a context as close as possible to actual use. Each subject in the group performs the essential tasks for which the user interface has been designed.
Moreover, when assessing the existing usability, testing is carried out on a commercial version of the product, on a real-life product, and not on a prototype. However, you can also apply the between subjects design as well as within groups design at the design stage of testing. In this case, testing is done on a prototype.
Real experience will be gained through direct interaction with the product. The test subject receives a set of instructions that are aimed at using typical software or website tasks.
And there is another critical factor in the between subjects design method, which will help to identify the presence of problems. During testing, the user himself controls what he will be like when he is alone with the software. And already on the basis of observations, errors, misunderstanding of the interface, dead ends, any events indicating difficulties in using the software can be noted.
At the end of the test, these various observations are “hot analyzed” with the user to better understand the root cause of the problem. These discussions usually lead to original solutions.
Finally, you can use the between subjects design to test hypotheses about user behavior, such as how they navigate the interface, what information they look for, or what commands they use more often.
The actual effect of Between Subjects Design
The use of the between subjects design has been shown to enhance business performance. Therefore, companies that underestimate the importance of design may be missing out on vital opportunities.
When used consistently across an enterprise, design can provide a number of business benefits. Among these benefits, we can find:
- increase in sales of your products or services
- improving your position in the market compared to your competitors
- greater customer loyalty and fewer complaints from them
- a stronger identity for your business
- the ability to create new products and services and open up new markets
- reduced time to market for new products and services
Attractiveness for clients
Effective use of design gives customers a reason to buy from you rather than your competitors. This is a valuable source of differentiation – a well-designed product or service will stand out from the competition.
The design also adds value to products and services. Customers are often willing to pay more for well-designed products that can provide them with benefits such as usability, enhanced functionality, and improved appearance.
Design for efficiency
However, design can not only improve the products and services you sell, but it can also improve the way your business operates – the efficiency of its processes, the profitability of the raw materials used, the quality of the packaging.
Design can increase the value of your products and services to customers and reduce the cost of production. For example, careful design of the manufacturing process can provide significant savings. It can also make processes and materials more efficient and environmentally friendly, helping companies comply with sustainability laws and regulations.
When should you use a Between Subjects Design (Example)?
There are several factors to consider when deciding whether to implement the subject design.
The first and probably the most important is the number of subjects and the duration of the experiment. If you are not limited in the number of subjects, then you can safely choose the between subject design. It is also worth considering the testing conditions.
For example, suppose the testing is based on specific actions of the subjects. In that case, the use of within subject design will be impossible because the repeated use of the same actions will badly affect the study`s objectivity. However, in the case of between subject design, this approach will only benefit.
Within Subjects Design and Between Subjects Design
Along with the methodology between subject design, it always goes within subject design. And therefore, it would be nice to consider them in a complex manner. Since we have already considered between subject design, the turn of the second component has come.
A within subjects design is a type of pilot project in which all participants are exposed to each condition. The term “condition” is used to describe the different levels of the independent variable.
For example, let’s say you are testing two versions of an interface. However, instead of dividing participants into two groups, you ask all participants to try the first option first and then the second. Then you compare the test results to determine which interface was more efficient and user-friendly.
Pros of the Within Subjects Design
One of the most significant advantages of within subjects design is that it does not require significant participants` pooling. In general, such an experiment in the within subjects design would require twice as many participants as the within subject design.
A within subjects design can also help reduce errors due to individual differences. In the within subjects design perspective, where individuals are arbitrarily assigned task conditions, fundamental differences between groups can influence the outcome. On the within subjects design, people are subject to all levels of conditions, and therefore the results are not distorted by individual differences. Each participant serves as their own baseline.
Cons of the Within Subjects Design
The main disadvantage of using a within subjects design is that an explicit action, which is that participants participate in one condition, can influence performance or behavior in all other conditions. This problem is known as the carry-over effect. The previous test experience can affect the participants’ performance.
Fatigue is another potential downside to using the within subjects design. Participants may feel drained, bored with the test, or simply become uninteresting after participating in several subsequent tests.
Finally, the experience gained can influence the effectiveness of subsequent tests. Participation in consecutive tests can help participants become more qualified and there will be no objectivity. This can distort results and interfere with determining whether a particular effect is due to different levels of testing or is simply a result of practice.
Mixed factorial design
Now it’s time to talk about a mixed factorial design. Let’s take a look at the varieties of factorial designs to understand the mixed type better.
For example, all participants can be tested either using the mobile version of the user interface or without using the mobile version of the user interface, and in the office or at home, during work or weekends, etc.
This would mean that each participant will be tested in one and only one condition. Alternatively, all participants could be tested both when using the mobile version of the user interface and without using the mobile version of the user interface, as well as during work or weekends. This would mean that each participant must be tested in all four conditions.
The mixed factorial design method consists of at least two independent parts: a factor within subjects and a factor between groups. For example, as part of a test, we can view the use of the mobile version of the user interface as a factor within the subject, testing the same participants both with the mobile version of the user interface and without the mobile version of the user interface.
But a strict balance and order must be observed when these conditions change. But at the same time, we can consider another factor, such as the environment, and test with each group or participant during business hours or on weekends. As a result, we get that each group or subject will be tested under two of the four conditions.
What is between groups research design?
An experimental approach to testing also includes a between groups research design. This approach is based on the fact that two or more groups participate in testing. These groups in each of their states are influenced by the independent variable.
It follows from this that we will have a new group of participants for each new test condition. We should form such groups by randomly distributing the subjects into groups.
Thanks to this approach, we will place each subject in equal conditions to get into one of the groups. And naturally, this will require a lot more people than with any other method described above.
Of course, like any other method, there are advantages and disadvantages.
The advantages we can attribute to the fact that using this method, we avoid the effect of fatigue or any other factor that will establish order in testing. This is achieved due to the fact that each subject participates in only one test condition.
However, there are also obvious disadvantages. Such as, for example, qualitative differences between the subjects, which can greatly affect the objective test results.
Summary
As you can see, testing can be done in differently ways depending on what goals you are pursuing. And each of these methods has its advantages and disadvantages – one will help to determine the difference between the conditions, but it will take a lot of time, the other will be able to show the differences in different approaches of the subjects.
Such an existing wide methodological base will be useful as a business owner, as well as a designer. The main thing is to know which method you need.
