Playtest day
This February morning, Maxime, who sees himself as a “passionate player,” has an appointment at Ubisoft. Not for a job interview, but to test a game that is still highly confidential. For more than two hours, he and other players will immerse themselves in what we call the “build” of the game and follow a program meticulously prepared by Ubisoft’s User Researchers. This team, composed of about twenty people, performs an important job: to challenge the game, for the first time during its production, with the intended audience.
Article written by Vincent Manilève on Stories!
USER RESEARCH LAB
“Our mission is to help designers create the experience they want to offer,” says Aurélien Fossard, Lead User Researcher at Ubisoft. It takes into account many dimensions, such as understanding player enjoyment and commitment. We are an empowering tool for designers. The laboratory is divided into several jobs to achieve this. User Researchers, analysts who will define and conduct the tests, make up 80% of the team. User Researcher Project Managers, on the other hand, relay between the production teams and the Lab. Moderators are responsible for providing logistical, technical, and infrastructural support to the laboratory. And finally, the Measure Specialist monitors the industry and deploys new tools to accompany analysts.
When it comes to working on worldwide projects, in this case, HD games (another team manages mobile games), the Paris User Research Lab collaborates mainly with Ubisoft’s Paris Studio.
“UR Labs depend above all on which specific Studio they are attached to”, explains Max Chauvin to Stories. As User Research Lab Director, his mission is to give the team’s experts the means to assist the productions with tests and innovative studies. The largest laboratory has about forty people dedicated to user research, and the smallest only has one person. The necessary teams carry out the work differently to perfectly adapt to each production’s needs. But we are dedicated to bringing as much homogeneity as possible since some of our reports can also influence Ubi’s Top Management’s strategic decisions.”
There are five test rooms in their premises that can accommodate 48 players, including one containing a corridor with tintless windows (allowing members of the production teams to attend the tests) and another “mimicking” the atmosphere of a family lounge for more casual games. Tests on specific design elements are frequently carried out with eight players, as Aurélien explains: “Fundamental work on user search has shown that mobilizing eight players can identify 80% to 90% of a feature’s usability problems. It is variable, of course, but it allows us to identify any existing concerns. Sometimes there are 16 in the Lab, for more comprehensive assessment tests, or even 32 when performing mirror tests with other Ubisoft Labs worldwide.
OUR MISSION IS TO HELP DESIGNERS CREATE THE EXPERIENCE THEY WANT TO OFFER. IT TAKES INTO ACCOUNT MANY DIMENSIONS, SUCH AS UNDERSTANDING PLAYER ENJOYMENT AND COMMITMENT. WE ARE A TOOL FOR EMPOWERING DESIGNERS
Aurélien Fossard
Lead User Researcher
On the day the Stories team visited the UR Lab, only three players are needed for the test’s first session to comply with the health guidelines. Florence Velut (User Researcher and Lead on this test), Agnès Bouttin (Associate User Researcher), and Laurent Cala (User Moderator Researcher) welcomed them, briefed them, and accompanied them to their test room. Each player will be observed for more than two hours by a referring analyst. Before the sessions start, Aurélien takes the time to explain the need to “guarantee the independence of players.” “Our goal is to inform them as much as possible about the testing process, but without skewing their experience. We want them to feel comfortable in our environment but keep their own perspective, stay themselves, play, and react naturally. At times, it can be complex when the tests last several days and they are necessarily required to talk with other players between sessions. That’s why, when they arrive, and until they sign a confidentiality agreement, players don’t know which game they’ll be testing. Sometimes when you recruit them, of course, they make deductions,” smiles Aurélien. “If asked if they like to dance, they will inevitably form their idea of the game.”
TEST PLAN
Maxime’s day at the Lab is not about dancing but about an unannounced game still in development that still requires a few things: the option to run is not yet available, nor is jumping. But the stakes are elsewhere: this test aims to answer very specific questions sent by the production studio, especially on sound coherence, user interfaces, and players’ ability to explore and orient themselves between main quests and side quests. “The URPM, the User Research Project Manager, works within the production team and relays with us,” explains Agnes.
They manage the test schedule of how many people are needed to complete it and what the team wants to test specifically. With this information in mind, the team begins to work on its methodology. Tests are based primarily on observations, discussions, and questions. Upstream exchanges with production teams provide a clear plan: which areas to explore, which missions to perform, what aspects of gameplay to test. But some tests sometimes require the development of new methodologies. “How to test a recommendation system when the algorithm has to learn from the player’s behaviors,” Aurélien asks. It requires different sessions. The same goes for battle royale games: how can we emulate a 100-player field? To test a game’s storytelling, the team created a tabletop role-playing game, including a deck of cards to test specific mechanics. “It’s interesting, especially when you want to test ideas or concepts because it allows you to do it as soon as possible, even before builds are available,” adds Agnes.
The team also ensures that the build, once available, allows testing to be carried out in optimal conditions, including exploring it beforehand to identify potential blockers and accompany players. “I remember a bug that deleted all the player’s progression after several days of gameplay, even though our objective was to test players’ progression. Sometimes it can be a problem.” Aurélien recalls. The need to work on the most stable game versions possible leads User Researchers to collaborate with Quality Control, a service that reviews game versions and tracks technical bugs.
Once the method is defined, player selection can begin. It depends on each test’s needs and the target audience for the game. “Sometimes you decide to remove the data collected from a player because they didn’t match the type of game experienced.” It is why players who are likely to be contacted complete detailed questionnaires about their gaming habits. “We can invite players who speak English, are used to playing with an Xbox controller, and who regularly play adventure action games if that’s our needs. We will also contact them for more details and then call the pre-selected players to fine-tune the profiles selected.”
EYE-TRACKING AND FOOTSTEPS
And that’s how Maxime was invited to this session. During the test, the player is encouraged to react as he would at home, with the knowledge that the game is not finished, some features are not implemented and that bugs can occur regularly. Some essential parts of the game are not accessible, and several characters still appear in the famous T poses, arms raised at ninety-degree angles. “You have to adapt to what’s possible when you have a game that is still in the early stages of development,” says Agnes. “We even need to add cheats to skip certain parts of the game that are still inaccessible. But for games at the end of development, sessions will be much freer for the players; we’ll really let them explore.”
“Ha, I think invisible walls sometimes block my movements,” says Maxime, after a few minutes spent on the game of the day. “I really enjoyed having this little sketch when it’s a secondary quest,” he adds later. “What did you think of this quest? Did you understand what you had to do? Was that clear enough to you?” Agnes asks him. “The interaction with this object still has some problems, but with a little patience, you can activate it,” she sometimes says to help the player.
“I’m taking part in tests because I like to review what’s coming out and see how the tests go,” says Maxime. Obviously, it isn’t like a regular gaming session; the builds are not representative of the final game. It’s like dining out but eating the food while it’s being cooked. You have to keep that in mind when you make your report: give your player impressions and ignore the context, knowing that what you have seen will continue to evolve.”
Even though the bulk of the test is relatively silent during the gaming sessions, Agnes benefits from other very interesting data. During a gameplay phase, in a menu, or during a dialogue, a camera films the player and tracks eye movements displayed using a small bubble on the Researcher’s Screen. It allows designers to make sure that the game elements attract the player’s attention as intended. “On our screen, you can also see which key the player presses in real-time,” adds Agnes. “It’s very convenient when a player can’t do an action: you can then look at what was happening on the controller. For example, if an action, usually assigned to a key in games, has to be pressed elsewhere to perform it, it can disrupt them. ”
UNDERSTAND THE PLAYER
After a global exploration of part of the game, Agnes asks Maxime to dive back into the universe and talk about a particular element: the main character’s footsteps. “The sound of footsteps on the sand works well, but it would work well for the grass as well,” Maxime replies. Although the sound of footsteps in the water has not been implemented yet, the player can appreciate the sound of steps on hard stones in a village or on the wooden planks of a bridge. Twenty minutes will also be spent analyzing interface elements, both in the menus and in the game itself, to ensure that each graphic element is understandable. “We have a whole set of questions and follow-ups to conduct our interviews and gather the most qualitative data possible,” adds Agnes. The real challenge is to have questions that won’t influence the players’ answers. We have to be careful to stay neutral.”
At around 12:30 p.m., after a little more than two hours of testing, the three players return home. Further tests will follow in the afternoon before the improvement work begins, and the report is written for the production team.
“The analysis and reporting work is fascinating; we try to look for similarities between what the players have said, to make the connection with the design of the game,” says Agnes. “We’re not just going to say that a player didn’t understand this or that aspect; we will try to explain why they did not understand. The report can take different forms: very condensed if it is addressed to the production as a whole and more detailed if it concerns a team and a particular aspect of the game, such as sound design. “Our role, even if we bring back feedback from players, is not to restrict creativity,” explains Aurélien. “We act as safety nets to allow game creators to test things freely. Analysts can then accompany the production team to understand the origin of the problem, make the right creative decision, maintain their choice, or adapt it.”
PLAY REMOTELY
In addition to the face-to-face testing on the day of our visit, we noticed other rooms and other analysts in the middle of work. Without players in front of them, though. They play from their homes, from a distance. A practice that is gradually taking hold at the Lab.
For a long time, remote studies were not a priority for the Lab’s teams, who preferred face-to-face testing. With “diary studies,” notebooks used to collect feedback, players can play whenever they want. This one form of testing, among others, and an approach already used by the UR team on mobile games, allows it to free itself from certain biases related to the sometimes solemn framework of laboratory tests and highly framed play rhythms. “However, it also raised questions,” adds Aurélien. “If you allow a player to access an HD game outside the Lab, this can lead to privacy issues. Other constraints are related to equipment and infrastructure. Do players have powerful enough PCs? Should we send the builds home?”
The health crisis has led them to re-consider these alternative solutions. The Parsec tool, also used by production teams and which allows remote play, has been set up for testing. “Our builds are also watermarked to prevent leaks,” explains Aurélien. “A few weeks after the start of the health crisis, this allowed us both to maintain our activity and take greater account of the player’s environment, playing from home. It also allows us to interact with players worldwide or even in France who can’t come to the Lab. “The players are necessarily more comfortable; we really stick to their natural playing context,” adds Agnes.
“I would even say that Covid-19 has accelerated the implementation of certain reflections,” says Max. “It has allowed us to focus on essential concepts. Planning ahead, the remote tests which are necessary in times of pandemic, may one day enable players to play when they want, to connect in the evening without having to worry about the opening hours of the Lab, which inevitably constrain the experience, making it challenging to balance features such as progression, economy, or difficulty.”
Other projects are also being carried out in the short and long term, including accessibility, an essential issue to enable everyone to immerse themselves without borders in a virtual universe. Reflections are also carried out on the tools used in the tests, collaborating with production teams, and around the user experience. “It’s a concept that goes beyond the game,” continues the Lab Director. “Because the gaming experience is not only limited to this one: it will also relate to what the player sees on forums, social networks, on streaming platforms, how they share it with their friends and the community. We were initially very design-oriented, but we have to open up our research field and collaborate more and more with other teams, from marketing experts to community developers. Simultaneously, we are pushing collaboration with other research centers, including those specializing in large volumes of data, to deliver even more comprehensive, relevant, and actionable analyses. That’s how we’ll understand how gamers consume and live video games.”
Written by Vincent Manilève. Read this article and more on Stories !
