The art of VR playtesting – Ganbatte’s first contact with external testers!
At Mimicry, we think we’ve become quite comfortable with our playtesting routine. Our last playtesting day happened to be two days before the end of the Summer Internship, making it our “End of the Summer Internship Milestone” to work towards. Therefore, it comes at no surprise that our interns Pedro and José took a very active role in defining and designing the playtest.
In any case, we would like to present our method and some thoughts on the art of playtesting.
The Test Plan
Having a concrete plan for a group of playtest sessions is very helpful. Our current playtest document goes a bit like this:
- List of each session, the responsible hosts, and testers. As Ganbatte is a multiplayer game, we find it necessary to have a group of multiple hosts to handle 4 people simultaneously.
- Check-list with all the required hardware, and game installation check-up.
- Script/Procedure where we write extensively what to say, when to say. This might come off a bit robotic, but it helps to create even access to information across the multiple groups of testers. It also helps to create a continuum, so that despite the rotation of the hosts, the playtests are similar in structure.
Getting People In
We relied both on our personal networks and the fact that we’re at the biggest startup incubator in our city (Instituto Pedro Nunes), to get our testers. Using Google Forms to build a signup form has proven very handy in order to organize the sessions.
Preparation
After the plan is made, preparations seem to be something unremarkable. However, something that we’ve learned from experience is: FREEZE THE TESTING BUILD SOME DAYS BEFORE THE PLAYTEST.
It’s always better to be told 100 times that something that you have already fixed in the current development branch needs to be fixed. Seriously, starting the playtests and having things that you thought were stable not working because someone messed up something the night before and didn’t go through internal functional testing before putting it in front of testers is terrible. It’s terrible because you’ll be completely stressed out trying to fix something during the playtest day, it’s terrible because you’re going to put your testers through a bad experience that could have been avoided, it’s terrible because those testers will probably not like to test again in the future. There’s no excuse for not being the most professional you can be.
Testing is serious (and fun) and an amazing opportunity to learn more about your game. Respect your work and your testers and your playtests will reward you with the keys to make your game better. Ignore at your own peril.
During the test
Even though this seems smooth sailing after a good preparation, it was interesting to notice that our (former) game dev rookies weren’t expecting the intensity of the sessions. We ended up having 5 sessions, with 4 players per session. This means that there was a constant demand for attention, being on our feet the whole day, explaining the same things over and over, hearing a lot of feedback, making personal notes. Despite that, there was still time to enjoy ourselves and enjoy the process.
This is why we are firm believers that the snacks for the testers should extend as snacks for the hosts! We definitely needed all those soda cans to keep up!
Gathering Information
These playtests we opted for 3 ways of gathering information:
- Active note taking during the session;
- Questionnaire;
- Final free-form interview.
Every one of these methods has its limitations. Since this was our first “big” public playtest, our main objectives had a lot to do with finding out how people interacted with the game. For this, our invasive note-taking and observation worked well. On the other hand, we are aware that the reactions towards the game might have been biased because around half of the testers were friends and we were present all along.
We look forward to doing a new public playtest event where no member of the development team is present during the experiment. So far, in our development we think our DIY approach has been adequate. The fact that we are based in a big business incubator is very helpful in order to get testers.
It’s also nice to be aware of the possibility of outsourcing a playtesting event, especially when we want to minimize your interference in the test results or we want to test multiplayer in various geographical locations. I’ve been testing and trying out Fishbowl VR (affiliated link), and the kind of service they offer would be perfect in order to set up a very low interference playtest with users from all over the world.
Analyzing, reflecting & improving
After the test, it’s time to look at the data and derive action points. It’s great to have all the team together analysing the sessions, and extracting takeaways. Our procedure is to first go over our personal notes and list all bugs, interaction problems, concerns. After that, we go over the questionnaire and one by one we check the values against our notes, adding more points to the list. After compiling this list, we create concrete action points.
We’re not following to the letter every piece of feedback we have been offered, but there was a lot of wisdom to be gained. Testers often don’t realise how important their experience is for helping us make the best possible games and experiences we can. We love them and we can’t thank them enough. For the team, this was an amazing opportunity to give everyone the experience of conducting a playtest event.
Having a playtesting event was also a good way of ending the Summer Internship. In general, it’s difficult to allow aspiring game developers to experience the whole process of making a game during an internship. I believe that being able to see people play the game you’ve been working on, analyzing the way they interact, listening to their feedback and defining the next steps, breeds love and stamina for the process of game development.
– Daniela
