Skip directly to search

Skip directly to content

 

A Basic Setup for Mass-Testing a Multiplayer Online Board Game

 
 

Testing | Sebastian Denzer |
19 October 2021

An essential aspect of software development is assuring the quality of the code one has developed. Testing is an important feedback element for developers, but testing a larger piece of software can be very time-consuming. As developers, we are basically looking for the most efficient approaches – and when it comes to testing, automation helps!

WHY TESTING IS IMPORTANT

The consequences of poorly tested software range from blemishes and malfunctions to a non-operable or non-functioning application. And users are merciless in their feedback on poorly tested software – and rightly so!

If software is already in live operation and you add even a small new feature, you still have to ensure that the entire application functions smoothly and as expected after the update. That means you need to test your app – again and again. Manual and automated testing are both common software testing mechanisms.

The traditional approach is to have everything tested by the QA team, who are manually testing each part of the application. They will not only find bugs but also expose other problems that automated testing cannot cover, like bad or misleading UX, missing dialogue or localisation issues. The downside of manual testing is that it takes individual testers hours and hours of work.

In many cases, automated testing can be a very efficient alternative and make your developers’ lives a lot easier – be it API testing or integration testing, where you write code to verify the functionality of certain functions, code parts, or entire software modules.

THE CHALLENGE: TESTING CATAN UNIVERSE

We have seen how essential testing is in general. Of course, it becomes even more important for a live product such as the multiplayer online board game Catan Universe, which we developed for our client United Soft Media GmbH in collaboration with Catan GmbH.

The “Universe” part of the name indicates a whole new level of complexity: the digital version of a whole universe of game variants and modes, including two games in one! Both come with different scenarios or maps, combinations of different game rules, settings for each game, etc.

Mass Testing
Two games in one: Catan – a board game for up to 6 players (left), Rivals for Catan – a card game for 2 players (right).

And on top of that, Catan Universe offers both a single-player and a multiplayer experience, with the latter one using two different matchmaking algorithms to connect multiple clients, i.e. “players”. The multiplayer part itself currently features games for 2-6 players. Behind the scenes, we are also looking into ways to increase this number even further. One day we might be able to allow 100 (in words: one hundred) and more players to play one game of Catan simultaneously.

OUR SOLUTION: AUTOMATED PLAY TESTING

Catan Universe is a long-running product, being first released in 2016 and enjoying continuous growth over the years. The overall variety of different game setups with all the different game situations – especially in the “endgame” phase which can last up to 1 hour – could not be covered manually by even the largest QA team. We needed a method to test all game permutations with their different modes and scenarios automatically. So, we built something that we call “Automated Play Testing”.

What helped us from the start is that we already had an AI (Artificial Intelligence) implementation for both single-player and multiplayer games: in the game, you can play against well-known Catan characters like Marianne or Louis who bring in their own distinct personalities and game approaches. And while the players enjoy competing against these characters, we also use them as our very own QA testers who can play the game the whole night – without needing pizza and caffeine. Of course, they can never be as clever or creative as human players, but they can cover each and every game situation, and they can do it repeatedly without ever getting bored!

Having our testers in place, what else do we need?

First, the game must start in some kind of “autopilot mode” in which it will simply log into a user account based on a config file containing the autopilot settings, e.g. described as JSON data.

{

"autoPilot": true,

"email": "autopilot@automail.com",

"password": "123456",

"autoPlay": false,

"turboMode": false

}

See more ˅
Example config file for auto pilot mode; this is just a subset of all the possible parameters that could be used to set up Automated Play Testing.

Secondly, we want to enter the autopilot mode only in certain situations, so we use command line parameters to control which special functions should be enabled. For example, we differentiate between “active” and “passive” game instances where one active player will invite other, passive players to a game.

CatanUniverse.exe /autopilot /config user1.json

See more ˅
Starting Catan Universe in autopilot mode as “user1” with command line parameters.

Once we can automatically log in, we should start giving the autopilot input on what it must react to or even start its own logic to communicate with other clients. These could be

  • friend or guild membership requests to build a player base we can interact with,
  • lobby invites for “custom matches” (inviting other players that we have in our friend / guild list),
  • or the game client could start the “auto-match” matchmaking (searching for any players that match your favourite game settings).

Autopilot mode: automatically logging in and starting a single-player game – without any user interaction.

We keep the autopilot code separate from our game logic code, so that none of our developers can accidentally break the game logic. Of course, we also want to strip all autopilot-related code from the final release build of the game since we do not want players to use the autopilot AI – they should play on their own! And finally, we definitely do not want to open any security issues by allowing automatic control of multiple game instances – think of DDOS (Distributed Denial of Service) attacks, for example.

Which brings us to the final puzzle piece: to be able to start as many game clients as we want, we need some external tool to allow so-called “multi-boxing”. This is a technique that some players of online games use to play with multiple characters simultaneously, e.g. for getting an increased amount of resources or experience – or only as an “additional challenge”, like playing chess against multiple opponents at once.

Multi-boxing software allows us to start an application multiple times on one single computer, giving each running instance a separate set of files to work with – in our case the user config file – so that they do not interfere with each other.

Now that we have the basic pipeline running with multiple AI players testing the game, we can consider some more advanced settings: it could help to implement a “turbo mode”. As the video below shows, the game is running at an incredibly fast speed. In addition, this also functions as stress test for the servers’ multiplayer session handling because the AI will send game actions to the server faster than any human being could ever play the game.

Autopilot mode: mass-testing games in “turbo mode.”

You may also want to add logging and tracking systems, so that you don’t have to visually check the hundreds of instances running. Actually, this system should run day and night in order to cover all game modes and situations, find any crashes or blockers and even identify situations where the AI stops playing the game because it could not find any valuable actions to perform, thus helping to improve the overall game experience.

Finally, that also means that you should integrate Automated Play Testing into your CI pipeline, so that it runs like a regular automated test on every new release you are planning.

CONCLUSION

We developed our Automated Play Testing solution in order to increase the quality and lower the costs for the extensive manual testing efforts required to validate the application. Although the initial implementation was cost-intensive, and the maintenance and adding of new features moderately increases the cost, the solution is more economical than the manual effort that would have been required for testing as well as ensuring and improving the quality of the product.

Sebastian Denzer

Senior Game Developer

Sebastian is a Game Developer with more than 15 years of experience in all areas of game development, from gameplay programming with a specialisation in game AI to graphics and sound programming to UI/UX and multiplayer/networking. Since 2019, he has been the Lead Developer for “Catan Universe”, the digital version of the famous board game. Within that role, he ensures that the vision of the product is kept in UX and UI and is always keen to bring the physical board game feeling – the ‘haptics’ – to its digital counterpart. Sebastian, also known as “Denzi”, basically spends his whole life with game development, as he uses pretty much all his spare time to create his own games.

 

Related Articles

  • 23 November 2021

    How User Experience Design is Increasing ROI

  • 16 November 2021

    Create Production-Ready, Automated Deliverables Using a Build Pipeline for Games – Part 1

  • 19 October 2021

    A Basic Setup for Mass-Testing a Multiplayer Online Board Game

  • 24 August 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 3

  • 20 July 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 2

  • 29 June 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 1

  • 15 May 2020

    Performance and security testing shifting left

  • 25 February 2020

    Challenges in creating relevant test data without using personally identifiable information

Most Popular Articles

11 Things I wish I knew before working with Terraform – part 2
 

Architecture | Julian Alarcon | 23 July 2019

11 Things I wish I knew before working with Terraform – part 2

11 Things I wish I knew before working with Terraform – part 1
 

Architecture | Julian Alarcon | 25 June 2019

11 Things I wish I knew before working with Terraform – part 1

EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 1
 

Software Engineering | Matjaz Bravc | 29 June 2021

EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 1

How User Experience Design is Increasing ROI
 

Next Gen Insights | Valentin Andrei Iordache | 23 November 2021

How User Experience Design is Increasing ROI

AWS Serverless with Terraform – Best Practices
 

Architecture | Vlad Cenan | 10 December 2019

AWS Serverless with Terraform – Best Practices

Kubernetes Design Principles Part 1
 

Architecture | Armen Kojekians | 30 April 2019

Kubernetes Design Principles Part 1

Elasticsearch and Apache Lucene: Fundamentals Behind the Relevance Score
 

Insights Through Data | Alveiro Garcia | 08 June 2021

Elasticsearch and Apache Lucene: Fundamentals Behind the Relevance Score

Create Production-Ready, Automated Deliverables Using a Build Pipeline for Games – Part 1
 

Software Engineering | Olaf Spiewok | 16 November 2021

Create Production-Ready, Automated Deliverables Using a Build Pipeline for Games – Part 1

Challenges in creating relevant test data without using personally identifiable information
 

Testing | Alex Gatu | 25 February 2020

Challenges in creating relevant test data without using personally identifiable information

 

Archive

  • 23 November 2021

    How User Experience Design is Increasing ROI

  • 16 November 2021

    Create Production-Ready, Automated Deliverables Using a Build Pipeline for Games – Part 1

  • 19 October 2021

    A Basic Setup for Mass-Testing a Multiplayer Online Board Game

  • 24 August 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 3

  • 20 July 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 2

  • 29 June 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 1

  • 08 June 2021

    Elasticsearch and Apache Lucene: Fundamentals Behind the Relevance Score

  • 27 May 2021

    Endava at NASA’s 2020 Space Apps Challenge

  • 27 January 2021

    Following the Patterns – The Rise of Neo4j and Graph Databases

  • 12 January 2021

    Data is Everything

  • 05 January 2021

    Closing-the-gap-between-the-product-owner-and-the-team-part-3

We are listening

How would you rate your experience with Endava so far?

We would appreciate talking to you about your feedback. Could you share with us your contact details?