Skip directly to search

Skip directly to content

 

An R&D Project on AI in 3D Asset Creation for Games

 
 

AI | Radu Orghidan |
17 May 2022

This blog article has been co-authored by Thomas Bedenk and Kai Wegner. We would also like to thank Oscar Michel for kindly helping us understand some of the more subtle aspects of the method and navigate through the technical intricacies of the code.

With our experience in building games and game services for the past 15 years, we know game production pipelines are about efficiency as much as creativity. Therefore, we think unlocking the capabilities of machine learning (ML) will elevate games production teams to new heights. Professional tools like Unity’s ArtEngine already make use of ML in interesting ways to improve texture work. We wanted to take it a step further and also look at mesh creation and alteration and see how this can either be used as creative input for artists or directly facilitate the creation of 3D assets.

At Endava, we help game companies create better games with our service offerings. We cover the full game development lifecycle, including concept, art, development, automated testing and DevOps. However, we also help our clients innovate and improve their own tools and pipeline. To showcase our advanced research and development (R&D) capabilities, we initiated a project that implements machine learning in the context of creating assets for game productions.

USING NEURAL NETWORKS FOR 3D MESH AND TEXTURE GENERATION

To investigate this, we formed a team of data scientists, ML ops, 3D artists and UI developers for about two months and started researching scientific papers that use neural networks to simultaneously generate texture and augment meshes.

After selecting StyleGAN, StyleCLIP and Text2Mesh as the most promising candidates, we built a working implementation of an AI-aided asset creation journey. Based on voice-guided user input, the solution transforms existing mesh data into new creations. We set out to evaluate where this approach is promising and where it hits limitations in its current form. We also improved the usability and applicability of this solution and evaluated how 3D artists would be able to work with this solution.

An R&D Project on AI in 3D Asset Creation for Games

Text2Mesh is a method that takes as an input a 3D mesh with a carefully constructed topology and augments its geometry and texture, striving to match the textual description provided by the user. The mesh evolution respects the semantic meaning of each part of the object and is determined by a set of parameters that model the accuracy of the details, the magnitude of the deformation or the number of iterations, among others. In the example below, it grew laces in the proper position and produced spiked soles for hiking boots.

An R&D Project on AI in 3D Asset Creation for Games

The ability of Text2Mesh to develop a meaningful geometry and texture in 3D makes it a powerful tool in the hands of our experienced data scientists and enables the creation of tools that can save many hours of work for visual artists.

PROVIDING CREATIVE INSPIRATION FOR THE ARTIST

We wanted to explore if 3D artists see value in using a neural network that specifically adds details to a basic 3D shape or its texture as an inspiration for creating interesting assets. How much influence would an artist want, and how would they want to interact?

Inspiration is often led by finding value in the unexpected. First and foremost, the solution is useful to create unexpected results. Based on the directional input given by a user, the solution adds features to the shape and texture of a mesh, which leads to interesting and inspirational outcomes. Have you ever wondered what a broccoli horse looks like? Me neither, but now we know for sure! Let’s have a look at some examples:

An R&D Project on AI in 3D Asset Creation for Games

We think the solution could likely be used with some additional development. The artists really valued not having to start with a blank page, and even the weirdest AI creation, like this cookie dough alien shape augmentation, can spark creativity. Our UI interface and some post-processing helped immensely to improve the accessibility of this tool for the 3D artists.

An R&D Project on AI in 3D Asset Creation for Games

For example, one of our artists used this boot that was augmented with certain patterns and shapes and turned it into a proper asset. Which leads us to the next segment.

ACCELERATING VARIED ASSET PRODUCTION

Building on the former approach, we wanted to find out how usable the created assets are out of the box as a starting point for high-quality game assets. What auto-processing could we add to facilitate pipeline integration and actually make the artist more productive, specifically when variations or skins of an asset are needed?

To make the assets more useful for production purposes, we built in algorithms to generate more useful results. This mostly meant ensuring that the topology of the mesh is fit for purpose, that the resulting UV map can be used in later stages of production and that the mesh transformation can be adjusted by the user. Also, we implemented a prototypical UI to make it easier for artists to run their own augmentations through the browser.

An R&D Project on AI in 3D Asset Creation for Games

For certain assets, the approach could already help make artists more efficient and give them a basis to work with. However, the method should allow for more specific direction and better control of the way it augments the models thematically. Moreover, there are clear limitations to the quality of the mesh output of the implementation. While there is a clear benefit to keeping the original mesh complexity because mapping and even animating the input object could be kept consistent, this is also why the augmentation of the mesh hits clear limits. To avoid this, we already recognise automatically when spikes in the mesh are appearing and avoid spending computational resources. It might be beneficial to look into a core method that is able to add complexity to the mesh where the objects are more strongly altered.

CREATING INDIVIDUAL 3D OBJECTS FOR IN-GAME CRAFTING MECHANICS

Could we even come close to using generated assets as part of a crafting or breeding mechanic to create individualised skins or even mesh variations for unique player items? Would a direct integration without manual editing be feasible?

In the near to mid-term future, we see this solution not only working as a tool for asset creation or alteration but also as a feature that can be directly integrated in a game. Think crafting or breeding mechanics where the player can combine the characteristics of two individual assets into a brand-new form.

An R&D Project on AI in 3D Asset Creation for Games

We think that in its current state, the selected method has only very limited use – and only if the game can accept the weird-looking results as part of its identity. However, we believe there is great potential to develop a custom solution specifically for these kinds of games where the uniqueness of an asset or item could have huge value for the game economy.

THE VALUE OF COLLABORATION AND EXPERIMENTAL RESEARCH

Applying machine learning in the context of game production has a bright future, and we are certain that the value it can create today is to be found in all stages of the software development lifecycle. If used to its full potential, it can create value especially where classic automation fails.

While this was an exciting experiment for our team – more than anything else – we see this as a way of showcasing how we are able to help our clients with their research and setting up interdisciplinary teams with great expertise in a wide range of capabilities stretching beyond core game development services.

Radu Orghidan

VP Cognitive Computing

Radu is a technical business consultant with in-depth knowledge of innovation management. He leads cross-functional teams to achieve cutting edge technical objectives for our clients. His projects use data acquired from different sensors such as depth sensing cameras, mobile robots or microphones to create systems that enhance a users’ abilities to understand and interact with the environment… or sometimes to create 3D scans of his two kids. In his free time Radu loves running outdoors, skiing and being in the snow, or eating seafood while sharing a good bottle of wine.

 

Related Articles

  • 17 May 2022

    An R&D Project on AI in 3D Asset Creation for Games

  • 30 April 2020

    AR & ML Deployment in the Wild – A Story About Friendly Animals

  • 01 October 2019

    Cognitive Computing Using Cloud-Based Resources II

  • 17 September 2019

    Cognitive Computing Using Cloud-Based Resources

  • 20 August 2019

    Extracting Data from Images in Presentations

 

From This Author

  • 11 February 2021

    Mapping the Future Applications of Artificial Intelligence

  • 08 July 2020

    A Virtual Hackathon Together with Microsoft

  • 30 April 2020

    AR & ML Deployment in the Wild – A Story About Friendly Animals

Most Popular Articles

11 Things I wish I knew before working with Terraform – part 2
 

Architecture | Julian Alarcon | 23 July 2019

11 Things I wish I knew before working with Terraform – part 2

11 Things I wish I knew before working with Terraform – part 1
 

Architecture | Julian Alarcon | 25 June 2019

11 Things I wish I knew before working with Terraform – part 1

EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 1
 

Software Engineering | Matjaz Bravc | 29 June 2021

EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 1

Elasticsearch and Apache Lucene: Fundamentals Behind the Relevance Score
 

Insights Through Data | Alveiro Garcia | 08 June 2021

Elasticsearch and Apache Lucene: Fundamentals Behind the Relevance Score

An R&D Project on AI in 3D Asset Creation for Games
 

AI | Radu Orghidan | 17 May 2022

An R&D Project on AI in 3D Asset Creation for Games

Scalable Microservices Architecture with .NET Made Easy – a Tutorial
 

Architecture | Matjaz Bravc | 25 January 2022

Scalable Microservices Architecture with .NET Made Easy – a Tutorial

Data is Everything
 

Insights Through Data | Adina Gabriela Stavar | 12 January 2021

Data is Everything

A Spatial Analysis of the Covid-19 Infection and Its Determinants
 

Insights Through Data | Stefana Belbe | 09 June 2022

A Spatial Analysis of the Covid-19 Infection and Its Determinants

Challenges in creating relevant test data without using personally identifiable information
 

Testing | Alex Gatu | 25 February 2020

Challenges in creating relevant test data without using personally identifiable information

 

Archive

  • 09 June 2022

    A Spatial Analysis of the Covid-19 Infection and Its Determinants

  • 17 May 2022

    An R&D Project on AI in 3D Asset Creation for Games

  • 07 February 2022

    Using Two Cloud Vendors Side by Side – a Survey of Cost and Effort

  • 25 January 2022

    Scalable Microservices Architecture with .NET Made Easy – a Tutorial

  • 04 January 2022

    Create Production-Ready, Automated Deliverables Using a Build Pipeline for Games – Part 2

We are listening

How would you rate your experience with Endava so far?

We would appreciate talking to you about your feedback. Could you share with us your contact details?