Nesting Nets of an Auto-Positioning Tool in Cascadeur

We already told about our auto-positioning tool in the program Cascadeurbut there are some more interesting details that we would like to share. In particular, we did not talk about how exactly we combine the work of several neural networks in one tool.

This article will consider the approach that allowed us to implement a fairly advanced functionality using only standard deep learning methods.

Formulation of the problem

We want to give the user the opportunity to pose quickly. He can control the positions of points of interest to him, and the tool will set the positions of the remaining points on them, keeping the pose realistic.

The use of fully-connected neural networks involves fixed input and output, so we made several neural networks with a different number of input points: 6, 15, 20, 28 points from all 43 points of the character. In the pictures below, the green points are those that are fed to the input of the neural network of the corresponding level of detail.

What is the problem of using levels of detail? If we want to move a point from the 4th level, then we need to apply all 28 points to the input. But we do not want to force the user to put them all. Our goal is to give him the opportunity to move only a few of them. How then to achieve a good result? Our solution involves nesting input, combining results and using a physical model.

Input Nesting

We have chosen levels of detail that have a special property of hierarchical nesting.

The set of input points of the neural network of each level contains all the points from the previous level and adds several new ones to them. This allows us to use the output from one network as input to the next.

Combining Results

Let’s take a look at the tool’s operation as an example: the user placed all 6 main points and decided to edit the orientation of the left brush due to additional brush points from the second level of detail.

As soon as you change one more point, except for the main 6, the tool remembers it and starts using the positions of other points in the calculation. The tool operates in several stages, depending on the edited points. In this case, the whole process is schematically depicted in the picture below.

First, the first level network is used – it exposes all 43 points of the character to 6 main ones. Then, in turn, networks of more detailed levels are called. Each subsequent one receives more and more detailed input data – either specified by the user or from the result of the previous level. Thus, we get the opportunity to use several neural networks with different details at the same time.

Physical correctness

Since machine learning models are imperfect, and our neural network predicts the global positions of points, the resulting pose will have an error in the length of the edges. This is corrected using an iterative physical process, which restores the length of the edges. If you reduce the number of iterations in the program settings, you can immediately see how this affects the final result.

This process is called after the work of each of the levels in order to prevent a situation when points from an incorrect pose are fed to the input of a neural network.

Conclusion and plans

So, the tool we created has proved its worth in practice. It helps us in creating animations at the very first stage, when it is necessary to see approximate poses. In the future, we plan to add support for custom humanoid skeletons, as well as make the tool more accurate and stable.

We are also exploring the opportunities that give us more universal deep learning approaches. For example, today you can restore parts of photos with specified characteristics, as well as transfer the style and other characteristics between images. In the future, we could also use this method when creating animations, for example, to add the desired characteristics to it or to a pose.

We continue to develop our auto-positioning tool. In the near future, Cascadeur will enter the stage of an open beta test. Be sure to follow the news on cascadeur.com and in the social networks of the project.

Learn more about Cascadeur and other studio projects. Banzai games:

Why 12 Disney Principles Are Not Enough
Cascadeur: The Falling Cat Problem
Physics in a Unity Project Using Mobile Fighting as an Example
Cascadeur: The Future of Game Animation
Artificial intelligence in the fighting game Shadow Fight 3

The Banzai Games team requires a Qt GUI programmer. You can read more about the vacancy here.

Similar Posts

Leave a Reply