Music visualizer based on the game Pong

Lately I've been experimenting with music visualizers. One of my favorites was inspired by the classic game Pong. In classic Pong, the ball is bounced off rackets in a constant rhythm. What if we synchronized the strokes with the beats of the music, making the rackets dance?

To make this possible, we will change the physics of the game so that the ball moves at a constant speed, and the rackets can move anywhere on their half of the screen.

We will also keep the following rules of the classic game:

  • The point of contact between the ball and the racket determines the angle of reflection
  • Rackets have no speed limit
  • The ball bounces off the top and bottom of the screen

Such physics provides us with the necessary number of degrees of freedom to move the rackets so that they hit the ball at the right moment.

A simple strategy to hit any timing is to keep your rackets close to the center. This gives us little horizontal space, but virtually infinite vertical space as the ball can bounce off the bottom and top edges of the screen. To achieve any desired shot duration, we can slow down the horizontal speed by hitting the ball more vertically. But while this proves that a solution exists for any input, it wouldn't be very interesting to look at.

What would make a visualization interesting? One desirable feature is the use of screen real estate; a game limited to a small area will seem cut-down and boring. Movement dynamics are also important; spectators love it when players make heroic efforts to barely reach the ball.

Unfortunately, it is difficult to write an algorithm that maximizes space usage while guaranteeing rules-based play. If the racket hits the ball far from the center of the screen, the ball may not have enough time to cross the screen for the next beat. Any movement of the rackets must take into account the potential impact on future moves. This is where the challenge lies: at what point should the racquets hit the ball on each beat to maximize space utilization while respecting the tempo of the composition and the physics of the game?

Optimization with given constraints

Fortunately, there has long been a field dedicated to optimizing the target (usable screen area) given variables (impact points) and the constraints of those variables (physics and pacing of the composition). If we write our requirements as a constrained optimization problem, we can use a standard solver to calculate the optimal racket placement rather than writing our own algorithm.

Formulating the problem as a constrained optimization computation gives us many advantages. If we change the physics, we only need to change the constraints without rewriting the algorithm. In addition, this will allow us to easily experiment with goals. Developing realistic animations is an iterative process made easier by a declarative style of constrained optimization.

Sounds great, but modeling 2D constraints is difficult; Is our game really 2D? It turns out that you only need to model the horizontal component! In our set of rules, the ball moves at a constant speed, so the vertical component of speed is clearly defined by the horizontal component. Thanks to the game simulation, we will be able to calculate the vertical position of the ball at any time.

Knowing the vertical position of the ball also gives us the vertical position of the rackets, because it must match the position of the ball (plus a small delta to get the desired angle). Between hits we simply interpolate the positions linearly to get a smooth motion. Thus, we only need the solver to calculate the horizontal positions and velocities at the moments the ball hits the rackets; the rest can be obtained from the game physics.

Ball flight diagram i

Constants

Let's start by defining the hard-coded parameters of the game:

$\begin{align} &\text{} W \in \mathbb{R}^+ \text{ - screen width} \\ &\text{} S \in \mathbb{R}^+ \text{ - speed ball} \end{align}$

Timings input data

Next we need to synchronize the time of the beats. Each

t

– this is the time when the ball should hit the racket. We get these times from MIDI files, although in the future I'd like to explore more automated ways to extract them from audio.

$\text{} T = {t_0, t_1, \ldots, t_{n}} \text{ - time of each beat}$

We take the differences between adjacent moments to get the duration of the ball's movement between hits. Each

d

is the time interval between two hits of the ball.

$\begin{align*} \text{} D &= {d_0, d_1, \ldots, d_{n-1}} \text{ - duration of each movement}\\ d_i &= t_{i+1} - t_ {i} \quad \text{for } i = 0,1, 2, \ldots, n-1 \\ \end{align*}$

All this is ready-made input data that we pass to the solver. Now let's define the variables that we need to optimize, as well as their relationships with other quantities.

Variables

It's worth remembering that we only need to optimize horizontal positions and speeds.

$\text{} P = {p_0, p_1, \ldots, p_{n-1}} \text{ - horizontal distance from the racket to the center}$

Each

$p_i$

denotes the horizontal distance from the racket to the center where it hits the ball the i-th time. Even indices (

$p_0$

,

$p_2$

…) refer to the left racket, and the odd ones (

$p_1$

,

$p_3$

…) – to odd.

$\text{} V = {v_0, v_1, \ldots, v_{n-1}} \text{ - horizontal speed of the ball}$

Multiple horizontal ball speeds. Each

$v_i$

denotes the horizontal speed of the ball after the i-th hit. To make it easier to create constraints, we make it always positive, regardless of whether the ball is moving left or right.

Restrictions

First we limit the paddles to their half of the screen:

$0 \leq p_i \leq \frac{W}{2}$

We have determined

$p$

as the distance from the center, so it is non-negative no matter which half of the screen the racket is on.

Next, we will limit the magnitude of the ball's horizontal speed to positive values ​​that do not exceed its total speed S.

$0 < v_i \leq S$

These two restrictions completely determine the physics of our game. To synchronize with the tempo of the music, we add another constraint:

$p_{i-1} + p_i = d_i v_i$

This ensures that the ball reaches the next racket at the right moment. The left side of the equation describes the total distance of movement horizontally (the sum of the distances to the center for two successive hits on the rackets), which should be equal to the product of the duration of the flight and the horizontal speed of the ball.

Target

We know that there is a degenerate solution in which the rackets do not move away from the center. They will need to be motivated to behave in the opposite way; maximize the distance to the center.

Our goal then is to maximize the sum of the distances from the racket to the center.

$\text{Maximize} \sum_{i=1}^n p_i$

Problem solution

After drawing up the mathematical formulation, all we have to do is solve it. All our constraints are linear, so any linear programming (LP) solver will suffice. I chose

CVXPY

for its simplicity and generality. CVXPY solves convex optimization problems, and LP problems are a subset of it. While we won't need the full range of solver capabilities, its flexibility to support more complex goals and constraints will make the creative exploration process easier.

In CVXPY we can express the above restrictions as follows:

Of course, the code matches the formulas. In fact, I found the code easier to understand than the formulas thanks to the clear variable names.

The solver returns the horizontal positions at which the rackets should hit the ball, as well as the horizontal speed of the ball, from which we can easily calculate the angles of reflection. This is sufficient to calculate vertical positions using simulation.

To animate the finished result, we use the positions of the ball and rackets at the time of each shot as key frames. Between hits, we interpolate positions to create smooth animations.

Conclusion

Visualization is an extremely multivariate task in which our eyes are the only judges. Most often, its results depend on what algorithms we know. I suspect there are many more constrained optimization-based renderers to be discovered; I'd love to know about them!

The code is open source: Github repository

Acknowledgments

Thanks to Pat O'Neill for proofreading drafts of this post.

Examples

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *