The math behind the rolling shutter phenomenon

5 min


image

I remember once saw the photo above on flickr and broke his brain, trying to understand what was wrong with her. The thing was that the propeller was spinning while the motion sensor in the camera was “reading the readings,” that is, during the camera’s exposure, there was some movement. This is really worth thinking about, let’s think together.

Many modern digital cameras use a CMOS sensor as their “sensitive” device, also known as active pixel sensor, which works by accumulating an electronic charge when light is incident on it. After a certain time has elapsed – exposure time – the charge line by line moves back to the chamber for further processing. After that, the camera scans the image, storing rows of pixels line by line. The image will be distorted if there was any movement during shooting. To illustrate, imagine shooting a rotating propeller. In the animations below, the red line corresponds to the current reading position, and the propeller continues to rotate as it reads. The part under the red line is the resulting image.

The first propeller makes 1/10 revolution during the exposure:

image

Subscribe to the channels:
@Ontol – the most interesting texts / videos of all time, affecting the picture of the world
@META LEARNING – where I share my most useful findings about education and the role of IT / games in education (as well as thoughts on this subject by Anton Makarenko, Seymour Peypert, Paul Graham, Joseph Liklider, Alan Kay)

The image is a little distorted, but nothing critical. Now the propeller will move 10 times faster, making a full rotation during the exposure:

image

It already looks like the picture that we saw at the beginning. Five times per exposure:

image

This is already a little too much, so you can move off the coils. Let’s have fun and see how different objects will look at different rotation speeds per exposure.
Exactly the same propeller:

image

Propeller with large blades:

image

Car wheel:

image

We can perceive rolling shutter effect as a kind of transformation of the coordinates of a real object from the “object space” into the “image space” of the distorted object. The animation below shows what happens to the Cartesian coordinate system as the number of revolutions increases. At low speeds, the deformation is negligible – the number increases to unity, and each side of the coordinate system sequentially moves to the right side of the image. This is a rather complex transformation for perception, but easy to understand.

image

Let the image be I (r, θ), the real (rotating) object will be f (r, θ), where (r, θ) are the 2D polar coordinates. We chose the polar coordinates for this task because of the rotational motion of objects.

The object rotates with an angular frequency ω, and the shutter moves along the image with a speed v vertically. In the position (r, θ) in the picture, the distance the shutter has passed since the beginning of the exposure is y = rsinθ, where the elapsed time from this moment is (rsinθ) / v. During this time, the object turned on (ω / v) rsinθ) radians. So we get

I (r, θ) = f (r, θ + (ω / v) rsinθ),

which is the required transformation. The coefficient ω / v is proportional to the number of rotations per exposure and parameterizes the transformation.

To get a deeper understanding of the obvious forms of propellers, we can consider an object consisting of P propellers where f is nonzero only for

θ = 2π / P, 4π / P … 2π = 2pπ / P for 1

Image I is nonzero for θ + (ω / v) rsinθ = 2pπ / P or

image

In the Cartesian coordinate system, it becomes

image

and helps us in explaining the reason why propellers take an S-shape – it’s just a function of arc tangent in image space. Cool. Below I built this function with a set of five propeller blades with slightly different initial shifts, you can see this on playback. They are very similar to the figures from the animations above.

image

Since we learned a little more about the process, can we fix corrupted photos? Using one of the images above, I can draw a line through it, rotate it back and paste these pixels into a new image. In the animation below, I scan the image on the left, marked with a red line, and then rotate the pixels along this line, getting a new image. So we can recreate the image of a real object, even if suddenly an annoying rolling shutter ruined your photo.

image

Eh, if I had better control of Photoshop, I would have extracted the propellers from the original photo on Flickr, edited and returned to the photo. I think I know what I’ll do in the future.

If you want to know the actual number of blades in the photo at the beginning of the post and the rotation speed, you can read this great post on tumblr daniel walshin which he gives a mathematical explanation.

He believes that we can calculate the number of blades by subtracting the “lower” blades from the “upper”, so we get three blades in that picture. We also know that the propeller scrolls approximately twice during the exposure time, so if we try to “cancel” the rotation at several different speeds, we get something like this:

image

I had to understand where the center of the propeller was, so I drew a circle. Apparently, the center should be somewhere nearby. Unfortunately, one blade is missing, but there is enough information for the image.

I found a place where everything intersects the most, therefore, at this rotation speed (2.39 revolutions per exposure), here’s what the original image and blades look like:

image
image

Unfortunately, the picture is not perfect, but at least very close to reality.

About the Author: Jason Cole is a graduate student from London who burns math, physics, and data visualization. Here him Web site. And the article was published here.

Discussion at Hacker News

add. video

Subscribe to the channels:
@Ontol – the most interesting texts / videos of all time, affecting the picture of the world
@META TRAINING, where I share my most useful findings about education and the role of IT / games in education (as well as thoughts on this subject by Anton Makarenko, Seymour Peypert, Paul Graham, Joseph Liklider, Alan Kay)

image

Learn the details of how to get a sought-after profession from scratch or Level Up in skills and salary by completing SkillFactory paid online courses:


Read more

  • Trends in Data Scenсe 2020
  • Data Science is dead. Long live Business Science
  • The coolest Data Scientist does not waste time on statistics
  • How to Become a Data Scientist Without Online Courses
  • Sorting cheat sheet for Data Science
  • Data Science for the Humanities: What is Data
  • Steroid Data Scenario: Introducing Decision Intelligence

0 Comments

Leave a Reply