How AI helps throw hats out of the window directly onto the heads of passers-by

I'm a simple person living in downtown New York. I put on my shoes one at a time, apologize when I accidentally push someone on the street, and use artificial intelligence, to throw caps on the heads of people who stand under the windows of my apartment. In short, the most ordinary guy.

A huge flow of people passes under my window every day. I see a sea of ​​bare heads under the hot sun. I believe that DropofaHat.zone will be the first of many “window shops.” Here, a busy New Yorker can book a 5-minute timeslot, pay for a cap, stand outside my window for 3 seconds, have the cap placed directly on his head, and get back into his extremely important, extremely stressful day—all within one New York minute.

How to use AI for window throws:

I dream that from all the city windows something will constantly fall on our heads. You can bring this dream closer. To do this, you will need a Raspberry Pi, an Adafruit stepper motor for the falling mechanism, a small piece of string, Roboflow for the artificial intelligence, and a very light but very cool product (like propeller caps).

How to open a window

It was a real challenge. My window only opens 4 inches. If I had not been able to solve this problem, my entire business would have had no chance. In theory there should have been some kind of key or screw that I should have removed to open the window wider, but I saw no sign of anything like that, other than a few very small cracks at the bottom.

If I could just look at what type of window I have, I would know what kind of lock goes with it. But everything turned out to be very confusing. I think I have a folding one (awning in the picture) with double glass. Maybe… Each window type can resemble many other types.

I ended up just Googling “window keys” and looking at all the ones that looked like they might match mine. Most looked like they needed a lock, and then I came across this weird shape.

I was mentally prepared to buy a dozen other key options, but this one fit the bill!

Choosing a hat

Next I had to decide what hats I was going to sell from the window. My window is located quite high. It should be a hat that won't hurt anyone if it falls and won't fly out onto the road.

I decided that I needed something, something futuristic. Something that will look beautiful when it gracefully falls out of the window onto your head.

Propeller cap! Among other things, it has a detail that symbolizes the flight that the cap is about to take.

Fall mechanism

This was the simplest thing I needed to do to get started. I had a Raspberry Pi and a stepper motor, so I decided to put them to work.

Imagining the ultra-sharp blades on the tiny motor cutting the rope, I realized it would be better to just wrap the rope around the stepper motor and make it move slightly. I had a huge camera stabilizer that I wanted to test for this purpose, and I was ready to stick it out the window when I realized that it could just hang a rope over the window.

I just copied this from Adafruit's stepper motor manual. This is a separate Python file on the Raspberry Pi that the computer will run when the AI ​​determines that someone is standing in the right place and ready to receive their hat.

“dropHat.py” on Raspberry Pi

    import time
    import board
    import digitalio

    enable_pin = digitalio.DigitalInOut(board.D18)
    coil_A_1_pin = digitalio.DigitalInOut(board.D4)
    coil_A_2_pin = digitalio.DigitalInOut(board.D17)
    coil_B_1_pin = digitalio.DigitalInOut(board.D23)
    coil_B_2_pin = digitalio.DigitalInOut(board.D24)

    enable_pin.direction = digitalio.Direction.OUTPUT
    coil_A_1_pin.direction = digitalio.Direction.OUTPUT
    coil_A_2_pin.direction = digitalio.Direction.OUTPUT
    coil_B_1_pin.direction = digitalio.Direction.OUTPUT
    coil_B_2_pin.direction = digitalio.Direction.OUTPUT

    enable_pin.value = True

    def forward(delay, steps):
        i = 0
        while i in range(0, steps):
            setStep(1, 0, 1, 0)
            time.sleep(delay)
            setStep(0, 1, 1, 0)
            time.sleep(delay)
            setStep(0, 1, 0, 1)
            time.sleep(delay)
            setStep(1, 0, 0, 1)
            time.sleep(delay)
            i += 1

    def setStep(w1, w2, w3, w4):
        coil_A_1_pin.value = w1
        coil_A_2_pin.value = w2
        coil_B_1_pin.value = w3
        coil_B_2_pin.value = w4

    # Run a full rotation (512 steps) with a 5 millsecond delay
    forward(5, int(512))

AI

I thought the AI ​​would be the hardest part, but it worked out surprisingly quickly. I installed a webcam above a window and wanted the result to be displayed on video in real time. I wanted the AI ​​to show me what it was seeing. This would allow for live broadcasting afterwards. Besides, it was very interesting to watch. This is what a webcam feed looks like.

I chose object detection as an initial model, and then recorded pedestrians without hats walking under my window for a couple of minutes. Then it's time to label the images. I wanted the model to tell me when someone was standing on the sidewalk right outside my window. I set the class and tooltip to “person” and most of the markup was done automatically for me. In other cases, I placed a frame around the person if he was on the desired part of the sidewalk, or marked it as zero if he was not.

You can view the tagged images here: https://universe.roboflow.com/test-y7opj/drop-of-aa-hat/browse.

Obviously the view from your window will be different from mine, so you'll have to spend a few minutes recording and uploading your own data.

The model seemed to work well even when it only had 133 labeled images of positive and zero values. Since I was only interested in a small section of the sidewalk, I added an image pre-processing step – cropping.

While this worked, I realized that I would occasionally bump the webcam and the image would shift, so a more general model was needed. I removed the cropping, and it turned out to be a good solution. The AI ​​would only “detect” the image when it was in the right place. Even though I only tried this with one “person” clue in the annotation, it worked for everyone who walked by.

Finally I had a working model. You can view it here: https://universe.roboflow.com/test-y7opj/drop-of-aa-hat/model/2

Now you need to run the Python program on the computer where the webcam is installed. I only need two things from this code:

1. Confirm that someone is standing in the correct place within 3 seconds.

2. Contact Raspberry Pi after 3 seconds.

On your computer with a webcam, install the pip output library and the SSH libraries that I used.

pip3 install inference-sdk
pip3 install opencv-python
pip3 install paramiko

This is the entire Python file. You will have to enter your own API key. I won't give you mine.

It works by calling the model every second, and if it confirms that someone is in that location for 3 seconds straight, it SSHs into my Raspberry Pi and runs the dropHat.py function.

import cv2
import time
import paramiko

from inference_sdk import InferenceHTTPClient

CLIENT = InferenceHTTPClient(
    api_url="https://detect.roboflow.com",
    api_key="API_KEY"
)

def ssh_execute(host, port, username, password, command):
    client = paramiko.SSHClient()
    client.load_system_host_keys()
    client.set_missing_host_key_policy(paramiko.WarningPolicy)

    try:
        client.connect(host, port=port, username=username, password=password)
        stdin, stdout, stderr = client.exec_command(command)
        
        # Print command output
        print(stdout.read().decode().strip())
        
        if stderr.read().decode().strip():
            print('Error:', stderr)
    finally:
        client.close()

video = cv2.VideoCapture(0)  # 0 usually refers to the webcam

consec_detections = 0
while True:
    ret, frame = video.read()

    # Directly pass the frames to the Roboflow model
    result = CLIENT.infer(frame, model_id="drop-of-a-a-hat/2")

    # Check if a prediction is made
    if 'predictions' in result and len(result['predictions'])>0:
        consec_detections += 1
    else:
        consec_detections = 0

    # If there are three consecutive detections, perform an action (like printing a message)
    if consec_detections >= 3:
        # DROP THE HAT
        ssh_execute('raspberry.local', 22, 'pi', 'raspberry', 'python3 dropHat.py')
        # Reset counter
        consec_detections = 0

    # Delay before the next frame for 1 second
    time.sleep(1)

Grand Design

And here is my big dream. Imagine a world where you can walk around New York City and everything you need (except bricks) falls out of the windows at you. At any moment, absolutely at any moment. I want to live in this world. That's why I posted this guide so that everyone can make their own window shop. Isn't this a great idea?