Skip to content

feat: add gesture-based drone control example script using MediaPipe#161

Open
sushantkumarkhobian-lab wants to merge 3 commits into
iamaisim:mainfrom
sushantkumarkhobian-lab:main
Open

feat: add gesture-based drone control example script using MediaPipe#161
sushantkumarkhobian-lab wants to merge 3 commits into
iamaisim:mainfrom
sushantkumarkhobian-lab:main

Conversation

@sushantkumarkhobian-lab

@sushantkumarkhobian-lab sushantkumarkhobian-lab commented Jun 23, 2026

Copy link
Copy Markdown

Description

This PR adds a Python-based gesture-controlled multirotor simulation example using AirSim (ProjectAirSim), MediaPipe, and OpenCV.

The system uses real-time webcam hand tracking to interpret predefined finger gestures and convert them into drone motion commands inside a simulated environment.

Features

  • Real-time hand tracking using MediaPipe Hands
  • Webcam-based input processing using OpenCV
  • Stable gesture filtering using short-term history smoothing
  • Gesture-based directional drone control (left, right, front, back)
  • Vertical control using hand gestures (ascend and descend)
  • Hover mode using a closed fist gesture
  • Instant landing / shutdown using a 3-finger gesture
  • Strict separation of horizontal and vertical motion to prevent altitude drift
  • Safe API shutdown sequence (disarm + disable control + disconnect)

Control Mapping

  • 1 finger up → Ascend
  • Thumb down → Descend
  • 1 finger Left → Move left
  • Thumb Right → Move right
  • 2 fingers up → Move forward
  • 2 fingers down → Move backward
  • Closed fist → Hover
  • 3 fingers up → Land + shutdown

Purpose

This example demonstrates a clean integration of computer vision and drone simulation control, enabling intuitive gesture-based navigation. It is designed as an educational and experimental interface for understanding real-time robotics control, motion mapping, and human–drone interaction systems using AirSim.

Dependencies

  • ProjectAirSim Python API
  • OpenCV (opencv-python)
  • MediaPipe (mediapipe)

@LucasJSch

Copy link
Copy Markdown
Collaborator

Nice work! I think it's a cool addition :).

Can you upload to this PR a video/screenshots of this script working?

@sushantkumarkhobian-lab

sushantkumarkhobian-lab commented Jun 25, 2026

Copy link
Copy Markdown
Author

Hi @LucasJSch! Thanks for the kind words.

I actually have a demo GIF hosted over in my main project repository readme AirSim-Gesture-Drone-Control. Here is a quick look at it in action:

Demo_Output

Let me know if you'd like me to add any additional screenshots!

@LucasJSch

Copy link
Copy Markdown
Collaborator

I have a question seeing your code and GIF. Are you using AirSim or ProjectAirsim?

@sushantkumarkhobian-lab

Copy link
Copy Markdown
Author

The script was originally written and tested using the legacy Microsoft AirSim Python API, which explains why you noticed that in the code and GIF.
I am actively transitioning my environment over to ProjectAirSim right now. If there are specific package imports, client initializations, or API changes required to make this fully compatible with ProjectAirSim standards, please let me know! I am more than happy to update the code in this PR to match your repository's requirements.

@sushantkumarkhobian-lab

Copy link
Copy Markdown
Author

Hi @LucasJSch, thanks for the review.

I have updated the implementation to be fully compatible with ProjectAirSim and cleaned up the control flow accordingly. The script has been simplified and stabilized, with proper handling of gesture-based navigation and shutdown behavior.

A short demo GIF of the working system is attached below for reference.

Screen Recording 2026-06-27 172415

Please let me know if any further changes are required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants