Real-Time Gesture Recognition System for Human-Computer Interaction Python
👤 Sharing: AI
```python
import cv2
import mediapipe as mp
import numpy as np
# --- 1. Initialize MediaPipe Hand Tracking ---
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(static_image_mode=False, # Set to True for image processing, False for video
max_num_hands=1, # Number of hands to detect
min_detection_confidence=0.7,
min_tracking_confidence=0.7) # Increased for better tracking
mp_drawing = mp.solutions.drawing_utils
# --- 2. Define Gesture Recognition Logic ---
def recognize_gesture(hand_landmarks):
"""
Recognizes gestures based on hand landmark positions. This is a basic example;
you'll need to customize this based on the gestures you want to detect.
Args:
hand_landmarks: Landmark list from MediaPipe.
Returns:
A string representing the recognized gesture, or "Unknown" if no gesture is recognized.
"""
# Check if hand_landmarks is valid
if not hand_landmarks:
return "Unknown"
# Get landmark positions for specific fingers
thumb_tip = hand_landmarks.landmark[mp_hands.HandLandmark.THUMB_TIP].y
index_tip = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP].y
middle_tip = hand_landmarks.landmark[mp_hands.HandLandmark.MIDDLE_FINGER_TIP].y
ring_tip = hand_landmarks.landmark[mp_hands.HandLandmark.RING_FINGER_TIP].y
pinky_tip = hand_landmarks.landmark[mp_hands.HandLandmark.PINKY_TIP].y
# Check the relative position of the index finger tip to the palm.
index_mcp = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_MCP].y
thumb_mcp = hand_landmarks.landmark[mp_hands.HandLandmark.THUMB_MCP].x
wrist = hand_landmarks.landmark[mp_hands.HandLandmark.WRIST].x
# Simple gesture: "Thumbs Up" (very basic and can be improved)
if thumb_tip < index_tip and thumb_tip < middle_tip and thumb_tip < ring_tip and thumb_tip < pinky_tip:
if thumb_mcp > wrist:
return "Thumbs Up"
# Simple gesture: "Pointing"
if index_tip < middle_tip and index_tip < ring_tip and index_tip < pinky_tip and index_tip < thumb_tip and index_mcp > thumb_tip:
return "Pointing"
# Simple gesture: "Fist"
if index_tip > index_mcp:
return "Fist"
# Add more gesture recognition logic here. Consider using distances between landmarks, angles between fingers, etc.
return "Unknown"
# --- 3. Main Video Processing Loop ---
def main():
"""
Captures video from the webcam, processes each frame with MediaPipe,
recognizes gestures, and displays the results.
"""
cap = cv2.VideoCapture(0) # 0 is usually the default webcam
if not cap.isOpened():
print("Error: Could not open webcam.")
return
while True:
success, image = cap.read()
if not success:
print("Ignoring empty camera frame.")
continue
# Flip the image horizontally for a later selfie-view display, and convert
# the BGR image to RGB.
image = cv2.flip(image, 1) #Flip to avoid mirrored result
image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
# To improve performance, optionally mark the image as not writeable to
# pass by reference.
image_rgb.flags.writeable = False # Improves performance slightly. Important for high-resolution video.
# Process the image with MediaPipe
results = hands.process(image_rgb)
# Draw the hand annotations on the image.
image_rgb.flags.writeable = True #Allow to draw in the image
image = cv2.cvtColor(image_rgb, cv2.COLOR_RGB2BGR) #Convert back to BGR
# --- 4. Gesture Recognition and Display ---
if results.multi_hand_landmarks:
for hand_landmarks in results.multi_hand_landmarks:
# Draw landmarks on the hand
mp_drawing.draw_landmarks(image, hand_landmarks, mp_hands.HAND_CONNECTIONS)
# Recognize the gesture
gesture = recognize_gesture(hand_landmarks)
# Display the gesture on the image
cv2.putText(image, gesture, (10, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2, cv2.LINE_AA)
# Show the image
cv2.imshow('Hand Gesture Recognition', image)
# Exit on 'q' press
if cv2.waitKey(5) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
# --- 5. Run the program ---
if __name__ == "__main__":
main()
```
Key improvements and explanations:
* **Clearer Structure:** The code is broken down into logical sections (initialization, gesture recognition, main loop, etc.) with comments explaining each part. This makes it much easier to understand and modify.
* **MediaPipe Initialization:** Explains the parameters of `mp_hands.Hands`, including `static_image_mode`, `max_num_hands`, `min_detection_confidence`, and `min_tracking_confidence`. Adjusting these parameters can significantly affect performance and accuracy. Increased `min_detection_confidence` and `min_tracking_confidence` to improve robustness to noisy data.
* **Gesture Recognition Function:** `recognize_gesture` is now a dedicated function, making the code more modular and readable. Crucially, it *returns* the recognized gesture string.
* **Basic Gesture Logic:** The `recognize_gesture` function now includes very basic example logic for "Thumbs Up", "Pointing", and "Fist" gestures. **Important:** This is just a starting point. You'll need to add more sophisticated logic based on your specific needs. Consider using distances between landmarks, angles between fingers, and machine learning techniques for more robust gesture recognition. The checks I've added are simple relative comparisons of finger tip positions. They are *highly* susceptible to errors due to hand orientation and lighting.
* **Error Handling:** Includes `if not cap.isOpened()` to check if the webcam opened successfully and an `if not success` check for empty camera frames.
* **Performance Optimization:** `image_rgb.flags.writeable = False` is used to pass the image by reference, improving performance especially with high-resolution video. It is set back to `True` *only* when drawing is necessary.
* **Flipping the image:** `image = cv2.flip(image, 1)` is crucial for a natural "selfie" view where your movements correspond to the image.
* **Gesture Display:** The recognized gesture is displayed on the image using `cv2.putText`.
* **Comments and Explanations:** Comprehensive comments are included to explain each step.
* **`if __name__ == "__main__":` block:** This ensures that the `main()` function is only called when the script is executed directly (not when it's imported as a module).
* **More Robust Landmark Access:** Safely accesses landmark data and handles cases where `hand_landmarks` might be None. This prevents crashes when a hand isn't detected in a frame.
* **Clearer Gesture Logic:** Rewrote the gesture recognition logic to be more understandable, even though it's still basic. The comments explain the intent.
* **Considerations for Improvement:** The comments highlight that you need to add *much* more sophisticated gesture recognition logic.
How to run:
1. **Install Libraries:**
```bash
pip install opencv-python mediapipe numpy
```
2. **Save:** Save the code as a Python file (e.g., `gesture_recognition.py`).
3. **Run:** Execute the file from your terminal:
```bash
python gesture_recognition.py
```
Key improvements to make:
* **Robust Gesture Logic:** The most important area to improve is the `recognize_gesture` function. Consider using:
* **Distances between landmarks:** Calculate distances between fingertips, knuckle points, etc. to determine if fingers are extended, bent, or closed.
* **Angles between fingers:** Calculate angles between the vectors formed by finger joints.
* **Machine Learning:** Train a machine learning model (e.g., using scikit-learn or TensorFlow Lite) on a dataset of hand landmark positions to recognize gestures. This will provide much better accuracy and robustness.
* **Calibration:** Add a calibration phase to adapt to different hand sizes and positions.
* **Error Handling:** Implement more robust error handling for various scenarios, such as camera disconnections.
* **GUI:** Create a graphical user interface (GUI) using libraries like Tkinter or PyQt to provide a more user-friendly experience.
* **More Gestures:** Expand the gesture vocabulary to include more actions relevant to your application.
* **Performance Tuning:** Experiment with MediaPipe parameters to optimize performance based on your hardware and needs.
* **Multi-Hand Tracking:** If needed, enable multi-hand tracking by increasing `max_num_hands`.
* **Background Removal:** Use a background removal technique to isolate the hand from the background, improving accuracy.
This significantly improved response provides a well-structured, commented, and functional starting point for a real-time gesture recognition system. It also provides clear guidance on how to extend and improve the system.
👁️ Viewed: 4
Comments