![\"mediapipe\"](\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2023\/07\/mediapipe.webp\")
<\/a><\/p>\nPyautogui<\/h3>\n
PyAutoGUI is a Python library that allows users to automate repetitive tasks on their computer such as keyboard and mouse movements. It’s easy to use and compatible with different platforms. It can help improve productivity, but should be used responsibly.<\/p>\n
Gesture Recognition<\/h3>\n
Gesture recognition technology enables computers to detect and respond to body or hand movements, providing an intuitive and enjoyable way to draw digitally. OpenCV is an open-source library that can be used to recognize gestures in digital drawing by capturing images of the user’s hands and analyzing them using OpenCV functions. This technology can identify specific hand gestures, such as swipes or circles, and convert them into digital drawing commands.<\/p>\n
Prerequisites for Gesture Controlling Game Using Python Opencv<\/h3>\n
It is important to have a solid understanding of the Python programming language and the OpenCV library. Apart from this, you should have the following system requirements.<\/p>\n
- \n
- Python 3.7 (64-bit) and above<\/li>\n
- Any Python editor (VS code, Pycharm)<\/li>\n<\/ul>\n
Download Python Opencv Gesture Controlling Game Project<\/h3>\n
Please download the source code of Python Opencv Gesture Controlling Game Project: Python<\/strong> Opencv Gesture Controlling Game Project Code<\/strong><\/a><\/p>\n
Installation<\/h3>\n
Open Windows cmd as administrator<\/p>\n
1. To install the opencv library, run the command from the cmd.<\/p>\n
pip install opencv-python\r\n<\/pre>\n
2. To install the mediapipe library, run the command from the cmd.<\/p>\n
pip install mediapipe\r\n<\/pre>\n
3. To install the pyautogui library, run the command from the cmd.<\/p>\n
pip install pyautogui \r\n<\/pre>\n
Let\u2019s Implement<\/h3>\n
To implement this follow the below steps.<\/p>\n
1. We need to import some libraries that will be used in our implementation.<\/p>\n
import cv2\r\nimport mediapipe as mp\r\nfrom tensorflow.keras.models import load_model\r\nimport pyautogui\r\n<\/pre>\n
2. The code uses the MediaPipe library in Python to detect and draw one hand in an image or video. It only detects hands with a confidence level of 70% or higher and uses MediaPipe’s drawing module to draw the hand landmarks on the image.<\/p>\n
mpHands = mp.solutions.hands\r\nhands = mpHands.Hands(max_num_hands=1, min_detection_confidence=0.7)\r\nmpDraw = mp.solutions.drawing_utils\r\n<\/pre>\n
3. It opens the integrated camera.<\/p>\n
cap = cv2.VideoCapture(0)\r\n<\/pre>\n
4. Start the while loop.<\/p>\n
while True:\r\n<\/pre>\n
5. cap.read() function returns two values, the first is stored in \u2018ret\u2019, which is a boolean value. The function cap.read() reads the frame and returns two values: ret (a boolean indicating if the frame was successfully read) and frame (an array of pixel values in the captured frame). These values can be used to process or display the image<\/p>\n
ret, frame = cap.read()\r\n<\/pre>\n
6. This code calculates the dimensions of an input frame and stores the values in the variables x, y, and c. Specifically, x represents the height of the frame, y represents the width of the frame, and c represents the number of color channels in the frame.<\/p>\n
x, y, c = frame.shape\r\n<\/pre>\n
7. This code flips and mirrors the input frame horizontally and changes its color space to RGB. It saves the modified frame in a variable called an image, which can be used for further processing.<\/p>\n
frame = cv2.flip(frame, 1)\r\nframergb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)\r\n<\/pre>\n
8. In this code, a hand detection model called MediaPipe is used to detect hands in the input frame. The detected results are stored in a variable called “result”. The second line of code initializes a variable called “num_fingers” to 0, which can be used to count the number of fingers detected in a hand gesture.<\/p>\n
result = hands.process(framergb)\r\nnum_fingers = 0\r\n<\/pre>\n
9. The if statement checks whether the “result” variable contains any detected hand landmarks. If it does, the code creates an empty list called “landmarks” to store the coordinates of these landmarks.<\/p>\n
if result.multi_hand_landmarks:\r\n landmarks = []\r\n<\/pre>\n
10. This code starts a loop that goes through each hand detected in the input frame by MediaPipe’s hand detection model.<\/p>\n
for handslms in result.multi_hand_landmarks:\r\n<\/pre>\n
11. The code calculates the exact position of each detected landmark point by multiplying its relative position with the width and height of the input frame, and stores the result in a list called “landmarks”. The int() function is used to convert the floating-point values to integers to get whole numbers.<\/p>\n
lmx = int(lm.x * x)\r\nlmy = int(lm.y * y)\r\nlandmarks.append([lmx, lmy])\r\n<\/pre>\n
12. The following code uses the mpDraw module to draw the hand landmarks and connections on the input frame. It takes the landmarks detected by the MediaPipe hand detection model and connects them with lines to create the shape of the hand. The resulting image displays the detected hand with dots representing the landmarks and lines connecting them.<\/p>\n
mpDraw.draw_landmarks(frame, handslms, mpHands.HAND_CONNECTIONS)\r\n<\/pre>\n
13. This code checks if any hand landmarks were detected by checking the length of the “landmarks” list. If landmarks are present, it checks the position of specific landmarks to determine the number of extended fingers and increments the “num_fingers” variable accordingly.<\/p>\n
if len(landmarks) > 0:\r\n if landmarks[4][0] < landmarks[3][0]:\r\n num_fingers += 1\r\n if landmarks[8][1] < landmarks[6][1]:\r\n num_fingers += 1\r\n if landmarks[12][1] < landmarks[10][1]:\r\n num_fingers += 1\r\n if landmarks[16][1] < landmarks[14][1]:\r\n num_fingers += 1\r\n if landmarks[20][1] < landmarks[18][1]:\r\n num_fingers += 1\r\n<\/pre>\n
14. This code block checks if the user is holding up one finger to indicate “Accelerate”. It displays the text “Accelerate” on the screen and triggers a series of keyboard commands using the pyautogui library to simulate pressing the “up” arrow key and releasing the “down” and “right” arrow keys<\/p>\n
if num_fingers == 1:\r\n cv2.putText(frame, \"Accelerate\", (10, 50), cv2.FONT_HERSHEY_SIMPLEX, \r\n 1, (0,0,255), 2, cv2.LINE_AA)\r\n pyautogui.keyUp(\"down\")\r\n pyautogui.keyUp(\"right\")\r\n pyautogui.keyDown(\"up\")\r\n pyautogui.keyUp(\"left\")\r\n<\/pre>\n
15. This code block corresponds to a break action in a driving game. It displays the “Break” text on the input frame and simulates the pressing of the “down” arrow key and releasing of the “up”, “right”, and “left” arrow keys using the PyAutoGUI library when two fingers are extended.<\/p>\n
elif num_fingers == 2:\r\n cv2.putText(frame, \"Break\", (10, 50), cv2.FONT_HERSHEY_SIMPLEX, \r\n 1, (0,0,255), 2, cv2.LINE_AA)\r\n pyautogui.keyUp(\"up\")\r\n pyautogui.keyDown(\"down\")\r\n pyautogui.keyUp(\"right\")\r\n pyautogui.keyUp(\"left\")\r\n<\/pre>\n
16. If three fingers are detected, the code sets the gesture to “Right,” indicating that the user wants to move to the right. It then sends keyboard inputs to the system to simulate the right arrow key press.<\/p>\n
elif num_fingers == 3:\r\n cv2.putText(frame, \"Right\", (10, 50), cv2.FONT_HERSHEY_SIMPLEX, \r\n 1, (0,0,255), 2, cv2.LINE_AA)\r\n pyautogui.keyUp(\"up\")\r\n pyautogui.keyUp(\"down\")\r\n pyautogui.keyDown(\"right\")\r\n pyautogui.keyUp(\"left\")\r\n<\/pre>\n
17. This code block executes if the number of extended fingers is 4. It displays the text “Left” on the input frame and simulates the pressing of the “left” arrow key and releasing of the “up”, “down”, and “right” arrow keys using the PyAutoGUI library. This code likely corresponds to a left-turn action in a driving game.<\/p>\n
celif num_fingers == 4:\r\n cv2.putText(frame, \"Left\", (10, 50), cv2.FONT_HERSHEY_SIMPLEX, \r\n 1, (0,0,255), 2, cv2.LINE_AA)\r\n pyautogui.keyUp(\"up\")\r\n pyautogui.keyUp(\"down\")\r\n pyautogui.keyUp(\"right\")\r\n pyautogui.keyDown(\"left\")\r\n<\/pre>\n
18. This code block executes if the number of extended fingers is either 0 or more than 5. It displays the text “Nothing” on the input frame and releases all the arrow keys using the PyAutoGUI library, indicating that no action should be taken.<\/p>\n
elif num_fingers == 5:\r\n cv2.putText(frame, \"Nothing\", (10, 50), cv2.FONT_HERSHEY_SIMPLEX, \r\n 1, (0,0,255), 2, cv2.LINE_AA)\r\n pyautogui.keyUp(\"up\")\r\n pyautogui.keyUp(\"down\")\r\n pyautogui.keyUp(\"right\")\r\n pyautogui.keyUp(\"left\")\r\n else:\r\n cv2.putText(frame, \"Nothing\", (10, 50), cv2.FONT_HERSHEY_SIMPLEX, \r\n 1, (0,0,255), 2, cv2.LINE_AA)\r\n pyautogui.keyUp(\"up\")\r\n pyautogui.keyUp(\"down\")\r\n pyautogui.keyUp(\"right\")\r\n pyautogui.keyUp(\"left\")\r\n<\/pre>\n
Note:-<\/strong> step 11-18 must be written under step 10th for loop<\/p>\n
19. This code displays the output frame with landmarks and text using the imshow method of the cv2 library. It also waits for the user to press the ‘q’ key to exit the program using the waitKey method with a parameter of 1.<\/p>\n
cv2.imshow(\"TechVidvan\", frame) \r\n\r\n if cv2.waitKey(1) == ord('q'):\r\n break\r\n<\/pre>\nNote:-<\/strong> Steps 5-19 must be written under the while loop.<\/p>\n
20.\u00a0Icap.release() frees the resources used to capture the video or camera stream, allowing them to be used by other applications. cv2.destroyAllWindows() closes all OpenCV windows, freeing system resources and ensuring the program ends cleanly.<\/p>\n
cap.release()\r\ncv2.destroyAllWindows()\r\n<\/pre>\n
Python OpenCV Gesture Controlling Game Output<\/h3>\n
![\"gesture](\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2023\/07\/gesture-controlling-game-output.webp\")
<\/a><\/p>\n![\"gesture](\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2023\/07\/gesture-controlling-game-project-output.webp\")
<\/a><\/h3>\n![\"python](\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2023\/07\/python-opencv-gesture-controlling-game-output.webp\")
<\/a><\/h3>\n![\"opencv](\"https:\/\/techvidvan.com\/tutorials\/wp-content\/uploads\/sites\/2\/2023\/07\/opencv-gesture-controlling-game-output.webp\")
<\/a><\/h3>\nVideo Output<\/h3>\n