Conception of a robotic arm teleoperation system
Project context
For 3 semesters, we have been asked to work by group on a project. We had to go from the definition of the state of art to the creation of a functional prototype.
The project that we chose consisted in the creation of a camera-based system that analyzes user gestures to control a robotic arm remotely. Furthermore, the user received visual feedback in a VR headset, which improves the user immersion.
My technical role was to design the distant communication between a user and the robotic arm.
Project details
I was in charge of the interfacing between the different devices used and of developing optimized programs to handle large data flows over long-distance transmissions. I decided to base the video stream transmission from the cameras to the VR headset on the UDP protocol to improve the communication fluidity. For the control of the robotic arm, I used the TCP protocol to ensure data integrity, for security reasons. The video stream that was sent to the VR headset was very heavy, so I optimized data transmission using multithreading and by dividing each image into smaller chunks.
During this project, I briefly had to use the C# programming language to develop the VR virtual environment. I also learned more about VPN technologies and how to use them to establish a connection through the internet. This project was an occasion for me to learn more about all the non-technical aspects of a project, like creating a business plan, a state of art, a Gantt chart planification, concessions due to time and budget constraints, …
We were able to successfully create a functional prototype by combining our diverse skills, such as programming, automation, systems, network, electronics, etc. We even had the occasion to present our work in a promotional video from Niryo, the manufacturer of the robotic arm we mainly used.
Project gallery
Test of data transmission of two video streams to a VR headset
Structure used to observe user gestures using two cameras
Niryo Ned2, main robotic arm used
Secondary robotic arm designed and used for testing purposes