An Environment-Aware Robot Arm Platform Using Low-Cost Sensors and Deep Learning

Abstract

The main aim of the thesis is to use low-cost hardware components to create an environment-aware robot arm system capable of analysing its workspace using vision sensors and ensure collision-free operation. The objective is not only to build a physical system, but the focus is to develop algorithms to understand the environment by using image and depth information from multiple cameras and allow the robot to calculate and execute safe movement trajectories in dynamically changing environment. In this thesis, we have developed an automatic camera-to-robot calibration system to perform camera internal and Eye-to-Hand calibrations. Furthermore, a visually based reactive-reflexive robot behaviour method allows the robot to find safe trajectories throughout a dynamically changing environment with a capability of quickly reacting to unexpectedly appearing close obstacles. The robot was also used to charge electric vehicles by using vision-based guidance autonomously. Eventually, the work evolved to using deep learning approaches to recognise the robot and estimate its 3D position with a simple 2D colour image used as an input. Multi-objective convolutional neural networks and transfer learning techniques allowed to expand the method to more robot types when having a limited amount of training data. The thesis concludes that commercially-available hardware can be integrated using advanced algorithms to precisely model the workspace of the robot and allow path planning and quick reactions to unexpected situations. Many 3D cameras as well as robots, including Universal Robots, Kuka iiwa LBR and Franka Emika Panda, were used to perform all the experiments in real-world conditions.