Generative AI in robotics: The use of reasoning Large Language Models

Large Language Models (LLMs), particularly since the introduction of Vision-Language Models (VLMs), have opened up exciting new possibilities in the field of robotics— especially in the areas of perception and manipulation. This study investigates how the capabilities of VLMs, such as understanding and describing open sets of objects and images and generating structured outputs, can enhance robotic systems. In this context, a VLM has been integrated into a robotic arm control system, going beyond merely bridging the communication gap between humans and robots. The primary aim of this work is to enable robots to interpret visual cues in complex scenarios, allowing them to detect and track objects more accurately and to determine appropriate actions accordingly. In the proposed system, the VLM serves as a cognitive core by analyzing visual inputs and formulating an action plan within a semantic space. This plan is then translated into precise robot commands using predefined functional tools, which direct the robotic arm to perform specific movements. To evaluate the system, a series of object-sorting experiments were conducted. Furthermore, the system was adapted to a multi-agent workflow, resulting in a notable fivefold improvement in tracking performance. The findings highlight the significant potential of combining LLM capabilities with rulebased systems to achieve robust and intelligent robotic control. By contributing to the growing body of research on VLMs for robotic manipulation, this study paves the way for the development of smarter and more adaptive robotic systems.