Automated generation of mobile UI layout files via a custom GUI element detection model trained with YOLOv5

In mobile application development, building a consistent user interface (UI) might be a costly and time-consuming process. This is especially the case if an organization has a separate team for each mobile platform such as iOS and Android. In this regard, one of the most parts of the UI design task is creating a graphical user interface (GUI). Accordingly, this study aims to employ the YOLOv5 to create a custom object detection model that recognizes GUI elements in a given UI image. In order to benchmark the newly trained YOLOv5 GUI element detection model, existing work from the literature and their datasets are considered and used for comparison purposes. Accordingly, this study makes use of 450 UI samples of the VINS dataset for testing, a similar amount for validation, and the rest for model training. Then the findings of this work are compared with another study that has used the single shot detector (SSD) algorithm and VINS dataset to train, validate and test its model, which showed that the proposed algorithm outperformed SSD's mean average precision (mAP) by 15.69%. Moreover, this study proposes a framework that utilizes this custom GUI element detection model to generate iOS and Android UI layout files of a given mobile UI design. Thus, this framework uses machine learning algorithms to explore and extract the attributes of the detected GUI elements. Then, the framework engages in these attributes to generate UI layout files for iOS and Android platforms. To evaluate this framework, 5 test images from the VINS dataset were randomly selected and given as input for the framework. The framework generated iOS and Android UI layout files for each test image. Lastly, the preview of the UI layout files has been captured as screenshots and compared with its input test image using image similarity metrics. The test results revealed that the proposed framework has obtained an average structural similarity (SSIM) score of 0.731 and 0.703 on generated iOS and Android UI designs respectively. Moreover, on average, the framework has yielded UI designs with pixel-based similarity of 83.41% for iOS and 81.29% for Android platforms on given test images.