Towards improving the robustness and generalizability of camera-based bird's eye view segmentation models

Bird's Eye View (BEV) representations are essential for many autonomous driving tasks, as they provide a structured understanding of the 3D spatial layout of the environment. Extracting BEV from camera images offers a cost-effective and scalable alternative to LiDAR. However, vision-only BEV models come with their own challenges: they often lack robustness when encountered with corrupted inputs (e.g., noisy images or camera failures) and struggle to generalize due to insufficient coverage of existing datasets. Both robustness and generalization are important for making autonomous driving systems safe and reliable for their deployment in real-world. In this thesis, we address both of these challenges. First, to improve the robustness of BEV perception, we propose utilizing the strong and generalizable features of large vision foundation models by integrating them into BEV perception using a parameter-efficient adaptation technique. Our experiments show improved robustness under various input corruptions, with additional gains observed by scaling the model size and input resolution. We also demonstrate faster convergence and reduced parameter requirements, highlighting the efficiency of our approach. Second, to tackle the issue of limited dataset coverage and the lack of flexible input control in multiview driving scene generation, we propose a guided layout generation pipeline that targets moderately challenging and underrepresented scene configurations. By analyzing the distribution of real-world driving scenes, we guide a generative model to synthesize scenes in these specific regions of the distribution. While augmenting the training set of BEV perception models with these guided samples does not yet yield performance gains, the generated layouts closely follow real-world configurations and mostly fall within the target region. This suggests promising potential for targeted synthetic data augmentation as a future direction for improving the generalization of BEV perception models.