Performance evaluation of llm based chatbots with E2e method:LLama-8b,LLama-7b,Gemma-7b and mistral-7b

This study investigates the performance of large language models (LLMs) within the context of customer support chatbots by employing an end-to-end (E2E) evaluation framework. Specifically, it compares three prominent open-source models (Gemma-7B, Mistral-7B, Llama-7B and Llama-8B) based on their ability to comprehend and respond to user queries in a meaningful and accurate manner. The chatbot application under review was designed to provide assistance on an educational content platform and was tested using over 3000 curated question-answer pairs. The evaluation combines both semantic and lexical metrics, using cosine similarity to measure the alignment of model responses with expert-written answers, and ROUGE metrics to assess word-level accuracy. Additionally, the study incorporates prompt engineering techniques and analyses how models handle random or off-topic inputs, providing a comprehensive view of their reliability and contextual sensitivity. Results indicate that Gemma-7B and Llama-8B performs most consistently across all metrics, while Mistral-7B offers balanced outputs with occasional variance. Llama-7B, although structurally robust, struggled to deliver semantically aligned and contextually appropriate responses. Overall, the findings highlight the practical implications of model selection for real-world chatbot deployments and demonstrate the importance of multi-dimensional evaluation methods when assessing LLM performance in customer interaction settings.