Select LLM and Develop System Prompts for Output Generation

This issue focuses on developing strategies for crafting effective system prompts by optimally combining user input, chat history, and retrieved context chunks. Additionally, this issue will involve the selection of the most suitable language model for generating high-quality answers.

System Prompt Engineering: Design effective system prompts that leverage both the user query, retrieved context, and insights from chat history to guide the language model in generating informative, accurate, and user-tailored responses.

LLM Selection: Choose a suitable open-source large language model (LLM) (e.g., Phi 4, Granite, Nous Hermes, Mistral, Falcon, DeepSeek-R1, Sailor2, etc.) based on performance, efficiency, compute requirements, etc.