If 2023 is the year of LLMs, then 2024 will be the year for LLM Applications. As LLMs continue to integrate into various applications ranging from chatbots and search engines to creative writing aids, the need to monitor and comprehend their behaviors intensifies.
Observability plays a crucial role in this context. It involves the systematic collection and analysis of data to enhance LLM performance, identify and correct biases, troubleshoot issues, and ensure AI systems are both reliable and trustworthy.
In this discussion, we will explore the concept of LLM observability in depth, focusing on how OpenTelemetry can fit into the world of LLM observability . Additionally, we will talk about challenges around modeling of prompts, completions, events, semantic conventions, and basically our path with the llm-sem-conv working group.
