Loki Logging update

[nuwangeek]

This pull request introduces significant improvements to logging infrastructure and configuration, replacing the loguru logger with a custom, asynchronous Loki logger across multiple modules. It also updates Docker and package management to support the new logging system, and enhances Grafana dashboard configuration for better observability.

Key changes include:

Logging Infrastructure Overhaul

• Replaced all loguru imports and usage with a new LokiLogger class, which sends logs asynchronously to Loki and provides compatibility methods for seamless migration. This affects files such as src/api_tool_indexer/main_indexer.py, src/api_tool_indexer/qdrant_manager.py, src/contextual_retrieval/bm25_search.py, and src/contextual_retrieval/config.py. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]
• Introduced a robust, thread-safe, and non-blocking LokiLogger implementation in grafana-configs/loki_logger.py, supporting loguru compatibility methods and asynchronous log delivery to avoid blocking the main application. [1] [2] [3] [4] [5]

Docker and Environment Configuration

• Updated docker-compose.yml and docker-compose-ec2.yml to mount the new loki_logger.py and ensure it is available in all relevant containers, and expanded the PYTHONPATH to include /app/src for proper module resolution. [1] [2] [3] [4] [5] [6]

Package and Dependency Management

• Removed the loguru dependency from the API tool indexer install script and replaced it with the requests package, which is required by the new logger.

Grafana Dashboard and Observability

• Enhanced the Grafana dashboard configuration by updating tags to better reflect the current stack and improving the Loki service name query for log filtering. [1] [2]

Code Cleanup

• Removed the now-obsolete generate_presigned_url.py script, as it is no longer needed in the codebase.

These changes collectively modernize the logging stack, improve reliability and observability, and streamline configuration for local and cloud deployments.