diff --git a/.claude/cicd_testing_strategy.md b/.claude/cicd_testing_strategy.md deleted file mode 100644 index 0ff4329d..00000000 --- a/.claude/cicd_testing_strategy.md +++ /dev/null @@ -1,118 +0,0 @@ -# CI/CD Testing Strategy for FLaME Research Project - -## Core Philosophy -For a PhD research project, CI/CD should focus on: -1. **Structural integrity** - Ensure code organization remains sound -2. **Core logic validation** - Test algorithms and data processing -3. **Interface contracts** - Verify APIs and module interfaces work -4. **Development velocity** - Fast feedback, not slow comprehensive tests - -## Identified Issues -The current failures are due to: -- Missing HuggingFace token in CI environment -- Ollama integration tests expecting local server -- Test isolation problems with dynamic module imports -- Authentication tests that exit with code 1 -- Hardcoded localhost configurations - -## Proposed CI/CD Architecture - -### 1. Three-Tier Testing Strategy - -**Tier 1: Lightning Tests (< 30 seconds)** -- Syntax validation -- Import checks -- Config parsing -- Core utilities - -**Tier 2: Core Tests (< 2 minutes)** -- Unit tests with mocked externals -- Prompt generation -- Data processing logic -- Task registry validation - -**Tier 3: Integration Tests (< 5 minutes)** -- Module discovery tests -- Multi-task orchestration -- Output format validation - -### 2. Environment Configuration - -Create separate configs: -- `configs/ci.yaml` - Mock models, no external deps -- `configs/test.yaml` - Minimal settings for fast tests -- Use environment detection: `CI=true` to auto-select - -### 3. Test Markers System - -Properly categorize tests: -```python -@pytest.mark.unit # No external deps -@pytest.mark.integration # Mocked external deps -@pytest.mark.smoke # Critical path only -@pytest.mark.slow # Skip in CI -@pytest.mark.requires_api # Skip without keys -``` - -### 4. Workflow Structure - -**PR Workflow (on every push):** -1. Syntax/Import Check (30s) -2. Unit Tests (1m) -3. Smoke Tests (30s) - -**Merge Workflow (on main branch):** -1. Full test suite -2. Code coverage report -3. Performance benchmarks - -**Nightly Workflow:** -1. Full integration tests -2. Dependency updates check -3. Security scanning - -### 5. Key Improvements Needed - -1. **Fix Authentication Tests** - - Don't exit(1) in tests - - Use pytest.skip for missing deps - - Mock HuggingFace login - -2. **Handle Ollama Tests** - - Mark as `@pytest.mark.requires_ollama` - - Skip in CI or use mock responses - - Document local testing setup - -3. **Module Discovery Tests** - - Run in isolated subprocess - - Clear module cache between tests - - Use pytest-xdist for parallel execution - -4. **Environment Variables** - ```yaml - env: - PYTEST_RUNNING: "1" - CI: "true" - FLAME_CONFIG: "ci" - HUGGINGFACEHUB_API_TOKEN: "mock-token-for-ci" - ``` - -### 6. Practical Implementation Steps - -1. Create CI-specific fixtures that always mock external services -2. Add `--ci` flag to pytest that forces all mocks -3. Split workflows into fast-feedback and comprehensive -4. Use GitHub Actions matrix for testing multiple Python versions -5. Cache dependencies aggressively -6. Add status badges to README - -## Benefits of This Approach - -1. **Fast Feedback**: Most tests complete in under 2 minutes -2. **Low Cost**: No API calls in CI -3. **Reliable**: No flaky external dependencies -4. **Informative**: Clear test categories and failure messages -5. **Maintainable**: Easy to add new tests in correct category -6. **Research-Friendly**: Focus on algorithm correctness, not production concerns - -This strategy balances the needs of academic research (correctness, reproducibility) with practical development concerns (speed, reliability). It's not overly complex but provides good coverage of critical functionality. \ No newline at end of file diff --git a/.claude/liteLLM/Batching Completion() liteLLM.md b/.claude/liteLLM/Batching Completion() liteLLM.md deleted file mode 100644 index b5dd8f4a..00000000 --- a/.claude/liteLLM/Batching Completion() liteLLM.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -created: 2025-05-26T16:02:00 (UTC -04:00) -tags: [] -source: https://docs.litellm.ai/docs/routing#max-parallel-requests-async -author: ---- - -# Batching Completion() | liteLLM - -> ## Excerpt -> LiteLLM allows you to: - ---- -LiteLLM allows you to: - -- Send many completion calls to 1 model -- Send 1 completion call to many models: Return Fastest Response -- Send 1 completion call to many models: Return All Responses - -## Send multiple completion calls to 1 model[](https://docs.litellm.ai/docs/routing#send-multiple-completion-calls-to-1-model "Direct link to Send multiple completion calls to 1 model") - -In the batch\_completion method, you provide a list of `messages` where each sub-list of messages is passed to `litellm.completion()`, allowing you to process multiple prompts efficiently in a single API call. - -[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/BerriAI/litellm/blob/main/cookbook/LiteLLM_batch_completion.ipynb) - -### Example Code[](https://docs.litellm.ai/docs/routing#example-code "Direct link to Example Code") - -``` -import litellmimport osfrom litellm import batch_completionos.environ['ANTHROPIC_API_KEY'] = ""responses = batch_completion( model="claude-2", messages = [ [ { "role": "user", "content": "good morning? " } ], [ { "role": "user", "content": "what's the time? " } ] ]) -``` - -## Send 1 completion call to many models: Return Fastest Response[](https://docs.litellm.ai/docs/routing#send-1-completion-call-to-many-models-return-fastest-response "Direct link to Send 1 completion call to many models: Return Fastest Response") - -This makes parallel calls to the specified `models` and returns the first response - -Use this to reduce latency - -- SDK -- PROXY - -### Example Code[](https://docs.litellm.ai/docs/routing#example-code-1 "Direct link to Example Code") - -``` -import litellmimport osfrom litellm import batch_completion_modelsos.environ['ANTHROPIC_API_KEY'] = ""os.environ['OPENAI_API_KEY'] = ""os.environ['COHERE_API_KEY'] = ""response = batch_completion_models( models=["gpt-3.5-turbo", "claude-instant-1.2", "command-nightly"], messages=[{"role": "user", "content": "Hey, how's it going"}])print(result) -``` - -### Output[](https://docs.litellm.ai/docs/routing#output "Direct link to Output") - -Returns the first response in OpenAI format. Cancels other LLM API calls. - -``` -{ "object": "chat.completion", "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": " I'm doing well, thanks for asking! I'm an AI assistant created by Anthropic to be helpful, harmless, and honest.", "role": "assistant", "logprobs": null } } ], "id": "chatcmpl-23273eed-e351-41be-a492-bafcf5cf3274", "created": 1695154628.2076092, "model": "command-nightly", "usage": { "prompt_tokens": 6, "completion_tokens": 14, "total_tokens": 20 }} -``` - -## Send 1 completion call to many models: Return All Responses[](https://docs.litellm.ai/docs/routing#send-1-completion-call-to-many-models-return-all-responses "Direct link to Send 1 completion call to many models: Return All Responses") - -This makes parallel calls to the specified models and returns all responses - -Use this to process requests concurrently and get responses from multiple models. - -### Example Code[](https://docs.litellm.ai/docs/routing#example-code-2 "Direct link to Example Code") - -``` -import litellmimport osfrom litellm import batch_completion_models_all_responsesos.environ['ANTHROPIC_API_KEY'] = ""os.environ['OPENAI_API_KEY'] = ""os.environ['COHERE_API_KEY'] = ""responses = batch_completion_models_all_responses( models=["gpt-3.5-turbo", "claude-instant-1.2", "command-nightly"], messages=[{"role": "user", "content": "Hey, how's it going"}])print(responses) -``` - -### Output[](https://docs.litellm.ai/docs/routing#output-1 "Direct link to Output") - -``` -[ JSON: { "object": "chat.completion", "choices": [ { "finish_reason": "stop_sequence", "index": 0, "message": { "content": " It's going well, thank you for asking! How about you?", "role": "assistant", "logprobs": null } } ], "id": "chatcmpl-e673ec8e-4e8f-4c9e-bf26-bf9fa7ee52b9", "created": 1695222060.917964, "model": "claude-instant-1.2", "usage": { "prompt_tokens": 14, "completion_tokens": 9, "total_tokens": 23 }}, JSON: { "object": "chat.completion", "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": " It's going well, thank you for asking! How about you?", "role": "assistant", "logprobs": null } } ], "id": "chatcmpl-ab6c5bd3-b5d9-4711-9697-e28d9fb8a53c", "created": 1695222061.0445492, "model": "command-nightly", "usage": { "prompt_tokens": 6, "completion_tokens": 14, "total_tokens": 20 }}, JSON: { "id": "chatcmpl-80szFnKHzCxObW0RqCMw1hWW1Icrq", "object": "chat.completion", "created": 1695222061, "model": "gpt-3.5-turbo-0613", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "Hello! I'm an AI language model, so I don't have feelings, but I'm here to assist you with any questions or tasks you might have. How can I help you today?" }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 13, "completion_tokens": 39, "total_tokens": 52 }}] -``` diff --git a/.claude/liteLLM/Reliability - Retries, Fallbacks liteLLM.md b/.claude/liteLLM/Reliability - Retries, Fallbacks liteLLM.md deleted file mode 100644 index 529cf910..00000000 --- a/.claude/liteLLM/Reliability - Retries, Fallbacks liteLLM.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -created: 2025-05-26T16:02:08 (UTC -04:00) -tags: [] -source: https://docs.litellm.ai/docs/routing#max-parallel-requests-async -author: ---- - -# Reliability - Retries, Fallbacks | liteLLM - -> ## Excerpt -> LiteLLM helps prevent failed requests in 2 ways: - ---- -LiteLLM helps prevent failed requests in 2 ways: - -- Retries -- Fallbacks: Context Window + General - -## Helper utils[](https://docs.litellm.ai/docs/routing#helper-utils "Direct link to Helper utils") - -LiteLLM supports the following functions for reliability: - -- `litellm.longer_context_model_fallback_dict`: Dictionary which has a mapping for those models which have larger equivalents -- `num_retries`: use tenacity retries -- `completion()` with fallbacks: switch between models/keys/api bases in case of errors. - -## Retry failed requests[](https://docs.litellm.ai/docs/routing#retry-failed-requests "Direct link to Retry failed requests") - -Call it in completion like this `completion(..num_retries=2)`. - -Here's a quick look at how you can use it: - -``` -from litellm import completionuser_message = "Hello, whats the weather in San Francisco??"messages = [{"content": user_message, "role": "user"}]# normal call response = completion( model="gpt-3.5-turbo", messages=messages, num_retries=2 ) -``` - -## Fallbacks (SDK)[](https://docs.litellm.ai/docs/routing#fallbacks-sdk "Direct link to Fallbacks (SDK)") - -### Context Window Fallbacks (SDK)[](https://docs.litellm.ai/docs/routing#context-window-fallbacks-sdk "Direct link to Context Window Fallbacks (SDK)") - -``` -from litellm import completionfallback_dict = {"gpt-3.5-turbo": "gpt-3.5-turbo-16k"}messages = [{"content": "how does a court case get to the Supreme Court?" * 500, "role": "user"}]completion(model="gpt-3.5-turbo", messages=messages, context_window_fallback_dict=fallback_dict) -``` - -### Fallbacks - Switch Models/API Keys/API Bases (SDK)[](https://docs.litellm.ai/docs/routing#fallbacks---switch-modelsapi-keysapi-bases-sdk "Direct link to Fallbacks - Switch Models/API Keys/API Bases (SDK)") - -LLM APIs can be unstable, completion() with fallbacks ensures you'll always get a response from your calls - -#### Usage[](https://docs.litellm.ai/docs/routing#usage "Direct link to Usage") - -To use fallback models with `completion()`, specify a list of models in the `fallbacks` parameter. - -The `fallbacks` list should include the primary model you want to use, followed by additional models that can be used as backups in case the primary model fails to provide a response. - -#### switch models[](https://docs.litellm.ai/docs/routing#switch-models "Direct link to switch models") - -``` -response = completion(model="bad-model", messages=messages, fallbacks=["gpt-3.5-turbo" "command-nightly"]) -``` - -#### switch api keys/bases (E.g. azure deployment)[](https://docs.litellm.ai/docs/routing#switch-api-keysbases-eg-azure-deployment "Direct link to switch api keys/bases (E.g. azure deployment)") - -Switch between different keys for the same azure deployment, or use another deployment as well. - -``` -api_key="bad-key"response = completion(model="azure/gpt-4", messages=messages, api_key=api_key, fallbacks=[{"api_key": "good-key-1"}, {"api_key": "good-key-2", "api_base": "good-api-base-2"}]) -``` - -[Check out this section for implementation details](https://docs.litellm.ai/docs/routing#fallbacks-1) - -## Implementation Details (SDK)[](https://docs.litellm.ai/docs/routing#implementation-details-sdk "Direct link to Implementation Details (SDK)") - -### Fallbacks[](https://docs.litellm.ai/docs/routing#fallbacks "Direct link to Fallbacks") - -#### Output from calls[](https://docs.litellm.ai/docs/routing#output-from-calls "Direct link to Output from calls") - -``` -Completion with 'bad-model': got exception Unable to map your input to a model. Check your input - {'model': 'bad-model'completion call gpt-3.5-turbo{ "id": "chatcmpl-7qTmVRuO3m3gIBg4aTmAumV1TmQhB", "object": "chat.completion", "created": 1692741891, "model": "gpt-3.5-turbo-0613", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "I apologize, but as an AI, I do not have the capability to provide real-time weather updates. However, you can easily check the current weather in San Francisco by using a search engine or checking a weather website or app." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 16, "completion_tokens": 46, "total_tokens": 62 }} -``` - -#### How does fallbacks work[](https://docs.litellm.ai/docs/routing#how-does-fallbacks-work "Direct link to How does fallbacks work") - -When you pass `fallbacks` to `completion`, it makes the first `completion` call using the primary model specified as `model` in `completion(model=model)`. If the primary model fails or encounters an error, it automatically tries the `fallbacks` models in the specified order. This ensures a response even if the primary model is unavailable. - -#### Key components of Model Fallbacks implementation:[](https://docs.litellm.ai/docs/routing#key-components-of-model-fallbacks-implementation "Direct link to Key components of Model Fallbacks implementation:") - -- Looping through `fallbacks` -- Cool-Downs for rate-limited models - -#### Looping through `fallbacks`[](https://docs.litellm.ai/docs/routing#looping-through-fallbacks "Direct link to looping-through-fallbacks") - -Allow `45seconds` for each request. In the 45s this function tries calling the primary model set as `model`. If model fails it loops through the backup `fallbacks` models and attempts to get a response in the allocated `45s` time set here: - -``` -while response == None and time.time() - start_time < 45: for model in fallbacks: -``` - -#### Cool-Downs for rate-limited models[](https://docs.litellm.ai/docs/routing#cool-downs-for-rate-limited-models "Direct link to Cool-Downs for rate-limited models") - -If a model API call leads to an error - allow it to cooldown for `60s` - -``` -except Exception as e: print(f"got exception {e} for model {model}") rate_limited_models.add(model) model_expiration_times[model] = ( time.time() + 60 ) # cool down this selected model pass -``` - -Before making an LLM API call we check if the selected model is in `rate_limited_models`, if so skip making the API call - -``` -if ( model in rate_limited_models): # check if model is currently cooling down if ( model_expiration_times.get(model) and time.time() >= model_expiration_times[model] ): rate_limited_models.remove( model ) # check if it's been 60s of cool down and remove model else: continue # skip model -``` - -#### Full code of completion with fallbacks()[](https://docs.litellm.ai/docs/routing#full-code-of-completion-with-fallbacks "Direct link to Full code of completion with fallbacks()") - -``` - response = None rate_limited_models = set() model_expiration_times = {} start_time = time.time() fallbacks = [kwargs["model"]] + kwargs["fallbacks"] del kwargs["fallbacks"] # remove fallbacks so it's not recursive while response == None and time.time() - start_time < 45: for model in fallbacks: # loop thru all models try: if ( model in rate_limited_models ): # check if model is currently cooling down if ( model_expiration_times.get(model) and time.time() >= model_expiration_times[model] ): rate_limited_models.remove( model ) # check if it's been 60s of cool down and remove model else: continue # skip model # delete model from kwargs if it exists if kwargs.get("model"): del kwargs["model"] print("making completion call", model) response = litellm.completion(**kwargs, model=model) if response != None: return response except Exception as e: print(f"got exception {e} for model {model}") rate_limited_models.add(model) model_expiration_times[model] = ( time.time() + 60 ) # cool down this selected model pass return response -``` diff --git a/.claude/liteLLM/Router - Load Balancing liteLLM.md b/.claude/liteLLM/Router - Load Balancing liteLLM.md deleted file mode 100644 index ffa14fd2..00000000 --- a/.claude/liteLLM/Router - Load Balancing liteLLM.md +++ /dev/null @@ -1,338 +0,0 @@ ---- -created: 2025-05-26T16:00:02 (UTC -04:00) -tags: [] -source: https://docs.litellm.ai/docs/routing#max-parallel-requests-async -author: ---- - -# Router - Load Balancing | liteLLM - -> ## Excerpt -> LiteLLM manages: - ---- -LiteLLM manages: - -- Load-balance across multiple deployments (e.g. Azure/OpenAI) -- Prioritizing important requests to ensure they don't fail (i.e. Queueing) -- Basic reliability logic - cooldowns, fallbacks, timeouts and retries (fixed + exponential backoff) across multiple deployments/providers. - -In production, litellm supports using Redis as a way to track cooldown server and usage (managing tpm/rpm limits). - -## Load Balancing[](https://docs.litellm.ai/docs/routing#load-balancing "Direct link to Load Balancing") - -(s/o [@paulpierre](https://www.linkedin.com/in/paulpierre/) and [sweep proxy](https://docs.sweep.dev/blogs/openai-proxy) for their contributions to this implementation) [**See Code**](https://github.com/BerriAI/litellm/blob/main/litellm/router.py) - -### Quick Start[](https://docs.litellm.ai/docs/routing#quick-start "Direct link to Quick Start") - -Loadbalance across multiple [azure](https://docs.litellm.ai/docs/providers/azure)/[bedrock](https://docs.litellm.ai/docs/providers/bedrock)/[provider](https://docs.litellm.ai/docs/providers/) deployments. LiteLLM will handle retrying in different regions if a call fails. - -- SDK -- PROXY - -``` -from litellm import Routermodel_list = [{ # list of model deployments "model_name": "gpt-3.5-turbo", # model alias -> loadbalance between models with same `model_name` "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") }}, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") }}, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-3.5-turbo", "api_key": os.getenv("OPENAI_API_KEY"), }}, { "model_name": "gpt-4", "litellm_params": { # params for litellm completion/embedding call "model": "azure/gpt-4", "api_key": os.getenv("AZURE_API_KEY"), "api_base": os.getenv("AZURE_API_BASE"), "api_version": os.getenv("AZURE_API_VERSION"), }}, { "model_name": "gpt-4", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-4", "api_key": os.getenv("OPENAI_API_KEY"), }},]router = Router(model_list=model_list)# openai.ChatCompletion.create replacement# requests with model="gpt-3.5-turbo" will pick a deployment where model_name="gpt-3.5-turbo"response = await router.acompletion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}])print(response)# openai.ChatCompletion.create replacement# requests with model="gpt-4" will pick a deployment where model_name="gpt-4"response = await router.acompletion(model="gpt-4", messages=[{"role": "user", "content": "Hey, how's it going?"}])print(response) -``` - -### Available Endpoints[](https://docs.litellm.ai/docs/routing#available-endpoints "Direct link to Available Endpoints") - -- `router.completion()` - chat completions endpoint to call 100+ LLMs -- `router.acompletion()` - async chat completion calls -- `router.embedding()` - embedding endpoint for Azure, OpenAI, Huggingface endpoints -- `router.aembedding()` - async embeddings calls -- `router.text_completion()` - completion calls in the old OpenAI `/v1/completions` endpoint format -- `router.atext_completion()` - async text completion calls -- `router.image_generation()` - completion calls in OpenAI `/v1/images/generations` endpoint format -- `router.aimage_generation()` - async image generation calls - -## Advanced - Routing Strategies ⭐️[](https://docs.litellm.ai/docs/routing#advanced---routing-strategies-%EF%B8%8F "Direct link to Advanced - Routing Strategies ⭐️") - -#### Routing Strategies - Weighted Pick, Rate Limit Aware, Least Busy, Latency Based, Cost Based[](https://docs.litellm.ai/docs/routing#routing-strategies---weighted-pick-rate-limit-aware-least-busy-latency-based-cost-based "Direct link to Routing Strategies - Weighted Pick, Rate Limit Aware, Least Busy, Latency Based, Cost Based") - -Router provides 4 strategies for routing your calls across multiple deployments: - -- Rate-Limit Aware v2 (ASYNC) -- Latency-Based -- (Default) Weighted Pick (Async) -- Rate-Limit Aware -- Least-Busy -- Custom Routing Strategy -- Lowest Cost Routing (Async) - -**🎉 NEW** This is an async implementation of usage-based-routing. - -**Filters out deployment if tpm/rpm limit exceeded** - If you pass in the deployment's tpm/rpm limits. - -Routes to **deployment with lowest TPM usage** for that minute. - -In production, we use Redis to track usage (TPM/RPM) across multiple deployments. This implementation uses **async redis calls** (redis.incr and redis.mget). - -For Azure, [you get 6 RPM per 1000 TPM](https://stackoverflow.com/questions/77368844/what-is-the-request-per-minute-rate-limit-for-azure-openai-models-for-gpt-3-5-tu) - -- sdk -- proxy - -``` -from litellm import Router model_list = [{ # list of model deployments "model_name": "gpt-3.5-turbo", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") "tpm": 100000, "rpm": 10000, }, }, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") "tpm": 100000, "rpm": 1000, },}, { "model_name": "gpt-3.5-turbo", "litellm_params": { # params for litellm completion/embedding call "model": "gpt-3.5-turbo", "api_key": os.getenv("OPENAI_API_KEY"), "tpm": 100000, "rpm": 1000, },}]router = Router(model_list=model_list, redis_host=os.environ["REDIS_HOST"], redis_password=os.environ["REDIS_PASSWORD"], redis_port=os.environ["REDIS_PORT"], routing_strategy="usage-based-routing-v2" # 👈 KEY CHANGE enable_pre_call_checks=True, # enables router rate limits for concurrent calls )response = await router.acompletion(model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}]print(response) -``` - -## Basic Reliability[](https://docs.litellm.ai/docs/routing#basic-reliability "Direct link to Basic Reliability") - -### Weighted Deployments[](https://docs.litellm.ai/docs/routing#weighted-deployments "Direct link to Weighted Deployments") - -Set `weight` on a deployment to pick one deployment more often than others. - -This works across **simple-shuffle** routing strategy (this is the default, if no routing strategy is selected). - -- SDK -- PROXY - -``` -from litellm import Router model_list = [ { "model_name": "o1", "litellm_params": { "model": "o1-preview", "api_key": os.getenv("OPENAI_API_KEY"), "weight": 1 }, }, { "model_name": "o1", "litellm_params": { "model": "o1-preview", "api_key": os.getenv("OPENAI_API_KEY"), "weight": 2 # 👈 PICK THIS DEPLOYMENT 2x MORE OFTEN THAN o1-preview }, },]router = Router(model_list=model_list, routing_strategy="cost-based-routing")response = await router.acompletion( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}])print(response) -``` - -### Max Parallel Requests (ASYNC)[](https://docs.litellm.ai/docs/routing#max-parallel-requests-async "Direct link to Max Parallel Requests (ASYNC)") - -Used in semaphore for async requests on router. Limit the max concurrent calls made to a deployment. Useful in high-traffic scenarios. - -If tpm/rpm is set, and no max parallel request limit given, we use the RPM or calculated RPM (tpm/1000/6) as the max parallel request limit. - -``` -from litellm import Router model_list = [{ "model_name": "gpt-4", "litellm_params": { "model": "azure/gpt-4", ... "max_parallel_requests": 10 # 👈 SET PER DEPLOYMENT }}]### OR ### router = Router(model_list=model_list, default_max_parallel_requests=20) # 👈 SET DEFAULT MAX PARALLEL REQUESTS # deployment max parallel requests > default max parallel requests -``` - -[**See Code**](https://github.com/BerriAI/litellm/blob/a978f2d8813c04dad34802cb95e0a0e35a3324bc/litellm/utils.py#L5605) - -### Cooldowns[](https://docs.litellm.ai/docs/routing#cooldowns "Direct link to Cooldowns") - -Set the limit for how many calls a model is allowed to fail in a minute, before being cooled down for a minute. - -- SDK -- PROXY - -``` -from litellm import Routermodel_list = [{...}]router = Router(model_list=model_list, allowed_fails=1, # cooldown model if it fails > 1 call in a minute. cooldown_time=100 # cooldown the deployment for 100 seconds if it num_fails > allowed_fails )user_message = "Hello, whats the weather in San Francisco??"messages = [{"content": user_message, "role": "user"}]# normal call response = router.completion(model="gpt-3.5-turbo", messages=messages)print(f"response: {response}") -``` - -**Expected Response** - -``` -No deployments available for selected model, Try again in 60 seconds. Passed model=claude-3-5-sonnet. pre-call-checks=False, allowed_model_region=n/a. -``` - -#### **Disable cooldowns**[](https://docs.litellm.ai/docs/routing#disable-cooldowns "Direct link to disable-cooldowns") - -- SDK -- PROXY - -``` -from litellm import Router router = Router(..., disable_cooldowns=True) -``` - -### Retries[](https://docs.litellm.ai/docs/routing#retries "Direct link to Retries") - -For both async + sync functions, we support retrying failed requests. - -For RateLimitError we implement exponential backoffs - -For generic errors, we retry immediately - -Here's a quick look at how we can set `num_retries = 3`: - -``` -from litellm import Routermodel_list = [{...}]router = Router(model_list=model_list, num_retries=3)user_message = "Hello, whats the weather in San Francisco??"messages = [{"content": user_message, "role": "user"}]# normal call response = router.completion(model="gpt-3.5-turbo", messages=messages)print(f"response: {response}") -``` - -We also support setting minimum time to wait before retrying a failed request. This is via the `retry_after` param. - -``` -from litellm import Routermodel_list = [{...}]router = Router(model_list=model_list, num_retries=3, retry_after=5) # waits min 5s before retrying requestuser_message = "Hello, whats the weather in San Francisco??"messages = [{"content": user_message, "role": "user"}]# normal call response = router.completion(model="gpt-3.5-turbo", messages=messages)print(f"response: {response}") -``` - -### \[Advanced\]: Custom Retries, Cooldowns based on Error Type[](https://docs.litellm.ai/docs/routing#advanced-custom-retries-cooldowns-based-on-error-type "Direct link to advanced-custom-retries-cooldowns-based-on-error-type") - -- Use `RetryPolicy` if you want to set a `num_retries` based on the Exception received -- Use `AllowedFailsPolicy` to set a custom number of `allowed_fails`/minute before cooling down a deployment - -[**See All Exception Types**](https://github.com/BerriAI/litellm/blob/ccda616f2f881375d4e8586c76fe4662909a7d22/litellm/types/router.py#L436) - -- SDK -- PROXY - -Example: - -``` -retry_policy = RetryPolicy( ContentPolicyViolationErrorRetries=3, # run 3 retries for ContentPolicyViolationErrors AuthenticationErrorRetries=0, # run 0 retries for AuthenticationErrorRetries)allowed_fails_policy = AllowedFailsPolicy( ContentPolicyViolationErrorAllowedFails=1000, # Allow 1000 ContentPolicyViolationError before cooling down a deployment RateLimitErrorAllowedFails=100, # Allow 100 RateLimitErrors before cooling down a deployment) -``` - -Example Usage - -``` -from litellm.router import RetryPolicy, AllowedFailsPolicyretry_policy = RetryPolicy( ContentPolicyViolationErrorRetries=3, # run 3 retries for ContentPolicyViolationErrors AuthenticationErrorRetries=0, # run 0 retries for AuthenticationErrorRetries BadRequestErrorRetries=1, TimeoutErrorRetries=2, RateLimitErrorRetries=3,)allowed_fails_policy = AllowedFailsPolicy( ContentPolicyViolationErrorAllowedFails=1000, # Allow 1000 ContentPolicyViolationError before cooling down a deployment RateLimitErrorAllowedFails=100, # Allow 100 RateLimitErrors before cooling down a deployment)router = litellm.Router( model_list=[ { "model_name": "gpt-3.5-turbo", # openai model name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), }, }, { "model_name": "bad-model", # openai model name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": "bad-key", "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), }, }, ], retry_policy=retry_policy, allowed_fails_policy=allowed_fails_policy,)response = await router.acompletion( model=model, messages=messages,) -``` - -### Caching[](https://docs.litellm.ai/docs/routing#caching "Direct link to Caching") - -In production, we recommend using a Redis cache. For quickly testing things locally, we also support simple in-memory caching. - -**In-memory Cache** - -``` -router = Router(model_list=model_list, cache_responses=True)print(response) -``` - -**Redis Cache** - -``` -router = Router(model_list=model_list, redis_host=os.getenv("REDIS_HOST"), redis_password=os.getenv("REDIS_PASSWORD"), redis_port=os.getenv("REDIS_PORT"), cache_responses=True)print(response) -``` - -**Pass in Redis URL, additional kwargs** - -``` -router = Router(model_list: Optional[list] = None, ## CACHING ## redis_url=os.getenv("REDIS_URL")", cache_kwargs= {}, # additional kwargs to pass to RedisCache (see caching.py) cache_responses=True) -``` - -## Pre-Call Checks (Context Window, EU-Regions)[](https://docs.litellm.ai/docs/routing#pre-call-checks-context-window-eu-regions "Direct link to Pre-Call Checks (Context Window, EU-Regions)") - -Enable pre-call checks to filter out: - -1. deployments with context window limit < messages for a call. -2. deployments outside of eu-region - -- SDK -- Proxy - -**1\. Enable pre-call checks** - -``` -from litellm import Router # ...router = Router(model_list=model_list, enable_pre_call_checks=True) # 👈 Set to True -``` - -**2\. Set Model List** - -For context window checks on azure deployments, set the base model. Pick the base model from [this list](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json), all the azure models start with `azure/`. - -For 'eu-region' filtering, Set 'region\_name' of deployment. - -**Note:** We automatically infer region\_name for Vertex AI, Bedrock, and IBM WatsonxAI based on your litellm params. For Azure, set `litellm.enable_preview = True`. - -[**See Code**](https://github.com/BerriAI/litellm/blob/d33e49411d6503cb634f9652873160cd534dec96/litellm/router.py#L2958) - -``` -model_list = [ { "model_name": "gpt-3.5-turbo", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "region_name": "eu" # 👈 SET 'EU' REGION NAME "base_model": "azure/gpt-35-turbo", # 👈 (Azure-only) SET BASE MODEL }, }, { "model_name": "gpt-3.5-turbo", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "gpt-3.5-turbo-1106", "api_key": os.getenv("OPENAI_API_KEY"), }, }, { "model_name": "gemini-pro", "litellm_params: { "model": "vertex_ai/gemini-pro-1.5", "vertex_project": "adroit-crow-1234", "vertex_location": "us-east1" # 👈 AUTOMATICALLY INFERS 'region_name' } } ]router = Router(model_list=model_list, enable_pre_call_checks=True) -``` - -**3\. Test it!** - -- Context Window Check -- EU Region Check - -``` -"""- Give a gpt-3.5-turbo model group with different context windows (4k vs. 16k)- Send a 5k prompt- Assert it works"""from litellm import Routerimport osmodel_list = [ { "model_name": "gpt-3.5-turbo", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE"), "base_model": "azure/gpt-35-turbo", }, "model_info": { "base_model": "azure/gpt-35-turbo", } }, { "model_name": "gpt-3.5-turbo", # model group name "litellm_params": { # params for litellm completion/embedding call "model": "gpt-3.5-turbo-1106", "api_key": os.getenv("OPENAI_API_KEY"), }, },]router = Router(model_list=model_list, enable_pre_call_checks=True) text = "What is the meaning of 42?" * 5000response = router.completion( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": text}, {"role": "user", "content": "Who was Alexander?"}, ],)print(f"response: {response}") -``` - -## Caching across model groups[](https://docs.litellm.ai/docs/routing#caching-across-model-groups "Direct link to Caching across model groups") - -If you want to cache across 2 different model groups (e.g. azure deployments, and openai), use caching groups. - -``` -import litellm, asyncio, timefrom litellm import Router # set os envos.environ["OPENAI_API_KEY"] = ""os.environ["AZURE_API_KEY"] = ""os.environ["AZURE_API_BASE"] = ""os.environ["AZURE_API_VERSION"] = ""async def test_acompletion_caching_on_router_caching_groups(): # tests acompletion + caching on router try: litellm.set_verbose = True model_list = [ { "model_name": "openai-gpt-3.5-turbo", "litellm_params": { "model": "gpt-3.5-turbo-0613", "api_key": os.getenv("OPENAI_API_KEY"), }, }, { "model_name": "azure-gpt-3.5-turbo", "litellm_params": { "model": "azure/chatgpt-v-2", "api_key": os.getenv("AZURE_API_KEY"), "api_base": os.getenv("AZURE_API_BASE"), "api_version": os.getenv("AZURE_API_VERSION") }, } ] messages = [ {"role": "user", "content": f"write a one sentence poem {time.time()}?"} ] start_time = time.time() router = Router(model_list=model_list, cache_responses=True, caching_groups=[("openai-gpt-3.5-turbo", "azure-gpt-3.5-turbo")]) response1 = await router.acompletion(model="openai-gpt-3.5-turbo", messages=messages, temperature=1) print(f"response1: {response1}") await asyncio.sleep(1) # add cache is async, async sleep for cache to get set response2 = await router.acompletion(model="azure-gpt-3.5-turbo", messages=messages, temperature=1) assert response1.id == response2.id assert len(response1.choices[0].message.content) > 0 assert response1.choices[0].message.content == response2.choices[0].message.content except Exception as e: traceback.print_exc()asyncio.run(test_acompletion_caching_on_router_caching_groups()) -``` - -## Alerting 🚨[](https://docs.litellm.ai/docs/routing#alerting- "Direct link to Alerting 🚨") - -Send alerts to slack / your webhook url for the following events - -- LLM API Exceptions -- Slow LLM Responses - -Get a slack webhook url from [https://api.slack.com/messaging/webhooks](https://api.slack.com/messaging/webhooks) - -#### Usage[](https://docs.litellm.ai/docs/routing#usage "Direct link to Usage") - -Initialize an `AlertingConfig` and pass it to `litellm.Router`. The following code will trigger an alert because `api_key=bad-key` which is invalid - -``` -from litellm.router import AlertingConfigimport litellmimport osrouter = litellm.Router( model_list=[ { "model_name": "gpt-3.5-turbo", "litellm_params": { "model": "gpt-3.5-turbo", "api_key": "bad_key", }, } ], alerting_config= AlertingConfig( alerting_threshold=10, # threshold for slow / hanging llm responses (in seconds). Defaults to 300 seconds webhook_url= os.getenv("SLACK_WEBHOOK_URL") # webhook you want to send alerts to ),)try: await router.acompletion( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Hey, how's it going?"}], )except: pass -``` - -## Track cost for Azure Deployments[](https://docs.litellm.ai/docs/routing#track-cost-for-azure-deployments "Direct link to Track cost for Azure Deployments") - -**Problem**: Azure returns `gpt-4` in the response when `azure/gpt-4-1106-preview` is used. This leads to inaccurate cost tracking - -**Solution** ✅ : Set `model_info["base_model"]` on your router init so litellm uses the correct model for calculating azure cost - -Step 1. Router Setup - -``` -from litellm import Routermodel_list = [ { # list of model deployments "model_name": "gpt-4-preview", # model alias "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-v-2", # actual model name "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") }, "model_info": { "base_model": "azure/gpt-4-1106-preview" # azure/gpt-4-1106-preview will be used for cost tracking, ensure this exists in litellm model_prices_and_context_window.json } }, { "model_name": "gpt-4-32k", "litellm_params": { # params for litellm completion/embedding call "model": "azure/chatgpt-functioncalling", "api_key": os.getenv("AZURE_API_KEY"), "api_version": os.getenv("AZURE_API_VERSION"), "api_base": os.getenv("AZURE_API_BASE") }, "model_info": { "base_model": "azure/gpt-4-32k" # azure/gpt-4-32k will be used for cost tracking, ensure this exists in litellm model_prices_and_context_window.json } }]router = Router(model_list=model_list) -``` - -Step 2. Access `response_cost` in the custom callback, **litellm calculates the response cost for you** - -``` -import litellmfrom litellm.integrations.custom_logger import CustomLoggerclass MyCustomHandler(CustomLogger): def log_success_event(self, kwargs, response_obj, start_time, end_time): print(f"On Success") response_cost = kwargs.get("response_cost") print("response_cost=", response_cost)customHandler = MyCustomHandler()litellm.callbacks = [customHandler]# router completion callresponse = router.completion( model="gpt-4-32k", messages=[{ "role": "user", "content": "Hi who are you"}]) -``` - -#### Default litellm.completion/embedding params[](https://docs.litellm.ai/docs/routing#default-litellmcompletionembedding-params "Direct link to Default litellm.completion/embedding params") - -You can also set default params for litellm completion/embedding calls. Here's how to do that: - -``` -from litellm import Routerfallback_dict = {"gpt-3.5-turbo": "gpt-3.5-turbo-16k"}router = Router(model_list=model_list, default_litellm_params={"context_window_fallback_dict": fallback_dict})user_message = "Hello, whats the weather in San Francisco??"messages = [{"content": user_message, "role": "user"}]# normal call response = router.completion(model="gpt-3.5-turbo", messages=messages)print(f"response: {response}") -``` - -## Custom Callbacks - Track API Key, API Endpoint, Model Used[](https://docs.litellm.ai/docs/routing#custom-callbacks---track-api-key-api-endpoint-model-used "Direct link to Custom Callbacks - Track API Key, API Endpoint, Model Used") - -If you need to track the api\_key, api endpoint, model, custom\_llm\_provider used for each completion call, you can setup a [custom callback](https://docs.litellm.ai/docs/observability/custom_callback) - -### Usage[](https://docs.litellm.ai/docs/routing#usage-1 "Direct link to Usage") - -``` -import litellmfrom litellm.integrations.custom_logger import CustomLoggerclass MyCustomHandler(CustomLogger): def log_success_event(self, kwargs, response_obj, start_time, end_time): print(f"On Success") print("kwargs=", kwargs) litellm_params= kwargs.get("litellm_params") api_key = litellm_params.get("api_key") api_base = litellm_params.get("api_base") custom_llm_provider= litellm_params.get("custom_llm_provider") response_cost = kwargs.get("response_cost") # print the values print("api_key=", api_key) print("api_base=", api_base) print("custom_llm_provider=", custom_llm_provider) print("response_cost=", response_cost) def log_failure_event(self, kwargs, response_obj, start_time, end_time): print(f"On Failure") print("kwargs=")customHandler = MyCustomHandler()litellm.callbacks = [customHandler]# Init Routerrouter = Router(model_list=model_list, routing_strategy="simple-shuffle")# router completion callresponse = router.completion( model="gpt-3.5-turbo", messages=[{ "role": "user", "content": "Hi who are you"}]) -``` - -## Deploy Router[](https://docs.litellm.ai/docs/routing#deploy-router "Direct link to Deploy Router") - -If you want a server to load balance across different LLM APIs, use our [LiteLLM Proxy Server](https://docs.litellm.ai/docs/simple_proxy#load-balancing---multiple-instances-of-1-model) - -## Debugging Router[](https://docs.litellm.ai/docs/routing#debugging-router "Direct link to Debugging Router") - -### Basic Debugging[](https://docs.litellm.ai/docs/routing#basic-debugging "Direct link to Basic Debugging") - -Set `Router(set_verbose=True)` - -``` -from litellm import Routerrouter = Router( model_list=model_list, set_verbose=True) -``` - -### Detailed Debugging[](https://docs.litellm.ai/docs/routing#detailed-debugging "Direct link to Detailed Debugging") - -Set `Router(set_verbose=True,debug_level="DEBUG")` - -``` -from litellm import Routerrouter = Router( model_list=model_list, set_verbose=True, debug_level="DEBUG" # defaults to INFO) -``` - -### Very Detailed Debugging[](https://docs.litellm.ai/docs/routing#very-detailed-debugging "Direct link to Very Detailed Debugging") - -Set `litellm.set_verbose=True` and `Router(set_verbose=True,debug_level="DEBUG")` - -``` -from litellm import Routerimport litellmlitellm.set_verbose = Truerouter = Router( model_list=model_list, set_verbose=True, debug_level="DEBUG" # defaults to INFO) -``` - -## Router General Settings[](https://docs.litellm.ai/docs/routing#router-general-settings "Direct link to Router General Settings") - -### Usage[](https://docs.litellm.ai/docs/routing#usage-2 "Direct link to Usage") - -``` -router = Router(model_list=..., router_general_settings=RouterGeneralSettings(async_only_mode=True)) -``` - -### Spec[](https://docs.litellm.ai/docs/routing#spec "Direct link to Spec") - -``` -class RouterGeneralSettings(BaseModel): async_only_mode: bool = Field( default=False ) # this will only initialize async clients. Good for memory utils pass_through_all_models: bool = Field( default=False ) # if passed a model not llm_router model list, pass through the request to litellm.acompletion/embedding -``` diff --git a/.claude/liteLLM/Streaming + Async liteLLM.md b/.claude/liteLLM/Streaming + Async liteLLM.md deleted file mode 100644 index b6243be4..00000000 --- a/.claude/liteLLM/Streaming + Async liteLLM.md +++ /dev/null @@ -1,128 +0,0 @@ ---- -created: 2025-05-26T16:00:07 (UTC -04:00) -tags: [] -source: https://docs.litellm.ai/docs/routing#max-parallel-requests-async -author: ---- -Streaming + Async -Feature LiteLLM SDK LiteLLM Proxy -Streaming ✅ start here ✅ start here -Async ✅ start here ✅ start here -Async Streaming ✅ start here ✅ start here -Streaming Responses - -LiteLLM supports streaming the model response back by passing stream=True as an argument to the completion function -Usage - -from litellm import completion -messages = [{"role": "user", "content": "Hey, how's it going?"}] -response = completion(model="gpt-3.5-turbo", messages=messages, stream=True) -for part in response: - print(part.choices[0].delta.content or "") - -Helper function - -LiteLLM also exposes a helper function to rebuild the complete streaming response from the list of chunks. - -from litellm import completion -messages = [{"role": "user", "content": "Hey, how's it going?"}] -response = completion(model="gpt-3.5-turbo", messages=messages, stream=True) - -for chunk in response: - chunks.append(chunk) - -print(litellm.stream_chunk_builder(chunks, messages=messages)) - -Async Completion - -Asynchronous Completion with LiteLLM. LiteLLM provides an asynchronous version of the completion function called acompletion -Usage - -from litellm import acompletion -import asyncio - -async def test_get_response(): - user_message = "Hello, how are you?" - messages = [{"content": user_message, "role": "user"}] - response = await acompletion(model="gpt-3.5-turbo", messages=messages) - return response - -response = asyncio.run(test_get_response()) -print(response) - -Async Streaming - -We've implemented an __anext__() function in the streaming object returned. This enables async iteration over the streaming object. -Usage - -Here's an example of using it with openai. - -from litellm import acompletion -import asyncio, os, traceback - -async def completion_call(): - try: - print("test acompletion + streaming") - response = await acompletion( - model="gpt-3.5-turbo", - messages=[{"content": "Hello, how are you?", "role": "user"}], - stream=True - ) - print(f"response: {response}") - async for chunk in response: - print(chunk) - except: - print(f"error occurred: {traceback.format_exc()}") - pass - -asyncio.run(completion_call()) - -Error Handling - Infinite Loops - -Sometimes a model might enter an infinite loop, and keep repeating the same chunks - e.g. issue - -Break out of it with: - -litellm.REPEATED_STREAMING_CHUNK_LIMIT = 100 # # catch if model starts looping the same chunk while streaming. Uses high default to prevent false positives. - -LiteLLM provides error handling for this, by checking if a chunk is repeated 'n' times (Default is 100). If it exceeds that limit, it will raise a litellm.InternalServerError, to allow retry logic to happen. - - SDK - PROXY - -import litellm -import os - -litellm.set_verbose = False -loop_amount = litellm.REPEATED_STREAMING_CHUNK_LIMIT + 1 -chunks = [ - litellm.ModelResponse(**{ - "id": "chatcmpl-123", - "object": "chat.completion.chunk", - "created": 1694268190, - "model": "gpt-3.5-turbo-0125", - "system_fingerprint": "fp_44709d6fcb", - "choices": [ - {"index": 0, "delta": {"content": "How are you?"}, "finish_reason": "stop"} - ], -}, stream=True) -] * loop_amount -completion_stream = litellm.ModelResponseListIterator(model_responses=chunks) - -response = litellm.CustomStreamWrapper( - completion_stream=completion_stream, - model="gpt-3.5-turbo", - custom_llm_provider="cached_response", - logging_obj=litellm.Logging( - model="gpt-3.5-turbo", - messages=[{"role": "user", "content": "Hey"}], - stream=True, - call_type="completion", - start_time=time.time(), - litellm_call_id="12345", - function_id="1245", - ), -) - -for chunk in response: - continue # expect to raise InternalServerError