🧩 Provider Abstraction Layer¶

Note

👋 Hey there! Siyarix is a personal passion project built by a single developer that is growing and under active development. Some of the architectural components and features described on this page might currently be Planned, Work in Progress, or basic implementations. Stay tuned as it evolves! 🚀

Welcome to the Provider Abstraction Layer! This component is the beating heart of our AI infrastructure. It smoothly decouples all AI-dependent components from specific model backends, making our system resilient, flexible, and fully provider-agnostic.

Think of it as an intelligent traffic controller for your AI models. It effortlessly manages 26 different provider profiles with features like automatic failover, circuit breaking, exponential backoff, and token usage tracking. Plus, it brings everything together under a single, unified OpenAICompat adapter.

Note

Provider states are intelligently persisted as JSON files. This ensures your AI configuration and cooldown statuses carry over seamlessly across different sessions!

🏗️ Architecture Overview¶

Here is a bird's-eye view of how the abstraction layer sits within the overall system:

┌──────────────────────────────────────────────────────┐
│                Consumer Layer                        │
│  (Planner, ChatSession, AutonomousExecutor, Swarm)   │
└─────────────────────────┬────────────────────────────┘
                          │
                          ▼
┌──────────────────────────────────────────────────────┐
│                   ProviderManager                    │
│                                                      │
│  • Provider selection (preference chain + scoring)   │
│  • Failover orchestration                            │
│  • Circuit breaking (record_failure)                 │
│  • Rate limiting                                     │
│  • DLP data redaction                                │
│  • Provider filtering by capability                  │
└──────┬───────────┬───────────┬───────────┬───────────┘
       │           │           │           │
       ▼           ▼           ▼           ▼
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│OpenAI    │ │Ollama    │ │LM Studio │ │llama.cpp │
│Compat    │ │Utils     │ │(OpenAI   │ │(OpenAI   │
│Adapter*  │ │(local)   │ │Compat)   │ │Compat)   │
└──────────┘ └──────────┘ └──────────┘ └──────────┘
       │           │
       ▼           ▼
  26 Provider Profiles (cloud, local, heuristic fallback)

Tip

* The OpenAICompat adapter (siyarix/chat/openai_compat.py) acts as a universal translator. It provides a standardized OpenAI-compatible API across all providers that support the OpenAI chat completions protocol!

🔌 The Provider Interface¶

Unlike traditional object-oriented systems that use heavy abstract base classes, we define providers using lightweight data profiles. This makes them incredibly fast to load and easy to configure.

You can find the core data models in the types module (siyarix/providers/types.py):

@dataclass
class ProviderProfile:
    name: str
    display_name: str
    provider_type: str           # "cloud" | "local" | "heuristic"
    base_url: str
    api_key_env: str | None      # Which environment variable holds the key?
    models: list[ModelInfo]
    capabilities: set[str]       # e.g., "chat", "embed", "function_calling", "vision"
    priority: int                # Position in the preference chain
    rate_limit: int | None = None
    timeout: int = 60

@dataclass
class ModelInfo:
    id: str
    display_name: str
    context_window: int
    max_output_tokens: int
    capabilities: set[str]       # What can this model do?

🎯 Provider Selection¶

Need a provider with specific superpowers? Filtering is built right in:

# Grab providers with the exact capabilities you need:
vision_providers = pm.get_providers_by_capability("vision")
free_providers = pm.get_providers_by_capability("free")
local_providers = pm.get_providers_by_capability("local")
fn_call_providers = pm.get_providers_by_capability("function_calling")

🌐 The 26 Provider Profiles¶

Our system is ready out-of-the-box to connect with an impressive array of AI backends.

☁️ Cloud Providers (API Key Required)¶

These providers require authentication but offer the most powerful, cutting-edge models.

Registry Name	SDK / API	Notable Models
`openai`	`openai`	GPT-4o, GPT-4-turbo, GPT-3.5-turbo
`gemini`	`google-generativeai`	Gemini 2.0 Flash, 1.5 Pro
`anthropic`	`anthropic`	Claude 3.5 Sonnet, Claude 3 Opus
`groq`	`groq`	Llama 3 70B, Mixtral 8x7B
`together`	`together`	Mixtral, DeepSeek, Llama variants
`openrouter`	`openai`	Multi-model router
`deepseek`	`openai`	DeepSeek-V2, DeepSeek-Coder
`xai`	`openai`	Grok
`mistral`	`mistralai`	Mistral Large, Mistral Small
`perplexity`	`openai`	Sonar, Sonar-Pro
`cerebras`	`openai`	Fast inference models
`fireworks`	`openai`	Open model serving
`zai`	`openai`	Z.A.I. models
`minimax`	`openai`	MiniMax models
`moonshot`	`openai`	Moonshot / Kimi
`nvidia`	`openai`	NVIDIA Nemotron
`huggingface`	`huggingface-hub`	Hugging Face Inference API
`azure`	`openai`	Azure OpenAI (personal, managed AD)
`opencode_zen`	`openai`	OpenCode Zen backend

💻 Local Providers (No API Key Required)¶

Run AI completely offline, free, and secure directly on your own hardware!

Registry Name	Default Endpoint	Notes
`ollama`	`http://localhost:11434`	The easiest way to pull and run open-weight models
`lmstudio`	`http://localhost:1234`	Awesome GUI + local API server
`llamacpp`	`http://localhost:8080`	Highly efficient CPU/GPU inference
`vllm`	Configurable	Built for high-throughput GPU serving
`localai`	`http://localhost:8080`	A seamless drop-in OpenAI replacement

🛠️ Heuristic Fallback¶

When all else fails, we have reliable, non-AI fallbacks.

Name	Description
`registry`	RegistryPlanner — A 100% rule-based system, always available and offline-safe.

🤝 OpenAICompat Adapter¶

Dealing with dozens of different API specifications is a headache. That's why we created the OpenAICompat adapter (siyarix/chat/openai_compat.py). It acts as a universal bridge for over 14+ providers.

By standardizing around the OpenAI chat completions protocol, you write your code once, and it automatically supports: - 💬 Chat completions & Streaming responses - 🧠 Embedding generation - ⚙️ Tool and function calling - 📋 Structured output (JSON mode)

# Look how easy it is to swap providers!
adapter = OpenAICompat(provider="openai", api_key=...)
adapter = OpenAICompat(provider="groq", api_key=...)
adapter = OpenAICompat(provider="deepseek", api_key=...)

🧠 ProviderManager¶

The ProviderManager (siyarix/providers/manager.py) is the smart coordinator of the entire system.

🚦 Selection Logic¶

How does it decide which AI to use? It evaluates several factors dynamically: 1. User Preference: Starts with your model_provider config. 2. Key Availability: Checks the CredentialStore for valid API keys. 3. Health Status: Performs quick connectivity tests. 4. Task Matching: Ensures the provider supports the requested feature (like vision or function calling). 5. Cooldowns: Actively avoids providers that recently failed. 6. Historical Reliability: Deprioritizes providers with high error rates.

⛓️ The Preference Chain¶

When set to model_provider = "auto", the system acts autonomously, marching down a prioritized list until it finds a working model:

Gemini → OpenAI → Anthropic → Groq → Together → OpenRouter → DeepSeek → xAI → Mistral → Perplexity → Cerebras → Fireworks → ZAI → MiniMax → Moonshot → NVIDIA → HuggingFace → Azure → OpenCode Zen → Ollama → LM Studio → llama.cpp → vLLM → LocalAI → Registry (Heuristic)

🔄 Failover Behavior¶

Info

Our architecture guarantees robust failovers. If an AI provider goes down, your app doesn't crash—it seamlessly routes to the next best option!

Request → Provider A (Preferred)
              │
              ├── ✅ Success → Return result, record success
              │
              └── ❌ Failure → Trigger record_failure()
                            │
                            ▼
                    ProviderStateManager enforces cooldown
                            │
                            ▼
                    Provider B (Next in chain)
                            │
                            ├── ✅ Success → Return result
                            │
                            └── ❌ Failure → Continue chain
                                          │
                                          ▼
                                  Registry (Heuristic fallback)

📊 Live Statistics¶

Want to know how your providers are performing?

stats = provider_manager.stats()
# You instantly get detailed usage stats, error rates, and current statuses!

⚡ Circuit Breaking¶

To protect your system from hanging endlessly on broken APIs, we use a Circuit Breaker pattern via ProviderManager.record_failure().

Parameter	Default	What it means
Failure threshold	3	How many consecutive errors trigger a circuit "trip" (opening the circuit).
Recovery timeout	60s	Time to wait before cautiously trying the provider again (half-open).
Cooldown duration	300s	Total time a provider is "benched" after failing hard.

state = provider_manager.record_failure("openai")  # Trips to OPEN after 3 strikes
state = provider_manager.record_success("gemini")  # Restores trust, moving back to CLOSED

💾 ProviderStateManager¶

Your system's memory isn't wiped when you restart. The ProviderStateManager saves the exact state of your providers into a simple JSON file (provider_state.json).

@dataclass
class ProviderState:
    provider_name: str
    circuit_state: str                # CLOSED | OPEN | HALF_OPEN
    failure_count: int
    last_failure: datetime | None
    cooldown_until: datetime | None
    rate_limited_until: datetime | None
    total_requests: int
    total_tokens: int
    total_cost: float

Tip

Because this is saved to disk, if a provider hits a rate limit right before you close your app, it will still correctly skip that provider when you reboot!

📈 UsageTracker¶

Keep your AI costs under control! The UsageTracker monitors every single token and penny.

Metric	How it's tracked
Prompt tokens	Per Request
Completion tokens	Per Response
Total tokens	Request + Response
Estimated cost	Based on the provider's specific rate card
Request count	Per Session
Latency	Per Request

tracker = UsageTracker()
tracker.record("openai", prompt_tokens=150, completion_tokens=450, latency=2.3)
summary = tracker.get_summary("openai")
# Instantly see exactly what you spent:
# UsageSummary(total_tokens=600, total_cost=0.009, avg_latency=2.3, request_count=1)

⏱️ Exponential Backoff¶

When an API blinks with a temporary error (like a 429 Rate Limit or 503 Server Error), we don't just hammer it. We respectfully back off using jittered exponential delays:

# Formula for our smart delays
backoff = min(2 ** attempt + random.uniform(0, JITTER), MAX_DELAY)

Attempt	Delay Range	Max Delay
1st Try	1.0 – 2.0s	30s
2nd Try	2.0 – 3.0s	30s
3rd Try	4.0 – 5.0s	30s
4th Try	8.0 – 9.0s	30s
5th+ Try	16.0 – 17.0s	30s

🦙 Ollama Utilities¶

Running models locally? siyarix/providers/ollama_utils.py makes it a breeze with dedicated helpers for Ollama:

🔍 Auto-Discovery: Automatically lists your downloaded models.
⬇️ Smart Pulling: Downloads missing models with live progress tracking.
❤️ Health Checks: Monitors your local endpoint's vitals.
💻 Hardware Awareness: Intelligently selects models that fit your specific GPU/CPU constraints.

🛡️ Security & Data Masking (DLP)¶

We take your data seriously. Before any information leaves your machine and hits a cloud provider, our DLPEngine (siyarix/dlp.py) scrubs it clean.

Warning

Never disable the DLPEngine in a production cloud environment!

Data Type	Sent to Cloud Provider	Behavior for Local Models
IP addresses	Masked as `10.x.x.x`	Sent safely unmasked
Credentials	`[REDACTED]`	Permanently redacted
API keys	`[REDACTED]`	Permanently redacted
Internal hostnames	Masked as `example.com`	Sent safely unmasked
JWTs / tokens	`[REDACTED]`	Permanently redacted

🏷️ ModelAliases¶

AI companies change their model names all the time. The ModelAliases system (siyarix/model_aliases.py) abstracts this away so your code never breaks when a new version drops!

Your Code Asks For	We Actually Route To
`gpt-4`	`gpt-4-turbo` or `gpt-4o` (automatically uses the best available)
`claude-3`	`claude-3-opus` or `claude-3-sonnet`
`gemini-pro`	`gemini-1.5-pro` or `gemini-2.0-flash`
`llama-3`	`llama-3-70b` or `llama-3-8b`
`mixtral`	`mixtral-8x7b` or `mixtral-8x22b`

🗺️ Component Relationships Map¶

Here is how all these incredible features tie together:

                   ┌────────────────────────┐
                   │     Consumer Layer     │
                   │  (Planner, Chat, etc.) │
                   └────────┬───────────────┘
                            │
                            ▼
                   ┌────────────────────────┐
                   │    ProviderManager     │
                   │                        │
                   │  ┌──────────────────┐  │
                   │  │ ⛓️ Pref. Chain    │  │
                   │  │  + Failover      │  │
                   │  └──────────────────┘  │
                   │  ┌──────────────────┐  │
                   │  │ ⚡ Circuit Break  │  │
                   │  │  (record_failure)│  │
                   │  └──────────────────┘  │
                   │  ┌──────────────────┐  │
                   │  │ ⏱️ Exp. Backoff   │  │
                   │  └──────────────────┘  │
                   │  ┌──────────────────┐  │
                   │  │ 🛡️ DLP Redaction  │  │
                   │  └──────────────────┘  │
                   │  ┌──────────────────┐  │
                   │  │ 🎯 Capability     │  │
                   │  │    Filtering     │  │
                   │  └──────────────────┘  │
                   └────────┬───────────────┘
                            │
              ┌─────────────┼─────────────┐
              ▼             ▼             ▼
    ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
    │ OpenAICompat │ │ ProviderState│ │ Ollama       │
    │ Adapter      │ │ Manager      │ │ Utils        │
    │ (14+ APIs)   │ │ (JSON File)  │ │ (Local AI)   │
    └──────────────┘ └──────────────┘ └──────────────┘
                            │
                            ▼
                   ┌────────────────────────┐
                   │     UsageTracker       │
                   │  (Tokens + Cost Per    │
                   │   Provider)            │
                   └────────────────────────┘