HybridInference
English 简体中文

Getting Started

  • Quickstart
    • What you need
    • Stage 1: prove the router works
      • Inspect the gateway yourself
    • Stage 2: continue into the Web and Admin Consoles
    • Stage 3: replace the fake provider with local inference
    • What the example contains
    • Where to go next
    • Stop, resume, and reset
    • Troubleshooting
      • Port already in use
      • Cannot connect to the Docker daemon
      • Empty /v1/models and routes_configured: 0
      • make up chose another distribution
      • Start over after a partial run
      • Inspect a failed service
  • Installation
    • What you need
    • Running with Docker
    • Development checkout (no Docker)
    • Configuration
      • Environment variables
      • Provider credentials
      • Where the config files live
    • Verifying the install
    • Troubleshooting
  • Configuration
    • What the gateway reads at startup
    • Where configuration lives
    • How a gateway finds its config
    • What a fresh clone does with no configuration at all
    • The model registry
    • The routing file
    • Settings that currently have no effect
    • Runtime configuration from the admin console
      • Providers tab
      • Routing tab
      • Settings tab
      • How the two layers combine
    • Environment variables
    • Running with your configuration

Concepts

  • Architecture
    • Design principles
    • The four layers
    • Request lifecycle
      • 1. Middleware
      • 2. Authentication and quota
      • 3. Model gate
      • 4. Router selection
      • 5. Dispatch, fallback, and the circuit breaker
      • 6. Logging
    • The routing engine in detail
      • Routers and strategies
      • Two layers of “strategy”
      • Execution and composition
      • provider versus endpoint_id
    • Adapters
      • Streaming timeouts
    • Configuration
    • Storage
    • Observability
    • HTTP surface
    • Deployment shape
  • Routing
    • Choosing a router per model
    • How FixedRouter picks an endpoint
      • Fallback
    • Endpoint health and circuit breaking
      • Admission, for router authors
    • Routes excluded from selection
    • Prefill-aware selection
    • Session affinity
    • Outbound concurrency
    • Queue-wait offload
      • Engine waits
      • Engine queue limit
      • Max offload input
    • RouteWise
    • Reserving upstream keys for a tier
    • API endpoints
    • Where the code lives
    • Adding a routing strategy
      • 1. Understand the contract
      • 2. Get the routes
      • 3. Write the strategy module
      • 4. Register it at import time
      • 5. Select it
      • 6. Test it
  • Routing Internals
    • How a request reaches an adapter
    • Leaves, pools, and dispatch instructions
    • Router types and execution boundaries
    • Full-pool RouteWise: available today
    • Fixed composition: explicitly enabled
    • Greedy with cloud RouteWise: future extension
  • The Public Path Table
    • The path table
      • Runtime route handler — the cloud agent proxy
      • Legacy beforeFiles compatibility
      • afterFiles — the gateway
    • Why /pgadmin is a route handler and not a rewrite
      • The trailing-slash interaction
    • Adding a public path
  • Glossary
    • Requests and models
    • Routes and providers
    • Choosing a route
    • Deployments

Guides

  • Adding a New Model
    • Where your model registry lives
    • Adding a model
    • Verify it through the gateway
    • Configuration Reference
      • Model fields
      • Route configuration
        • Custom embeddings paths
      • Supported adapter kinds
      • Weights across routes
    • Troubleshooting
      • Model not appearing in /v1/models
      • Authentication failures
    • See also
  • Adding a New Local Model
    • Overview
    • Private Server (No Public Internet)
    • Step 1: Start the Local Model Server
    • Step 2: Add the Model to Your Registry
      • Naming a Route in the Dashboard
    • Step 3: Add Optional Remote Fallbacks
    • Step 4: Restart the Gateway
    • Step 5: Verify Through the Gateway
    • Routing Notes
      • Prioritizing Decode on an sglang Route
      • Recording a Prefix-Cache Miss on an sglang Route
    • Troubleshooting
      • Model Does Not Appear in /v1/models
      • Gateway Cannot Reach the Local Server
      • Requests Fail After Registration
    • See Also
  • Writing a Provider Adapter
    • Step 1: decide whether you need an adapter at all
    • Step 2: write the adapter (custom protocols only)
    • Step 3: add the model to your registry
    • Step 4: configure environment variables
    • Step 5: verify it
    • Deployment-local adapters
    • BaseAdapter API reference
    • Advanced features
      • Multi-modal support
      • Tool / function calling
      • Structured output (JSON mode)
      • Limiting concurrent requests
    • Examples
    • Troubleshooting
      • Response format errors
      • Streaming issues
    • Best practices
    • See also
  • Routing through OpenRouter
    • The default catalog
    • Route syntax
      • kind: openrouter
      • kind: openrouter[<slug>]
      • Sort policy
    • What the adapter sends
    • Cost accounting
    • API keys
    • Error handling
    • Troubleshooting
  • Serving a Model from an HPC Cluster
    • 1. Allocate a node
    • 2. Make the container image available offline
    • 3. Start the model server
    • 4. Tunnel it to the gateway host
    • 5. Register the route
    • 6. Release everything when you are done
  • Claude Code Setup
    • Setup
    • Model Families and Aliases
    • Usage and Verification
    • Troubleshooting
    • Uninstall
  • Docs RAG Assistant
    • How it works
    • Setting it up
      • Rebuilding the index
      • Deploying the index
    • Endpoints
    • Turning it off
    • Configuration
    • Limitations
    • Code map

Customizing a Deployment

  • Distribution customization
    • What you can customize
    • Configuration-only customization
      • Where your files live
      • The distribution manifest
      • Activating a manifest
      • Identity, and what the manifest must not contain
      • Runtime branding
      • Feature settings and their limits
      • When changes take effect
      • The example distribution
    • Custom public pages and backend code
    • Upgrade checklist
      • An existing source fork
      • Before promotion
  • Site UI Modules
    • What a module can and cannot change
    • What a module may import
    • A module is a directory
    • Document language and page titles
    • Building an image with a module
      • Local development
    • Assets
    • Styling the shared forms
    • When a module component fails
    • Compatibility
    • Testing a module
  • Backend Extensions
    • Loading an extension
    • Cloud Agent access
    • Quota reporting
      • Run a local quota source

Operations

  • Deployment Guide
    • What the stack is
      • Putting it on the public internet
        • What your proxy still has to do
      • Keeping it private: an SSH tunnel
    • Everyday operations
      • What a change actually requires
    • Configuration
      • Environment
      • Config file resolution
      • Local inference servers
    • Health checks
      • Marking monitor traffic
    • Alerting
      • What an auth alert tells you
      • Silencing alerts from the dashboard
    • The first admin account
    • Database
      • pgAdmin (optional)
    • Troubleshooting
      • A service will not start
      • A monitor or service account is suddenly getting 429s
      • Resetting the stack
      • Rebuilding after code changes
  • Releases and upgrades
    • Release identity and support
      • Include the version in a bug report
    • Compatibility and breaking changes
    • Upgrade and rollback
  • Database
    • Is a database required?
    • Connection settings
    • How the schema is created
      • Secrets for deleting accounts
    • What the tables hold
    • Request logging and privacy
      • Which session a request belongs to
    • Backup
    • Reset
    • Inspecting the database
    • Optional: pgAdmin
      • Starting and restarting it
      • Authentication
      • Registering the database
      • Resetting pgAdmin
  • Trusted Proxies and Client IPs
    • The server underneath
    • Trust configuration
      • Trusted proxies
      • Direct private client networks
      • Cloudflare-header-authorized peers
      • Trust flags
      • Why this matters
    • Resolution order
      • Example: multi-hop chain
      • Example: attacker prepends fake addresses
      • What counts as routable
    • The “unknown” outcome
    • Cloudflare Pseudo IPv4
    • Bucketing: why IPv6 folds to a /64
      • An IPv6 client on an IPv4-only origin is normal
    • What is logged
    • What is persisted
      • Retention is yours to set
    • Testing your setup
  • Automation score (human vs. script)
    • Inputs
    • The signals
      • 1. turn_pattern — user turns
      • 2. prompt_size_dispersion — length of user turn
      • 3. client_tool_prior — user-agent
      • 4. daily_activity_shape — daily activity
      • 5. tool_call_human_tell — supporting
      • 6. agent_opener_override — supporting
      • 7. user_message_shape — user-message size & entropy
    • Combining the signals
    • Bands
    • Caveats
    • Reproducing it from the CLI
    • Code map

Contributing

  • Contributing
    • Getting set up
    • Your first contribution
    • Repository layout
    • Quality gates
    • Tests
      • Frontend
    • Documentation
      • Writing a page
      • Translations
    • Proposing a change
      • What CI will run
  • Translating the Docs
    • Translating into Chinese, Japanese or Korean
    • How a translation reaches the published site
    • The check that catches a reverted translation
    • Chinese terminology

Project

  • About
    • The project and the lab
    • See it running
    • Getting in touch
HybridInference
  • Search

© Copyright 2026, The HybridInference contributors.

Built with Sphinx using a theme provided by Read the Docs.