Self-Hosting Compute Community

Overview

Compute Community allows you to run your own server to provide AI compute resources to your network. After setting up your server, you'll have an API key and gateway URL that you can share with friends to let them access your GPU resources.

Prerequisites

To run the Compute Community server, you'll need:

• Docker Desktop
• WSL (Windows Subsystem for Linux) if running on Windows
• NVIDIA GPU with compatible drivers
• NVIDIA Container Toolkit
• ngrok (for exposing local server)

Docker Configuration

Ensure Docker is installed and properly configured with NVIDIA support. The server uses vLLM for LLM inference. Make sure your model fits on your GPU's available memory.

Running the Server

Use Docker to run the vLLM server with your chosen model:

docker run --runtime nvidia --gpus all ^
    -p 8000:8000 ^
    --ipc=host ^
    vllm/vllm-openai:latest ^
    --model Qwen/Qwen2.5-14B-Instruct-AWQ ^
    --gpu-memory-utilization 0.90 ^
    --max_model_len 16384 ^
    --api-key YOUR_API_KEY

ngrok Setup

To expose your local server to the internet, you'll need to set up ngrok:

1. Create an ngrok account

   Sign up on ngrok

2. Authenticate with your ngrok authtoken

   Find your authtoken in your ngrok dashboard and set it up:

   ngrok authtoken YOUR_AUTH_TOKEN

3. Create a static domain (recommended)

   Create a static domain in your ngrok dashboard, such as your-domain.ngrok-free.app

4. Start the ngrok tunnel

   ngrok http --url=YOUR_STATIC_DOMAIN 8000

Sharing Your Server

After setting up your server, you can share the following details with your friends:

• Gateway URL: Your ngrok URL (e.g., https://your-domain.ngrok-free.app)
• API Key: The API key you specified when running the Docker container
• Model Name: The model you're serving (e.g., Qwen/Qwen2.5-14B-Instruct-AWQ)

They can add these details in the Compute Community settings page to connect to your server.