This guide tell about online deployment using Hugging Face Inference Endpoints.

1. Online deployment

Use Inference Endpoints to deploy the model online when local resources are insufficient.

Deployment Steps

1. Access Deployment Interface

Visit Hugging Face Inference Endpoints to start

2. Configuration

• Hardware configuration

Choose an instance, here are our recommended choices for different model sizes:

Take 7B model as an example, we choose “Nvidia L40S 1GPU 48G” instance:

• Container Configuration

Set Max Number of Tokens (per Query) to 32768

Set Max Batch Prefill Tokens to 32768

Set Max Input Length (per Query) to 32767

• Environment Variables

Add CUDA_GRAPHS=0 to avoid launching failed. Check this issue for details.