What This Design Does:

Serve a large language model (LLM) with GPUs in Google Kubernetes Engine (GKE) mode. Create a GKE Standard cluster that uses multiple L4 GPUs and prepares the GKE infrastructure to serve any of the following models: 1. Falcon 40b. 2. Llama 2 70b

Design Snapshot

Serve an LLM with multiple GPUs in GKE design snapshot

Caveats and Consideration:

Depending on the data format of the model, the number of GPUs varies. In this design, each model uses two L4 GPUs.

Compatibility:

Design Details

Related Designs

deployment
vault-0.30.1.tgz
MESHERY40c7
`