Open in Playground Download design mesheryctl design import -f https://raw.githubusercontent.com/meshery/meshery.io/master/catalog/9d3b33bf-eaf6-4d06-9dd6-64aa725ab383/0.0.1/design.yml

What This Design Does:

Serve a large language model (LLM) with GPUs in Google Kubernetes Engine (GKE) mode. Create a GKE Standard cluster that uses multiple L4 GPUs and prepares the GKE infrastructure to serve any of the following models: 1. Falcon 40b. 2. Llama 2 70b

Design Snapshot

Serve an LLM with multiple GPUs in GKE design snapshot

Caveats and Consideration:

Depending on the data format of the model, the number of GPUs varies. In this design, each model uses two L4 GPUs.

Compatibility:

Design Details

Related Designs

deployment
vault-0.30.1.tgz
MESHERY40c7
`