Catalog Details

  • CATEGORY

    deployment

  • UUID

    9d3b33bf-eaf6-4d06-9dd6-64aa725ab383

  • CREATED BY

  • CREATED AT

    March 07, 2024

  • VERSION

    0.0.1

  • MODELS

Design Snapshot

Related Designs

deployment
vault-0.30.1.tgz
MESHERY40c7

What This Design Does:

Serve a large language model (LLM) with GPUs in Google Kubernetes Engine (GKE) mode. Create a GKE Standard cluster that uses multiple L4 GPUs and prepares the GKE infrastructure to serve any of the following models: 1. Falcon 40b. 2. Llama 2 70b

Caveats and Consideration:

Depending on the data format of the model, the number of GPUs varies. In this design, each model uses two L4 GPUs.

Compatibility: