Open in Playground Download design mesheryctl design import -f https://raw.githubusercontent.com/meshery/meshery.io/master/catalog/4085f7a4-34e4-45a5-b00b-7fb1c19543a7/0.0.9/design.yml

What This Design Does:

This distributed web crawler system is designed as a cloud-native, scalable solution for large-scale web data extraction. Built using Kubernetes-native components, it follows microservices architecture principles to ensure high availability, fault tolerance, and horizontal scalability.

Design Snapshot

Caveats and Consideration:

Configuration Dependencies: Container Images: Replace your-registry/crawler-worker:latest and your-registry/crawler-api:latest with actual image locations Database Credentials: Currently using hardcoded passwords (crawlerpass) (should be moved to Kubernetes Secrets) Service Discovery: Assumes internal DNS resolution works correctly between services Storage Limitations: PostgreSQL Single Instance: No built-in high availability or replication configured No Persistent Volumes: Redis instances will lose data on pod restarts

Compatibility:

Design Details

• CATEGORY

  deployment

• CREATED BY

  Ahmed Hossam

• CREATED AT

  July 29, 2025

• VERSION

  0.0.9

• MODELS

  Aws-elasticache-controller , Aws-ecr-controller , Aws-eks-controller , Aws-opensearchservice-controller , Aws-rds-controller , Aws-s3-controller , Psmdb-operator

Related Designs

deployment

vault-0.30.1.tgz

MESHERY40c7

`

« Mount(Pod -> PersistentVolume) cert-manager-v1.18.2.tgz »