Load Balancing

### Setup #### Secrets ##### radosgw Keyring ``` ceph-authtool --create-keyring /etc/pve/priv/ceph.client.radosgw.keyring ``` ###### On each node ln -s /etc/pve/priv/ceph.client.admin.keyring /etc/ceph/ceph.client.admin.keyring ln -s /etc/pve/priv/ceph.client.radosgw.keyring /etc/ceph/ceph.client.radosgw.keyring ##### Node keys ceph-authtool /etc/pve/priv/ceph.client.radosgw.keyring -n client.radosgw.px-m-40 --gen-key ceph-authtool /etc/pve/priv/ceph.client.radosgw.keyring -n client.radosgw.px-m-41 --gen-key ceph-authtool /etc/pve/priv/ceph.client.radosgw.keyring -n client.radosgw.px-m-42 --gen-key ceph-authtool /etc/pve/priv/ceph.client.radosgw.keyring -n client.radosgw.px-m-43 --gen-key ceph-authtool /etc/pve/priv/ceph.client.radosgw.keyring -n client.radosgw.px-m-44 --gen-key ceph-authtool /etc/pve/priv/ceph.client.radosgw.keyring -n client.radosgw.px-m-45 --gen-key ##### Privileges ###### create the privilege tokens ceph-authtool -n client.radosgw.px-m-40 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/pve/priv/ceph.client.radosgw.keyring ceph-authtool -n client.radosgw.px-m-41 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/pve/priv/ceph.client.radosgw.keyring ceph-authtool -n client.radosgw.px-m-42 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/pve/priv/ceph.client.radosgw.keyring ceph-authtool -n client.radosgw.px-m-43 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/pve/priv/ceph.client.radosgw.keyring ceph-authtool -n client.radosgw.px-m-44 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/pve/priv/ceph.client.radosgw.keyring ceph-authtool -n client.radosgw.px-m-45 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/pve/priv/ceph.client.radosgw.keyring ###### Add the newly minted auth tokens to the cluster Using the admin keyring, add the newly minted tokens to the cluster. ceph -k /etc/pve/priv/ceph.client.admin.keyring auth add client.radosgw.px-m-40 -i /etc/pve/priv/ceph.client.radosgw.keyring ceph -k /etc/pve/priv/ceph.client.admin.keyring auth add client.radosgw.px-m-41 -i /etc/pve/priv/ceph.client.radosgw.keyring ceph -k /etc/pve/priv/ceph.client.admin.keyring auth add client.radosgw.px-m-42 -i /etc/pve/priv/ceph.client.radosgw.keyring ceph -k /etc/pve/priv/ceph.client.admin.keyring auth add client.radosgw.px-m-43 -i /etc/pve/priv/ceph.client.radosgw.keyring ceph -k /etc/pve/priv/ceph.client.admin.keyring auth add client.radosgw.px-m-44 -i /etc/pve/priv/ceph.client.radosgw.keyring ceph -k /etc/pve/priv/ceph.client.admin.keyring auth add client.radosgw.px-m-45 -i /etc/pve/priv/ceph.client.radosgw.keyring added key for client.radosgw.px-m-40 added key for client.radosgw.px-m-41 added key for client.radosgw.px-m-42 added key for client.radosgw.px-m-43 added key for client.radosgw.px-m-44 added key for client.radosgw.px-m-45

#### Update /etc/services ``` radosgw 7480/tcp # Ceph Rados gw ``` #### Increase Systemic Limits ### Installation and Service Enablement

## Setting up the external gateway ### DNS Records #### wildcard and A records I chose `dog.wolfspyre.io` as the root subdomain. Since I want to facilitate my offsite hosts to be able to access this as well, I need to enable external and internal resolution of the endpoints. ##### external records ``` # wolfspyre.io dog IN A 108.221.46.29 *.dog IN A 108.221.46.29 ``` ##### internal records ``` # wolfspyre.io dog IN NS ns01.wolfspyre.io. dog IN NS ns02.wolfspyre.io. dog IN NS ns03.wolfspyre.io. ``` ``` # dog.wolfspyre.io @ IN A 198.18.1.33 * IN A 198.18.1.33 skwirreltrap IN A 198.18.198.1 atticus IN A 198.18.198.2 evey IN A 198.18.198.3 px-m-40 IN A 198.18.198.40 px-m-41 IN A 198.18.198.41 px-m-42 IN A 198.18.198.42 px-m-43 IN A 198.18.198.43 px-m-44 IN A 198.18.198.44 px-m-45 IN A 198.18.198.45 ``` ### firewall adjustment - I needed to permit traffic from internal hosts to the VIP on `tcp:443` - I needed to permit traffic from the firewalls to the proxmox nodes on `TCP:7480` ### OPNsense Haproxy config - - - #### Real Servers (Backends) ------ ###### Main info - Name or Prefix : px-m-40-7080 - Description : px-m-40-rados - Type : static ###### Static Server - FQDN or IP : px-m-40.dog.wolfspyre.io - Port : 7480 - Mode : active [default] - Multiplexer Protocol : auto-selection [recommended] - Prefer IP Family : prefer IPv4 ###### Common Options - SSL : [ ] - SSL SNI : px-m-40.dog.wolfspyre.io - Verify SSL Certificate : [ ] - SSL Verify CA : Nothing Selected - SSL Verify CRL : None - SSL Client Certificate : None - Max Connections : N/A - Weight : N/A - Check Interval : N/A - Down Interval : N/A - Port to check : N/A - Source address : 198.18.198.1 - Option pass-through : N/A ------ #### Backend Pools - - - - advanced mode : [ x ] - Enabled : [ x ] - Name : PXMCeph-S3-Pool - Description : Proxmox Ceph S3 Backend Pool - Mode : HTTP (Layer 7) [default] - Balancing Algorithm : Source-IP Hash [default] - Random Draws : 2 - Proxy Protocol : none - Servers : - pxm-40-8006 - pxm-41-8006 - pxm-42-8006 - pxm-43-8006 - pxm-44-8006 - pxm-45-8006 - FastCGI Application : none - Resolver : none - Resolver Options : none - Prefer IP Family : prefer IPv4 - Source address : 198.19.198.1 - Enable Health Checking : [x] ###### Health Checking - Health Monitor : PXM UI Port 8006 Check - Log Status Changes : a - Check Interval : a - Down Interval : a - Unhealthy Threshold : a - Healthy Threshold : a - E-Mail Alert : none ###### HTTP(S) settings - Enable HTTP/2 : [ ] - HTTP/2 without TLS : [ ] - Advertise Protocols (ALPN) : - HTTP/1.1 - HTTP/1.0 ###### Persistence - Persistence type : Stick-table persistence [default] ##### Stick-table persistence - Table type : none - Stored data types : Connection count - Expiration time : 30m - Size : 50k - Cookie name : none - Cookie length : none - Connection rate period : 60s - Session rate period : 60s - HTTP request rate period : 60s - HTTP error rate period : 60s - Bytes in rate period : 60s - Bytes out rate period : 60s ###### Basic Authentication - Enable : [ ] - Allowed Users : Nothing selected - Allowed Groups : Nothing selected ###### Tuning Options - Connection Timeout : 20s - Check Timeout : 10s - Server Timeout : 20s - Retries : 1 - Option pass-through : none - Default for server : none - Use Frontend port : [ ] - HTTP reuse : Always - Enable Caching : [ X ] ###### Rules - Select Rules : noneyet ###### Error Messages - Select Error Messages : Nothing selected ### Condition COND:HostEndsWith-dog_wolfspyre_io COND:HostMatches-dog_wolfspyre_io ### Rules RUL-AllowHTTPReq-EndsWith-dog_wolfspyre_io ### Health Check readiness check on TCP:7480 ### Backend Pool Ceph-S3-VIP-Pool ### Frontend Pools #### Internal pool #### External pool