Skip to main content
  1. 2023/
  2. Posts from January/

Getting things in place

Back to the top...

Stuff yer gonna need: #

Proxmox Ceph Cluster #

It kinda goes without saying. If you want to use the Ceph storage amalgam provided with Proxmox to enable this, you’ll need a functional Proxmox cluster..

This means at least two, realistically three nodes.

A Front-End gateway. #

What’s going to respond to your HTTP(s) traffic?

Ceph’s RADOS gateway endpoint doesn’t provide any intrinsic request routing or load balancing. To maintain a durable front door, and reliable service, you’ll need a load balancer/front end.

How you facilitate this is up to you.

In my environment, I already have a functional, highly available HAProxy rig, backended by OPNsense… so I use that.


You’ll need DNS set up to point everything to the right place.

That will mean having:

Wildcard and A records #

I chose as the root subdomain. Since I want to facilitate my offsite hosts to be able to access this as well, I need to enable external and internal resolution of the endpoints.

External records #
Public-Facing DNS records:
dog   IN A
*.dog IN A
internal records #
NS entries for the new subdomain
dog IN NS
dog IN NS
dog IN NS sub-zone

@ IN A

skwirreltrap IN A atticus IN A evey IN A

px-m-40 IN A px-m-41 IN A px-m-42 IN A px-m-43 IN A px-m-44 IN A px-m-45 IN A

  •         IN A</code></pre>


  • At minimum, you’ll want a wildcard ssl certificate for your s3 apex. (in my case *

  • You may also want a wildcard ssl cert for s3 websites. But I’m not really sure (at the moment) how this works :)

Stuff ya may wanna read: #

I’m going to lay out RADOS in alignment with the failure boundaries already established within my existing cluster. You may have different needs.

flowchart RL
  subgraph rm0["fa:fa-bolt Realm (namespace)"]
    subgraph zzz["RADOS Traffic Flow"]
     direction LR 
    subgraph zg0["Zone Group: Barn" ]
        subgraph zzy["Note:"]
          zzya["Zone Groups contain one or more zones. They must have one master zone."]
          direction LR 
      subgraph z0["Zone - PXM Master" ]
        subgraph zzx["Note:"]
          zzxa["Zones define an isolation/replication boundary."]
          direction LR 
        subgraph n40["Physical host: px-m-40"]
          n40v198["Node 40 Ceph Network<br>"]
          r40a["RADOS OSG Process 40A<br>"]
        subgraph n41["Physical host: px-m-41"]
          n41v198["Node 41 Ceph Network<br>"]
          r41a["RADOS OSG Process 41A<br>"]
        subgraph n42["Physical host: px-m-42"]
          n42v198["Node 42 Ceph Network<br>"]
          r42a["RADOS OSG Process 42A<br>"]
        subgraph n43["Physical host: px-m-43"]
          n43v198["Node 43 Ceph Network<br>"]
          r43a["RADOS OSG Process 43A<br>"]
        subgraph n44["Physical host: px-m-44"]
          n44v198["Node 44 Ceph Network<br>"]
          r44a["RADOS OSG Process 44A<br>"]
        subgraph n45["Physical host: px-m-45"]
          n45v198["Node 45 Ceph Network<br>"]
          r45a["RADOS OSG Process 45A<br>"]
  subgraph world["public requests"]
    direction BT
    usera["User Requests"]
    userb["from outside"]
    userc["the cluster"]
  subgraph op["OPNSense Cluster"]
    direction BT
    subgraph OPNHAP["OPNSense HAProxy"]
      direction BT
      zvip0["https://* <br>"]
    opv1["OPNSense VIP Network<br>"]
    opv198["OPNSense CEPH Network <br>"]
    opv2["OPNSense Public Network"]
  r40a -.-> n40v198 --- n40 --> opv198 
  r41a -.-> n41v198 --- n41 --> opv198 
  r42a -.-> n42v198 --- n42 --> opv198 
  r43a -.-> n43v198 --- n43 --> opv198 
  r44a -.-> n44v198 --- n44 --> opv198 
  r45a -.-> n45v198 --- n45 --> opv198 
  opv198 -.-> opv1 -.-> zvip0 ===> r40a & r41a & r42a & r43a & r44a & r45a

   usera & userb & userc -.- world ---> opv2 -.- opv1 -.-> zvip0 ---> world