Introduction⌗

Following on from the Ceph Upmap Balancer Lab this will run a similar test but using the crush-compat balancer mode instead. This can either be done as a standalone lab, or as a follow on to the Upmap lab.

Using the Ceph Octopus lab setup previously with RadosGW nodes, this will attempt to simulate a cluster where OSD utilisation is skewed. In the cluster, each node has an extra 50G OSD to help try and skew the usage percentages on the OSDs.

This is the current configuration of the cluster

Ceph Upman Test Cluster
Ceph crush-compat Test Cluster

Test Setup⌗

In summary, the cluster will have data added to it using s3cmd. There will be two buckets created, one for ISO’s and other large images, and another for loads of photos and other small files.

Requirements⌗


# ceph balancer off
# ceph balancer status
  • Example output. The key is to ensure "active": false
    
{
    "active": false,
    "last_optimize_duration": "0:00:00.012292",
    "last_optimize_started": "Fri Jan 15 09:50:40 2021",
    "mode": "upmap",
    "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect",
    "plans": []
}

Setup s3cmd⌗

  • Install s3cmd

# dnf install s3cmd


# cat <<EOF > ~/.ceph_s3cfg
[default]
access_key = 4IMSY2D3RPWW7VB7ECPB
secret_key = 7AuFEvU9HKa4WB6BjfuTlZEDv6t1oHKhQ01zmIDo
host_base = ceph-rgw01.ceph.lab
# If wildcard DNS is configured
# host_bucket = %(bucket)s.ceph-rgw01.ceph.lab
# If no wildcrds DNS
host_bucket = ceph-rgw01.ceph.lab
# If a proxy host is needed to reach the RGW nodes
#proxy_host = 192.168.0.15
#proxy_port = 8888
# If SSL has not been enabled on RGW
use_https = False
human_readable_sizes = True
EOF

Modify ~/.ceph_s3cfg as required. Alternatively modify ~/.s3cfg if this will be the only cluster s3cmd will connect to.

  • Create two buckets


# s3cmd -c ~/.ceph_s3cfg mb s3://isos-2
# s3cmd -c ~/.ceph_s3cfg mb s3://photos-2

Seed Test Data⌗

  • Sync local directories via s3cmd

This will probably take a while, depending on the amount of data, speed of the Ceph drives etc. May be useful to setup a custom Grafana Dashboard to ensure Crush-Compat is working as intended (if this has not already been done).


# s3cmd -c ~/.ceph_s3cfg sync /var/lib/libvirt/images/iso/ s3://isos-2/
# s3cmd -c ~/.ceph_s3cfg sync ~/Photos s3://photos-2/
  • Check files are accessible

# s3cmd -c ~/.ceph_s3cfg ls s3://isos-2/
2021-01-14 10:30    858783744  s3://isos-2/CentOS-7-x86_64-GenericCloud.qcow2
  • Check the size of the buckets

# s3cmd -c ~/.ceph_s3cfg du s3://isos-2/ 
12G      12 objects s3://isos-2/

Setup Custom Grafana Dashboard⌗

Using either the default Grafana setup or the monitoring set up here, log into Grafana as admin

  • Import a new dashboard from this JSON

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": "-- Grafana --",
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 0,
  "id": 16,
  "links": [],
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Dashboard1",
      "decimals": 0,
      "description": "Space used on each OSD represented as a percentage of total space available",
      "fill": 0,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 24,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 4,
      "legend": {
        "alignAsTable": true,
        "avg": false,
        "current": true,
        "max": false,
        "min": false,
        "rightSide": true,
        "show": true,
        "sort": "current",
        "sortDesc": true,
        "total": false,
        "values": true
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "(ceph_osd_stat_bytes_used{instance=~\"$mgr\"}  / ceph_osd_stat_bytes{instance=~\"$mgr\"} * 100)",
          "instant": false,
          "legendFormat": "{{ ceph_daemon }}",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "OSD Space Used",
      "tooltip": {
        "shared": true,
        "sort": 2,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": 25,
        "min": null,
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "percent",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "decimals": null,
          "format": "short",
          "label": "",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    },
    {
      "aliasColors": {},
      "bars": true,
      "dashLength": 10,
      "dashes": false,
      "datasource": "Dashboard1",
      "description": "Each bar indicates the number of OSD's that have a PG count in a specific range as shown on the x axis.",
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 8,
        "w": 24,
        "x": 0,
        "y": 7
      },
      "hiddenSeries": false,
      "id": 2,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
      },
      "lines": false,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "dataLinks": []
      },
      "percentage": false,
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "ceph_osd_numpg",
          "instant": true,
          "legendFormat": "PGs per OSD",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeRegions": [],
      "timeShift": null,
      "title": "Distribution of PGs per OSD",
      "tooltip": {
        "shared": false,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "buckets": 20,
        "min": null,
        "mode": "histogram",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "short",
          "label": "# of OSDs",
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "refresh": "10s",
  "schemaVersion": 22,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "allValue": null,
        "current": {
          "text": "ceph-mon02:9283",
          "value": "ceph-mon02:9283"
        },
        "datasource": "Dashboard1",
        "definition": "label_values(ceph_osd_stat_bytes_used, instance)",
        "hide": 0,
        "includeAll": false,
        "label": "mgr",
        "multi": false,
        "name": "mgr",
        "options": [],
        "query": "label_values(ceph_osd_stat_bytes_used, instance)",
        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
        "tagValuesQuery": "",
        "tags": [],
        "tagsQuery": "",
        "type": "query",
        "useTags": false
      }
    ]
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ]
  },
  "timezone": "",
  "title": "Upmap Dashboard",
  "uid": "yU-29mBMk",
  "version": 5
}
  • The new dashboard should look something like this
Grafana Dashboard
Grafana Dashboard

Enable Crush-Compat Balancer⌗

Steps below will be following the instructions from Ceph docs to enable the crush-compat balancer.

Current State of the Cluster⌗

With 3 OSDs above 85% utilisation and the lowest OSDs at ~50% utilisation. This is due to the lab being a follow on from the upmap balancer lab.

Pre CRUSH compat
OSD utilisation before crush-compat is enabled
  • Most utilised OSDs

# ceph osd df | sort -nr -k 17 | head -n 5
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META      AVAIL    %USE   VAR   PGS  STATUS
14    hdd  0.00490   1.00000  5.0 GiB  4.5 GiB  3.5 GiB  149 KiB  1024 MiB  510 MiB  90.04  1.58   16      up
22    hdd  0.00490   1.00000  5.0 GiB  4.4 GiB  3.4 GiB  164 KiB  1024 MiB  630 MiB  87.69  1.54   19      up
29    hdd  0.00490   1.00000  5.0 GiB  4.3 GiB  3.3 GiB  1.1 MiB  1023 MiB  708 MiB  86.16  1.52   20      up
12    hdd  0.00490   1.00000  5.0 GiB  4.2 GiB  3.2 GiB  661 KiB  1023 MiB  787 MiB  84.63  1.49   21      up
13    hdd  0.00490   1.00000  5.0 GiB  4.2 GiB  3.2 GiB  137 KiB  1024 MiB  829 MiB  83.79  1.47   18      up
  • Least utilised OSDs

# ceph osd df | sort -n -k 17 | head -n 7
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP     META      AVAIL    %USE   VAR   PGS  STATUS
34    hdd  0.04880   1.00000   50 GiB   25 GiB   24 GiB  1.9 MiB  1022 MiB   25 GiB  49.85  0.88  155      up
32    hdd  0.04880   1.00000   50 GiB   25 GiB   24 GiB  3.4 MiB  1021 MiB   25 GiB  50.50  0.89  142      up
24    hdd  0.00980   1.00000   10 GiB  5.1 GiB  4.1 GiB  206 KiB  1024 MiB  4.9 GiB  51.04  0.90   26      up
33    hdd  0.04880   1.00000   50 GiB   26 GiB   25 GiB  2.4 MiB  1022 MiB   24 GiB  51.29  0.90  148      up
38    hdd  0.04880   1.00000   50 GiB   26 GiB   25 GiB  1.5 MiB  1023 MiB   24 GiB  52.13  0.92  152      up

Starting the crush-compat Balancer⌗

  • Set the balancer mode to crush-compat

# ceph balancer mode crush-compat
  • Confirm this has been set

# ceph balancer status

  • Example output

    
{
    "active": false,
    "last_optimize_duration": "0:00:00.012292",
    "last_optimize_started": "Fri Jan 15 09:50:40 2021",
    "mode": "crush-compat",
    "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect",
    "plans": []
}

  • Start the balancer


# ceph balancer on
  • Check the balancer is running

# ceph balancer status
  • Example output
    
{
    "active": true,
    "last_optimize_duration": "0:00:00.446920",
    "last_optimize_started": "Fri Jan 15 13:42:59 2021",
    "mode": "crush-compat",
    "optimize_result": "Optimization plan created successfully",
    "plans": []
}

At this point the cluster will start rebalancing data. ceph -s will show PGs being remapped to optimise data placement. The grafana graph should start to show the OSD utilisation percentages averaging out.

  • Once the balancer has completed, it will show a message similar to below. All I/O was stopped on the cluster during the rebalance.

# ceph balancer status
  • Example output
    
{
    "active": true,
    "last_optimize_duration": "0:00:00.775223",
    "last_optimize_started": "Fri Jan 15 14:05:14 2021",
    "mode": "crush-compat",
    "optimize_result": "Unable to find further optimization, change balancer mode and retry might help",
    "plans": []
}

Summary⌗

Data utilisation across the cluster is a bit better, with the highest OSD usage at 87% and lowest at 49%.

Post crush-compat
Post crush-comapt OSD Utilisation

Although this doesn’t seem as efficient as the upmap balancer, it may be the test at fault for not starting from scratch, as when changing the balancer mode back to upmap produces this is the message:


{
    "active": true,
    "last_optimize_duration": "0:00:00.027171",
    "last_optimize_started": "Fri Jan 15 14:11:23 2021",
    "mode": "upmap",
    "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect",
    "plans": []
}

Considering it unlikely that there would be drives 10x larger than others in the same CRUSH hierarchy in a production environment, the distribution of data across the available OSDs appears fairly balanced.

Cleanup⌗

To clean up this lab, use the script here.