Akemi

Ceph集群配置方式介绍

2025/07/01

集群配置方式

ceph通过以下来源来获得它的配置 :

  • 编译默认值
  • 集中式数据库(推荐)
  • 本地主机上的配置文件(不推荐)
  • 环境变量
  • 命令行参数(推荐)
  • 运行时参数

集群配置文件ceph.conf

每个ceph节点都会存储一个本地集群配置文件,默认位置在/etc/ceph/ceph.conf

这是cephadm创建的初始ceph配置文件

示例文件/usr/share/doc/ceph/sample.ceph.conf

  • 全局配置 → 组件级配置 → 实例级配置
  • 下层配置覆盖上层同名配置
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
/usr/share/doc/ceph/sample.ceph.conf
去除掉注释与空行后基本是这样
[global] # ceph全局配置
;public network = 192.168.0.0/24
;cluster network = 192.168.0.0/24
pid file = /var/run/ceph/$name.pid
;max open files = 131072
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
;cephx require signatures = true
cephx cluster require signatures = true
cephx service require signatures = false
;keyring = /etc/ceph/$cluster.$name.keyring
;osd pool default size = 3
;osd pool default min size = 2
;osd pool default pg num = 128
;osd pool default pgp num = 128
;osd pool default crush rule = 0
;osd crush chooseleaf type = 1
;log file = /var/log/ceph/$cluster-$name.log
;log to syslog = true
;ms bind ipv6 = true
[mon] # MON全局配置
;mon initial members = mycephhost
;mon host = cephhost01,cephhost02
;mon addr = 192.168.0.101,192.168.0.102
;mon data = /var/lib/ceph/mon/$name
;mon clock drift allowed = .15
;mon osd full ratio = .95
;mon osd nearfull ratio = .85
;mon osd down out interval = 600
;mon osd report timeout = 300
;debug ms = 1
;debug mon = 20
;debug paxos = 20
;debug auth = 20
;[mon.alpha] # 多实例配置
; host = alpha
; mon addr = 192.168.0.10:6789
;[mon.beta]
; host = beta
; mon addr = 192.168.0.11:6789
;[mon.gamma]
; host = gamma
; mon addr = 192.168.0.12:6789
[mds]
;keyring = /var/lib/ceph/mds/$name/keyring
;mds standby replay = true
; mds logging to debug issues.
;debug ms = 1
;debug mds = 20
;debug journaler = 20
;mds cache memory limit = 2G
;[mds.alpha]
; host = alpha
;[mds.beta]
; host = beta
[osd]
;osd data = /var/lib/ceph/osd/$name
;osd recovery max active = 3
;osd max backfills = 5
;osd max scrubs = 2
;osd mkfs type = btrfs
;osd mount options btrfs = noatime,nodiratime
;osd journal = /var/lib/ceph/osd/$name/journal
;osd check for log corruption = true
;osd journal size = 2048 ; journal size, in megabytes
;journal dio = false
;debug ms = 1
;debug osd = 20
;debug filestore = 20
;debug journal = 20
;filestore max sync interval = 5
;filestore btrfs snap = false
;filestore flusher = true
;filestore queue max ops = 500
;filestore merge threshold = -10
;filestore split multiple = 2
;filestore op threads = 4
;osd crush update on start = false
;[osd.0]
; host = delta
;[osd.1]
; host = epsilon
;[osd.2]
; host = zeta
;[osd.3]
; host = eta
[client]
rbd cache = true
;rbd cache size = 33554432
;rbd cache max dirty = 25165824
;rbd cache target dirty = 16777216
;rbd cache max dirty age = 1.0
;rbd cache writethrough until flush = true
;admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok
[client.radosgw.gateway] # 对象存储网关配置
;rgw data = /var/lib/ceph/radosgw/$name
;host = ceph-radosgw
;keyring = /etc/ceph/ceph.client.radosgw.keyring
;rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
;log file = /var/log/ceph/client.radosgw.gateway.log
;rgw print continue = false
;rgw dns name = radosgw.ceph.internal

集中式数据库(常用)

  • 动态调整运行中集群的配置,修改配置后无需重启服务
  • 避免读取静态文件,直接查询运行时配置数据库
  • 配置变更自动同步到所有相关节点
  • 配置审计与版本控制
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
使用ceph config命令查询数据库,查看配置信息

ceph config ls 列出所有配置设置
ceph config ls | wc -l
2214

ceph config help "setting" 查看特定配置设置的帮助
ceph config help mon_host
mon_host - list of hosts or addresses to search for a monitor
(str, basic)
Default:
Can update at runtime: false
Services: [common]

This is a list of IP addresses or hostnames that are separated by commas, whitespace, or semicolons. Hostnames are resolved via DNS. All A and AAAA records are included in the search list.

ceph config dump 查看集群配置数据库设置
ceph config dump
WHO MASK LEVEL OPTION VALUE RO
global basic container_image quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f *
mon advanced auth_allow_insecure_global_id_reclaim false
mon advanced public_network 192.168.10.0/24 *
mgr advanced mgr/cephadm/container_init True *
mgr advanced mgr/cephadm/migration_current 6 *
mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://ansible:9093 *
mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false *
mgr advanced mgr/dashboard/GRAFANA_API_URL https://ansible:3000 *
mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://ansible:9095 *
mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY 80MD2T0S7FG1Z2GUR72Q *
mgr advanced mgr/dashboard/RGW_API_SECRET_KEY SifSg4rfkqeWbEsrjEqXqGyBtDmIrDTE0yKaDgLU *
mgr advanced mgr/dashboard/ssl_server_port 8443 *
mgr advanced mgr/orchestrator/orchestrator cephadm
osd advanced osd_memory_target_autotune true
client.rgw.realm.zone.cephadm-1.qfehta basic rgw_frontends beast port=80 *
client.rgw.realm.zone.cephadm-2.vluxwt basic rgw_frontends beast port=80 *
client.rgw.realm.zone.cephadm-3.ekrysr basic rgw_frontends beast port=80 *

ceph config show $type.$id 查看特定守护进行的数据库配置。使用show-with-defaults包含缺省设置

ceph config get $type.$id 获取特定的配置
ceph config set $type.$id 设置特定的配置

使用assimilation-conf子命令将文件中的配置应用到正在运行的集群中

使用服务配置文件

即之前使用cephadm bootstrap部署集群时,使用的/etc/ceph/initial-config-primary-cluster.yaml服务配置文件,一般来说都是用来调整集群整体配置的情况下

  • 支持滚动更新
  • 安全迁移 MON 或 MDS 等关键服务
  • 版本控制集群拓扑,快速重建/克隆集群环境

运行时状态覆盖配置设置(临时测试)

适合需要临时绕过常规配置的场景,允许管理员在不修改持久化配置的情况下,针对特定守护进程或当前状态进行动态调整

特性:

  • 仅支持单守护进程
  • 优先级最高
  • 重启守护进程即恢复
  • 支持通配符

适用场景:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
紧急故障排除
ceph daemon osd.0 config set debug_osd 20/20 *# 临时开启详细日志*

性能压测调优
ceph daemon osd.1 config set osd_recovery_max_active 0 *# 临时禁用恢复限流*

规避配置冲突
ceph tell mon.* injectargs '--mon_osd_full_ratio 0.97' # 调整OSD存储阈值到97%
绕过 [global] 或 [mon] 节的限制

A/B 测试
# 对半数 OSD 应用新参数
for i in {0..32}; do
if (( i % 2 == 0 )); then
ceph daemon osd.$i config set osd_recovery_sleep 0.05
fi
done

规避重启MDS服务
ceph tell mds.alpha config set mds_cache_memory_limit 8G
设置内存缓存上限为8GB,阻止其重启

使用方式

1
2
3
4
5
6
7
8
9
10
ceph tell <type>.<id> config命令

获取特定守护进程运行时的配置
ceph tell <type>.<id> config get

设定特定守护进程运行时的配置
ceph tell <type>.<id> config set

使用通配符
ceph tell osd.* config get debug_ms
CATALOG
  1. 1. 集群配置方式
  2. 2. 集群配置文件ceph.conf
  3. 3. 集中式数据库(常用)
  4. 4. 使用服务配置文件
  5. 5. 运行时状态覆盖配置设置(临时测试)