集群配置方式 ceph通过以下来源来获得它的配置 :
编译默认值
集中式数据库(推荐)
本地主机上的配置文件(不推荐)
环境变量
命令行参数(推荐)
运行时参数
集群配置文件ceph.conf 每个ceph节点都会存储一个本地集群配置文件,默认位置在/etc/ceph/ceph.conf
这是cephadm创建的初始ceph配置文件
示例文件/usr/share/doc/ceph/sample.ceph.conf
全局配置 → 组件级配置 → 实例级配置
下层配置覆盖上层同名配置
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 /usr/share/doc/ceph/sample.ceph.conf 去除掉注释与空行后基本是这样 [global] ;public network = 192.168.0.0/24 ;cluster network = 192.168.0.0/24 pid file = /var/run/ceph/$name .pid ;max open files = 131072 auth cluster required = cephx auth service required = cephx auth client required = cephx ;cephx require signatures = true cephx cluster require signatures = true cephx service require signatures = false ;keyring = /etc/ceph/$cluster .$name .keyring ;osd pool default size = 3 ;osd pool default min size = 2 ;osd pool default pg num = 128 ;osd pool default pgp num = 128 ;osd pool default crush rule = 0 ;osd crush chooseleaf type = 1 ;log file = /var/log/ceph/$cluster -$name .log ;log to syslog = true ;ms bind ipv6 = true [mon] ;mon initial members = mycephhost ;mon host = cephhost01,cephhost02 ;mon addr = 192.168.0.101,192.168.0.102 ;mon data = /var/lib/ceph/mon/$name ;mon clock drift allowed = .15 ;mon osd full ratio = .95 ;mon osd nearfull ratio = .85 ;mon osd down out interval = 600 ;mon osd report timeout = 300 ;debug ms = 1 ;debug mon = 20 ;debug paxos = 20 ;debug auth = 20 ;[mon.alpha] ; host = alpha ; mon addr = 192.168.0.10:6789 ;[mon.beta] ; host = beta ; mon addr = 192.168.0.11:6789 ;[mon.gamma] ; host = gamma ; mon addr = 192.168.0.12:6789 [mds] ;keyring = /var/lib/ceph/mds/$name /keyring ;mds standby replay = true ; mds logging to debug issues. ;debug ms = 1 ;debug mds = 20 ;debug journaler = 20 ;mds cache memory limit = 2G ;[mds.alpha] ; host = alpha ;[mds.beta] ; host = beta [osd] ;osd data = /var/lib/ceph/osd/$name ;osd recovery max active = 3 ;osd max backfills = 5 ;osd max scrubs = 2 ;osd mkfs type = btrfs ;osd mount options btrfs = noatime,nodiratime ;osd journal = /var/lib/ceph/osd/$name /journal ;osd check for log corruption = true ;osd journal size = 2048 ; journal size, in megabytes ;journal dio = false ;debug ms = 1 ;debug osd = 20 ;debug filestore = 20 ;debug journal = 20 ;filestore max sync interval = 5 ;filestore btrfs snap = false ;filestore flusher = true ;filestore queue max ops = 500 ;filestore merge threshold = -10 ;filestore split multiple = 2 ;filestore op threads = 4 ;osd crush update on start = false ;[osd.0] ; host = delta ;[osd.1] ; host = epsilon ;[osd.2] ; host = zeta ;[osd.3] ; host = eta [client] rbd cache = true ;rbd cache size = 33554432 ;rbd cache max dirty = 25165824 ;rbd cache target dirty = 16777216 ;rbd cache max dirty age = 1.0 ;rbd cache writethrough until flush = true ;admin socket = /var/run/ceph/$cluster -$type .$id .$pid .$cctid .asok [client.radosgw.gateway] ;rgw data = /var/lib/ceph/radosgw/$name ;host = ceph-radosgw ;keyring = /etc/ceph/ceph.client.radosgw.keyring ;rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock ;log file = /var/log/ceph/client.radosgw.gateway.log ;rgw print continue = false ;rgw dns name = radosgw.ceph.internal
集中式数据库(常用)
动态调整运行中集群的配置,修改配置后无需重启服务
避免读取静态文件,直接查询运行时配置数据库
配置变更自动同步到所有相关节点
配置审计与版本控制
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 使用ceph config命令查询数据库,查看配置信息 ceph config ls 列出所有配置设置 ceph config ls | wc -l 2214 ceph config help "setting" 查看特定配置设置的帮助 ceph config help mon_host mon_host - list of hosts or addresses to search for a monitor (str, basic) Default: Can update at runtime: false Services: [common] This is a list of IP addresses or hostnames that are separated by commas, whitespace, or semicolons. Hostnames are resolved via DNS. All A and AAAA records are included in the search list. ceph config dump 查看集群配置数据库设置 ceph config dump WHO MASK LEVEL OPTION VALUE RO global basic container_image quay.io/ceph/ceph@sha256:a0f373aaaf5a5ca5c4379c09da24c771b8266a09dc9e2181f90eacf423d7326f * mon advanced auth_allow_insecure_global_id_reclaim false mon advanced public_network 192.168.10.0/24 * mgr advanced mgr/cephadm/container_init True * mgr advanced mgr/cephadm/migration_current 6 * mgr advanced mgr/dashboard/ALERTMANAGER_API_HOST http://ansible:9093 * mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false * mgr advanced mgr/dashboard/GRAFANA_API_URL https://ansible:3000 * mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://ansible:9095 * mgr advanced mgr/dashboard/RGW_API_ACCESS_KEY 80MD2T0S7FG1Z2GUR72Q * mgr advanced mgr/dashboard/RGW_API_SECRET_KEY SifSg4rfkqeWbEsrjEqXqGyBtDmIrDTE0yKaDgLU * mgr advanced mgr/dashboard/ssl_server_port 8443 * mgr advanced mgr/orchestrator/orchestrator cephadm osd advanced osd_memory_target_autotune true client.rgw.realm.zone.cephadm-1.qfehta basic rgw_frontends beast port=80 * client.rgw.realm.zone.cephadm-2.vluxwt basic rgw_frontends beast port=80 * client.rgw.realm.zone.cephadm-3.ekrysr basic rgw_frontends beast port=80 * ceph config show $type .$id 查看特定守护进行的数据库配置。使用show-with-defaults包含缺省设置 ceph config get $type .$id 获取特定的配置 ceph config set $type .$id 设置特定的配置 使用assimilation-conf子命令将文件中的配置应用到正在运行的集群中
使用服务配置文件 即之前使用cephadm bootstrap部署集群时,使用的/etc/ceph/initial-config-primary-cluster.yaml服务配置文件,一般来说都是用来调整集群整体配置的情况下
支持滚动更新
安全迁移 MON 或 MDS 等关键服务
版本控制集群拓扑,快速重建/克隆集群环境
运行时状态覆盖配置设置(临时测试) 适合需要临时绕过常规配置的场景,允许管理员在不修改持久化配置的情况下,针对特定守护进程或当前状态进行动态调整
特性:
仅支持单守护进程
优先级最高
重启守护进程即恢复
支持通配符
适用场景:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 紧急故障排除 ceph daemon osd.0 config set debug_osd 20/20 * 性能压测调优 ceph daemon osd.1 config set osd_recovery_max_active 0 * 规避配置冲突 ceph tell mon.* injectargs '--mon_osd_full_ratio 0.97' 绕过 [global] 或 [mon] 节的限制 A/B 测试 for i in {0..32}; do if (( i % 2 == 0 )); then ceph daemon osd.$i config set osd_recovery_sleep 0.05 fi done 规避重启MDS服务 ceph tell mds.alpha config set mds_cache_memory_limit 8G 设置内存缓存上限为8GB,阻止其重启
使用方式
1 2 3 4 5 6 7 8 9 10 ceph tell <type >.<id > config命令 获取特定守护进程运行时的配置 ceph tell <type >.<id > config get 设定特定守护进程运行时的配置 ceph tell <type >.<id > config set 使用通配符 ceph tell osd.* config get debug_ms