etcd更换磁盘

字数统计: 802阅读时长: 4 min

 2026/07/29 

起因是通过Grafana仪表盘查看到集群etcd频繁进行出现切换的情况changes(etcd_server_leader_changes_seen_total{cluster=~“$cluster”}[1d])
怀疑是磁盘性能不足导致etcd频繁切换主节点
所以把磁盘从机械换成SSD，etcd挂载的是一块单独的分区，隶属于vgs k8s的lv etcd

当前使用的是/dev/mapper/k8s-etcd即vdb的分卷，现在准备改成vdc

lsblk
NAME                                             MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                                               11:0    1  470K  0 rom  
vda                                              252:0    0   50G  0 disk 
├─vda1                                           252:1    0    1G  0 part /boot
├─vda2                                           252:2    0   19G  0 part 
│ ├─bigcloud--enterprise--linux--for--euler-root 253:0    0   47G  0 lvm  /
│ └─bigcloud--enterprise--linux--for--euler-swap 253:1    0    2G  0 lvm  
└─vda3                                           252:3    0   30G  0 part 
  └─bigcloud--enterprise--linux--for--euler-root 253:0    0   47G  0 lvm  /
vdb                                              252:16   0  300G  0 disk 
├─k8s-etcd                                       253:2    0   50G  0 lvm  /var/lib/etcd
├─k8s-kubelet                                    253:3    0   50G  0 lvm  /var/lib/kubelet
└─k8s-containerd                                 253:4    0  200G  0 lvm  /var/lib/containerd
vdc                                              252:32   0   50G  0 disk 
└─vdc1                                           252:33   0   50G  0 part 
  └─etcd--new-etcd--new                          253:5    0   50G  0 lvm

实施前etcd备份

master节点上进行etcd备份作为保底——如果最后etcd集群意外崩溃了，可以使用这一份进行恢复

sh /etc/kubernetes/etcd/backup-etcd.sh
#!/bin/sh

bashpath=$(cd `dirname $0`; pwd)

now=`date +%Y%m%d_%H_%M_%S`

backpath=/var/lib/containerd/etcdbackup
respath=/var/lib/etcd/
mkdir -p /var/lib/containerd/etcdbackup

etcdendpoints="localhost:2379"
etcdcmd="/usr/bin/etcdctl"
backupfile="$now.snapshot.db"

cp -rf  $bashpath/etcdctl /usr/bin/
chmod +x /usr/bin/etcdctl
export ETCDCTL_API=3
$etcdcmd --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key  --endpoints $etcdendpoints snapshot save $backpath/$backupfile


if [ `whoami` != 'root' ];then
    echo "Must be root run this scripts!!" >> /var/log/backup/messages
    exit
fi

ago=`date -d "-30 day" +%Y%m%d`
if [ ! -d $backpath ];then
    echo "This path [${$backpath}] not exist, please check." >> /var/log/backup/messages
    exit
fi
for i in `ls $backpath`
do
# Get datestamp and check it. For example: 20160526_11_00_00.bak.tar.gz
    datestamp=`echo $i | awk -F'_' '{print $1}'`
    check=`echo "$datestamp" | grep "^[0-9]\{8\}$"`
    if [ "$check" != '' ];then
        # Remove old files.
        if [ "$datestamp" -lt "$ago" ];then
                /bin/rm -rf $backpath/$i
        fi
    fi
done

etcd重启与磁盘更换

3个master节点依次执行，凭借高可用能力轮流进行切换挂载

# 新磁盘分区
fdisk /dev/vdc
p
n
w
pvcreate /dev/vdc1
vgcreate etcd-new /dev/vdc1
lvcreate -n etcd-new -l 100%FREE etcd-new
# 格式化
mkfs.xfs /dev/mapper/etcd--new-etcd--new
# 拿到UUID
blkid | grep etcd--new
/dev/mapper/etcd--new-etcd--new: UUID="8022f448-60ea-4b57-a896-48adadf431c3" BLOCK_SIZE="512" TYPE="xfs"

# 停etcd
mkdir -p /data/etcdbackup/{yaml,data}
# 停止etcd
mv /etc/kubernetes/manifests/etcd.yaml /data/etcdbackup/yaml/
kubectl -n kube-system get po -owide | grep etcd
# 备份数据
cp -a /var/lib/etcd/* /data/etcdbackup/data/

# 修改fstab的挂载，注释掉原本的挂载，然后添加新挂载
umount /var/lib/etcd
cat /etc/fstab
#UUID=d93f8875-0980-4c6a-9850-2d8cb6cd1df2 /var/lib/etcd xfs defaults 1 2
UUID=8022f448-60ea-4b57-a896-48adadf431c3 /var/lib/etcd xfs defaults 1 2

mount -a

# 恢复原有数据
cp -a /data/etcdbackup/data/* /var/lib/etcd/
mv /data/etcdbackup/yaml/etcd.yaml /etc/kubernetes/manifests/

集群状态验证

export ETCDCTL_API=3
etcdctl -w table --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --endpoints=https://xxx:2379,https://xxx:2379,https://xxx:2379 endpoint status

# 检查是否存在告警 
etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key --endpoints=https://xxx:2379,https://xxx:2379,https://xxx:2379 alarm list

截图由于可能涉敏就不发了

原文作者：王盛

原文链接：https://akemi.zj.cn/2026/07/29/Etcd-Replace-Disk/

发表日期：July 29th 2026, 5:20:00 pm

更新日期：July 29th 2026, 5:22:28 pm