文档版本 v3.7-DRAFT 处于 草稿 状态。如需获取最新的稳定版文档，请参阅 v3.6。
将 etcd 从 v3.6 降级到 v3.5
关于将 etcd 从 v3.6 降级到 v3.5 的流程、清单和注意事项
通常情况下，从 etcd v3.6 降级到 v3.5 可以实现零停机、滚动式降级：
- 逐个停止 etcd v3.6 进程，并将其替换为 etcd v3.5 进程
- 启用降级后，v3.6 中的新功能将不再对集群可用
在开始降级之前，请通读本指南其余部分以做好准备。
降级检查清单
从 v3.6 到 v3.5 的主要变更（破坏性变化）：
命令行参数的差异
如果在 v3.6 配置中使用了以下任意参数，请确保在降级到 v3.5 时移除、重命名或更改其默认值。
注意： 此差异基于版本 v3.6.0 和 v3.5.18。实际差异取决于您使用的补丁版本，请先通过 diff <(etcd-3.6/bin/etcd -h | grep \\-\\-) <(etcd-3.5/bin/etcd -h | grep \\-\\-) 命令进行核对。
# flags not available in v3.5
-etcd --discovery-token ''
-etcd --discovery-endpoints ''
-etcd --discovery-dial-timeout '2s'
-etcd --discovery-request-timeout '5s'
-etcd --discovery-keepalive-time '2s'
-etcd --discovery-keepalive-timeout '6s'
-etcd --discovery-insecure-transport 'true'
-etcd --discovery-insecure-skip-tls-verify 'false'
-etcd --discovery-cert ''
-etcd --discovery-key ''
-etcd --discovery-cacert ''
-etcd --discovery-user ''
-etcd --discovery-password ''
-etcd --feature-gates
-etcd --log-format
# same flag with different names
-etcd --bootstrap-defrag-threshold-megabytes
+etcd --experimental-bootstrap-defrag-threshold-megabytes
-etcd --compaction-batch-limit
+etcd --experimental-compaction-batch-limit
-etcd --compact-hash-check-time
+etcd --experimental-compact-hash-check-time
-etcd --compaction-sleep-interval
+etcd --experimental-compaction-sleep-interval
-etcd --corrupt-check-time
+etcd --experimental-corrupt-check-time
-etcd --enable-distributed-tracing
+etcd --experimental-enable-distributed-tracing
-etcd --distributed-tracing-address
+etcd --experimental-distributed-tracing-address
-etcd --distributed-tracing-instance-id
+etcd --experimental-distributed-tracing-instance-id
-etcd --distributed-tracing-sampling-rate
+etcd --experimental-distributed-tracing-sampling-rate
-etcd --distributed-tracing-service-name
+etcd --experimental-distributed-tracing-service-name
-etcd --downgrade-check-time
+etcd --experimental-downgrade-check-time
-etcd --max-learners
+etcd --experimental-max-learners
-etcd --memory-mlock
+etcd --experimental-memory-mlock
-etcd --peer-skip-client-san-verification
+etcd --experimental-peer-skip-client-san-verification
-etcd --snapshot-catchup-entries
+etcd --experimental-snapshot-catchup-entries
-etcd --warning-apply-duration
+etcd --experimental-warning-apply-duration
-etcd --warning-unary-request-duration
+etcd --experimental-warning-unary-request-duration
-etcd --watch-progress-notify-interval
+etcd --experimental-watch-progress-notify-interval
# equivalent flags of v3.6 feature gates
-etcd --feature-gates=CompactHashCheck=true
+etcd --experimental-compact-hash-check-enabled=true
-etcd --feature-gates=InitialCorruptCheck=true
+etcd --experimental-enable-initial-corrupt-check=true
-etcd --feature-gates=LeaseCheckpoint=true
+etcd --experimental-enable-lease-checkpoint=true
-etcd --feature-gates=LeaseCheckpointPersist=true
+etcd --experimental-enable-lease-checkpoint-persist=true
-etcd --feature-gates=StopGRPCServiceOnDefrag=true
+etcd --experimental-stop-grpc-service-on-defrag=true
-etcd --feature-gates=TxnModeWriteWithSharedBuffer=false
+etcd --experimental-txn-mode-write-with-shared-buffer=false
# same flag different defaults
-etcd --snapshot-count=10000
+etcd --snapshot-count=100000
-etcd --v2-deprecation='write-only'
+etcd --v2-deprecation='not-yet'
-etcd --discovery-fallback='exit'
+etcd --discovery-fallback='proxy'
Prometheus 指标的变化
# metrics not available in v3.5
-etcd_network_known_peers
-etcd_server_feature_enabled
服务器降级检查清单
降级要求
为了确保滚动降级顺利进行，运行中的集群必须处于健康状态。在继续操作前，请使用 etcdctl endpoint health 命令检查集群健康状况。
准备工作
在降级 etcd 之前，务必先在预发布环境中测试依赖 etcd 的服务，然后再将降级部署到生产环境。
开始之前，下载快照备份。如果降级过程中出现问题，可以使用此备份回滚至当前的 etcd 版本。
开始之前，下载 etcd v3.5 的最新发布版本。
混合版本
降级期间，etcd 集群支持不同版本的成员共存，并以最低公共版本的协议运行。一旦通过 etcdctl downgrade enable 3.5 启用降级，即视为完成降级。内部会将整个集群版本设置为目标降级版本，用于控制报告的版本号和支持的功能。
回滚
在降级 etcd 集群之前，请创建并下载 etcd 集群的快照备份。如有需要，可使用该快照将集群恢复到降级前的状态。如果用户在降级过程中遇到问题，应首先识别并解决根本原因。
如果在运行 etcdctl downgrade enable 后已开始降级，且集群仍处于混合版本状态——即至少有一个成员仍在 v3.6 上运行，则用户可通过运行 etcdctl downgrade cancel 取消正在进行的降级过程，并使用原始的 v3.6 二进制文件重新启动所有已降级的成员。
当所有成员都已降级到 v3.5 后，集群被视为完全降级。如果用户希望在完整降级完成后返回原始版本，则应遵循官方的升级指南，以确保一致性并避免数据损坏。
降级流程
本示例演示如何在本地机器上将一个由 3 个成员组成的 v3.6 etcd 集群进行降级。
步骤 1：检查降级要求
集群是否健康且正在运行 v3.6.x？
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint health
<<COMMENT
localhost:2379 is healthy: successfully committed proposal: took = 2.118638ms
localhost:22379 is healthy: successfully committed proposal: took = 3.631388ms
localhost:32379 is healthy: successfully committed proposal: took = 2.157051ms
COMMENT
curl http://localhost:2379/version
<<COMMENT
{"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
COMMENT
curl http://localhost:22379/version
<<COMMENT
{"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
COMMENT
curl http://localhost:32379/version
<<COMMENT
{"etcdserver":"3.6.0-alpha.0","etcdcluster":"3.6.0","storage":"3.6.0"}
COMMENT
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 2 | 10 | 10 | | | false |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 10 | 10 | | | false |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.6.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 10 | 10 | | | false |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
步骤 2：从主节点下载快照备份
下载快照备份，以便在出现问题时提供降级恢复路径。
步骤 3：验证降级目标版本
在启用降级之前，请先验证降级目标版本：
- 我们仅支持一次降级一个次版本。例如，不允许从 v3.6 降级到 v3.4。
- 请在验证成功之前不要进入下一步。
etcdctl downgrade validate 3.5
<<COMMENT
Downgrade validate success, cluster version 3.6
COMMENT
步骤 4：启用降级
etcdctl downgrade enable 3.5
<<COMMENT
Downgrade enable success, cluster version 3.6
COMMENT
启用降级后，集群将开始使用 v3.5 协议（即降级目标版本）。此外，etcd 将自动迁移 schema 至目标降级版本，此过程通常非常迅速。在进入下一步之前，请通过检查端点状态确认所有服务器的存储版本均已迁移到 v3.5。
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 2 | 12 | 12 | | 3.5.0 | true |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 12 | 12 | | 3.5.0 | true |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 12 | 12 | | 3.5.0 | true |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
注意：一旦启用降级，即使所有服务器仍在运行 v3.6 二进制文件，集群也将继续使用 v3.5 协议运行，除非使用 etcdctl downgrade cancel 取消降级。
步骤 5：停止一个现有的 etcd 服务器
在停止服务器之前，请检查其是否为 Leader。我们建议最后再降级 Leader 节点。
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 2 | 12 | 12 | | 3.5.0 | true |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 12 | 12 | | 3.5.0 | true |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 2 | 12 | 12 | | 3.5.0 | true |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
如果待停止的服务器是 Leader，可以在停止前使用 move-leader 命令将 Leader 角色转移到其他服务器，以避免部分停机时间。
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 move-leader 91bc3c398fb3c146
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 13 | 13 | | 3.5.0 | true |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 3 | 13 | 13 | | 3.5.0 | true |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 13 | 13 | | 3.5.0 | true |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
每当一个 etcd 进程停止时，集群中的其他成员会记录预期的错误日志。这是正常现象，因为集群成员之间的连接已（临时）中断。
{"level":"warn","ts":"2025-02-28T17:35:43.795069Z","caller":"etcdserver/cluster_util.go:259","msg":"failed to reach the peer URL","address":"http://127.0.0.1:12380/version","remote-member-id":"8211f1d0f64f3269","error":"Get \"http://127.0.0.1:12380/version\": dial tcp 127.0.0.1:12380: connect: connection refused"}
{"level":"warn","ts":"2025-02-28T17:35:43.795149Z","caller":"etcdserver/cluster_util.go:160","msg":"failed to get version","remote-member-id":"8211f1d0f64f3269","error":"Get \"http://127.0.0.1:12380/version\": dial tcp 127.0.0.1:12380: connect: connection refused"}
{"level":"warn","ts":"2025-02-28T17:35:44.368651Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"8211f1d0f64f3269","rtt":"483.01µs","error":"dial tcp 127.0.0.1:12380: connect: connection refused"}
{"level":"warn","ts":"2025-02-28T17:35:44.368726Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"8211f1d0f64f3269","rtt":"735.659µs","error":"dial tcp 127.0.0.1:12380: connect: connection refused"}
步骤 6：使用相同的配置重启 etcd 服务器（移除或替换 v3.5 中已废弃或变更的参数）
使用相同的配置但替换为新的 etcd 二进制文件来重启 etcd 服务器。
-etcd-3.6/bin --name s1 \
+etcd-3.5/bin --name s1 \
--data-dir /tmp/etcd/s1 \
--listen-client-urls http://localhost:2379 \
--advertise-client-urls http://localhost:2379 \
--listen-peer-urls http://localhost:2380 \
--initial-advertise-peer-urls http://localhost:2380 \
--initial-cluster s1=http://localhost:2380,s2=http://localhost:22380,s3=http://localhost:32380 \
--initial-cluster-token tkn \
--initial-cluster-state existing
验证每个成员以及整个集群在使用新的 v3.5 etcd 二进制文件后是否恢复正常运行：
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.5.18 | | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 14 | 14 | | | false |
| localhost:22379 | 91bc3c398fb3c146 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | true | false | 3 | 14 | 14 | | 3.5.0 | true |
| localhost:32379 | fd422379fda50e48 | 3.6.0-alpha.0 | 3.5.0 | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 14 | 14 | | 3.5.0 | true |
+-----------------+------------------+---------------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
<<COMMENT
localhost:22379 is healthy: successfully committed proposal: took = 4.650967ms
localhost:2379 is healthy: successfully committed proposal: took = 4.634377ms
localhost:32379 is healthy: successfully committed proposal: took = 5.047777ms
COMMENT
注意：你会看到 v3.5 服务器的 DOWNGRADE ENABLED 显示为 false，这是因为降级信息在 v3.5 的状态端点中未实现；此时集群的降级仍处于启用状态。
步骤 7：对剩余成员重复 步骤 5 和 步骤 6
当所有成员都完成降级后，检查集群的健康状况和状态，确认所有成员的次版本均为 v3.5，且存储版本为空：
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint status -w=table
<<COMMENT
+-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| ENDPOINT | ID | VERSION | STORAGE VERSION | DB SIZE | IN USE | PERCENTAGE NOT IN USE | QUOTA | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | DOWNGRADE TARGET VERSION | DOWNGRADE ENABLED |
+-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
| localhost:2379 | 8211f1d0f64f3269 | 3.5.18 | | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 26 | 26 | | | false |
| localhost:22379 | 91bc3c398fb3c146 | 3.5.18 | | 20 kB | 16 kB | 20% | 0 B | true | false | 3 | 26 | 26 | | | false |
| localhost:32379 | fd422379fda50e48 | 3.5.18 | | 20 kB | 16 kB | 20% | 0 B | false | false | 3 | 26 | 26 | | | false |
+-----------------+------------------+---------+-----------------+---------+--------+-----------------------+-------+-----------+------------+-----------+------------+--------------------+--------+--------------------------+-------------------+
COMMENT
etcdctl endpoint health --endpoints=localhost:2379,localhost:22379,localhost:32379
<<COMMENT
localhost:22379 is healthy: successfully committed proposal: took = 4.650967ms
localhost:2379 is healthy: successfully committed proposal: took = 4.634377ms
localhost:32379 is healthy: successfully committed proposal: took = 5.047777ms
COMMENT
curl http://localhost:2379/version
<<COMMENT
{"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
COMMENT
curl http://localhost:22379/version
<<COMMENT
{"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
COMMENT
curl http://localhost:32379/version
<<COMMENT
{"etcdserver":"3.5.18","etcdcluster":"3.5.0"}
COMMENT
在 Leader 节点的日志中，你应该能看到类似以下的消息：
{"level":"info","ts":"2025-02-28T17:59:50.019862Z","caller":"etcdserver/server.go:2749","msg":"the cluster has been downgraded","cluster-version":"3.5.0"}