1、Codis 集群演化与 Redis 异步迁移Codis 作者 王乃峥自我介绍王乃峥 spinlock9Codis 作者之一曾就职于豌豆荚、今日头条从事 NoSQL 的设计与实践GitHub:Codis 集群演化背景与挑战Redis+Twemproxy2012 2015Redis Cluster2010 20152015 Present背景与挑战访问方式兼容所有（语言）客户端性能高吞吐、低延迟容量支持 GB 到 TB 的水平扩展能力高可用2014.7 2015.2Codis 1.9+Redis 2.8.13开源2015.3 2015.7Codis 2.0+Redis 2.8.21迭代2015.1

2、2 PresentCodis 3.x+Redis 3.2.11完善发展历程FeatureRedis ClusterCodis扩容缩容YESYES平滑扩容缩容NOYESPipelineNOYESDatabaseNOYESPasswordNOYESMGET/MSETSLOTYESPUB/SUBSLOTNOTransactionSLOTNO方案比较架构设计架构设计优势架构简单模块化设计稳定性强扩展性强易于维护劣势组件过多、部署复杂中间件性能损失牺牲部分功能高吞吐指令流水实现效率+优化 GC特性介绍并发多连接单连接 Redis多连接 SSD(RocksDB)特性介绍多 DB 支持默认 16 个 Dat

3、abase访问控制指令黑名单SessionAuth Proxy 独立配置ProductAuth Codis 集群共享特性介绍读写分离 跨机房优化牺牲一致性写：写主读：同 IP 同 DC 跨 DC特性介绍高可用特性介绍兼容 Redis升级到 Redis-4.2做到最小修改兼容分片策略未来规划完善 Codis完善 K8S 支持提升自动化程度实现 Transaction 等集成 Sentinel 功能更大的副本容量单副本 100TB 以上更高的集群吞吐QPS:500 1000w/s更复杂的访问模式普通业务 低延迟数据平台 高吞吐更大的挑战Codis+RocksDB开源方案 Pika Qihoo360

4、/PikaRedis 异步迁移RedisCodis同步迁移MIGRATE host port key|destination-dbtimeout COPY REPLACE KEYS keyRESTORE key ttl serialized-value REPLACESLOTSMGRTONEhost port timeout keySLOTSMGRTTAGONEhost port timeout keySLOTSMGRTSLOThost port timeout slotSLOTSMGRTTAGSLOT host port timeout slotSLOTSRESTOREkey ttl val

5、 key ttl val.实际使用存在哪些问题？同步迁移SourceTargetredis-cliENCODEDECODEDELETE MIGRATEMIGRATERESTORERESTOREOKOKOKOK(1)(1)(2)(2)(4)(4)(3)(3)反序列化本地删除序列化同步传输同步阻塞存在哪些问题？降低服务质量阻塞服务误触 HA、丢失数据同步迁移SourceTarget序列化网络反序列化本地删除CPUQPS存在哪些问题？限制数据规模实际建议 2MB（RESP 限制 多个简单、高效指令平摊 CPU 开销处理读写冲突、错误恢复批量迁移（Batch）异步删除BIO 线程解决思路指令分解支持

6、AUTH、SELECT 指令指令翻译SET(String/Encoded)ZADD/SADD/HMSETRPUSH平摊开销 单指令 us 级别异步迁移O(NlogN)O(N)SourceTargetredis-cliMIGRATEMIGRATE-ASYNCASYNCOKOK10M 10M ZsetZsetBIO ThreadBIO ThreadAUTH PASSWDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY EXPIREEXPIRE*KEY TTLKEY TTLcursorcursorZADDZADD*KEY SCORE2 KEY SCORE2 MEM

7、BER2MEMBER2异步删除ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1MEMBER1异步 IO+指令流水异步迁移SourceTargetAUTH PASSWDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY EXPIREEXPIRE*KEY TTLKEY TTLZADDZADD*KEY SCORE2 KEY SCORE2 MEMBER2MEMBER2ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1MEMBER1SourceTargetAUTH PASSWDAUTH PASSWDSELECT DB

8、SELECT DBDELDEL*KEYKEY EXPIREEXPIRE*KEY TTLKEY TTLZADDZADD*KEY SCORE2 KEY SCORE2 MEMBER2MEMBER2ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1MEMBER1 等价异步 IO+指令流水连接复用流量控制 发送窗口批量迁移（Batch）异步迁移SourceTargetredis-cliMIGRATEMIGRATE-ASYNCASYNCOKOK10M 10M ZsetZsetAUTH PASSWDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY

9、 EXPIREEXPIRE*KEY TTLKEY TTLcursorcursorZADDZADD*KEY SCORE2 KEY SCORE2 MEMBER2MEMBER2异步删除ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1MEMBER1BIO ThreadBIO Thread脏数据清理-错误处理主动清理过期清理异步迁移SourceTargetredis-cliMIGRATEMIGRATE-ASYNCASYNCOKOK10M 10M ZsetZsetAUTH PASSWDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY EXPI

10、REEXPIRE*KEY KEY TTLTTLcursorcursorZADDZADD*KEY SCORE2 KEY SCORE2 MEMBER2 with MEMBER2 with RTT*3RTT*3异步删除ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1 with MEMBER1 with RTT*3RTT*3BIO ThreadBIO Thread迁移状态机发送窗口初始化确定传输模式：简单 or 分片？异步迁移SourceTargetredis-cliMIGRATEMIGRATE-ASYNCASYNCOKOK10M 10M ZsetZsetAUTH PASS

11、WDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY EXPIREEXPIRE*KEY TTLKEY TTLcursorcursor异步删除SPDCEZADDZADD*KEY SCORE2 KEY SCORE2 MEMBER2 with MEMBER2 with RTT*3RTT*3ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1 with MEMBER1 with RTT*3RTT*3BIO ThreadBIO Thread迁移状态机连接初始化（Start）REPLACE 语义异步迁移SourceTargetredis-cliM

12、IGRATEMIGRATE-ASYNCASYNCOKOK10M 10M ZsetZsetAUTH PASSWDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY EXPIREEXPIRE*KEY TTLKEY TTLcursorcursor异步删除SPDCEZADDZADD*KEY SCORE2 KEY SCORE2 MEMBER2 with MEMBER2 with RTT*3RTT*3ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1 with MEMBER1 with RTT*3RTT*3BIO ThreadBIO Thread

13、迁移状态机发送分片（Chunked）临时 TTL=RTT x 3异步迁移SourceTargetredis-cliMIGRATEMIGRATE-ASYNCASYNCOKOK10M 10M ZsetZsetAUTH PASSWDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY EXPIREEXPIRE*KEY TTLKEY TTLcursorcursorZADDZADD*KEY SCORE2 KEY SCORE2 MEMBER2 with MEMBER2 with RTT*3RTT*3异步删除ZADDZADD*KEY SCORE1 KEY SCORE1 MEM

14、BER1 with MEMBER1 with RTT*3RTT*3SPDCEBIO ThreadBIO Thread迁移状态机迁移完成重置 TTL异步迁移SourceTargetredis-cliMIGRATEMIGRATE-ASYNCASYNCOKOK10M 10M ZsetZsetAUTH PASSWDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY EXPIREEXPIRE*KEY KEY TTLTTLcursorcursorZADDZADD*KEY SCORE2 KEY SCORE2 MEMBER2 with MEMBER2 with RTT*3RT

15、T*3异步删除ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1 with MEMBER1 with RTT*3RTT*3SPDCEBIO ThreadBIO Thread迁移状态机标记完成（Done）异步删除异步迁移SourceTargetredis-cliMIGRATEMIGRATE-ASYNCASYNCOKOK10M 10M ZsetZsetAUTH PASSWDAUTH PASSWDSELECT DBSELECT DBDELDEL*KEYKEY EXPIREEXPIRE*KEY KEY TTLTTLcursorcursorZADDZADD*KEY SCORE2

16、 KEY SCORE2 MEMBER2 with MEMBER2 with RTT*3RTT*3异步删除ZADDZADD*KEY SCORE1 KEY SCORE1 MEMBER1 with MEMBER1 with RTT*3RTT*3SPDCEBIO ThreadBIO Thread迁移间隙Key Missing迁移过程中读-迁冲突写-迁冲突Key Missing访问冲突迁移间隙Asking/MovedTryAgain迁移过程中阻塞，无法访问同步迁移STCSETE:ASKINGSTCASKING&SETOK(1)(2)迁移间隙迁移过程中读-迁出冲突读-迁入冲突写-迁冲突Key Missing异步迁移STCSETE:TRYAGAINMIGRATINGSTCSETE:ASKINGSTCASKING&SETOK(1)(2)(3)写-迁冲突MIGRATING_KEYSIMPORTING_KEYSTryAgain基于 Redis-3.2.8同步 vs 异步基于 Redis-3.2.8同步过程同步 vs 异步基于 Redis-3.2.8异步过程同步 vs 异步基于 Redis-4.2(unstable)PR#3997同步 vs 异步Hash10MZset10M