v5.4.15-alpha

RELEASE NOTE

Important New Feature

• Add backup capability：global binlog files can be uploaded to OSS in real time and restored to local from OSS for recovery
• Greatly improved performance：the maximum EPS capacity is increased from 100w/s to 200w/s, and the maximum write throughput capacity is increased from 250M/s to 500M/s

Feature enhancement

• Add built-in automatic cleaning capability：binlog file on the local disk can be automatically cleared periodically when the total file size exceeds a specified threshold
• Reduced memory usage by 40%：all kinds of optimization such as Metadata persistence, buffer size optimization, memory/disk swap policy optimization, and serialization/deserialization optimization
• More Strong data consistency: During DDL changes, the data reformat module supports reformating the data type and precision of binlog events
• New historical record clearing capability: Supports periodic clearing of ddl marking information and cluster scheduling history information
• Optimized log clearing policy: Optimized the clearing policy for the memory dump files (*.hprof) to reduce the threshold of the number of reserved files and avoid the risk of disk explosion
• Optimized the health checking policy for cdc processes: Change the health checking method from the 'jps' command to 'ps -ef' to solve the host cpu jitter problem caused by jps in high load scenarios
• Optimized CDC recovery policy: fix the issue that When a 2PC transaction spans multiple DN binlog files during recovery, data loss may occur in extreme cases
• Add hot effect capability for parameters: automatically scans the binlog_system_config configuration table and config.properties file, and automatically resets memory parameter values
• Optimized performance for binlog data filtering : optimize the TxnBuffer data structure to solve the performance bottleneck caused by the high-frequency remove operation on the refList
• Added blacklist configuration for database/table: supports filtering the data for some database or table which configured in the blacklist

发布说明

重要功能新增

• 新增binlog备份恢复能力：binlog文件可实时上传至OSS、以及从OSS恢复到本地
• 同步性能大幅提升：最大EPS能力从100w/s提升至200w/s，最大写入吞吐能力从250M/s提升至500M/s

特性更新

• 新增binlog自动清理能力：本地磁盘binlog文件大小超过指定阈值，支持可定时自动清理
• 内存占用降低40%：支持对元数据进行持久化、缓冲区大小优化、内存/磁盘swap策略优化、序列化/反序列化优化等
• 更高数据一致性保证：DDL变更过程中，数据整形模块新增支持对binlog event的数据类型和精度进行整形
• 新增历史记录清理能力：支持对ddl打标信息、集群调度历史信息等进行定时清理
• 日志清理策略优化：内存dump文件(*.hprof)清理策略优化，降低保留文件个数阈值，规避磁盘打爆风险
• 进程探活定时任务优化：进程探活方式，由jps命令调整为ps -ef，解决高负载场景下，因jps导致宿主机cpu抖动问题
• CDC Recovery策略优化：解决在链路恢复过程中，当某个2PC事务跨多个DN binlog文件时，极端情况下可能导致的数据丢失问题
• 新增参数热生效能力：自动扫描binlog_system_config配置表和 config.properties文件，自动reset内存参数值
• binlog数据过滤性能优化：优化TxnBuffer数据结构，解决因对refList进行高频remove操作导致的性能瓶颈问题
• 新增库表黑名单能力：全局binlog中支持针对指定库表的数据进行过滤