Skip to content

Commit

Permalink
5.4.13 release
Browse files Browse the repository at this point in the history
  • Loading branch information
lulu2panpan committed Mar 30, 2022
1 parent d402a37 commit 1433cec
Show file tree
Hide file tree
Showing 312 changed files with 49,737 additions and 17,888 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ See [Quick Start](https:/ApsaraDB/galaxysql#to-quick-start-with-pola

PolarDB-X is fully compatible with MySQL binary log related commands, such as `SHOW BINARY LOGS`, `SHOW BINLOG EVENTS`, etc. All commands can be found in the [official MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/binary-log-formats.html).

3. PolarDB-X as MySQL Slave (WIP)
3. PolarDB-X as MySQL Slave

PolarDB-X supports MySQL `CHANGE MASTER TO` command, you can use PolarDB-X as MySQL slave, refer to MySQL [official document](https://dev.mysql.com/doc/refman/8.0/en/change-master-to.html) for command details.
PolarDB-X supports MySQL Replica related commands, such as `CHANGE MASTER TO``START/STOP/RESET SLAVE``SHOW SLAVE STATUS`, etc. you can use PolarDB-X as MySQL slave, refer to [Replica Reference Manual](https://github.com/ApsaraDB/galaxycdc/tree/main/polardbx-cdc-rpl/README.md) for command details.

See [more docs](https:/ApsaraDB/galaxysql#quick-start) about PolarDB-X.

Expand Down
5 changes: 4 additions & 1 deletion docker/admin/app.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ function start_app() {
sh /home/admin/bin/jdk8.sh
chmod 755 /home/admin/polardbx-binlog.standalone/bin/*
sudo -E su admin -c 'sh /home/admin/polardbx-binlog.standalone/bin/daemon.sh start'
if [ -f "/home/admin/drds-worker/bin/globalSchedule.sh" ]; then
sh /home/admin/drds-worker/bin/globalSchedule.sh
fi
}

function stop_app() {
Expand All @@ -25,8 +28,8 @@ function config_env() {

if [ ! -f "/home/admin/env/env.properties" ]; then
touch /home/admin/env/env.properties
echo "ins_ip=$(hostname -i)" >> /home/admin/env/env.properties
fi
echo "ins_ip=$(hostname -i)" >> /home/admin/env/env.properties
chown -R admin:admin /home/admin/.
}

Expand Down
38 changes: 37 additions & 1 deletion docker/bin/cdc_log_cleaner.sh
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,7 @@ clean_log() {
print "dir[$clean_log_path] removed!" $2
sudo rm -rvf $clean_log_path
else
if [[ ${clean_log_path: -6} == ".hprof" ]] || [[ "$clean_log_path" == *"gc.log-20"* ]] || [[ "$clean_log_path" == *"console.log-20"* ]]; then
if [[ ${clean_log_path: -6} == ".hprof" ]] || [[ "$clean_log_path" == *"gc.log-20"* ]] || [[ "$clean_log_path" == *"console.log-20"* ]] || [[ ${clean_log_path: -4} == ".tmp" ]]; then
sudo rm -fv $clean_log_path
elif [[ ${clean_log_path: -4} == ".pid" ]]; then
print "skip the ${clean_log_path} file"
Expand Down Expand Up @@ -168,6 +168,9 @@ do
process_path="$cdc_log_path/polardbx-binlog"
clean_process_log $process_path $clean_level

process_path_2="$cdc_log_path/polardbx-rpl"
clean_db_log $process_path_2 $clean_level

else
print "clean finish! current usage[$use GB] `date`===================="
exit
Expand All @@ -186,3 +189,36 @@ sudo find /var/log/ -type f -size +500M -exec cp /dev/null {} \;
#use=`df -h $cdc_log_path|sed '1d'|awk '{print $5+0}'`
get_dir_size
print "clean finish! current useage $use GB `date`===================="


########################### binlog-rpl log 清理 #############################
# 删除超过 30 天未更新过的文件和文件夹
day=30
maxGb=30
folder=/home/admin/logs/polardbx-rpl

# -path 排除 folder 本身
find $folder -path "$folder" -type d -mtime +$day -exec rm -rf {} \;
find $folder -type f -mtime +$day -exec rm -rf {} \;
# statistic.log, position.log, commit.log,gc.log 总是保存 30 天,因为每个文件大小比较固定,每天约 5M

while [ $day -ge 1 ]
do
day=`expr $day - 1`
rpl_log_size=`du -sb $folder | awk '{print $1}'`
rpl_log_size_g=`expr $rpl_log_size / 1024 / 1024 / 1024`
echo "rpl log size: " $rpl_log_size_g GB

if [ $rpl_log_size_g -ge $maxGb ];then
# 如果还是很大,删除所有已经归档的 default.*.gz 和 meta.*.gz 文件
echo remove $day day .gz log files
find $folder -type f -mtime +$day -name 'default.*.gz' -exec ls -Shl {} +;
find $folder -type f -mtime +$day -name 'meta.*.gz' -exec ls -Shl {} +;

find $folder -type f -mtime +$day -name 'default.*.gz' -exec rm -rf {} \;
find $folder -type f -mtime +$day -name 'meta.*.gz' -exec rm -rf {} \;
else
echo break
break
fi
done
4 changes: 2 additions & 2 deletions docs/zh_CN/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ GalaxyCDC 组件内置于 PolarDB-X 实例中,提供完全兼容 MySQL Binary

PolarDB-X 完全兼容 MySQL Binlog 相关指令,例如 `SHOW BINARY LOGS``SHOW BINLOG EVENTS` 等,全部相关命令可参考 [MySQL 官方文档](https://dev.mysql.com/doc/refman/8.0/en/binary-log-formats.html)

3. 将 PolarDB-X 作为 MySQL 的备库(开发中)
3. 将 PolarDB-X 作为 MySQL 的备库

PolarDB-X 支持 MySQL `CHANGE MASTER TO` 命令,可将 PolarDB-X 直接作为 MySQL 的备库,命令详细说明请参考 [MySQL 官方文档](https://dev.mysql.com/doc/refman/8.0/en/change-master-to.html)
PolarDB-X 支持 MySQL Replica 相关指令,例如 `CHANGE MASTER TO``START/STOP/RESET SLAVE``SHOW SLAVE STATUS`,可将 PolarDB-X 直接作为 MySQL 的备库,命令详细说明请参考 [MySQL 官方文档](https://dev.mysql.com/doc/refman/8.0/en/change-master-to.html)

可在 [PolarDB-X 介绍页](https:/ApsaraDB/galaxysql#quick-start) 查看更多文档。

Expand Down
8 changes: 8 additions & 0 deletions polardbx-cdc-assemble/assembly.xml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@
<source>target/classes/logback.xml</source>
<outputDirectory>polardbx-binlog.standalone/conf</outputDirectory>
</file>
<file>
<source>target/classes/rpl-logback.xml</source>
<outputDirectory>polardbx-binlog.standalone/conf</outputDirectory>
</file>
<file>
<source>../polardbx-cdc-common/target/classes/config.properties</source>
<outputDirectory>polardbx-binlog.standalone/conf</outputDirectory>
Expand All @@ -30,6 +34,10 @@
<source>../polardbx-cdc-common/target/classes/metrics.txt</source>
<outputDirectory>polardbx-binlog.standalone/conf</outputDirectory>
</file>
<file>
<source>target/classes/git.properties</source>
<outputDirectory>polardbx-binlog.standalone/conf</outputDirectory>
</file>
</files>

<dependencySets>
Expand Down
37 changes: 36 additions & 1 deletion polardbx-cdc-assemble/bin/process-check.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,40 @@
#!/usr/bin/env bash
PATH="/opt/taobao/java/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin"
# This file is used by cron table

WORKER_BIN=/home/admin/polardbx-binlog.standalone/bin
needStart=0;

PID_FILE="daemon.pid";
executeShell="daemon.sh";

sh /home/admin/polardbx-binlog.standalone/bin/check-deamon.sh
if [ ! -f "$WORKER_BIN"/${PID_FILE} ];then
echo "${PID_FILE} not exist"
needStart=1;
else
pid=`cat $WORKER_BIN/${PID_FILE}`
echo "current ${PID_FILE} value : $pid";
if [ -z "$pid" ]; then
echo "${PID_FILE} exist, but pid is empty"
needStart=1;
else
runningPid=`jps | grep "$pid" | awk '{print $1}'`;
echo "Running pid: $runningPid"
if [ "$pid" != "$runningPid" ]; then
echo "pid not empty, but process not exist"
needStart=1;
fi
fi
fi

runningCount=`jps | grep DaemonBootStrap | wc -l`
echo "Running ${PID_FILE} count: $runningCount"
if [ "$runningCount" -gt 1 ];then
echo "Running ${PID_FILE} count more than 1";
needStart=1;
fi

if [ "$needStart" -eq 1 ];then
echo "Retarting ${PID_FILE}";
sh $WORKER_BIN/${executeShell} restart > /dev/null 2>&1 &
fi
125 changes: 125 additions & 0 deletions polardbx-cdc-assemble/bin/start_rpl.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
#!/bin/bash

### ====================================================================== ###
## ##
## Polardbx-rpl Task Startup Script ##
## ##
### ====================================================================== ###
### 2020-08-08 by chengjin
### make some different

TASK_ID=$1
TASK_NAME=$2
MEMORY=3072
PERM_MEMORY=256
BASE_HOME=$HOME/polardbx-binlog.standalone
LOG_DIR=$HOME/logs/polardbx-rpl/$TASK_NAME

#get param from 16th to end
JVM_PARAMS=""

usage() {
echo "please set startup Args String"
exit 1
}

if [ $# -lt 1 ]; then
usage
fi

if [ $(whoami) == "root" ]; then
echo DO NOT use root user to launch me.
exit 1
fi

export LD_LIBRARY_PATH=${BASE_HOME}/lib:${LD_LIBRARY_PATH}
export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK
export LANG=zh_CN.GB18030

if [ -f ${HOME}/bin/jdk8.sh ]; then
sudo sh ${HOME}/bin/jdk8.sh
fi

logback_configurationFile=${BASE_HOME}/conf/rpl-logback.xml

JAVA_OPTS="${JAVA_OPTS} -server -Xms${MEMORY}m -Xmx${MEMORY}m -Xss1m -Djute.maxbuffer=10240000 -DtaskId=$TASK_ID -Dlogback.configurationFile=$logback_configurationFile"
if [[ ! "$JVM_PARAMS" =~ "PermSize" ]]; then
JAVA_OPTS="${JAVA_OPTS} -XX:PermSize=${PERM_MEMORY}m -XX:MaxPermSize=${PERM_MEMORY}m"
fi
#JAVA_OPTS="${JAVA_OPTS} -XX:NewSize=128m -XX:MaxNewSize=256m"
JAVA_OPTS="${JAVA_OPTS} -XX:+UseParallelGC"
JAVA_OPTS="${JAVA_OPTS} -XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=2 -XX:NewRatio=1 -XX:ParallelGCThreads=6"
JAVA_OPTS="${JAVA_OPTS} -XX:-OmitStackTraceInFastThrow"

JAVA_OPTS="${JAVA_OPTS} -Djava.net.preferIPv4Stack=true"
JAVA_OPTS="${JAVA_OPTS} -XX:+PrintGCDetails"
JAVA_OPTS="${JAVA_OPTS} -XX:+PrintGCDateStamps"
JAVA_OPTS="${JAVA_OPTS} -XX:+DisableExplicitGC"
JAVA_OPTS="${JAVA_OPTS} -Xloggc:${LOG_DIR}/gc.log"
JAVA_OPTS="${JAVA_OPTS} -Dmemory=${MEMORY}"
JAVA_OPTS="${JAVA_OPTS} -Djava.util.prefs.systemRoot=${HOME}/.java -Djava.util.prefs.userRoot=${HOME}/.java/.userPrefs -Dfile.encoding=UTF-8"
JAVA_OPTS="${JAVA_OPTS} -Ddaemon.home.dir=${BASE_HOME}"

if [ -f /home/admin/env.properties ]; then
for line in `cat /home/admin/env.properties`
do
JAVA_OPTS="${JAVA_OPTS} -D$line"
done
fi

if [ ! -d ${HOME}/.java ]; then
mkdir -p "${HOME}/.java"
fi

if [ ! -d ${HOME}/.java/.userPrefs ]; then
mkdir -p "${HOME}/.java/.userPrefs"
fi

if [ ! -d ${HOME}/.java/.systemPrefs ]; then
mkdir -p "${HOME}/.java/.systemPrefs"
fi

JAVA_OPTS="${JAVA_OPTS} -XX:+HeapDumpOnOutOfMemoryError"
JAVA_OPTS="${JAVA_OPTS} ${JVM_PARAMS}"

## set java path
TAOBAO_JAVA="/opt/taobao/java/bin/java"
ALIBABA_JAVA="/usr/alibaba/java/bin/java"
if [ -f $TAOBAO_JAVA ]; then
JAVA=$TAOBAO_JAVA
elif [ -f $ALIBABA_JAVA ]; then
JAVA=$ALIBABA_JAVA
else
JAVA=$(which java)
if [ ! -f $JAVA ]; then
echo "Cannot find a Java JDK. Please set either set JAVA or put java (>=1.7) in your PATH." 2>&2
exit 1
fi
fi

#BUILDER IN classpath
CLASSPATH="${BASE_HOME}/conf";

if [ -f ${HOME}/env/env.properties ]; then
CORES=$(sed '/^cpu_cores=/!d;s/.*=//' ${HOME}/env/env.properties)
if [ -n "$CORES" ]; then
JAVA_OPTS="${JAVA_OPTS} -XX:ActiveProcessorCount=$CORES"
fi
CLASSPATH="${HOME}/env:$CLASSPATH"
fi

for jar in $(ls ${BASE_HOME}/lib/*.jar); do
CLASSPATH="${CLASSPATH}:""${jar}"
done

if [ ! -d ${LOG_DIR} ]; then
mkdir -p ${LOG_DIR}
fi
defaultLog=${LOG_DIR}/default.log

#change user.dir to /home/admin
cd $HOME

#Start Java Process
echo "$(date +"%Y-%m-%d-%H:%M:%S.%N") ${JAVA} ${JAVA_OPTS} -classpath ${CLASSPATH}:. com.aliyun.polardbx.rpl.RplTaskEngine "taskId=${TASK_ID}"" >>$defaultLog
${JAVA} ${JAVA_OPTS} -classpath ${CLASSPATH}:. com.aliyun.polardbx.rpl.RplTaskEngine "taskId=${TASK_ID} taskName=${TASK_NAME}" 1>>$defaultLog 2>&1 &
37 changes: 37 additions & 0 deletions polardbx-cdc-assemble/bin/stop_rpl.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/bin/bash

### ====================================================================== ###
## ##
## Polardbx-rpl Task Stop Script ##
## ##
### ====================================================================== ###

TASK_ID=$1

BASE_HOME="/home/admin/polardbx-binlog.standalone"

usage() {
echo "please set stop Args taskId"
exit 1
}

if [ $# -lt 1 ]; then
usage
fi

if [ $(whoami) == "root" ]; then
echo DO NOT use root user to launch me.
exit 1
fi


runningCount=`ps -ef | grep RplTaskEngine | grep taskId=${TASK_ID} | wc -l`
echo "taskId: ${TASK_ID} running count: $runningCount"

if [ "$runningCount" -ge 1 ];then
echo "taskId: ${TASK_ID} already running, kill all and restart"
for pid in `ps -ef | grep RplTaskEngine | grep taskId=${TASK_ID} | awk '{print $2}'` ; do kill -9 $pid ; done
else
echo "no pid for $TASK_ID to stop."
fi

39 changes: 38 additions & 1 deletion polardbx-cdc-assemble/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>polardbx-cdc</artifactId>
<groupId>com.aliyun.polardbx</groupId>
<version>5.4.12-SNAPSHOT</version>
<version>5.4.13-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>polardbx-cdc-assemble</artifactId>
Expand Down Expand Up @@ -42,6 +42,43 @@
<finalName>polardbx-binlog</finalName>

<plugins>
<plugin>
<groupId>pl.project13.maven</groupId>
<artifactId>git-commit-id-plugin</artifactId>
<version>4.0.5</version>
<executions>
<execution>
<goals>
<goal>revision</goal>
</goals>
</execution>
</executions>
<configuration>
<verbose>false</verbose>
<dateFormat>yyyy-MM-dd HH:mm:ssZ</dateFormat>
<generateGitPropertiesFile>true</generateGitPropertiesFile>
<generateGitPropertiesFilename>${project.build.outputDirectory}/git.properties
</generateGitPropertiesFilename>
<!--git描述配置,可选;由JGit提供实现;-->
<gitDescribe>
<!--是否生成描述属性-->
<skip>false</skip>
<!--提交操作未发现tag时,仅打印提交操作ID,-->
<always>false</always>
<!--提交操作ID显式字符长度,最大值为:40;默认值:7;
0代表特殊意义;后面有解释;
-->
<abbrev>7</abbrev>
<!--构建触发时,代码有修改时(即"dirty state"),添加指定后缀;默认值:"";-->
<dirty>-dirty</dirty>
<!--always print using the "tag-commits_from_tag-g_commit_id-maybe_dirty" format, even if "on" a tag.
The distance will always be 0 if you're "on" the tag.
-->
<forceLongFormat>false</forceLongFormat>
</gitDescribe>
</configuration>
</plugin>

<plugin>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
Expand Down
Loading

0 comments on commit 1433cec

Please sign in to comment.