AWS 到腾讯云跨机房同步失败

[marsmay]

AWS 机器网卡情况：

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 172.20.2.72  netmask 255.255.255.0  broadcast 172.20.2.255
        inet6 fe80::8ea:82ff:fe26:a6c  prefixlen 64  scopeid 0x20<link>
        ether 0a:ea:82:26:0a:6c  txqueuelen 1000  (Ethernet)
        RX packets 17881949292  bytes 7405855014622 (6.7 TiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 12977732802  bytes 3690611677897 (3.3 TiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 6395733  bytes 954360270 (910.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6395733  bytes 954360270 (910.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

腾讯云机器网卡情况：

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.94  netmask 255.255.224.0  broadcast 172.16.31.255
        ether 52:54:00:79:0d:3f  txqueuelen 1000  (Ethernet)
        RX packets 40574811  bytes 57368455301 (53.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7592708  bytes 517480379 (493.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 285  bytes 29470 (28.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 285  bytes 29470 (28.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

两台机器都有外网IP，且安全组允许所有流量互通；
AWS 机器为主，腾讯云机器为从；两台机器均未配置绑定的网卡，看源码应该是监听 0.0.0.0 ;
在 AWS 上 dump 了文件以后，打包传到腾讯云解压，然后用 info 文件里的日志节点信息设置同步；
两台机器 pika 版本均为 2.2.6，同步命令为腾讯云上执行 slaveof aws_remote_ip 9231 145190 4056595;

AWS 机器的 pika info：

# Server
pika_version:2.2.6
pika_git_sha:e646201f53c5584294bdb2dece7b073f0d0e69b2
pika_build_compile_date: Nov  3 2017
os:Linux 3.10.0-514.16.1.el7.x86_64 x86_64
arch_bits:64
process_id:29692
tcp_port:9232
thread_num:6
sync_thread_num:6
uptime_in_seconds:6477562
uptime_in_days:76
config_file:/data/pika_slave/pika.conf

# Data
db_size:67671659064
db_size_human:64536M
compression:snappy
used_memory:2773284976
used_memory_human:2644M
db_memtable_usage:452626080
db_tablereader_usage:2320658896

# Log
log_size:25554088962
log_size_human:24370M
safety_purge:write2file147036
expire_logs_days:7
expire_logs_nums:300
binlog_offset:147137 42799283

# Clients
connected_clients:1

# Stats
total_connections_received:117489
instantaneous_ops_per_sec:1671
total_commands_processed:10050538356
is_bgsaving:No, 20180403090939, 0
is_slots_reloading:No, , 0
is_scaning_keyspace:No
is_compact:No
compact_cron:
compact_interval:

# Replication(MASTER/SLAVE)
role:slave
master_host:172.20.2.234
master_port:9231
master_link_status:up
slave_read_only:1
repl_state: 3
connected_slaves:1
slave0:ip=172.16.0.103,port=9231,state=offline

# Keyspace
# Time:1970-01-01 08:00:00
kv keys:0
hash keys:0
list keys:0
zset keys:0
set keys:0

腾讯云机器的 pika info：

# Server
pika_version:2.2.6
pika_git_sha:e646201f53c5584294bdb2dece7b073f0d0e69b2
pika_build_compile_date: Oct 11 2017
os:Linux 3.10.0-514.21.1.el7.x86_64 x86_64
arch_bits:64
process_id:31954
tcp_port:9231
thread_num:6
sync_thread_num:6
uptime_in_seconds:35331
uptime_in_days:2
config_file:/data/pika/pika.conf

# Data
db_size:88547551541
db_size_human:84445M
compression:snappy
used_memory:4216
used_memory_human:0M
db_memtable_usage:4120
db_tablereader_usage:96

# Log
log_size:8340108
log_size_human:7M
safety_purge:write2file145180
expire_logs_days:7
expire_logs_nums:100
binlog_offset:145190 4056595

# Clients
connected_clients:1

# Stats
total_connections_received:5
instantaneous_ops_per_sec:0
total_commands_processed:5
is_bgsaving:No, , 0
is_slots_reloading:No, , 0
is_scaning_keyspace:No
is_compact:No
compact_cron:
compact_interval:

# Replication(SLAVE)
role:slave
master_host:34.213.*.*
master_port:9231
master_link_status:down
slave_read_only:1
repl_state: 1

# Keyspace
# Time:1970-01-01 08:00:00
kv keys:0
hash keys:0
list keys:0
zset keys:0
set keys:0

AWS 机器的 pika 日志文件设置为 300，目前差值还远未达到；
腾讯云机器的日志提示不断重复：

W0404 01:38:19.298636 31971 pika_trysync_thread.cc:87] Reply from master after trysync: ERR AlreadyExist
W0404 01:38:19.298703 31971 pika_trysync_thread.cc:116] trysync, error: ERR AlreadyExist
W0404 01:38:22.837440 31971 pika_trysync_thread.cc:87] Reply from master after trysync: ERR AlreadyExist
W0404 01:38:22.837538 31971 pika_trysync_thread.cc:116] trysync, error: ERR AlreadyExist
W0404 01:38:26.385805 31971 pika_trysync_thread.cc:87] Reply from master after trysync: ERR AlreadyExist
W0404 01:38:26.385891 31971 pika_trysync_thread.cc:116] trysync, error: ERR AlreadyExist

觉得可能是 master 没有正确识别 slave 的外网 IP，因为 info 里看到 slave 信息里是从的内网 IP；
想把从实例绑定到外网 IP，但找了一下，pika 只能绑定网卡，不能绑定 IP，网卡信息上面有，没有办法实现目的；
现在需要怎么实现跨机房，各自有内网地址的同步呢？或者说这个问题有没有什么其他解决办法？

[derbyguyb]

相同问题，请教有什么办法解决吗？查看slave的机器，rsync使用内网ip监听，看了master的log，也是往slave的内网ip发送，导致全量同步失败

[fancy-rabbit]

试试看最新 master 分支，merge 了我修复这个问题的代码
但是如果是 NAT 环境下，可能不一定行（我没有环境可供测试）
多网卡不绑定情况下已经 ok 了

[nickyang4github]

同样遇到这个问题，master是3.0.4，slave是3.0.6

[AlexStocks]

比这个晚的 issue 472 都已经被 close，现在 close 掉这个 issue。