mc mirror takes long time when target bucket contains a lot of files

[archsh]

Expected behavior

Using mc mirror to mirror local folder to a target bucket, it should be exit soon if local folder is empty or few files.

Actual behavior

It took a long time to start transfer files if the target bucket contains a lot of objects.

Steps to reproduce the behavior

mc mirror --json ./localfolder target/bucket/path

mc --version

• (paste output of mc --version)
  mc version RELEASE.2024-01-05T05-04-32Z (commit-id=59eca9fea8984adec1e8e7a1c95d0ea23107ceff)

System information

CentOS release 6.9 (Final) x64

[zveinn]

Yes, mc mirror does a full source to target comparison when it's ran. This is due to the --remove flag. However, it might be possible to speed up the mirror given that the --remove flag is not present. I will look into this in the coming weeks, no promises on delivery date though since my schedule is rather full.

[PrisedRabbit]

Same here. It takes a lot of time to mirror folder with 20 000+ files ( using aws cli solved the problem )

[zveinn]

I will be looking into this next week, if there is any additional information you want to share, then now is the time :)

[karsapeng]

helloHello, I wrote a script to use “mc mirror --remove --overwrite --limit-download $LIMIT_DOWNLOAD --limit-upload $LIMIT_UPLOAD minio-master minio-slave” to synchronize the data of the master node. Even if the data of the two minio nodes are consistent, if the master node is shut down and restarted due to a fault, I will change the direction of the synchronization and transfer the data to the slave node. Then the master will run “mc mirror --remove --overwrite --limit-download $LIMIT_DOWNLOAD --limit-upload $LIMIT_UPLOAD minio-slave minio-master”for synchronization. However, even if the data of the two minio nodes is consistent, minio-master will perform a full synchronization, which will take up a lot of io and cause the server to crash. My mc version is mc:RELEASE.2024-08-26T10-49-58Z

[karsapeng]

Hello, has this problem been solved?