Restore from increment causes mysqld to segault

[FrankelJb]

Creating the issue here because I'm not sure where else to ask for assistance.

I've been trying all day to test my backups and I want to restore my maria instance from its incrementals. I first run
mariabackup --prepare --target-dir base/'2021-11-25_23-00-01'/backup/, the preparation works. However, when I try and prepare the first incremental backup I get this stack trace:

[00] 2021-11-30 14:20:25 open files limit requested 0, set to 1024                                                                                              
[00] 2021-11-30 14:20:25 This target seems to be already prepared.                                                                                              
[00] 2021-11-30 14:20:25 mariabackup: using the following InnoDB configuration for recovery:                                                                    
[00] 2021-11-30 14:20:25 innodb_data_home_dir = .                                                                                                               
[00] 2021-11-30 14:20:25 innodb_data_file_path = ibdata1:12M:autoextend                                                                                         
[00] 2021-11-30 14:20:25 innodb_log_group_home_dir = /mnt/data/mariadb-restore/backup/incr/2021-11-25_23-00-01/2021-11-26_23-00-01/backup/                      
[00] 2021-11-30 14:20:25 InnoDB: Using Linux native AIO                                                                                                         
[00] 2021-11-30 14:20:25 mariabackup: Generating a list of tablespaces                                                                                          
211130 14:20:25 [ERROR] mysqld got signal 11 ;                                                                                                                  
This could be because you hit a bug. It is also possible that this binary                                                                                       
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.5.12-MariaDB
key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=1
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6047 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
addr2line: 'mariabackup': No such file
mariabackup(my_print_stacktrace+0x41)[0x55fa9f8b1ab1]
Printing to addr2line failed
mariabackup(handle_fatal_signal+0x498)[0x55fa9f3fb368]
??:0(__restore_rt)[0x7ffa86442a20]
:0(__memcmp_avx2_movbe)[0x7ffa86050cfe]
addr2line: 'mariabackup': No such file
mariabackup(+0xc9b52b)[0x55fa9f67c52b]
mariabackup(+0xd87358)[0x55fa9f768358]
mariabackup(+0xc9cea7)[0x55fa9f67dea7]
mariabackup(+0xce72e3)[0x55fa9f6c82e3]
mariabackup(+0x6a22cb)[0x55fa9f0832cb]
mariabackup(+0x6a0a7e)[0x55fa9f081a7e]
mariabackup(+0x6a0dc7)[0x55fa9f081dc7]
mariabackup(+0x6acbb8)[0x55fa9f08dbb8]
mariabackup(main+0x194)[0x55fa9f048444]
??:0(__libc_start_main)[0x7ffa85f1bb75]
addr2line: 'mariabackup': No such file
mariabackup(_start+0x2e)[0x55fa9f07da9e] 
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...

I'm using kubernetes to run the maria instance and I've tried to create a duplicate environment to restore and I've tried running the prepare directly on the host with the same results. This leads me to believe that there is an issue with the incremental. Any help/guidance on this will be greatly appreciated.

[s0kr8s]

I ran into this exact issue and could not find any solutions anywhere online other than "submit a bug report," which was less than inspiring because the process of installing a development version of MariaDB with debug symbols is no light undertaking.

So, after several painful days of trial and error I'm happy to share my solution to prevent headaches for anyone else out there struggling with the same obscure issue.

In my case, there was a single table causing all the problems, and it had its ROW_FORMAT set to COMPRESSED. Apparently MariaDB is getting ready to deprecate this feature, so I guess they're not attending to its nuances very carefully over in mariabackup development, hence the obscure bug.

As soon as I changed that table with:

ALTER TABLE my_table ROW_FORMAT=default;

The crashes stopped happening during incremental backups. Keep in mind that this assumes your default row format is set to dynamic, as per the new default guidelines for any version of MariaDB >= 10.2.2.

If you need to find which tables are set to compressed, try:

SELECT table_name from information_schema.tables WHERE create_options LIKE '%COMPRESSED%';