Sanitizer CHECK failed: ((allocated_for_dlsym)) < ((kDlsymAllocPoolSize)) (0x402, 0x400)) with preload

[hjl-tools]

Bugzilla Link	52278
Resolution	FIXED
Resolved on	Nov 20, 2021 06:16
Version	unspecified
OS	Linux
Blocks	#51489
CC	@delcypher,@nickdesaulniers,@Lekensteyn,@tstellar,@vitalybuka

Extended Description

With glibc 2.34 on Linux/x86-64, LLVM 13.0.0 rc1 gave me:

[hjl@gnu-skx-1 gcc]$ cat x.c
#include <assert.h>

struct A {
char a[3];
int b[3];
};

volatile int ten = 10;

attribute((noinline)) void foo(int index, int len) {
volatile struct A str[len] attribute((aligned(32)));
assert(!((long) str & 31L));
str[index].a[0] = '1'; // BOOM
}

int main(int argc, char **argv) {
foo(ten, ten);
return 0;
}
[hjl@gnu-skx-1 gcc]$ clang -O0 -fsanitize=address x.c -shared-libasan -m32
[hjl@gnu-skx-1 gcc]$ LD_PRELOAD=/export/users/hjl/build/gnu/tools-build/gcc-debug/build-x86_64-linux/x86_64-pc-linux-gnu/32/libsanitizer/libclang_rt.asan-i386.so ./a.out
AddressSanitizer: CHECK failed: asan_malloc_linux.cpp:46 "((allocated_for_dlsym)) < ((kDlsymAllocPoolSize))" (0x402, 0x400) (tid=3485465)

[hjl@gnu-skx-1 gcc]$

depending on the directory length where libclang_rt.asan-i386.so is placed.

[hjl-tools]

A simpler reproducer:

[hjl@gnu-skx-1 gcc]$ LD_PRELOAD=/tmp/export-users-hjl-build-gnu-tools-build-gcc-debug-build-x86_64-linux-x86_64-pc-linux-gnu/libclang_rt.asan-i386.so ./a.out
AddressSanitizer: CHECK failed: asan_malloc_linux.cpp:46 "((allocated_for_dlsym)) < ((kDlsymAllocPoolSize))" (0x405, 0x400) (tid=3485517)

[hjl@gnu-skx-1 gcc]$

[hjl-tools]

In glibc 2.34, dlsym does

1. malloc 1
2. malloc 2
3. free pointer from malloc 1
4. free pointer from malloc 2

so that
static void DeallocateFromLocalPool(const void *ptr) {
// Hack: since glibc 2.27 dlsym no longer uses stack-allocated memory to store // error messages and instead uses malloc followed by free. To avoid pool
// exhaustion due to long object filenames, handle that special case here.
uptr prev_offset = allocated_for_dlsym - last_dlsym_alloc_size_in_words;
void prev_mem = (void)&alloc_memory_for_dlsym[prev_offset];
if (prev_mem == ptr) {
REAL(memset)(prev_mem, 0, last_dlsym_alloc_size_in_words * kWordSize);
allocated_for_dlsym = prev_offset;
last_dlsym_alloc_size_in_words = 0;
}
}

has a memory leak.

[hjl-tools]

A patch

[vitalybuka]

Reproducible by bots on Ubuntu 21.10 https://lab.llvm.org/staging/#/builders/19/builds/654/steps/8/logs/stdio

[vitalybuka]

Fixed with cb0e14c

But if we want to cherry-pick we need 4 patches.

1da33a5 [NFC][sanitizer] Move GET_MALLOC_STACK_TRACE closer to the use
64d4420 [NFC][lsan] Simplify root_regions initialization
651797f [NFC][asan][memprov] Remove dlsym hack from posix_memalign
cb0e14c [sanitizer] Switch dlsym hack to internal_allocator

[vitalybuka]

And

diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_allocator_dlsym.h b/compiler-rt/lib/sanitizer_common/sanitizer_allocator_dlsym.h
index 92b1373ef84d..8ae6f7cce75b 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_allocator_dlsym.h
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_allocator_dlsym.h
@@ -25,7 +25,7 @@ struct DlSymAllocator {
return !SANITIZER_FUCHSIA && UNLIKELY(Details::UseImpl());
}

• static bool PointerIsMine(const void *ptr) {

• static bool PointerIsMine(void *ptr) {
  // Fuchsia doesn't use dlsym-based interceptors.
  return !SANITIZER_FUCHSIA &&
  UNLIKELY(internal_allocator()->FromPrimary(ptr));

[vitalybuka]

This is not a regression in LLVM, it's changes in glibc.

I am not sure if it safe to cherry-pick all these patches into release today. Maybe after a couple of weeks?

If someone can test, maybe this trivial change is safe and good enough?
kDlsymAllocPoolSize = 1024
to
kDlsymAllocPoolSize = 2048

[hjl-tools]

This is not a regression in LLVM, it's changes in glibc.

I am not sure if it safe to cherry-pick all these patches into release
today. Maybe after a couple of weeks?

If someone can test, maybe this trivial change is safe and good enough?
kDlsymAllocPoolSize = 1024
to
kDlsymAllocPoolSize = 2048

I tested this on release/13.x branch on Fedora 35 with glibc 2.34:

diff --git a/compiler-rt/lib/asan/asan_malloc_linux.cpp b/compiler-rt/lib/asan/asan_malloc_linux.cpp
index c6bec8551bc5..b4552428f425 100644
--- a/compiler-rt/lib/asan/asan_malloc_linux.cpp
+++ b/compiler-rt/lib/asan/asan_malloc_linux.cpp
@@ -30,7 +30,7 @@ using namespace __asan;
 
 static uptr allocated_for_dlsym;
 static uptr last_dlsym_alloc_size_in_words;
-static const uptr kDlsymAllocPoolSize = 1024;
+static const uptr kDlsymAllocPoolSize = 2048;
 static uptr alloc_memory_for_dlsym[kDlsymAllocPoolSize];
 
 static inline bool IsInDlsymAllocPool(const void *ptr) {

It fixes the problem.

[mgorny]

Reopening for 13.x backport.

The change from 1024 to 2048 fixes most of the test failures for me (on Gentoo amd64) but not all of them. I'm trying 4096 now.

[mgorny]

The smallest power of 2 that I've been able to get AddressSanitizer-i386-linux :: TestCases/Linux/long-object-path.cpp to pass is 8192.

[tstellar]

The smallest power of 2 that I've been able to get AddressSanitizer-i386-linux :: TestCases/Linux/long-object-path.cpp to pass is 8192.

@vitalybuka Is it OK to bump the size this high on the release/13.x branch?

[vitalybuka]

Yes

[tstellar]

@mgorny Can you prepare a branch with this change and push it to a github fork?

[mgorny]

Sure: release/13.x...mgorny:13-kdlsymalloc

[tstellar]

/branch mgorny/llvm-project/13-kdlsymalloc

[tstellar]

Merged: d96358a