Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Plugin fails to start on Grafana 9.1.5: "Unrecognized remote plugin message" #1507

Closed
alecxvs opened this issue Sep 16, 2022 · 4 comments
Closed

Plugin fails to start on Grafana 9.1.5: "Unrecognized remote plugin message" #1507

alecxvs opened this issue Sep 16, 2022 · 4 comments

Comments

@alecxvs
Copy link
Contributor

alecxvs commented Sep 16, 2022

Describe the bug

logger=plugin.manager t=2022-09-16T17:53:51.112365641Z level=error msg="Could not start plugin" pluginId=alexanderzobnin-zabbix-datasource err="Unrecognized remote plugin message: \n\nThis usually means that the plugin is either invalid or simply\nneeds to be recompiled to support the latest protocol."

Expected behavior
Plugin and datasource to work

Screenshots
If applicable, add screenshots to help explain your problem.

Network data
If it's related to metric data visualization would be great to get the raw query and response for the network request (check this in browser dev tools network tab, there you can see metric requests, please include the request body and request response)

Software versions

Grafana Zabbix Grafana-Zabbix Plugin
9.1.5 n/a 4.2.10
@alecxvs
Copy link
Contributor Author

alecxvs commented Sep 22, 2022

I didn't realize the logging changed in the recent version, a simple plugin:debug filter doesn't work for this error.

logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.373315125Z level=debug msg="plugin process exited" path=/var/lib/grafana/plugins/alexanderzobnin-zabbix-app/zabbix-plugin_linux_amd64 pid=15 error="exit status 2"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371881036Z level=debug msg="goroutine 1 [running]:"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371914786Z level=debug msg=runtime.systemstack_switch()
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371926476Z level=debug msg="\t/usr/local/go/src/runtime/asm_amd64.s:350 fp=0xc000054780 sp=0xc000054778 pc=0x46a180"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371937716Z level=debug msg=runtime.main()
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372001826Z level=debug msg="\t/usr/local/go/src/runtime/proc.go:174 +0x7b fp=0xc0000547e0 sp=0xc000054780 pc=0x43ae3b"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372012126Z level=debug msg=runtime.goexit()
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372021557Z level=debug msg="\t/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000547e8 sp=0xc0000547e0 pc=0x46c3a1"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372030917Z level=debug msg=
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372078967Z level=debug msg="rax    0x0"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372091297Z level=debug msg="rbx    0xf"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372179198Z level=debug msg="rcx    0xffffffffffffffff"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372193388Z level=debug msg="rdx    0x6"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372203008Z level=debug msg="rdi    0xf"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372218458Z level=debug msg="rsi    0xf"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372294078Z level=debug msg="rbp    0x6"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372329108Z level=debug msg="rsp    0x7ffe3bef7870"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372398089Z level=debug msg="r8     0x7ffe3bef7940"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372414719Z level=debug msg="r9     0x7fb6969050c0"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372424309Z level=debug msg="r10    0x8"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.372465819Z level=debug msg="r11    0x246"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.3724939Z level=debug msg="r12    0x203000"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37250657Z level=debug msg="r13    0x6"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37251549Z level=debug msg="r14    0x122f800"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37252523Z level=debug msg="r15    0x7fb66fbdc068"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37259423Z level=debug msg="rip    0x7fb6967df19f"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37260808Z level=debug msg="rflags 0x246"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37261751Z level=debug msg="cs     0x33"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37262754Z level=debug msg="fs     0x0"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37263748Z level=debug msg="gs     0x0"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370919349Z level=debug msg="0x00007ffe3bef7780:  0x00007fb696767e50  0x00007fb6967efc90 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370934009Z level=debug msg="0x00007ffe3bef7790:  0x0000000000000000  0x00007ffe3bef7b40 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370943989Z level=debug msg="0x00007ffe3bef77a0:  0x2525252525252525  0x2525252525252525 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37101136Z level=debug msg="0x00007ffe3bef77b0:  0x000000ffffffffff  0x000000ffffffffff "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37102217Z level=debug msg="0x00007ffe3bef77c0:  0x000000ffffffffff  0x000000ffffffffff "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37103237Z level=debug msg="0x00007ffe3bef77d0:  0x415353454d5f434c  0x505f434c00534547 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37104218Z level=debug msg="0x00007ffe3bef77e0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.37109352Z level=debug msg="0x00007ffe3bef77f0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371137411Z level=debug msg="0x00007ffe3bef7800:  0x0000000000000000  0x00007ffe3bef7aa0 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371187021Z level=debug msg="0x00007ffe3bef7810:  0x0000000000416046 <runtime.heapBits.forwardOrBoundary+0x0000000000000046>  0x00007fb66fbdec00 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371226291Z level=debug msg="0x00007ffe3bef7820:  0xb431c00000000000  0xb4b3800000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371239291Z level=debug msg="0x00007ffe3bef7830:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371310042Z level=debug msg="0x00007ffe3bef7840:  0x77394b7473643870  0x69684d6849322d69 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371323622Z level=debug msg="0x00007ffe3bef7850:  0x754b7a6356504e47  0x726e523058726662 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371379872Z level=debug msg="0x00007ffe3bef7860:  0x655731624d6f7456  0x00007fb6967df191 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371398012Z level=debug msg="0x00007ffe3bef7870: <0x6f47715f74702d4d  0x546979687072396b "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371408193Z level=debug msg="0x00007ffe3bef7880:  0x5938355f64626c52  0x74476662306c3931 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371429663Z level=debug msg="0x00007ffe3bef7890:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371472153Z level=debug msg="0x00007ffe3bef78a0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371492033Z level=debug msg="0x00007ffe3bef78b0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371503993Z level=debug msg="0x00007ffe3bef78c0:  0x0000000000004000  0x0000000000000400 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371513433Z level=debug msg="0x00007ffe3bef78d0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371522893Z level=debug msg="0x00007ffe3bef78e0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371570964Z level=debug msg="0x00007ffe3bef78f0:  0x0000000000000000  0x61add4a94f3f0c00 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371584004Z level=debug msg="0x00007ffe3bef7900:  0x00007fb69674d740  0x0000000000000006 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371661744Z level=debug msg="0x00007ffe3bef7910:  0x0000000000203000  0x0000000000203000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371681424Z level=debug msg="0x00007ffe3bef7920:  0x000000000122f800  0x00007fb696792276 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371718115Z level=debug msg="0x00007ffe3bef7930:  0x00007ffe3bef7b50  0x00007fb69677c7f7 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371783455Z level=debug msg="0x00007ffe3bef7940:  0x0000000000000020  0x69684d6849322d69 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371813275Z level=debug msg="0x00007ffe3bef7950:  0x754b7a6356504e47  0x726e523058726662 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371825985Z level=debug msg="0x00007ffe3bef7960:  0x655731624d6f7456  0x7937594d384a6f61 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.371835535Z level=debug msg=
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.368694085Z level=debug msg="SIGABRT: abort"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.368803356Z level=debug msg="PC=0x7fb6967df19f m=0 sigcode=18446744073709551610"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.368836646Z level=debug msg=
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.368903316Z level=debug msg="goroutine 0 [idle]:"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.368944397Z level=debug msg="runtime: unknown pc 0x7fb6967df19f"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369005247Z level=debug msg="stack: frame={sp:0x7ffe3bef7870, fp:0x0} stack=[0x7ffe3b6f8e18,0x7ffe3bef7e40)"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369045457Z level=debug msg="0x00007ffe3bef7770:  0x0000000000000001  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369104498Z level=debug msg="0x00007ffe3bef7780:  0x00007fb696767e50  0x00007fb6967efc90 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369143688Z level=debug msg="0x00007ffe3bef7790:  0x0000000000000000  0x00007ffe3bef7b40 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369202398Z level=debug msg="0x00007ffe3bef77a0:  0x2525252525252525  0x2525252525252525 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369242328Z level=debug msg="0x00007ffe3bef77b0:  0x000000ffffffffff  0x000000ffffffffff "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369300689Z level=debug msg="0x00007ffe3bef77c0:  0x000000ffffffffff  0x000000ffffffffff "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369339809Z level=debug msg="0x00007ffe3bef77d0:  0x415353454d5f434c  0x505f434c00534547 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369397479Z level=debug msg="0x00007ffe3bef77e0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.36944279Z level=debug msg="0x00007ffe3bef77f0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.36950319Z level=debug msg="0x00007ffe3bef7800:  0x0000000000000000  0x00007ffe3bef7aa0 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.36954571Z level=debug msg="0x00007ffe3bef7810:  0x0000000000416046 <runtime.heapBits.forwardOrBoundary+0x0000000000000046>  0x00007fb66fbdec00 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369604681Z level=debug msg="0x00007ffe3bef7820:  0xb431c00000000000  0xb4b3800000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369643291Z level=debug msg="0x00007ffe3bef7830:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369696391Z level=debug msg="0x00007ffe3bef7840:  0x77394b7473643870  0x69684d6849322d69 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369738052Z level=debug msg="0x00007ffe3bef7850:  0x754b7a6356504e47  0x726e523058726662 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369795422Z level=debug msg="0x00007ffe3bef7860:  0x655731624d6f7456  0x00007fb6967df191 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369871903Z level=debug msg="0x00007ffe3bef7870: <0x6f47715f74702d4d  0x546979687072396b "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369909843Z level=debug msg="0x00007ffe3bef7880:  0x5938355f64626c52  0x74476662306c3931 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.369969883Z level=debug msg="0x00007ffe3bef7890:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370013693Z level=debug msg="0x00007ffe3bef78a0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370079184Z level=debug msg="0x00007ffe3bef78b0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370123054Z level=debug msg="0x00007ffe3bef78c0:  0x0000000000004000  0x0000000000000400 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370188755Z level=debug msg="0x00007ffe3bef78d0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370233805Z level=debug msg="0x00007ffe3bef78e0:  0x0000000000000000  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370349306Z level=debug msg="0x00007ffe3bef78f0:  0x0000000000000000  0x61add4a94f3f0c00 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370421786Z level=debug msg="0x00007ffe3bef7900:  0x00007fb69674d740  0x0000000000000006 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370511077Z level=debug msg="0x00007ffe3bef7910:  0x0000000000203000  0x0000000000203000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370595417Z level=debug msg="0x00007ffe3bef7920:  0x000000000122f800  0x00007fb696792276 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370610217Z level=debug msg="0x00007ffe3bef7930:  0x00007ffe3bef7b50  0x00007fb69677c7f7 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370645898Z level=debug msg="0x00007ffe3bef7940:  0x0000000000000020  0x69684d6849322d69 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370723258Z level=debug msg="0x00007ffe3bef7950:  0x754b7a6356504e47  0x726e523058726662 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370782768Z level=debug msg="0x00007ffe3bef7960:  0x655731624d6f7456  0x7937594d384a6f61 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370822399Z level=debug msg="runtime: unknown pc 0x7fb6967df19f"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370836519Z level=debug msg="stack: frame={sp:0x7ffe3bef7870, fp:0x0} stack=[0x7ffe3b6f8e18,0x7ffe3bef7e40)"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.370846369Z level=debug msg="0x00007ffe3bef7770:  0x0000000000000001  0x0000000000000000 "
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.364965371Z level=debug msg="runtime/cgo: pthread_create failed: Operation not permitted"
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.343401191Z level=debug msg="plugin started" path=/var/lib/grafana/plugins/alexanderzobnin-zabbix-app/zabbix-plugin_linux_amd64 pid=15
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.343490772Z level=debug msg="waiting for RPC address" path=/var/lib/grafana/plugins/alexanderzobnin-zabbix-app/zabbix-plugin_linux_amd64
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:57:03.341290087Z level=debug msg="starting plugin" path=/var/lib/grafana/plugins/alexanderzobnin-zabbix-app/zabbix-plugin_linux_amd64 args=[/var/lib/grafana/plugins/alexanderzobnin-zabbix-app/zabbix-plugin_linux_amd64]
logger=plugin.alexanderzobnin-zabbix-datasource t=2022-09-22T17:56:59.380429694Z level=warn msg="plugin failed to exit gracefully"

@alecxvs
Copy link
Contributor Author

alecxvs commented Sep 22, 2022

Related?
elastic/apm-server#6238
moby/moby#42680
moby/moby#42681

@alecxvs
Copy link
Contributor Author

alecxvs commented Sep 22, 2022

Docker 20.10.10 is the minimum version to upgrade to for fixing this particular issue (if you're running Grafana in a docker container): runtime/cgo: pthread_create failed: Operation not permitted

After upgrading, the plugin seems to be loading again.

@SeanMica
Copy link

Docker 20.10.10 is the minimum version to upgrade to for fixing this particular issue (if you're running Grafana in a docker container): runtime/cgo: pthread_create failed: Operation not permitted

After upgrading, the plugin seems to be loading again.

I had this issue too. After updating Docker from 20.10.3 to 20.10.18, this issue is fixed.

Thank you!

@SanderDelden SanderDelden mentioned this issue Oct 12, 2022
Closed
@alecxvs alecxvs closed this as completed Oct 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alecxvs @SeanMica