-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[LLM INFER] Append attn #9244
base: develop
Are you sure you want to change the base?
[LLM INFER] Append attn #9244
Conversation
…nto append_attn
Change-Id: Ibe8920ba41ea9775e676b05b12dc01cb9da95b5e
…nto append_attn
…nto append_attn
Thanks for your contribution! |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #9244 +/- ##
===========================================
+ Coverage 52.81% 53.08% +0.26%
===========================================
Files 660 657 -3
Lines 107281 106801 -480
===========================================
+ Hits 56660 56690 +30
+ Misses 50621 50111 -510 ☔ View full report in Codecov by Sentry. |
4d69d01
to
4a4a4b4
Compare
static_cast<uint8_t>(quant_value2 + 128.0f); | ||
} | ||
// write k | ||
// 大分块 lane_id / 4 / 2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
中文注释删一删
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
太多了,留着无伤大雅吧
50a48de
to
3789175
Compare
PR types
New features
PR changes
Others
Description
大模型推理attention组网重构,新的append_attn方案相比旧方案有10%到90%的性能提升。
目前已支持了llama/qwen/qwen-moe/mixtral的推理。
使用方式,原推理脚本的 --block_attn选项改为--append_attn即可。
TODO: