New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

自定义词表问题 #5

Open

18140663659 opened this issue Aug 10, 2023 · 0 comments

18140663659 commented Aug 10, 2023

我看到关闭的issues中有一个关于如何生成自定义词表的问题，我尝试按照其中提到的方法自己裁剪原生bloom的词表，遇到了以下的问题：

按造[samsha1971]提供的代码，报错：

这个应该是merges属性导致的问题 2. 如果将t["model"]["merges"] = [] 设置为空，encode的结果也为空；

所以想请问，手动重构词表怎么处理这些问题？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment