Add intfloat/multilingual-e5-large-instruct to model zoo

[david-sitsky]

Description

@frankfliu asked I create an issue to track this which was originally reported by me on Slack: https://deepjavalibrary.slack.com/archives/C01AURG857U/p1727308663498229.

Can we please add intfloat/multilingual-e5-large-instruct to the model zoo? It is considered the best E5 model in terms of MTEB scoring and would be highly valuable. Thank you.

[frankfliu]

@xyang16

[david-sitsky]

@xyang16 - any ideas when intfloat/multilingual-e5-large-instruct will be added to the model zoo? I am pretty keen to try this out..

[xyang16]

@david-sitsky I just uploaded this model to model zoo.

[david-sitsky]

@xyang16 - many thanks. I just tried accessing it with DJL 0.30.0, using either OnnxRuntime or PyTorch, but it still seems to fail. Any ideas?

Caused by: java.lang.IllegalArgumentException: Invalid djl URL: djl://ai.djl.huggingface.onnxruntime/intfloat/multilingual-e5-large-instruct
	at ai.djl.repository.RepositoryFactoryImpl$DjlRepositoryFactory.newInstance(RepositoryFactoryImpl.java:262)
	at ai.djl.repository.RepositoryFactoryImpl.newInstance(RepositoryFactoryImpl.java:64)
	at ai.djl.repository.Repository.newInstance(Repository.java:90)
	at ai.djl.repository.zoo.DefaultModelZoo.parseLocation(DefaultModelZoo.java:66)
	at ai.djl.repository.zoo.DefaultModelZoo.<init>(DefaultModelZoo.java:47)
	at ai.djl.repository.zoo.Criteria$Builder.optModelUrls(Criteria.java:577)

and

Caused by: java.lang.IllegalArgumentException: Invalid djl URL: djl://ai.djl.huggingface.pytorch/intfloat/multilingual-e5-large-instruct
	at ai.djl.repository.RepositoryFactoryImpl$DjlRepositoryFactory.newInstance(RepositoryFactoryImpl.java:262)
	at ai.djl.repository.RepositoryFactoryImpl.newInstance(RepositoryFactoryImpl.java:64)
	at ai.djl.repository.Repository.newInstance(Repository.java:90)
	at ai.djl.repository.zoo.DefaultModelZoo.parseLocation(DefaultModelZoo.java:66)
	at ai.djl.repository.zoo.DefaultModelZoo.<init>(DefaultModelZoo.java:47)

[david-sitsky]

My apologies.. I forgot to delete the ~/.djl.ai/cache directory.

[david-sitsky]

@xyang16 - despite zapping the cache dir, I still see the same errors. Also when I download https://mlrepo.djl.ai/model/nlp/text_embedding/ai/djl/huggingface/pytorch/models.json.gz, it seems to be missing intfloat/multilingual-e5-large-instruct.

Am I missing something or has this not been done yet?

[xyang16]

@david-sitsky Sorry. I have uploaded again. Please try again.

[david-sitsky]

@xyang16 - many thanks for that. It works for PyTorch but not for OnnxRuntime. The HF repo has the ONNX model too - can that be fixed? This works for existing E5 models.

Caused by: java.lang.IllegalArgumentException: Invalid djl URL: djl://ai.djl.huggingface.onnxruntime/intfloat/multilingual-e5-large-instruct
	at ai.djl.repository.RepositoryFactoryImpl$DjlRepositoryFactory.newInstance(RepositoryFactoryImpl.java:262)
	at ai.djl.repository.RepositoryFactoryImpl.newInstance(RepositoryFactoryImpl.java:64)
	at ai.djl.repository.Repository.newInstance(Repository.java:90)
	at ai.djl.repository.zoo.DefaultModelZoo.parseLocation(DefaultModelZoo.java:66)
	at ai.djl.repository.zoo.DefaultModelZoo.<init>(DefaultModelZoo.java:47)