Documentation on --vectors-loc (for init-model) doesn't match read_vector()?

[mrxiaohe]

Your Environment

• Operating System: Windows 10
• Python Version Used: 3.7
• spaCy Version Used: 2.1.8
• Environment Information:

For init-model, the documentation for the option --vectors-loc says that the file should be a tab-separated file in Word2Vec format where the first column contains the word and the remaining columns the values.

The function definition for read_vectors() appears to suggest that the function expects the first row to contain the dimensions of the vectors (this line: shape = tuple(int(size) for size in next(f).split())), followed by a word2vec formatted table?

In addition, the line pieces = line.rsplit(" ", vectors_data.shape[1]) does not split the rows (at least on my Windows computer) if the vector file is indeed tab separated. It only works if I replace the space " " with a tab "\t" (pieces = line.rsplit("\t", vectors_data.shape[1])

Not sure if I misunderstood the formatting of the vectors (a distinct possibility!). Thanks!

def read_vectors(vectors_loc):
    # temp fix to avoid import issues cf https:/explosion/spaCy/issues/4200
    from tqdm import tqdm

    f = open_file(vectors_loc)
    shape = tuple(int(size) for size in next(f).split())
    vectors_data = numpy.zeros(shape=shape, dtype="f")
    vectors_keys = []
    for i, line in enumerate(tqdm(f)):
        line = line.rstrip()
        pieces = line.rsplit(" ", vectors_data.shape[1])
        word = pieces.pop(0)
        if len(pieces) != vectors_data.shape[1]:
            msg.fail(Errors.E094.format(line_num=i, loc=vectors_loc), exits=1)
        vectors_data[i] = numpy.asarray(pieces, dtype="f")
        vectors_keys.append(word)
    return vectors_data, vectors_keys

[honnibal]

I think there might be some small differences in the word2vec formats produced by different tools. I know the init-model command works directly with the files produced by FastText, as those are the ones I normally use. But it could be that the Mikolov word2vec implementation produces a slightly different format.

I think you're right that the description in the docs needs to be updated though, thanks!

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.