Displacy no support for utf-8

[avramandrei]

I get the following error when I try to use displacy to evaluate a Romanian corpus with entities.

Traceback (most recent call last):
  File "C:\Users\avramus\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\avramus\AppData\Local\Programs\Python\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\avramus\AppData\Local\Programs\Python\Python36\lib\site-packages\spacy\__main__.py", line 35, in <module>
    plac.call(commands[command], sys.argv[1:])
  File "C:\Users\avramus\AppData\Local\Programs\Python\Python36\lib\site-packages\plac_core.py", line 328, in call
    cmd, result = parser.consume(arglist)
  File "C:\Users\avramus\AppData\Local\Programs\Python\Python36\lib\site-packages\plac_core.py", line 207, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "C:\Users\avramus\AppData\Local\Programs\Python\Python36\lib\site-packages\spacy\cli\evaluate.py", line 77, in evaluate
    ents=render_ents,
  File "C:\Users\avramus\AppData\Local\Programs\Python\Python36\lib\site-packages\spacy\cli\evaluate.py", line 89, in render_parses
    file_.write(html)
  File "C:\Users\avramus\AppData\Local\Programs\Python\Python36\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u0103' in position 469: character maps to <undefined>

It looks like you don't open the html files for the entities and parses in UTF-8. I modified the code to open them in UTF-8 and it worked.

[ines]

Just to confirm: What code / command did you run here? It looks like the error actually occurred in cli.evaluate, not in displacy directly? And that's also where the file is opened incorrectly?

Edit: Looks like the problem is in cli.evaluate, so I'm just fixing that.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.