Force UTF8 for all text reading / writing.

[LeifW]

Replaced all usages of readFile, writeFile, hGetLine, and hPutStrLn from
Prelude with versions from
System.IO.UTF8

Fixes #94

Tested with export LANG=C. Was able to load a unicode-containing .idr
file, compile and run it, use interactive vim stuff like
case-splitting and proof search, and could also :addproof from the repl.

Technically, only the change to readFile in Idris/Chaser.hs is necessary
to fix #94. The rest are mainly for consistency. I am a little leary
of changing the output encoding to unicode when the system encoding
might not be that on the things that don't deal with idris code, e.g.
the .c, .java, .pom, etc output files, that are then run through gcc,
javac, whatever.

Also, this might address the issue fixed by #1334, and make #1334
redunant?

[Melvar]

As far as I can see, System.IO.UTF8 seems to always assume that strings were read as bytes erroneously, however my system will read them as UTF-8 perfectly correctly, so it would try to double-decode in that case, which would almost certainly cause a lot of failures. To test this, try using a UTF-8 locale and using non-ascii in your file or queries. Calling System.IO.UTF8.getLine in GHCi on my system and feeding it τι throws an exception ("*** Exception: Enum.toEnum{Word8}: tag (964) is outside of bounds (0,255))

If you want to force UTF-8 everywhere, I believe the correct approach is to set the handle encoding with hSetEncoding after opening every handle that will be used for text.

[edwinb]

What's the status of this? There seems to be a merge conflict at the moment in any case...

[LeifW]

This worked for me in my testing, but the extra encoding / decoding the IO methods from the utf8-string library seems weird. I'll redo this by adding a readUtf8File utility function to replace readFile that ignores LANG and simply sets the file handle to UTF8. #94 (comment)

[david-christiansen]

Hi @LeifW, how's the re-do coming? Or is it done? I just want to make sure we're not waiting on each other here :-)

[david-christiansen]

@LeifW Should I close this PR now? You mention that it should be re-done, but there hasn't been any further activity for some time.

[LeifW]

Closing for now; not sure I'll be able to get to it soon, and haven't seen complaints about this behaviour lately (not having UTF8 in the stdlib source helps). Will re-open when I have something ready for review & merge.

Now that the rainy season's started and the days are shorter, maybe I'll be spending more time inside in front of a computer...