Render emojis to PDF

[asinghvi17]

Basically, this pr introduces an emojifont font in the opening to solve any potential bbox / char shape mismatch issues. All emojis are rendered with this font. It also changes a lot of the plot! signatures to convert_arguments which allows the user to update with any supported type.

This is not yet complete, but I did this in the web editor so putting it out there.

[asinghvi17]

I guess this is basically what I wanted to do. The framework for the Cairo special case is there but commented out, since I tried to use Cairo to draw colored emojis and it didn't (even though the docs explicitly list colored emojis as being supported). We could still draw the SVGs using Rsvg there though.

[fatteneder]

I think direct rendering of emojis will also require support for variation sequences from FreeTypeAbstraction.jl to extract the code point from the font files. Without that, you don't even get passed layout_formatted_text.

Also: The font not working could be related to the note made on the release page of OpenMoji, but not 100% sure: https:/hfg-gmuend/openmoji/tree/master/font

⚠️ The OpenMoji fonts work in all operating systems, but will currently only show color glyphs in Mozilla Firefox and Adobe CC. This is not a limitation of the generated fonts, but of the operating systems and applications.

Don't quote me on this, but I think I remember reading somewhere that unicode standards can have different implementations and that Firefox's and Adobe's implementations did have something in common or so. Perhaps that is the reason ...

We could still draw the SVGs using Rsvg there though.

Why not use PNGs instead? We already use those for GLMakie rendering.

[fatteneder]

Don't quote me on this, but I think I remember reading somewhere that unicode standards can have different implementations and that Firefox's and Adobe's implementations did have something in common or so. Perhaps that is the reason ...

I think this is what I remembered:

SVG in Open Type is a standard by Adobe and Mozilla for color OpenType and Open Font Format fonts.

Taken from https:/DeeDeeG/noto-color-emoji-font#examples

[fatteneder]

I spent some more time on trying to understand how unicode variation sequences work and how to possibly render emojis from a font file. Here is a brief summary:

• FreeType's variation sequence interface (https://freetype.org/freetype2/docs/reference/ft2-glyph_variants.html) as provided by FreeType.jl works or at least returns no error codes when used with the Noto Emoji Font https:/googlefonts/noto-emoji (see example below). However, using it with the .ttf files provided by OpenMoji I could not query any variation selectors at all (FT_Face_GetVariantSelectors(face) returns a null pointer).
• As mentioned above and also discussed here Improve/fix OpenMoji Colorfont (and Black) hfg-gmuend/openmoji#93 the OpenMoji files seem to be not working perfectly any app. I am still wondering how Firefox is able to render them, given that the ttf files miss the variation selectors. But perhaps they can use the underlying tables directly, because they know the right glyph index already? Wild speculations here ...
• Assuming variation selectors would work, there is then the question on how to render the glyphs. FreeType supports rendering to bitmaps and SVGs, the latter requiring an extra dependency that takes care of the rendering itself. From what I read SVGs are superior to bitmaps because they allow arbitrary scaling. Not sure if possible and how much work it would be to set this up from within Julia using Rsvg and FreeType alone (one has to set some callbacks in FreeType, see https://freetype.org/freetype2/docs/reference/ft2-svg_fonts.html)

Experiments

Here are my experiments with the FreeType library. Mainly adding it here for my own reference. Interesting note: the germany flag below is automatically rendered, whereas in my editor I entered it through the unicode sequence \u1f1e9\u1f1ea.

using FreeTypeAbstraction
using FreeType

# Unicode variation sequence methods imported from `FreeType.jl`.
#
# function FT_Face_GetCharVariantIndex(face, charcode, variantSelector)
#     ccall((:FT_Face_GetCharVariantIndex, libfreetype), FT_UInt, (FT_Face, FT_ULong, FT_ULong), face, charcode, variantSelector)
# end
#
# function FT_Face_GetCharVariantIsDefault(face, charcode, variantSelector)
#     ccall((:FT_Face_GetCharVariantIsDefault, libfreetype), FT_Int, (FT_Face, FT_ULong, FT_ULong), face, charcode, variantSelector)
# end
#
# function FT_Face_GetVariantSelectors(face)
#     ccall((:FT_Face_GetVariantSelectors, libfreetype), Ptr{FT_UInt32}, (FT_Face,), face)
# end
#
# function FT_Face_GetVariantsOfChar(face, charcode)
#     ccall((:FT_Face_GetVariantsOfChar, libfreetype), Ptr{FT_UInt32}, (FT_Face, FT_ULong), face, charcode)
# end
#
# function FT_Face_GetCharsOfVariant(face, variantSelector)
#     ccall((:FT_Face_GetCharsOfVariant, libfreetype), Ptr{FT_UInt32}, (FT_Face, FT_ULong), face, variantSelector)
# end

function get_variant_selectors(font::FTFont)
  selectors = UInt32[]
  ptr_selectors = FT_Face_GetVariantSelectors(font)
  ptr_selectors == Ptr{UInt32}() && return selectors
  idx = 1
  v = unsafe_load(ptr_selectors, idx)
  while v != 0
    push!(selectors, v)
    idx += 1
    v = unsafe_load(ptr_selectors, idx)
  end
  return selectors
end


function get_charcodes_of_variant(font::FTFont, variantSelector::UInt32)
  charcodes = UInt32[]
  ptr_charcodes = FT_Face_GetCharsOfVariant(font, variantSelector)
  ptr_charcodes == Ptr{UInt32}() && return charcodes
  idx = 1
  v = unsafe_load(ptr_charcodes, idx)
  while v != 0
    push!(charcodes, v)
    idx += 1
    v = unsafe_load(ptr_charcodes, idx)
  end
  return charcodes
end


function get_variantselectors_of_charcode(font::FTFont, charcode::UInt32)
  selectors = UInt32[]
  ptr_selectors = FT_Face_GetVariantsOfChar(font, charcode)
  ptr_selectors == Ptr{UInt32}() && return selectors
  idx = 1
  v = unsafe_load(ptr_selectors, idx)
  while v != 0
    push!(selectors, v)
    idx += 1
    v = unsafe_load(ptr_selectors, idx)
  end
  return selectors
end


function get_charcode_variantselector_default(font::FTFont, charcode::UInt32, variantSelector::UInt32)
  return FT_Face_GetCharVariantIsDefault(font, charcode, variantSelector)
end


function get_charcode_variantselector_index(font::FTFont, charcode::UInt32, variantSelector::UInt32)
  code = FT_Face_GetCharVariantIndex(font, charcode, variantSelector)
  return code == 0 ? nothing : code
end

# fontname = "/home/florian/.fonts/openmoji/OpenMoji-Black.ttf"
# fontname= "/home/florian/.fonts/openmoji/OpenMoji-Color.ttf"
# fontname= "/tmp/OpenMoji-Color.ttf"
fontname = "/home/florian/.fonts/NotoColorEmoji.ttf"
font = FTFont(fontname)
display(font)

selectors = get_variant_selectors(font)
# println(selectors)
charsofvariant = get_charcodes_of_variant.(Ref(font), selectors)
# println(length.(charsofvariant))
# println.(charsofvariant)
# variantsofchar = get_variantselectors_of_charcode.(Ref(font), first(charsofvariant))

### returns non-sense
# s = first(selectors)
# c = first(first(get_charcodes_of_variant.(Ref(font), selectors)))

emoji_flagde = "🇩🇪"
cp_emoji_flagde = [ convert(UInt32, codepoint(emoji_flagde[i])) for i in eachindex(emoji_flagde) ]
println(cp_emoji_flagde)
# println(cp_emoji_flagde[1] in first(charsofvariant))
# println(cp_emoji_flagde)
# println(s in selectors)
# println(get_charcode_variantselector_index.(Ref(font), cp_emoji_flagde, selectors[1]))

emoji_info = '\U2139'
cu_info = convert(UInt32, emoji_info)
# println(convert(UInt32,emoji_info))
println(cu_info in first(charsofvariant))
vs_glyphindex_info = get_charcode_variantselector_index(font, cu_info, first(selectors))
ft_face = getfield(font, :ft_ptr)
err = FreeType.FT_Load_Glyph(ft_face, vs_glyphindex_info, FreeType.FT_LOAD_NO_SCALE)
display(err)

# summary:
# it appears that glyph loading itself is not implemented in FreeTypeAbstraction,
# alhtough, this sounds wrong because how would people then load all the characters for
# text rendering? Only with FT_Load_Char?
# I think the problem is that the variation sequence methods of FreeType return
# a glyph index in the end, instead of a character code (makes sense since we started
# with a VS character code), but FT_Load_Char expects a character code.

[fatteneder]

Moving forward, we should settle on using PNGs and/or SVGs for now.

Ideally, I would like to only use one format at all, but from what I understand right now GLMakie prefers PNGs and CairoMakie SVGs, correct?

If it is possible to use PNGs also with CairoMakie, then we should do so for the moment. Reasons are that

1. frankly we have more important features to implement for a release (global slide elements, performance issues, etc.) and
2. we have PNGs already working for GLMakie.

[asinghvi17]

I think that last commit might be unnecessary - CairoMakie should be able to render this as is without any external intervention. It's only if you want SVG output in vector graphics that you need this kind of overload.

[fatteneder]

Using only

function CairoMakie.draw_plot(scene::Scene, screen::CairoMakie.CairoScreen, fmttxt::T) where T <: FormattedText

    txt = fmttxt.plots[1]
    CairoMakie.draw_plot(scene, screen, txt)

    if length(fmttxt.plots) > 1
        scttr = fmttxt.plots[2]
        CairoMakie.draw_plot(scene, screen, scttr)
    end
end

gives me the following error:

Stacktrace

julia> include("examples/mwe.jl")
ERROR: LoadError: MethodError: no method matching draw_marker(::Cairo.CairoContext, ::Matrix{ColorTypes.RGBA{Float32}}, ::Vec{2, Float32}, ::Vec2{Float64}, ::ColorTypes.RGBA
{Float32}, ::Float32, ::Vec2{Float64}, ::Quaternionf)
Closest candidates are:
  draw_marker(::Any, ::Circle, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:281
  draw_marker(::Any, ::GeometryBasics.HyperRectangle, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:305
  draw_marker(::Any, ::Char, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any, ::Any) at ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:231
Stacktrace:
  [1] (::CairoMakie.var"#30#31"{Scene, Cairo.CairoContext, typeof(identity), StaticArrays.SMatrix{4, 4, Float32, 16}, StaticArrays.SMatrix{4, 4, Float32, 16}, FreeTypeAbstra
ction.FTFont, Symbol, Symbol})(point::Point{2, Float32}, col::ColorTypes.RGBA{Float32}, markersize::Vec{2, Float32}, strokecolor::ColorTypes.RGBA{Float32}, strokewidth::Floa
t32, m::Matrix{ColorTypes.RGBA{Float32}}, mo::Vec{2, Float32}, rotation::Quaternionf)
    @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:224
  [2] macro expansion
    @ ~/.julia/packages/Makie/bJ9rD/src/utilities/utilities.jl:198 [inlined]
  [3] broadcast_foreach(::CairoMakie.var"#30#31"{Scene, Cairo.CairoContext, typeof(identity), StaticArrays.SMatrix{4, 4, Float32, 16}, StaticArrays.SMatrix{4, 4, Float32, 16
}, FreeTypeAbstraction.FTFont, Symbol, Symbol}, ::Vector{Point{2, Float32}}, ::ColorTypes.RGBA{Float32}, ::Vector{Vec{2, Float32}}, ::ColorTypes.RGBA{Float32}, ::Float32, ::
Vector{Matrix{ColorTypes.RGBA{Float32}}}, ::Vector{Vec{2, Float32}}, ::Quaternionf)
    @ Makie ~/.julia/packages/Makie/bJ9rD/src/utilities/utilities.jl:184
  [4] draw_atomic_scatter(scene::Scene, ctx::Cairo.CairoContext, transfunc::Function, colors::ColorTypes.RGBA{Float32}, markersize::Vector{Vec{2, Float32}}, strokecolor::Col
orTypes.RGBA{Float32}, strokewidth::Float32, marker::Vector{Matrix{ColorTypes.RGBA{Float32}}}, marker_offset::Vector{Vec{2, Float32}}, rotations::Quaternionf, model::StaticA
rrays.SMatrix{4, 4, Float32, 16}, positions::Vector{Point{2, Float32}}, size_model::StaticArrays.SMatrix{4, 4, Float32, 16}, font::FreeTypeAbstraction.FTFont, markerspace::S
ymbol, space::Symbol)
    @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:209
  [5] draw_atomic(scene::Scene, screen::CairoMakie.CairoScreen{Cairo.CairoSurfaceBase{UInt32}}, primitive::Scatter)
    @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/primitives.jl:201
  [6] draw_plot(scene::Scene, screen::CairoMakie.CairoScreen{Cairo.CairoSurfaceBase{UInt32}}, primitive::Scatter{Tuple{Vector{Point{2, Float32}}}})
    @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/infrastructure.jl:251
  [7] draw_plot(scene::Scene, screen::CairoMakie.CairoScreen{Cairo.CairoSurfaceBase{UInt32}}, fmttxt::Combined{MakieSlides.formattedtext, Tuple{Markdown.Paragraph}})
    @ MakieSlides ~/wd/MakieSlides.jl/src/formattedtext.jl:262
  [8] cairo_draw(screen::CairoMakie.CairoScreen{Cairo.CairoSurfaceBase{UInt32}}, scene::Scene)
    @ CairoMakie ~/.julia/packages/CairoMakie/gU5me/src/infrastructure.jl:192
  [9] save(name::String, presentation::Presentation; aspect::Tuple{Int64, Int64})
    @ MakieSlides ~/wd/MakieSlides.jl/src/MakieSlides.jl:234
 [10] save(name::String, presentation::Presentation)
    @ MakieSlides ~/wd/MakieSlides.jl/src/MakieSlides.jl:219
 [11] top-level scope
    @ ~/wd/MakieSlides.jl/examples/mwe.jl:159
 [12] include(fname::String)
    @ Base.MainInclude ./client.jl:451
 [13] top-level scope
    @ REPL[20]:1
in expression starting at /home/florian/wd/MakieSlides.jl/examples/mwe.jl:159

I thought that was the problem you mentioned here #15 (comment).

But now that you say it, it might be enough to just implement the draw_marker method for Matrix{RGBAf} overload.

[asinghvi17]

Yeah that's what would have to happen. We could implement it here for now and upstream it to CairoMakie...see the implementation of the fast path in the heatmap draw_atomic for how you need to do it.

[fatteneder]

Got it to work.
Only thing is that I now convert the Matrix{RGBAf} pixels first to Matrix{ARGB32} and then create a CairoSurface with it.
I tried to do avoid this by using Cairo.move_to and Cairo.set_source_rgba but could not get it work. Do you know of a more efficient solution?

[asinghvi17]

AFAIK that solution is optimal since Cairo wants UInt32 color. It would have been converted either way.

Overall it looks good to me. Thanks for pushing it this far! I am on vacation now but feel free to upstream the method you implemented to CairoMakie.