Generate types with better support for nested shapes

[baioc]

Solves #143 by:

• Flattening nested types generated in the same file/module:
  This way, equal types used in different places only need to be generated once (and the typechecker knows they're the same).

• Adding type aliases to every single field and type parameter:
  Since type aliases need to be in the toplevel for the typechecker to see them, we need a hierarchical naming scheme.
  The current scheme is not perfect: there could be conflicts if consumers use field names with double underscores.
  Also, accessing the types that parameterize a collection is a bit awkward, since we lack names and resort to numeric indexes.

[baioc]

In this first draft, we can use a constructor with a not-that-mangled name.

inner_t = outer_t._nested_t

inner = inner_t(my_field=3)
outer: outer_t = outer_t(nested=inner)

Using that as a type annotation still doesn't work, since we're accessing it from a nested class and that is a known limitation of Pyre.

Another problem is that the additional field which contains the first-class type is added for every field in generated shapes.
This is unnecessary in most cases all cases, since we could use the __annotations__ meta-attribute for that same information.

Using __annotations__ won't help with the annotations, however, as the static type checker requires our inner class definition to be in the global namespace. The solution appears to be changing codegen so as to "flatten" classes in the same generated module.
This is what I'll work on next. Let me know if I'm missing anything.

[vmagro]

since we're accessing it from a nested class and that is a known limitation of Pyre.

I'm curious, is it because it's being generated with a metaclass? Would it work if we did the codgen for the types inner class in shape.bzl directly? I only implemented it in a metaclass because it was a lot faster (to write) at the time.

Flattening the generated classes all into the module seems to be a good solution (/ the only solution). If we do that can they still be referenced by the nested names, or do they all have to get readable names (set in shape.bzl)?

[baioc]

I'm curious, is it because it's being generated with a metaclass?

AFAIK, Pyre simply doesn't check for types aliases that are not in the global namespace.
That said, using the mangled name does work, even if it is in a nested class.

Flattening the generated classes all into the module seems to be a good solution (/ the only solution). If we do that can they still be referenced by the nested names, or do they all have to get readable names (set in shape.bzl)?

Actually, it seems we don't need to flatten the classes themselves if we provide aliases in the global namespace.
Therefore we can keep the "real" type with a mangled name, but add readable aliases in the toplevel of the generated code:

[vmagro]

Therefore we can keep the "real" type with a mangled name, but add readable aliases in the toplevel of the generated code

That sounds fine, how are you thinking of providing the readable names? Just the field name?

[baioc]

That sounds fine, how are you thinking of providing the readable names? Just the field name?

As of the latest commit, and since they have to be in the toplevel, the naming scheme for type aliases is:

typeof_<field name>[__<inner field> ... [__<innermost field>]]
      ^--- single underscore             ^--- double underscore

So it is still possible to have conflicting aliases if you're using weird field names with double underscores.

It gets a bit weird with collection types: since there is no field name we can associate to them, we use 0-based indexes.
But this is needed because otherwise there would be no way to annotate those types. So having an outer shape like this:

deep_t = shape.dict(str, shape.list(shape.shape(very_deep=bool)))
inner_t = shape.shape(
    first = deep_t,
    second = deep_t,
)
outer_t = shape.shape(
    nested = inner_t,
)

We can have annotations for the types of every single field, of every single nested shape:
(it would be possible to only generate aliases for non-primitive fields, but I think that would be more awkward)

import outer  # assuming we generated the `outer_t` shape in `outer.py`

outer_t = outer.outer_t
inner_t = outer.typeof_nested
assert outer.typeof_nested__first == outer.typeof_nested__second  # they should be the same
deep_t = outer.typeof_nested__first
deepest_t = outer.typeof_nested__first__1__0  # 1 to get the dict's value type, 0 for the type of the list

deeper: deepest_t = deepest_t(very_deep=True)
deep: deep_t = {'key': (deeper, deeper, deeper)}
inner: inner_t = inner_t(first=deep, second=deep)
outer: outer_t = outer_t(nested=inner)

This runs and typechecks.

[vmagro]

Seems like you're on the right track, but this is breaking the build now.
Looking at the CI run it seems like buck build //images/appliance:stable-build-appliance is a working repro

[baioc]

Oh, I didn't realize python_data needed to be changed as well. Should be fixed now.
Speaking of which, it looks like _python_data does pretty much the same thing as _loader with a bit of extra stuff, we should probably refactor this.

[vmagro]

I think this seems pretty reasonable - I'm not in love with the generated code, but digging into it should only be necessary in very narrow circumstances and other than that the public api is the same

I just left one comment of something that I think can be improved, but I'll import and land the PR when that is addressed :)

[facebook-github-bot]

@vmagro has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

[vmagro]

c1e4f75 changes the way codegen works to accomplish this same result without the associated difficulty in starlark code