Skip to content

Commit

Permalink
bevy_reflect: Improve serialization format even more (#5723)
Browse files Browse the repository at this point in the history
> Note: This is rebased off #4561 and can be viewed as a competitor to that PR. See `Comparison with #4561` section for details.

# Objective

The current serialization format used by `bevy_reflect` is both verbose and error-prone. Taking the following structs[^1] for example:

```rust
// -- src/inventory.rs

#[derive(Reflect)]
struct Inventory {
  id: String,
  max_storage: usize,
  items: Vec<Item>
}

#[derive(Reflect)]
struct Item {
  name: String
}
```

Given an inventory of a single item, this would serialize to something like:

```rust
// -- assets/inventory.ron

{
  "type": "my_game::inventory::Inventory",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "inv001",
    },
    "max_storage": {
      "type": "usize",
      "value": 10
    },
    "items": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "my_game::inventory::Item",
          "struct": {
            "name": {
              "type": "alloc::string::String",
              "value": "Pickaxe"
            },
          },
        },
      ],
    },
  },
}
```

Aside from being really long and difficult to read, it also has a few "gotchas" that users need to be aware of if they want to edit the file manually. A major one is the requirement that you use the proper keys for a given type. For structs, you need `"struct"`. For lists, `"list"`. For tuple structs, `"tuple_struct"`. And so on.

It also ***requires*** that the `"type"` entry come before the actual data. Despite being a map— which in programming is almost always orderless by default— the entries need to be in a particular order. Failure to follow the ordering convention results in a failure to deserialize the data.

This makes it very prone to errors and annoyances.


## Solution

Using #4042, we can remove a lot of the boilerplate and metadata needed by this older system. Since we now have static access to type information, we can simplify our serialized data to look like:

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        name: "Pickaxe"
      ),
    ],
  ),
}
```

This is much more digestible and a lot less error-prone (no more key requirements and no more extra type names).

Additionally, it is a lot more familiar to users as it follows conventional serde mechanics. For example, the struct is represented with `(...)` when serialized to RON.

#### Custom Serialization

Additionally, this PR adds the opt-in ability to specify a custom serde implementation to be used rather than the one created via reflection. For example[^1]:

```rust
// -- src/inventory.rs

#[derive(Reflect, Serialize)]
#[reflect(Serialize)]
struct Item {
  #[serde(alias = "id")]
  name: String
}
```

```rust
// -- assets/inventory.ron

{
  "my_game::inventory::Inventory": (
    id: "inv001",
    max_storage: 10,
    items: [
      (
        id: "Pickaxe"
      ),
    ],
  ),
},
```

By allowing users to define their own serialization methods, we do two things:

1. We give more control over how data is serialized/deserialized to the end user
2. We avoid having to re-define serde's attributes and forcing users to apply both (e.g. we don't need a `#[reflect(alias)]` attribute).

### Improved Formats

One of the improvements this PR provides is the ability to represent data in ways that are more conventional and/or familiar to users. Many users are familiar with RON so here are some of the ways we can now represent data in RON:

###### Structs

```js
{
  "my_crate::Foo": (
    bar: 123
  )
}
// OR
{
  "my_crate::Foo": Foo(
    bar: 123
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Foo",
  "struct": {
    "bar": {
      "type": "usize",
      "value": 123
    }
  }
}
```

</details>

###### Tuples

```js
{
  "(f32, f32)": (1.0, 2.0)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "(f32, f32)",
  "tuple": [
    {
      "type": "f32",
      "value": 1.0
    },
    {
      "type": "f32",
      "value": 2.0
    }
  ]
}
```

</details>

###### Tuple Structs

```js
{
  "my_crate::Bar": ("Hello World!")
}
// OR
{
  "my_crate::Bar": Bar("Hello World!")
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::Bar",
  "tuple_struct": [
    {
      "type": "alloc::string::String",
      "value": "Hello World!"
    }
  ]
}
```

</details>

###### Arrays

It may be a bit surprising to some, but arrays now also use the tuple format. This is because they essentially _are_ tuples (a sequence of values with a fixed size), but only allow for homogenous types. Additionally, this is how RON handles them and is probably a result of the 32-capacity limit imposed on them (both by [serde](https://docs.rs/serde/latest/serde/trait.Serialize.html#impl-Serialize-for-%5BT%3B%2032%5D) and by [bevy_reflect](https://docs.rs/bevy/latest/bevy/reflect/trait.GetTypeRegistration.html#impl-GetTypeRegistration-for-%5BT%3B%2032%5D)).

```js
{
  "[i32; 3]": (1, 2, 3)
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "[i32; 3]",
  "array": [
    {
      "type": "i32",
      "value": 1
    },
    {
      "type": "i32",
      "value": 2
    },
    {
      "type": "i32",
      "value": 3
    }
  ]
}
```

</details>

###### Enums

To make things simple, I'll just put a struct variant here, but the style applies to all variant types:

```js
{
  "my_crate::ItemType": Consumable(
    name: "Healing potion"
  )
}
```

<details>
<summary>Old Format</summary>

```js
{
  "type": "my_crate::ItemType",
  "enum": {
    "variant": "Consumable",
    "struct": {
      "name": {
        "type": "alloc::string::String",
        "value": "Healing potion"
      }
    }
  }
}
```

</details>

### Comparison with #4561

This PR is a rebased version of #4561. The reason for the split between the two is because this PR creates a _very_ different scene format. You may notice that the PR descriptions for either PR are pretty similar. This was done to better convey the changes depending on which (if any) gets merged first. If #4561 makes it in first, I will update this PR description accordingly.

---

## Changelog

* Re-worked serialization/deserialization for reflected types
* Added `TypedReflectDeserializer` for deserializing data with known `TypeInfo`
* Renamed `ReflectDeserializer` to `UntypedReflectDeserializer` 
* ~~Replaced usages of `deserialize_any` with `deserialize_map` for non-self-describing formats~~ Reverted this change since there are still some issues that need to be sorted out (in a separate PR). By reverting this, crates like `bincode` can throw an error when attempting to deserialize non-self-describing formats (`bincode` results in `DeserializeAnyNotSupported`)
* Structs, tuples, tuple structs, arrays, and enums are now all de/serialized using conventional serde methods

## Migration Guide

* This PR reduces the verbosity of the scene format. Scenes will need to be updated accordingly:

```js
// Old format
{
  "type": "my_game::item::Item",
  "struct": {
    "id": {
      "type": "alloc::string::String",
      "value": "bevycraft:stone",
    },
    "tags": {
      "type": "alloc::vec::Vec<alloc::string::String>",
      "list": [
        {
          "type": "alloc::string::String",
          "value": "material"
        },
      ],
    },
}

// New format
{
  "my_game::item::Item": (
    id: "bevycraft:stone",
    tags: ["material"]
  )
}
```

[^1]: Some derives omitted for brevity.
  • Loading branch information
MrGVSV committed Sep 20, 2022
1 parent bc863ce commit 9321741
Show file tree
Hide file tree
Showing 22 changed files with 1,555 additions and 913 deletions.
68 changes: 24 additions & 44 deletions assets/scenes/load_scene_example.scn.ron
Original file line number Diff line number Diff line change
Expand Up @@ -3,61 +3,41 @@
entity: 0,
components: [
{
"type": "bevy_transform::components::transform::Transform",
"struct": {
"translation": {
"type": "glam::f32::vec3::Vec3",
"value": (0.0, 0.0, 0.0),
},
"rotation": {
"type": "glam::f32::sse2::quat::Quat",
"value": (0.0, 0.0, 0.0, 1.0),
},
"scale": {
"type": "glam::f32::vec3::Vec3",
"value": (1.0, 1.0, 1.0),
},
},
"bevy_transform::components::transform::Transform": (
translation: (
x: 0.0,
y: 0.0,
z: 0.0
),
rotation: (0.0, 0.0, 0.0, 1.0),
scale: (
x: 1.0,
y: 1.0,
z: 1.0
),
),
},
{
"type": "scene::ComponentB",
"struct": {
"value": {
"type": "alloc::string::String",
"value": "hello",
},
},
"scene::ComponentB": (
value: "hello",
),
},
{
"type": "scene::ComponentA",
"struct": {
"x": {
"type": "f32",
"value": 1.0,
},
"y": {
"type": "f32",
"value": 2.0,
},
},
"scene::ComponentA": (
x: 1.0,
y: 2.0,
),
},
],
),
(
entity: 1,
components: [
{
"type": "scene::ComponentA",
"struct": {
"x": {
"type": "f32",
"value": 3.0,
},
"y": {
"type": "f32",
"value": 4.0,
},
},
"scene::ComponentA": (
x: 3.0,
y: 4.0,
),
},
],
),
Expand Down
29 changes: 23 additions & 6 deletions crates/bevy_reflect/bevy_reflect_derive/src/impls/enums.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ pub(crate) fn impl_enum(reflect_enum: &ReflectEnum) -> TokenStream {
enum_name_at,
enum_field_len,
enum_variant_name,
enum_variant_index,
enum_variant_type,
} = generate_impls(reflect_enum, &ref_index, &ref_name);

Expand Down Expand Up @@ -53,12 +54,13 @@ pub(crate) fn impl_enum(reflect_enum: &ReflectEnum) -> TokenStream {
}
});

let string_name = enum_name.to_string();
let typed_impl = impl_typed(
enum_name,
reflect_enum.meta().generics(),
quote! {
let variants = [#(#variant_info),*];
let info = #bevy_reflect_path::EnumInfo::new::<Self>(&variants);
let info = #bevy_reflect_path::EnumInfo::new::<Self>(#string_name, &variants);
#bevy_reflect_path::TypeInfo::Enum(info)
},
bevy_reflect_path,
Expand Down Expand Up @@ -136,6 +138,14 @@ pub(crate) fn impl_enum(reflect_enum: &ReflectEnum) -> TokenStream {
}
}

#[inline]
fn variant_index(&self) -> usize {
match self {
#(#enum_variant_index,)*
_ => unreachable!(),
}
}

#[inline]
fn variant_type(&self) -> #bevy_reflect_path::VariantType {
match self {
Expand Down Expand Up @@ -254,6 +264,7 @@ struct EnumImpls {
enum_name_at: Vec<proc_macro2::TokenStream>,
enum_field_len: Vec<proc_macro2::TokenStream>,
enum_variant_name: Vec<proc_macro2::TokenStream>,
enum_variant_index: Vec<proc_macro2::TokenStream>,
enum_variant_type: Vec<proc_macro2::TokenStream>,
}

Expand All @@ -267,13 +278,21 @@ fn generate_impls(reflect_enum: &ReflectEnum, ref_index: &Ident, ref_name: &Iden
let mut enum_name_at = Vec::new();
let mut enum_field_len = Vec::new();
let mut enum_variant_name = Vec::new();
let mut enum_variant_index = Vec::new();
let mut enum_variant_type = Vec::new();

for variant in reflect_enum.variants() {
for (variant_index, variant) in reflect_enum.variants().iter().enumerate() {
let ident = &variant.data.ident;
let name = ident.to_string();
let unit = reflect_enum.get_unit(ident);

enum_variant_name.push(quote! {
#unit{..} => #name
});
enum_variant_index.push(quote! {
#unit{..} => #variant_index
});

fn for_fields(
fields: &[StructField],
mut generate_for_field: impl FnMut(usize, usize, &StructField) -> proc_macro2::TokenStream,
Expand Down Expand Up @@ -301,9 +320,6 @@ fn generate_impls(reflect_enum: &ReflectEnum, ref_index: &Ident, ref_name: &Iden
enum_field_len.push(quote! {
#unit{..} => #field_len
});
enum_variant_name.push(quote! {
#unit{..} => #name
});
enum_variant_type.push(quote! {
#unit{..} => #bevy_reflect_path::VariantType::#variant
});
Expand Down Expand Up @@ -342,7 +358,7 @@ fn generate_impls(reflect_enum: &ReflectEnum, ref_index: &Ident, ref_name: &Iden
});

let field_ty = &field.data.ty;
quote! { #bevy_reflect_path::NamedField::new::<#field_ty, _>(#field_name) }
quote! { #bevy_reflect_path::NamedField::new::<#field_ty>(#field_name) }
});
let arguments = quote!(#name, &[ #(#argument),* ]);
add_fields_branch("Struct", "StructVariantInfo", arguments, field_len);
Expand All @@ -358,6 +374,7 @@ fn generate_impls(reflect_enum: &ReflectEnum, ref_index: &Ident, ref_name: &Iden
enum_name_at,
enum_field_len,
enum_variant_name,
enum_variant_index,
enum_variant_type,
}
}
5 changes: 3 additions & 2 deletions crates/bevy_reflect/bevy_reflect_derive/src/impls/structs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,14 +51,15 @@ pub(crate) fn impl_struct(reflect_struct: &ReflectStruct) -> TokenStream {
}
});

let string_name = struct_name.to_string();
let typed_impl = impl_typed(
struct_name,
reflect_struct.meta().generics(),
quote! {
let fields = [
#(#bevy_reflect_path::NamedField::new::<#field_types, _>(#field_names),)*
#(#bevy_reflect_path::NamedField::new::<#field_types>(#field_names),)*
];
let info = #bevy_reflect_path::StructInfo::new::<Self>(&fields);
let info = #bevy_reflect_path::StructInfo::new::<Self>(#string_name, &fields);
#bevy_reflect_path::TypeInfo::Struct(info)
},
bevy_reflect_path,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,15 @@ pub(crate) fn impl_tuple_struct(reflect_struct: &ReflectStruct) -> TokenStream {
}
});

let string_name = struct_name.to_string();
let typed_impl = impl_typed(
struct_name,
reflect_struct.meta().generics(),
quote! {
let fields = [
#(#bevy_reflect_path::UnnamedField::new::<#field_types>(#field_idents),)*
];
let info = #bevy_reflect_path::TupleStructInfo::new::<Self>(&fields);
let info = #bevy_reflect_path::TupleStructInfo::new::<Self>(#string_name, &fields);
#bevy_reflect_path::TypeInfo::TupleStruct(info)
},
bevy_reflect_path,
Expand Down
5 changes: 1 addition & 4 deletions crates/bevy_reflect/bevy_reflect_derive/src/utility.rs
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,7 @@ where
let mut bitset = BitSet::default();

member_iter.fold(0, |next_idx, member| match member {
ReflectIgnoreBehavior::IgnoreAlways => {
bitset.insert(next_idx);
next_idx
}
ReflectIgnoreBehavior::IgnoreAlways => next_idx,
ReflectIgnoreBehavior::IgnoreSerialization => {
bitset.insert(next_idx);
next_idx + 1
Expand Down
51 changes: 48 additions & 3 deletions crates/bevy_reflect/src/enums/dynamic_enum.rs
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ impl From<()> for DynamicVariant {
pub struct DynamicEnum {
name: String,
variant_name: String,
variant_index: usize,
variant: DynamicVariant,
}

Expand All @@ -96,6 +97,30 @@ impl DynamicEnum {
) -> Self {
Self {
name: name.into(),
variant_index: 0,
variant_name: variant_name.into(),
variant: variant.into(),
}
}

/// Create a new [`DynamicEnum`] with a variant index to represent an enum at runtime.
///
/// # Arguments
///
/// * `name`: The type name of the enum
/// * `variant_index`: The index of the variant to set
/// * `variant_name`: The name of the variant to set
/// * `variant`: The variant data
///
pub fn new_with_index<I: Into<String>, V: Into<DynamicVariant>>(
name: I,
variant_index: usize,
variant_name: I,
variant: V,
) -> Self {
Self {
name: name.into(),
variant_index,
variant_name: variant_name.into(),
variant: variant.into(),
}
Expand All @@ -117,6 +142,18 @@ impl DynamicEnum {
self.variant = variant.into();
}

/// Set the current enum variant represented by this struct along with its variant index.
pub fn set_variant_with_index<I: Into<String>, V: Into<DynamicVariant>>(
&mut self,
variant_index: usize,
name: I,
variant: V,
) {
self.variant_index = variant_index;
self.variant_name = name.into();
self.variant = variant.into();
}

/// Create a [`DynamicEnum`] from an existing one.
///
/// This is functionally the same as [`DynamicEnum::from_ref`] except it takes an owned value.
Expand All @@ -129,8 +166,9 @@ impl DynamicEnum {
/// This is functionally the same as [`DynamicEnum::from`] except it takes a reference.
pub fn from_ref<TEnum: Enum>(value: &TEnum) -> Self {
match value.variant_type() {
VariantType::Unit => DynamicEnum::new(
VariantType::Unit => DynamicEnum::new_with_index(
value.type_name(),
value.variant_index(),
value.variant_name(),
DynamicVariant::Unit,
),
Expand All @@ -139,8 +177,9 @@ impl DynamicEnum {
for field in value.iter_fields() {
data.insert_boxed(field.value().clone_value());
}
DynamicEnum::new(
DynamicEnum::new_with_index(
value.type_name(),
value.variant_index(),
value.variant_name(),
DynamicVariant::Tuple(data),
)
Expand All @@ -151,8 +190,9 @@ impl DynamicEnum {
let name = field.name().unwrap();
data.insert_boxed(name, field.value().clone_value());
}
DynamicEnum::new(
DynamicEnum::new_with_index(
value.type_name(),
value.variant_index(),
value.variant_name(),
DynamicVariant::Struct(data),
)
Expand Down Expand Up @@ -226,6 +266,10 @@ impl Enum for DynamicEnum {
&self.variant_name
}

fn variant_index(&self) -> usize {
self.variant_index
}

fn variant_type(&self) -> VariantType {
match &self.variant {
DynamicVariant::Unit => VariantType::Unit,
Expand All @@ -237,6 +281,7 @@ impl Enum for DynamicEnum {
fn clone_dynamic(&self) -> DynamicEnum {
Self {
name: self.name.clone(),
variant_index: self.variant_index,
variant_name: self.variant_name.clone(),
variant: self.variant.clone(),
}
Expand Down
Loading

0 comments on commit 9321741

Please sign in to comment.