Testing generated code in-depth

[jonaslagoni]

What we are trying to solve?

Modelina is aiming to be the defacto standard for generating data models, whether it is to generate them in templates or somewhere else.

Imagine Modelina has to support +50 different inputs, +50 different outputs, with +100 different configurations in each output language. How do we ensure that we always produce the expected data models and they work at runtime?

Currently, we only test that we generate expected output through snapshot matching and that they can compile/transpile. None of which ensures that the generated output is semantically correct.

For example, say we generate Java data models that overwrite the hashcode method, we want to ensure that it does exactly what it is supposed to do at runtime. And that is that one instance of a model should match against another instance when they are created with the same values. This means we need to be able to write tests such as the following:

@Test
public void testHashcode_Symmetric() {
    Person x = new Person("Foo Bar");  // equals and hashCode check name field value
    Person y = new Person("Foo Bar");
    Assert.assertTrue(x.hashCode() == y.hashCode());
}
@Test
public void testHashcode_NonSymmetric() {
    Person x = new Person("Foo Bar");
    Person y = new Person("Foo");
    Assert.assertFalse(x.hashCode() == y.hashCode());
}

So we need to figure out how we from TS can run and potentially generate tests in all of the output languages 🤯 But don't worry, we start small and keep iterating.

GSOC status

Difficulty: HARD (MEDIUM, if you are already well established in the output languages)

What you will learn:

• What Modelina is and how it works.
  • This includes the basics of AsyncAPI, JSON Schema, and OpenAPI.
• GitHub CI and workflows used by Modelina.
• How to work with code generation.
• How you can test code generation in other projects such as code templates (same principles apply).
• How to create basic projects and tests in Java
• How to create basic projects and tests in C#
• How to create basic projects and tests in Go

Requirements:

• You should have basic knowledge of TypeScript as this is the language Modelina is written in.
• Hunger for working in multiple languages and with code generation.
  • To solve this issue you need to like working in other languages since the tests will be written in the output language and not TS. You don't need to know the language, but be willing to learn!

[ritik307]

@jonaslagoni What I understood from the above proposal is that we are currently using snapshot matching to test our code but now we need to surpass that approach to find an approach through which we can ensure our generated output also works basically how to execute the output code base. Kindly correct me if I am wrong 😊.
The thing I am not clear with is how we will be able to handle 50+ different inputs and generate output for them?

[jonaslagoni]

Kindly correct me if I am wrong 😊.

Nothing to correct, you are on point 👍

The thing I am not clear with is how we will be able to handle 50+ different inputs and generate output for them?

Do you mean in general how Modelina will support it? Or how you can handle that in testing?

[ritik307]

@jonaslagoni I am concerned about how we will be able to handle it in the testing part.

[ron-debajyoti]

we only test that we generate expected output through snapshot matching and that they can compile/transpile. None of which ensures that the generated output actually works

Oh, this is one of the things you asked recently in the current PR for enabling TS input!

The thing I am not clear with is how we will be able to handle 50+ different inputs and generate output for them?

@ritik307 like @jonaslagoni said I feel like we need to start with C#, Java, and Go first as these are the languages we currently support. We can extend the testing support gradually as we support more languages.

[ritik307]

@ritik307 like @jonaslagoni said I feel like we need to start with C#, Java, and Go first as these are the languages we currently support. We can extend the testing support gradually as we support more languages.

Got it. Thanks for the help 😊

[jonaslagoni]

Well, that is an engineering challenge that we need to solve. How do we want to do it exactly? Let's see what we figure out. The entire GSOC for us (or at least for me) is to try not to give you a task that you can follow to the letter.

Instead, we will try to solve it together and break it down, so you in the future can apply the principles to any engineering challenge you face 🙂

@ritik307 like @jonaslagoni said I feel like we need to start with C#, Java, and Go first as these are the languages we currently support. We can extend the testing support gradually as we support more languages.

Exactly 👌 We start small and keep integrating the changes into the library. Eventually, we will have a framework in place that supports all inputs and outputs, but we start with 1 and work our way there 🙂

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[jonaslagoni]

Still relevant.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity 😴

It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation.

There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model.

Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here.

Thank you for your patience ❤️

[jonaslagoni]

As this is already well under way, I am going to close this issue, we still have some runtimes to setup, but overall it's moving in the right direction 👌