Does this support $ref?

[dkengaroo]

Hiya,

I have a schema with
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "title",
"type": "object",
"item": {
"type": "object",
"properties": {
"attribute": {
"$ref": "#/definitions/schema2"
}
},
"required": [
"attribute_info"
]
}
....
something like that.

I kept the form simple with the "*" and a submit button. However the form doesn't render properly where the $ref is.

am I doing something wrong?

[fiddur]

No. References are planned to be implemented, but it isn't yet.

[dkengaroo]

Cool! Great to know :) 👍

[dkengaroo]

Would you happen to know when the implementation is planned to happen?

[fiddur]

Hard to say right now. We might need it in the project we're working on at Textalk, in that case it will be implemented before november. Otherwise it is more uncertain.

If someone else wants to have a go at it, go ahead :) I guess it would be best to follow the reference while recursing into that level when expanding the form, keeping in mind that references could be cyclic. Probably finding a good JSON-Pointer parser...

[dkengaroo]

darn.
I hope that the project will need it :)

I tried the generator with a schema that has a nested object.
didn't work either :[ I'm guessing this only works with flattened schemas?

[torstenrudolf]

@dkengaroo do you mean something like this? (That works fine for me)

The Schema:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "required": ["name"],
    "properties": {
        "name": {
            "$schema": "http://json-schema.org/draft-04/schema#",
            "title": "Person Name",
            "description": "A person's name",
            "type": "object",
            "required": ["familyName", "givenName"],
            "properties": {
                "familyName": {
                    "type": "string", "maxLength": 30, "minLength": 1,
                    "title": "Surname"
                },
                "givenName": {
                    "type": "string", "maxLength": 20, "minLength": 1,
                    "title": "First Name"
                },
                "middleName": {
                    "type": "string", "maxLength": 20, "minLength": 1,
                    "title": "Middle Name"
                }
            }
        }
    }
}

[dkengaroo]

Hiya,

the code you provided works for me too!
However, when I do this:

 "$schema": "http://json-schema.org/draft-04/schema#",
    "type": "object",
    "required": ["name"],
    "properties": {
        "name": {
            "$schema": "http://json-schema.org/draft-04/schema#",
            "title": "Person Name",
            "description": "A person's name",
            "type": "object",
            "properties": {
                "groups":{
                    "type": "object",
                    "required": ["familyName", "givenName"],
                    "properties": {
                        "familyName": {
                            "type": "string", "maxLength": 30, "minLength": 1,
                            "title": "Surname"
                        },
                        "givenName": {
                            "type": "string", "maxLength": 20, "minLength": 1,
                            "title": "First Name"
                        },
                        "middleName": {
                            "type": "string", "maxLength": 20, "minLength": 1,
                            "title": "Middle Name"
                        }

                    }
                }
            }
        }
    }

It works as well. However the word "groups" only shows up as "g" in the form

Does it have issues because of something when it is traversing?

[torstenrudolf]

When I put your example in http://textalk.github.io/angular-schema-form/examples/bootstrap-example.html it works just fine, doesn't it?
Are you using the most recent release v0.7.2? I think there was a problem with default title #67

[dkengaroo]

Hey!
it works when I get v0.7.2! :)
should have done an update earlier...

[eskatos]

$ref support would be a real 👍

[dkengaroo]

whoa! Nice!
👍

pretty spiffy :)

[kuldeepcode]

Could this issue be re-opened, and re-labeled as an enhancement / improvement request? If the maintainers would be receptive to including it, we may be willing to code the improvement and/or have it done.

[nicklasb]

Hi, with regards to the implementation of that, it is quite complicated, if not impossible to solve in a completely general matter, as schemas aren't always exposed as files.

However, a midway is to not overthink it, but to leave some of that to the user. I think the python jsonschema library solves this quite nicely by letting the user implement a uri handler for each namespace.
So when the validator encounters a $ref with an unknown namespace, it checks the if the resolver handles it, which in turn tries its handlers. And if any does, it returns the schema(a dict in python).
In the case below, it is against a file, but it could of course be against an array, object properties or whatever.

As you see, the handler is very easy to implement, and the same should work in JavaScript.

def mbe_uri_handler(self, uri):
    """ Handle the mbe:// references """
    # Use urlparse to parse the file location from the URI
    _file_location = self.json_schema_folder + urlparse(uri).netloc
    return json.loads(open(_file_location, "r", encoding="utf-8").read())



def __init__(self)
    .....
    ..... 
    _resolver = RefResolver(base_uri="",
                           handlers={"mbe": self.mbe_uri_handler}, referrer=None, cache_remote=True)
    self.d4validator = Draft4Validator({}, resolver=_resolver)

So, for example, if I have schemas referencing other scemas, I use a specific namespace. The code below, for example, references another file, and is resolved by the uri handler:

    "schemaId": {
        "description": "The schema id of the node schema, a uuid.",
        "$ref" :"mbe://custom.json#properties/uuid"
    },

I could the declare another handler for another namespace, like "obj:", that references to schemas stored in an objects' properties(like a wannabe dictionary). And I can then, in a schema, use "$ref" :"obj://customer#properties/address_postal" and let the handler figure out that "customer" means the schema stored in $scope.object.customer or whatever.

Docs:
https://python-jsonschema.readthedocs.org/en/latest/references/
Resolver code:
https:/Julian/jsonschema/blob/master/jsonschema/validators.py#L219

[nicklasb]

The same pattern could obviously then apply to the forms as $ref could be used there as well.
It would be great to have inheritance there as well.

[Anthropic]

This is really important for me on a project I am using angular-schema-form for, I wish I had time to work on it myself. I just haven't found a good json resolver that can output a full schema with predefined base URI yet.

That said json-refs is working minus a base URI.

TV4 has JavaScript based code for handling it with a base URI defined, key code is in the function
ValidatorContext.prototype.resolveRefs
and just below it
ValidatorContext.prototype.getSchema

But just does validation and wont output a compiled schema for use with angular-schema-form

Hope that helps someone :)

[nicklasb]

I would think that as long as angular-schema-form is provided a way to get its hands on the referenced JSON structure, finding something in said structure should be fairly trivial. The same method and $ref-syntax could be used to extend forms, I presume.

It is important for me as well to get this working, I do have some time to work on it and make a pull request, but would first like to have some kind of discussion on what would be an acceptable approach.
I am also not completely familiar with the direction of the project; what would be a angular-schema-form-istic solution?

[nicklasb]

If there aren't any major objections I could write something that uses the TV4-solution that Anthropic suggested. I suppose that there could be some value in having at least similar interfaces?

[davidlgj]

@nicklasb check out #118 there seems to be a solution using json-refs, or at least people that have got it working. It feels like "resolving" all the refs in a schema before rendering it with angular schema form is the more flexible solution, but I'm open to changing my mind. Would that work for you?

[fiddur]

I think this must be implemented inside schema form, and it must be done with infinite recursion in mind.

Consider this schema:

{
  "$ref": "#/definitions/sub",
  "definitions": {
    "sub": {
      "type": "object",
      "properties": { "sub": {"$ref": "#/definitions/sub" } },
      "additionalProperties": false
    }
  }
}

This will work in tv4, but it can't be expanded before usage. You can try it in http://json-schema-validator.herokuapp.com/ with data: { "sub": { "sub": { "sub": {} } } }

With this in mind, a form definition can't be generated from the schema without also considering the model; only if the sub-object has the key "sub", THEN the reference must be followed and generate a sub-form.

[davidlgj]

@fiddur yes that is certainly a show stopper for resolving all refs outside of angular schema form, but do we need to support this? Is there a real world example where infinitively recursive forms is useful?

Not generating a form for it though unless its in the model? what does that even mean? How should that be represented in the actual DOM/HTML? Currently an object as in your example is just a fieldset or a section (just a div) or not part of it at all if you point out the specific properties in your form definition.

[eskatos]

Any tree-like structure could be some real world example candidate.
First thing that comes to my mind : directories/folders

[fiddur]

Yes, there are relevant real world examples. A bit too long to put here though… But imagine for example a json schema representing a tree structure. On each level, there might be a name, some data, and possibly another container for the next level, something like:

Schema:
{
  "$ref": "#/definitions/child",
  "definitions": {
    "child": {
      "type": "object",
      "properties": {
        "name": { "type": "string" },
        "children": {
          "type": "array",
          "items": { "$ref": "#/definitions/child" }
        }
      },
      "additionalProperties": false
    }
  }
}

Model:
{
  "name": "Grandfather",
  "children": [
    {
      "name": "My Father",
      "children": [
        { "name": "Me" },
        { "name": "My Sister" }
      ]
    },
    {
      "name": "My Uncle"
    }
  ]
} 

It is not an uncommon structure.

In this case I would want the form to display the above data with name as simple fields, just that the leaf would have an empty array. Adding an item in that array would give the empty form for name and another empty array for children.

[davidlgj]

@eskatos and @fiddur yes a tree structure is a relevant example but IMHO not best solved this way. By rendering it with with schema form it would just be a russion doll of forms withing fieldsets within forms within fieldsets. Each child getting smaller and smaller.

The only way to get around that is to make a new form type (an add-on), that renders a tree properly (which also could add features ). This is possible to do today without $ref support because it's only the add-on that needs to handle the $refs in that case. And it can still use the standard way to render each node in the tree.

That said if you then had such a recursive structure + some other refs that you do want to have resolved you wouldn't be able to resolve them before hand. It might still be a good idea to actually support $ref in the schema, probably in the same fashion as tv4 does it.

[seriousme]

Ok, I did some testing and came up with the following http://jsfiddle.net/jklu/6e4bujwy/
So if I can have a tv4 instance containing the current schema while walking the schema its easy to automatically follow refs.
As tv4 allows multiple schemas using tv4.addSchema its also possible to preload external schema to be used in external refs. tv4 can take care of cyclic checks etc.

Hope this helps..

[urpro-bas]

Ok from seriousme's idea I have created a service that can fetch all dependencies in a tv4 instance. It can then also replace the refs in the schema with actual object references that can be followed. It's still just a quick implementation but let me know if you have any comments. Check it out here: http://jsfiddle.net/9sjaqka7/

The functions:
fetchSchemaWithDependencies takes an uri to a schema and returns a promise for a schemaSetobject. schemaSet.schema is the schema on the uri and schemaSet.tv4 contains the tv4 instance with all the dependencies.
refToObjRef takes a schemaSet and converts the refs to object references.

As far as I figured out it supports all kinds of references, including external and cyclic. Keep in mind though angular-schema-form does not support cylic references.

[seriousme]

Nice, the core schema (http://json-schema.org/draft-04/schema") does not offer CORS headers (and you probably don't want to be depending on some other site either) so it would be nice if you could hand the app a list with aliases e.g. { "http://json-schema.org/draft-04/schema": "myMetaSchema.json" } or { "http://json-schema.org/draft-04/schema": myMetaSchema } to solve that one ;-).

[urpro-bas]

I am not sure, it feels kind of hacky, isn't this what the $schema and id properties are for? xml should have similar problems do you know how javascript xml/xsd handlers solve this problem?

[eskatos]

@urpro-bas XML/XSD handlers solve this using "resolvers" which are pretty much like what @seriousme suggested.

[seriousme]

@urpro-bas that is also why validators like tv4 have the addSchema(uri,url) method ;-)

[mike-marcacci]

Hey all, just to make sure I'm clear on this, is this feature request to support $ref in the traversal/form-generation logic, or support fetching external schemas? The way I see it these are two different features.

The former (support for $ref) would be extremely useful even without the latter. I often define many sub-schemas within one file which can then be referenced after the hash.

The latter (support for external schemas) could be accomplished by pre-registering schemas, much the way jjv does. All schemas could be registered with the schema-form directive with an optional sf-schemas attribute. The current sf-schema attribute could continue to work as it currently does, and also accept a string reference to any registered schema.

This would offload the dozens of inevitable edge-cases that will surface when fetching schemas cross-origin, with special authentication methods, etc.

$scope.schemas = {
    'http://example.com/schemas/one.json': { ... },
    'http://example.com/schemas/two.json': { ... }
}

$scope.mySchema = { ... };

<!-- direct reference -->
<form sf-schema="mySchema" sf-form="form" sf-model="model"></form>

<!-- direct reference w/ external schemas -->
<form sf-schemas="schemas" sf-schema="mySchema" sf-form="form" sf-model="model"></form>

<!-- ID reference w/ external schemas -->
<form sf-schemas="schemas" sf-schema="'http://example.com/schemas/one.json'" sf-form="form" sf-model="model"></form>

[urpro-bas]

@mike-marcacci in some sense my code example supported both, The fetchSchemaWithDependencies method fetched the nessary external schemas, while refToObjRef replaced the ref objects with the object it was referencing, thus easing the traversal of the schema and also implicitly enabling form generation from schema's containing refs.

I agree that those are separate features, but I think are both are necessary features. Finding dependended schemas and fetching prevents you from having to maintain a list of dependencies. I am not sure if a solution without the edge cases is better than no solution at all.

But I also think all those feature start to drift away from the core of angular-schema-form. Maybe a separate schema manager project would be an idea. This could then be used by validators and other users of json schemas.

I am not sure about all this, I am not really experienced.

[seriousme]

I'm no angular guru either, but I'd think that one could hand the whole schema management to tv4.
For backwards compatibility and ease of use, the user could still inject a schema, but now also a tv4 object. Experienced users can then fillup the tv4 object outside angular-schema-form and inject it into the scope.

My 2cts ;-)

[itozapata]

Is there any sort of update on this one as far as when proper $ref support is coming?

[AleksueiR]

👍

[davidlgj]

@itozapata @AleksueiR We need it soonish at Textalk, so I will be implementing it. But I don't know exactly when yet.

[mlegenhausen]

+1

[nicklasb]

I just realized i had spent several hundred hours building this huge solution around believing local refs where supported, thought this was about remote ones. :-/
I also realized that I could work around that, but damn i got scared there for a while.. :-)

WRT rendering, only the "next" level should be rendered. There is no point in rendering all possibilities.

What I also realized would be very useful, when doing this anyway, would be to be able to specify a sub-schema path for a key that defines where in a schema the.

For example, "#/definitions/myCar" would specify that I am editing a myCar, not the entire schema, and #/definitions/myTypeWheels that are referenced within myType should be resolved in the context of the parent schema.

This because many JSON schemas doesn't specify a single structure, but are just holders of definitions. I.e. they have no "properties", just a "definitions" and are resources to other schemas.

[AndreNel7]

Example of "$ref" workaround, with "allOf" workaround:

paste following in test.html (based on: http://schemaform.io/examples/custom-validators.html)

<!DOCTYPE html>
<html>
<head>
<title>Custom validators, async validators etc</title>
<link rel="stylesheet" href="./dist/bootstrap/3.3.4/css/bootstrap.min.css">
<link rel="stylesheet" href="./dist/bootstrap/3.3.4/css/bootstrap-theme.min.css">
</head>
<body ng-app="test" class="container" ng-controller="TestCtrl">

<h3>ASF (Angular Schema Form)</h3>
<form name="theForm">
<div sf-schema="schema" sf-form="form" sf-model="model"></div>
<div>
The form is <em ng-show="theForm.$pristine">pristine</em><em ng-show="theForm.$dirty">dirty</em>
and <em ng-show="theForm.$valid">valid</em><em ng-show="!theForm.$valid">invalid</em>.
</div>
<div>{{prettyModel}}</div>
</form>

<script type="text/javascript" src="./bower_components/tv4/tv4.js"></script> <!-- Tiny Validator for Schema v4 - https:/geraintluff/tv4 -->
<script type="text/javascript" src="./bower_components/angular/angular.min.js"></script>
<script type="text/javascript" src="./bower_components/angular-sanitize/angular-sanitize.min.js">
</script>

<script type="text/javascript" src="./bower_components/objectpath/lib/ObjectPath.js"></script>

<script type="text/javascript" src="./dist/schema-form.js"></script>
<script type="text/javascript" src="./dist/bootstrap-decorator.min.js"></script>

<script type="text/javascript" src="./json-refs-standalone.js"></script> <!-- https:/whitlockjc/json-refs -->

<script type="text/javascript" src="./jquery-2.1.3.js"></script> <!-- for json extend / merge -->
<script type="text/javascript" src="./jQuery.extendext.min.js"></script> <!-- for json extend / merge - with array extend (instead of array overwrite) - https:/mistic100/jQuery.extendext -->

<script>
    function getObject(in_obj, in_prop, in_val, in_path) // http://stackoverflow.com/questions/15523514/find-by-key-deep-in-nested-json-object // http://jsfiddle.net/FM3qu/7/
    {
        if (!in_path) in_path = '';
        var result = null;
        if (in_obj instanceof Array)
        {
            for (var i = 0; i <in_obj.length; i++)
            {
                result = getObject(in_obj[i], in_prop, in_val);
                if (result)
                {
                    break;
                }
            }
        }
        else
        {
            for (var prop in in_obj)
            {
                //console.log(prop + ': ' + in_obj[prop] + ' -> ' + in_prop);
                if (prop == in_prop)
                {
                    //console.log(prop);
                    if (in_val)
                    {
                        if (in_obj[prop] == in_val)
                        {
                            return in_obj;
                        }
                    }
                    else
                    {
                        return in_path += '.' + prop; // return path rather than object
                        //console.log(in_path);
                        //return in_obj;
                    }
                }
                if (in_obj[prop] instanceof Object || in_obj[prop] instanceof Array)
                {
                    //console.log(in_path + '.' + prop);
                    result = getObject(in_obj[prop], in_prop, in_val, in_path + '.' + prop);
                    if (result)
                    {
                        break;
                    }
                }
            }
        }
        return result;
    }

    var defRefJsonSchema =
    {
        "$schema": "http://json-schema.org/draft-04/schema#",

        "definitions": {
            "address": {
                "type": "object",
                "properties": {
                    "street_address": { "type": "string" },
                    "city": { "type": "string" },
                    "state": { "type": "string" }
                },
                "required": ["street_address", "city", "state"]
            },
            "country": {
                "type": "object",
                "properties": {
                    "country": { "type": "string" },
                    "country-dial-code": { "type": "integer" },
                    "country-short": { "type": "string" }
                },
                "required": ["country", "country-dial-code", "country-short"]
            }
        },
        "type": "object",
        "properties": {
            "billing_address": { "allOf": [{ "type": "object", "properties": { "billing_id": { "type": "number" } } }, { "$ref": "#/definitions/address" }, { "$ref": "#/definitions/country" }] },
        "shipping_address": { "allOf": [ {"type": "object", "properties": { "shipping_id": { "type": "number" } } }, { "$ref": "#/definitions/address" } ] },
        }
    };
    // ___________________
    //| "$ref" workaround |
    //|___________________|
    JsonRefs
    .resolveRefs(defRefJsonSchema, {
        "filter": ['relative', 'local']
    })
    .then(function (res) // https:/whitlockjc/json-     refs/blob/master/docs/API.md#module_JsonRefs.resolveRefs
    {
        var sjson = JSON.stringify(res.resolved);

        // ____________________
        //| "allOf" workaround |
        //|____________________|
        while (sjson.indexOf('allOf') !== -1) // loop until all 'allOf' have been replaced
        {
            // Replace allOf with combined JSON object
            //console.log('---------------------allOf-------------------------------');
            //console.log(res.resolved);
            //var aAllOf = getObject(res.resolved, 'allOf').allOf;
            var aAllOfPath = getObject(res.resolved, 'allOf'); // hacked getObject() to return path to sub-object (from inside the main-object) rather than the sub-object itself
            //console.log(aAllOfPath);
            var aAllOf = eval('res.resolved' + aAllOfPath); // get sub-object from path in main-object
            //console.log(aAllOf);

            //var result = $.extend(true, {}, aAllOf[0], aAllOf[1]); // resultant object from combining allOf-path's objects
            var result = {}; // resultant object from combining allOf-path's objects
            for (var i = 0, length = aAllOf.length; i <length; i++)
            {
                //console.log(aAllOf[i]);
                //$.extend(true, result, aAllOf[i]);
                $.extendext(true, 'extend', result, aAllOf[i]); // https:/mistic100/jQuery.extendext
            }
            console.log(JSON.stringify(result));

            eval('delete res.resolved' + aAllOfPath); // delete allOf
            //console.log('res.resolved' + aAllOfPath.split('.allOf')[0]);
            eval('res.resolved' + aAllOfPath.split('.allOf')[0] + ' = ' + JSON.stringify(result)); // put combined obj in place of deleted allOf
            //console.log(res.resolved);
            sjson = JSON.stringify(res.resolved);
        }
        //console.log(sjson)

        buildASF(res.resolved);
    }, function (err)
    {
        console.log(err.stack);
    });

    function buildASF(jsonSchema)
    {
        angular.module('test', ['schemaForm']).controller('TestCtrl', function ($scope, $q, $timeout)
        {
            $scope.schema = jsonSchema;

            $scope.form = [
                "*",
                {
                    "type": "submit",
                    "title": "OK"
                }
            ];

            $scope.model =
            {
                "billing_address": { "billing_id": 111, "street_address": "123 Home", "city":"centurion", "state":"Gauteng", "country":"South Africa", "country-short":"ZA", "country-dial-code":27 },
                "shipping_address": { "shipping_id": 222, "street_address": "7 Ship" }
            };

            $scope.$watch('model', function (value)
            {
                if (value)
                {
                    $scope.prettyModel = JSON.stringify(value, undefined, 2);
                }
            }, true);
        });
    }
</script>

</body>
</html>

[mariusheil]

Any updates on this issue?

[Anthropic]

@mariusheil I have one more failing test to fix before I start taking a look at it, Although anything implemented would not be much beyond what @AndreNel7 posted above initially, just internally. It could be a while before we can fully support recursive form generation.

[Anthropic]

@mariusheil the second alpha has $ref pre-processing

[Anthropic]

This is now in 1.0.0-alpha.2 and has passing tests