Semantic Graphs for Interoperability between Websites and AI Assistants

[AdamSobieski]

Introduction

Semantic Web technologies could serve as a bridge or glue between websites and interoperating AI assistants.

Use Cases

It would enable accessibility scenarios for end-users to be able to use natural language with interoperating AI assistants to ask questions about, to engage in dialogues about, and to interact with documents' components (e.g., images, animations, videos, 3D graphics, and charts).

Use cases of interest also include using interoperable AI assistants as natural-language user interfaces for:

1. interacting with intricate charts, diagrams, and schematics (e.g., subway maps, engine schematics)
2. Web-based CAD and CAE applications
3. Web-based scientific and educational computer simulations
4. Web-based interactive fiction and videogames

Objectives

By using Semantic Web technologies:

1. APIs specific to AI assistants (which might version) could be abstracted over or encapsulated
2. websites could work with multiple, ideally all, AI assistants
3. internationalization
4. extensibility

Capturing Components' Visual Representations

Interoperating multimodal AI systems could request and receive visual representations of Web documents' components to answer questions about and to engage in dialogues about them.

Navigating from Visual Representations to Components

An end-user might ask a question about a chart or a diagram to an AI assistant; its response might include an image with a selection of content made; then, developers might want to obtain, e.g., to provide the AI assistant with, information about any components in that selection, navigating from visual representations into semantic graphs describing components.

If points on components' visual representations are involved, one could utilize elementFromPoint() or elementsFromPoint(). For 3D-graphics scenarios, there are techniques involving raycasting.

Describing Components

To enhance Q&A and dialogue about Web documents' components, pertinent data and functionalities could accompany their captured visual representations.

With respect to describing Web document' components, alongside visual representations, one could additionally provide sets of components':

1. described semantic classes
2. natural-language descriptions
3. described data or file attachments
4. described relationships to other components
5. described readable/writeable properties
6. described invokable functions or methods
7. described constraints which define when the component is valid

A useful model to consider is that of a semantic graph of interrelated components. Components could be subcomponents of one another, contain one another, be parts of one another, or be interrelated in other ways.

API Sketch

There are many possibilities with respect to JavaScript APIs and object models for using semantic graphs.

For a list of JavaScript libraries, see: https://rdf.js.org/ .

Here is a sketch of an API and object model, expressed using TypeScript:

type IRI =
{
    readonly nodeType: 1;

    readonly namespace: string;
    readonly id: string;
}

type BlankNode =
{
    readonly nodeType: 2;

    readonly id: string;
}

type Variable =
{
    readonly nodeType: 3;

    readonly id: string;
}

type Literal =
{
    readonly nodeType: 4;

    readonly value: string;
    readonly dataType: IRI;
    readonly lang: string | null;
    readonly dir: 'ltr' | 'rtl' | null;
}

type TripleGround =
{
    readonly nodeType: 5;
    readonly isGround: true;

    /** subject */
    readonly 0: IRI | BlankNode;
    /** predicate */
    readonly 1: IRI;
    /** object */
    readonly 2: IRI | BlankNode | Literal | TripleGround;

    readonly subject: IRI | BlankNode
    readonly predicate: IRI
    readonly object: IRI | BlankNode | Literal | TripleGround;
}

type TripleNotGround =
{
    readonly nodeType: 5;
    readonly isGround: false;

    /** subject */
    readonly 0: IRI | BlankNode | Variable;
    /** predicate */
    readonly 1: IRI | Variable;
    /** object */
    readonly 2: IRI | BlankNode | Variable | Literal | Triple;

    readonly subject: IRI | BlankNode | Variable;
    readonly predicate: IRI | Variable;
    readonly object: IRI | BlankNode | Variable | Literal | Triple;
}

type Triple = | TripleGround | TripleNotGround;

type GraphGroundOnly = Iterable<TripleGround> &
{
    readonly isGroundOnly: true;
    readonly isValidOnly: boolean;
    readonly isReadOnly: boolean;

    readonly isGround: true;
    readonly isValid: boolean;
    readonly isEmpty: boolean;

    readonly variables: ReadonlyArray<Variable> & { readonly length: 0 };

    readonly validators: ReadonlyArray<Validator>;
    readonly provenance: Provenance | null;

    readonly baseIRI: string | null;
    readonly name: IRI | BlankNode | null;

    add(subject: IRI | BlankNode, predicate: IRI, object: IRI | BlankNode | Literal | TripleGround): boolean;
    contains(subject: IRI | BlankNode, predicate: IRI, object: IRI | BlankNode | Literal | TripleGround): boolean;
    remove(subject: IRI | BlankNode, predicate: IRI, object: IRI | BlankNode | Literal | TripleGround): boolean;
    replace(map: ReadonlyMap<Variable, IRI | BlankNode | Variable | Literal | TripleGround>, options?: GraphCreationOptions): Graph;
    update(update: Update): boolean;
}

type GraphNotGroundOnly = Iterable<Triple> &
{
    readonly isGroundOnly: false;
    readonly isValidOnly: boolean;
    readonly isReadOnly: boolean;

    readonly isGround: boolean;
    readonly isValid: boolean;
    readonly isEmpty: boolean;

    readonly variables: ReadonlyArray<Variable>;

    readonly validators: ReadonlyArray<Validator>;
    readonly provenance: Provenance | null;

    readonly baseIRI: string | null;
    readonly name: IRI | BlankNode | Variable | null;

    add(subject: IRI | BlankNode | Variable, predicate: IRI | Variable, object: IRI | BlankNode | Variable | Literal | Triple): boolean;
    contains(subject: IRI | BlankNode | Variable, predicate: IRI | Variable, object: IRI | BlankNode | Variable | Literal | Triple): boolean;
    remove(subject: IRI | BlankNode | Variable, predicate: IRI | Variable, object: IRI | BlankNode | Variable | Literal | Triple): boolean;
    replace(map: ReadonlyMap<Variable, IRI | BlankNode | Variable | Literal | Triple>, options?: GraphCreationOptions): Graph;
    update(update: Update): boolean;
}

type Graph = | GraphGroundOnly | GraphNotGroundOnly;

declare var Graph:
{
    readonly IRI: 1;
    readonly BLANK: 2;
    readonly VARIABLE: 3;
    readonly LITERAL: 4;
    readonly TRIPLE: 5;
}

class GraphCreationOptions
{
    isGroundOnly?: true | false;
    isValidOnly?: true | false;
    isReadOnly?: true | false;

    baseIRI?: string;
    name?: IRI | BlankNode | Variable;

    validators?: Iterable<Validator>;
}

type Update =
{
    readonly delete: Graph;
    readonly insert: Graph;
    readonly where: Graph;
}

type Validator = Graph &
{
    validate(graph: Graph): boolean;
    validate(graph: Graph, update: Update): boolean;
}

class ValidatorCreationOptions { }

type Reasoner = Graph &
{
    reason(source: Graph, options?: GraphCreationOptions): Graph;
}

class ReasonerCreationOptions { }

type Provenance =
{
    reasoner: Reasoner;
    source: Graph;
}

type SemanticsImplementation =
{
    createIRI(iri: string): IRI;
    createIRI(namespace: string, id: string): IRI;
    createBlank(id: string): BlankNode;
    createVariable(id: string): Variable;
    createLiteral(value: boolean | number | bigint | string, datatype?: IRI, lang?: string, dir?: 'ltr' | 'rtl'): Literal;
    createTriple(subject: IRI | BlankNode | Variable, predicate: IRI | Variable, object: IRI | BlankNode | Variable | Literal | Triple): Triple;
    createGraph(triples?: Iterable<Triple>, options?: GraphCreationOptions): Graph;
    createUpdate(deletions: Graph, insertions: Graph, where?: Graph): Update;
    createValidator(graph: Graph, options?: ValidatorCreationOptions): Validator;
    createReasoner(graph: Graph, options?: ReasonerCreationOptions): Reasoner;
}

type Semantics =
{
    readonly implementation: SemanticsImplementation;

    open(): void;

    getDescription(element: Element): GraphGroundOnly;
    hasDescription(element: Element): boolean;
    setDescription(element: Element, graph: Graph): void;
    updateDescription(element: Element, update: Update): boolean;
    deleteDescription(element: Element): void;

    close(): void;
}

Examples

Example 1

This example shows constructing and storing a knowledge graph about a custom chart element.

<custom-chart id="chart-123" src="specification.json" />

let ex = 'http://www.example.org/ns#'
let rdf = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#';
let xsd = 'http://www.w3.org/2001/XMLSchema#';
let dcterms = 'http://purl.org/dc/terms/';
let doco = 'http://purl.org/spar/doco/';
let fno = 'https://w3id.org/function/ontology#';

let element = document.getElementById('chart-123');
assert(element !== null);

let i = window.semantics.implementation;
let graph = i.createGraph([], { isReadOnly: false, isGroundOnly: true });
assert(graph.isGroundOnly === true);

let component = element.id === null ? i.createBlank(nextIdentifier()) : i.createIRI(element.baseURI, element.id);

// described semantic classes

graph.add(component, i.createIRI(rdf, 'type'), i.createIRI(doco, 'Figure'));
graph.add(component, i.createIRI(rdf, 'type'), i.createIRI(ex, 'BarChart'));

// natural-language descriptions

graph.add(component, i.createIRI(dcterms, 'description'), i.createLiteral('This is a bar chart showing the average temperature in the United States, per year, between the years of 1920 and 2020.', i.createIRI(rdf, 'langString'), 'en'));

// described data or file attachments

var data1 = i.createBlank(nextIdentifier());

graph.add(component, i.createIRI(ex, 'hasData'), data1);
graph.add(data1, i.createIRI(rdf, 'type'), i.createIRI(ex, 'Data'));
graph.add(data1, i.createIRI(dcterms, 'description'), i.createLiteral('This is the data of the bar chart in comma-separated values format (.CSV).', i.createIRI(rdf, 'langString'), 'en'));
graph.add(data1, i.createIRI(dcterms, 'format'), i.createLiteral('text/csv'));
graph.add(data1, i.createIRI(rdf, 'value'), i.createLiteral('...'));

var data2 = i.createBlank(nextIdentifier());

graph.add(component, i.createIRI(ex, 'hasData'), data2);
graph.add(data2, i.createIRI(rdf, 'type'), i.createIRI(ex, 'Data'));
graph.add(data2, i.createIRI(dcterms, 'description'), i.createLiteral('This is the data of the bar chart in spreadsheet format (.XSLX).', i.createIRI(rdf, 'langString'), 'en'));
graph.add(data2, i.createIRI(dcterms, 'format'), i.createLiteral('application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'));
graph.add(data2, i.createIRI(rdf, 'value'), i.createLiteral('...'));

// described relationships to other components

graph.add(component, i.createIRI(ex, 'navigationProperty'), i.createIRI(ex, 'hasSubcomponent'));
graph.add(i.createIRI(ex, 'hasSubcomponent'), i.createIRI(rdf, 'type'), i.createIRI(rdf, 'Property'));
graph.add(i.createIRI(ex, 'hasSubcomponent'), i.createIRI(dcterms, 'description'), i.createLiteral('This is a description of a relation for AI systems.', i.createIRI(rdf, 'langString'), 'en'));

var labels = ['point-1920', 'point-1921', /* ... */ 'point-2020'];

for (const label of labels)
{
    var subcomponent = i.createBlank(nextIdentifier());

    graph.add(component, i.createIRI(ex, 'hasSubcomponent'), subcomponent);
    graph.add(subcomponent, i.createIRI(rdf, 'type'), i.createIRI(ex, 'BarChartPoint'));
    graph.add(subcomponent, i.createIRI(ex, 'hasLabel'), i.createLiteral(label));
    //...
}

graph.add(component, i.createIRI(ex, 'navigationProperty'), i.createIRI(ex, 'hasSupercomponent'));
graph.add(i.createIRI(ex, 'hasSupercomponent'), i.createIRI(rdf, 'type'), i.createIRI(rdf, 'Property'));
graph.add(i.createIRI(ex, 'hasSupercomponent'), i.createIRI(dcterms, 'description'), i.createLiteral('This is a description of a relation for AI systems.', i.createIRI(rdf, 'langString'), 'en'));

var supercomponent = i.createIRI(window.document.documentURI);
graph.add(component, i.createIRI(ex, 'hasSupercomponent'), supercomponent);
graph.add(supercomponent, i.createIRI(rdf, 'type'), i.createIRI(ex, 'Document'));
graph.add(supercomponent, i.createIRI(rdf, 'hasLabel'), i.createLiteral('super'));

// described invokable functions or methods

var function1 = i.createBlank(nextIdentifier());

graph.add(component, i.createIRI(ex, 'hasFunction'), function1);
graph.add(function1, i.createIRI(rdf, 'type'), i.createIRI(fno, 'Function'));
graph.add(function1, i.createIRI(fno, 'name'), i.createLiteral('pan'));
graph.add(function1, i.createIRI(dcterms, 'description'), i.createLiteral('This is a description for AI systems of the pan function.', i.createIRI(rdf, 'langString'), 'en'));

var list1 = i.createBlank(nextIdentifier());
var parameter1 = i.createBlank(nextIdentifier());

graph.add(function1, i.createIRI(fno, 'expects'), list1);
graph.add(list1, i.createIRI(rdf, 'type'), i.createIRI(rdf, 'List'));
graph.add(list1, i.createIRI(rdf, 'first'), parameter1);
graph.add(parameter1, i.createIRI(rdf, 'type'), i.createIRI(fno, 'Parameter'));
graph.add(parameter1, i.createIRI(fno, 'name'), i.createLiteral('direction'));
graph.add(parameter1, i.createIRI(dcterms, 'description'), i.createLiteral('This is a description for AI systems of the pan function\'s direction parameter, which should be from an enumeration of values { left, right }.', i.createIRI(rdf, 'langString'), 'en'));
graph.add(parameter1, i.createIRI(fno, 'required'), i.createLiteral(true, i.createIRI(xsd, 'boolean')));

var list2 = i.createBlank(nextIdentifier());
var output1 = i.createBlank(nextIdentifier());

graph.add(function1, i.createIRI(fno, 'returns'), list2);
graph.add(list2, i.createIRI(rdf, 'type'), i.createIRI(rdf, 'List'));
graph.add(list2, i.createIRI(rdf, 'first'), output1);
graph.add(output1, i.createIRI(rdf, 'type'), i.createIRI(fno, 'Output'));

//...

window.semantics.open();
window.semantics.setDescription(element, graph);
window.semantics.close();

Example 2

let ex = 'http://www.example.org/ns#'
let rdf = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#';
let xsd = 'http://www.w3.org/2001/XMLSchema#';
let dcterms = 'http://purl.org/dc/terms/';

let i = window.semantics.implementation;
let graph = i.createGraph([], { isGroundOnly: true, isReadOnly: false, name: i.createIRI('http://www.namedgraph.org/ns#graph-123') });
assert(graph.isGroundOnly === true);
assert(graph.name !== null);

var element = window.document.getElementById('widget-123');
assert(element !== null);

let component = element.id === null ? i.createBlank(nextIdentifier()) : i.createIRI(element.baseURI, element.id);

graph.add(graph.name, i.createIRI(ex, 'about'), component);

graph.add(component, i.createIRI(rdf, 'type'), i.createIRI(ex, 'Widget'));
graph.add(component, i.createIRI(ex, 'color'), i.createLiteral('blue'));
graph.add(component, i.createIRI(ex, 'mass'), i.createLiteral(1234, i.createIRI(xsd, 'unsignedInt')));
graph.add(component, i.createIRI(dcterms, 'description'), i.createLiteral('This is a long-form description of a widget instance for an interoperating AI-assistant technology.'));

window.semantics.open();
window.semantics.setDescription(element, graph);
window.semantics.close();

Example 3

let ex = 'http://www.example.org/ns#'
let rdf = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#';
let xsd = 'http://www.w3.org/2001/XMLSchema#';

let i = window.semantics.implementation;
let update = i.createUpdate(i.createGraph([], { isGroundOnly: false, isReadOnly: false }), i.createGraph([], { isGroundOnly: false, isReadOnly: false }), i.createGraph([], { isGroundOnly: false, isReadOnly: false }));
assert(update.where.isGroundOnly === false);
assert(update.delete.isGroundOnly === false);
assert(update.insert.isGroundOnly === false);

let x = i.createVariable('x');

update.where.add(x, i.createIRI(rdf, 'type'), i.createIRI(ex, 'Widget'));
update.where.add(x, i.createIRI(ex, 'color'), i.createLiteral(oldColor));
update.delete.add(x, i.createIRI(ex, 'color'), i.createLiteral(oldColor));
update.insert.add(x, i.createIRI(ex, 'color'), i.createLiteral(newColor));

window.semantics.open();
window.semantics.updateDescription(element, update);
window.semantics.close();

Ontologies

The ontologies sketched, above, are intended to be placeholders for illustration purposes.

Charts, for instance, can have more features than semantic classes, natural-language descriptions, data, relationships to their data points, and invokable functions for interaction. Projects like Vega and Data Navigator indicate that charts can also have readable and writeable properties as well as other subcomponents including signals, scales, projections, axes, legends, and marks.

Graph Updating

The second example, above, shows that semantic diffs or deltas could be used to efficiently update components' knowledge-graph representations, e.g., in the event that a component's property's value changed.

There is an interesting relationship between semantic diffs or deltas and SPARQL's DELETE/INSERT operation.

See, for example:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX ex:  <http://example.org/>
PREFIX doc: <http://example.org/document.html#>

DELETE { doc:widget-123 ex:color '#000000' }
INSERT { doc:widget-123 ex:color '#FF0000' }

and:

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX ex:  <http://example.org/>

DELETE { ?x ex:color '#000000' }
INSERT { ?x ex:color '#FF0000' }
WHERE
{
  ?x rdf:type ex:Widget .
  ?x ex:guid '2aaa8ca1-f801-49b7-a1a2-5432479bba9e' .
} 

Protocols

Semantic Web technologies could be utilized to define communication protocols, these including sets of message classes, where individual messages would be semantic graphs capable of being validated.

Webpages and AI assistants initiating interoperation could, then, negotiate between (versions of) communication protocols with which to recognize and validate messages. Approaches like these would enable the simultaneous availability of different (versions of) communication protocols between websites and AI assistants.

Example 4

Inspired by the Web Messaging API, perhaps communication channels could be opened between websites and (browser-integrated) artificial-intelligence assistants for specified protocols in a manner resembling:

var channel = window.assistant.openChannelForProtocol('http://example.org/2024/protocol-123/#');

if(channel != null)
{
  channel.onmessage = (event) => {
      switch(event.class)
      {
        case 'http://example.org/2024/protocol-123/#messageClass1':
          messageHandler1(event.graph);
          break;
        case 'http://example.org/2024/protocol-123/#messageClass2':
          messageHandler2(event.graph);
          break;
        ...
      }
    };

  channel.postMessage(...);
}

Semantic protocol definitions could be used to constrain and secure communications. In addition to defining message classes with which to validate messages, protocol definitions could specify rules (including time-based), valid sequences of classes of messages, valid state transitions, and so forth.

In addition to both websites and artificial-intelligence assistants being capable of verifying messages and conformance with semantically-defined protocols, Web browsers, as a communication-mediating software, could be configured to verify their conformance with protocols as messages were exchanged.

Adapters

Interoperating artificial-intelligence systems needn't generate semantic-graph-based messages for websites directly. These systems might, instead, generate natural language, special strings, XML, JSON, or (Python) source code.

Software adapters would, as considered, accompany artificial-intelligence systems, mapping semantic messages and any captured multimedia for them to implementation-dependent prompt components and/or API calls while also recognizing, validating, and translating systems' implementation-dependent responses and actions intended for websites (e.g., navigations or invocations / tool uses) to semantic messages.

In this way, implementation-dependent APIs, which might version, could be abstracted over and websites would be able to interoperate with multiple, ideally all, artificial-intelligence assistants.

Development and Debugging

It could prove useful for Web developers to be able to obtain and explore components' semantic graphs when inspecting documents' elements via Web browsers' F12 developer tools.

More broadly, it could prove useful for developers to be able to log and/or to display on a timeline those semantic-graph-based messages exchanged between websites and interoperating artificial-intelligence assistants using F12 developer tools or IDEs.

Selections and Groups

It may be the case that selections and groups of multiple components should be capable of being formed and interacted with via natural-language user interfaces and that these selections and groups could also have semantic classes, natural-language descriptions, described data or file attachments, relationships to other things, readable/writeable properties, invokable functions or methods, and constraints.

Natural-language Search

In addition to navigating from components to interrelated components, e.g., to and through subcomponents, there are natural-language searches to consider. That is, artificial-intelligence assistants may be expected to be able to intelligently search for, retrieve, jump to, and select user-described contents in documents for subsequent Q&A, dialogue, or interaction.

See Also

• https://www.w3.org/reports/ai-web-impact/
• https://w3ctag.github.io/design-principles/
• https://www.w3.org/TR/mediacapture-fromelement/
• https://screen-share.github.io/element-capture/
• https:/WICG/canvas-place-element
• https://www.w3.org/TR/rdf12-concepts/
• https://www.w3.org/TR/shacl/
• https://w3c.github.io/rdf-canon/spec/
• https://www.w3.org/TR/sparql11-overview/
• https://www.w3.org/DesignIssues/Diff
• https://json-ld.org/
• https://json-ld.org/spec/latest/json-ld-rdf/
• https://fno.io/spec/
• https://www.frank.computer/data-navigator/
• https:/EmergenceAI/Agent-E

Conclusion

What do you think about these uses of Semantic Web technologies for interoperability between websites and AI assistants?

Thank you for any comments on these ideas.

[domoritz]

CC @frankelavsky who works on visualization accessibility and created data navigator.

[domoritz]

I love the idea of using a graph structure to provide access to semantic information about e.g. charts. Two notes

1. In Data Navigator, each node has an associated region on screen. I think that would be useful for being able to navigate from a visual into the semantic graph. Would this make sense to add to the example?
2. There is a tradeoff between a static, declarative specification of the graph and building it at runtime on request (which is what I think this proposal does, no?). A static version can be validated more easily and also doesn't reveal to the website that someone is using accessibility or AI technologies that are navigating the graph.

[frankelavsky]

There is an ongoing discussion over in aria about how best to structure data visualizations, and I think that this idea you've proposed @AdamSobieski actually goes even further than charts and graphs, which I love.

But in that other discussion I mentioned that it might be useful to generate some structure as-needed, since not all semantic relationships are known until the graph is created and someone interacts with it. That being said, I think that the core idea of providing a graph with as many known or possible semantic relationships is great and the use-case for AI + intelligent agents is solid. I'd love to see a working prototype of this!

[AdamSobieski]

@frankelavsky, thank you for that hyperlink to that ARIA discussion. Yes, I am hoping to find general solutions which can also enable AI-assistant interoperability with 3D illustrations and, eventually, CAD/CAE, computer simulations, interactive fiction, and videogames.

@domoritz, to your first point, an end-user might ask a question about a chart or a diagram to an AI assistant; its response might include an image with a selection of content made; then, developers might want to obtain, e.g., to provide the AI assistant with, information about those components in that selection, navigating "from a visual into the semantic graph".

If points on components' visual representations are involved, one could utilize elementFromPoint() or elementsFromPoint(). I can put some more thought into 2D-canvas scenarios. For 3D-graphics scenarios, there are techniques involving raycasting.

Did I understand your first point?

To your second point, brainstorming, one means of providing runtime-dynamic knowledge graphs while also ensuring privacy with respect to the presence of interoperating accessibility technologies would be for documents' elements/components to be able to write their overriding, custom knowledge graphs into a browser-held lookup table.

For example, a custom element, <custom-chart id="chart-123">, could do something like the following to place or to refresh a custom knowledge graph in a browser-held lookup table:

var elem = document.getElementById('chart-123');

// build knowledge graph here

window.semantics.setDescription(elem, graph);

Accessibility and AI-assistant technologies would then be able to interact with browsers, unbeknownst to webpages, to access and to read these lookup tables which map some elements/components to their custom knowledge graphs.

Interfaces for accessing browser-held lookup tables should include synchronization constructs so that, for instance, webpages could request locks on such tables to create or update the descriptions of multiple elements/components before releasing those locks.

var elem = document.getElementById('chart-123');

// build knowledge graph here

window.semantics.open();
window.semantics.setDescription(elem, graph);
window.semantics.close();

Browser-held lookup tables could protect privacy about the presence of accessibility or AI-assistant technologies while allowing these technologies to:

1. obtain custom knowledge-graph descriptions of elements/components
2. navigate to and through elements'/components' custom subcomponents

It would prove difficult to guarantee privacy about the presence of accessibility or AI-assistant technologies while allowing these technologies to:

3. invoke elements'/components' exported functions/methods

So, if these capabilities for interacting with documents' elements/components were desired to be made available, a user permission might be required.

[AdamSobieski]

Ok. I revised the initial post based on the feedback thus far.

This new revision also broaches including graph updates – diffs or deltas of knowledge graphs – as first-class objects. As a feature, this would simplify updating components' descriptions when only small subgraphs changed, e.g., in response to a component's attribute's or property's value having been modified. In the event of such an update to a component, it would only need only to send a small graph-update object rather than needing to re-create and send its entire modified knowledge-graph description.