Proposal to solve publish/subscribe confusion #618
Comments
|
@fmvilas Awesome proposal! I love it ❤️ However, I have some problems with some parts of this proposal, maybe some things are obvious for you and for other people, but some things surprise me and I think their assumptions are wrong. The main problem - message(s) in the channelWhy do you want to define the message(s) in the channel, not as currently in the operation level? Consider the first example you gave: asyncapi: 3.0.0
channels:
userSignedUp:
address: user/signedup
message:
payload:
type: object
properties:
email:
type: string
format: email
operations:
sendUserSignedUpEvent:
channel: userSignedUp
action: send
I think that in this case, message(s) should be only defined on the operation, not on the channel. Channels should be then treated only as "interface", which you must "implement" by the operations (also if you need by multiple operations). For me, in the current proposal, the operation is "hardcoded" to the given channel by the message. If I'm not right, please clarify the Splitting servers to the two groupsI like idea to clarify that given server is remote and application is connecting to it and another server "runs" the app, but... did you consider having Servers and remotes in the channelI mentioned about it the first section:
but I think that you should clarify this thing in the proposal, because I know that you copied this from existing PR, but people can be surprised 😄 Adding request/reply supportHere I'm just going to think out loud because I'm not an expert in this topic. Maybe instead of an extra action type like Example: operations:
checkPayment:
channel: payment
action: receive
then:
2XX: ... # "reply" operation when given operations success.
4XX:... # "reply" operation when given operations fails.In the example above, I use HTTP statuses to describe the state of the operation: success or failure. Probably something better can be invented. It seems more generic to me. The idea itself is taken from CQRS. 👏🏼 for the proposal. If we go along with this (and I think we will after a dozen or so revisions), then every tool will have to be re-written from scratch 😂 |
|
Thanks for the quick feedback, Maciej! 🙌
At least 2 reasons come to my mind:
Precisely. Interfaces let you define the "shape" of the implementation and that's what we're doing here.
How is that hardcoded? We're just standardizing what messages can go through a channel and I think this is actually one of the biggest values. It's exactly how it is today. In v2, the message is tied to an operation and a channel at the same time in the same place.
Mind clarifying this? I'm not sure I understand your question/concern.
Sorry, that was just an experiment I was doing and forgot to delete it before publishing. Thanks for the heads up.
You can define as many operations as you want in a given channel :) operations:
mySendOperation:
channel: myChannel
...
myReceiveOperation:
channel: myChannel
...
myOtherReceiveOperation:
channel: myChannel
...
I did, and instantly discarded it 😄 When you're referencing a server or a remote, you can clearly know by the URL if it's either a server o a remote. E.g.: $ref: 'common.asyncapi.yaml#/components/servers/production' # I know it's a server just by looking at the URL.
$ref: 'common.asyncapi.yaml#/components/remote/production' # I know it's a remote just by looking at the URL.In general, just because two things have the same shape, it doesn't mean they have to be merged. Structurally they look the same but semantically they're completely different and subject to evolve in different directions in the future.
Not a concern of this proposal (my example was just to showcase that something can be done) but I'll bear with you. It seems what you're trying to define there is workflow language. And that's a totally different thing altogether 😄 Thanks a lot, Maciej! I hope this clarifies things a bit more :) |
|
I think this is a very positive proposal and helps with a lot of the issues around re-use of channels in typical microservice documentation etc. A quick initial question. My understanding is server is something we have and remote is something we use. With that assumption, should there be an explicit connection between server/remote and channel? If i a channel becomes a reusable asset that I can define in a central file, where it is shared by producer and consumer, where is that channel hosted? Do I want to define the remote that hosts my channel in every file, even if I don't re-declare the channel there to somehow indicate they are hosted there? On Identifiers, I think there may be a generally useful concept here. Having been working with $ref whilst I try and put together an SNS/SQS binding, it struggles where you want to imply resource uses resource b - defined elsewhere - instead of pull in definition of resource b. |
|
@fmvilas Thanks for explanation. Some things are clear but
Yes, I agree but currently you define given message(s) for particular action/operation (publish/subscribe), even if you have defined explicit operation in the channel level. I can understand the reason for having messages in the channel level, but that's the biggest problem to understand: whether a given message(s) by default is/are associated with Also you wrote:
Yep, but you may end up with a message defined at the channel level that fits only one operation, and for the rest you have to define the message at the operation level. So what it value for the message in the channel level? This can only cause problems in understanding what message an operation takes. If an operation will have the ability to define messages which will override those on the channel level then I can accept this, but I opt for having messages only on the operation level. Can you give me an episode of |
There's an ongoing proposal for that. I didn't want to include it here on purpose to avoid bloating this issue so much: #531
That's the whole point. Not everyone is interested in operations. Many just want a menu of channels and messages and they don't care who sends or receives them.
The value is the one I described above. Those who want a menu of channels and their associated messages. Many people don't want to describe applications but their sources of data (topics aka channels).
Yes. An operation has the ability to override a channel's
It wasn't proposed like this explicitly but here's a conversation with @dalelane about it: https://youtu.be/Qsu_yC-5YYM?t=535. I recommend you watch the whole episode. |
|
Is there a reason we have a The reason behind this is that I hate you have to manually match channel id's, if it can be avoided. Gonna go with your examples to show the changes, remember in this case This will help us in tooling, as:
And it still enables all the same things, but without complexity. The
|
Because many people don't want to use AsyncAPI to define applications but a menu of channels, messages, servers, etc. Systems like event gateways are not interested in operations at all but in which channels and messages are available. |
@fmvilas forgot to add We can of course just keep the |
|
@jonaslagoni You can read my comment about matching channels to the servers and why operating on the id (names) of the server/channel is better option rather than refs (and plain Server/Channel Object) -> #531 (comment) @fmvilas Thanks, I'm starting to see that. With the proposal, we have an options:
Additionally, if messages were not defined at the channel level, but only at the operation level, we would have to have a tool that would retrieve these messages from operations - most likely we will have to have such a tool due to the fact that operations will be able to override messages from the channel.
The second option is better and more consistency. I don't know how to handle situation when you have defined on the channel level the 5 messages (by |
|
@jonaslagoni you're right. Actually, the In general, I'd not worry about tooling so much. Let's put the focus on the user experience. If it's better for the user, cool let's do it, if it's not, let's change it. Our decisions should be driven by users and not tooling difficulty/complexity. That said, there may be some good for UX in what you're proposing, especially that it would work with existing JSON Schema tools in editors. |
@magicmatatjahu Yes, we'd need such a tool but some people might not need it. E.g., those who don't use the
Maybe we should make the channel's channels:
myChannel:
address: my/channel
messages:
- myMessage
- myMessage2
- myMessage3
operations:
sendMyMessage2:
channel: myChannel
messages:
- myMessage2
messages:
myMessage:
payload:
...
myMessage2:
payload:
...
myMessage3:
payload:
...This would make it possible to check if |
|
@fmvilas Yes, it's a solution. However I see in this solution and with referencing the channels by If I have a situation when a given broker has in "menu" some channel and then I connect this channel in operation (in some app connected to broker), how do I know if a channel is unique in the system? To describe this in more detail, I will use an example. I have one broker and one application that is connected to the broker: The
|
|
@magicmatatjahu I think you're getting a few things wrong here. Let me recap: # broker.yaml
asyncapi: 3.0.0
info:
title: Broker
version: 1.0.0
servers:
brokerServer:
...
channels:
someChannel1:
...
someChannel2:
...
This definition is not semantically valid. This is not a menu, it's an application definition. Menus can only use # broker.yaml
asyncapi: 3.0.0
info:
title: Broker
version: 1.0.0
components:
servers:
brokerServer:
...
channels:
someChannel1:
...
someChannel2:
...Also, your broker is not a server of your applications but a remote so it should be like this: # broker.yaml
asyncapi: 3.0.0
info:
title: Broker
version: 1.0.0
components:
remotes:
broker:
...
channels:
someChannel1:
...
someChannel2:
...Now we got a broker menu. Let's define app1.yaml: asyncapi: 3.0.0
info:
title: App1
version: 1.0.0
remotes:
broker: # Notice how I'm using a remote and it's referencing a remote too.
$ref: 'broker.yaml#/components/remotes/broker'
channels:
channel1: # Notice this Id is different than the one in broker.yaml. Ids are local to the AsyncAPI file.
$ref: 'broker.yaml#/components/channels/someChannel1'
channel2:
$ref: 'broker.yaml#/components/channels/someChannel2'
operations:
someOperation1:
channel: channel1
someOperation2:
channel: channel2This is how you know that App1 is using |
|
@fmvilas Thanks for explanation, it makes sense 😅 Even if you are using the local id of the channels, you should have information (especially in tooling like our cupid) that the given channel is unique and for this you can use the channel's channel1 -> broker.yaml#/components/channels/someChannel1 -> channel's address I edited my previous comment, maybe you read it too late. |
|
Yes, the address can be the global id. |
I agree with @iancooper on both points, it's a positive proposal and I have concerns about the definition of server and remote. Because depending on the perspective, the same underlying object is a both a For example: a websocket client connecting to a node.js application that spawns an internal websocket server. The node.js application considers the websocket server an Concerns
This seems contrary our aims of addressing the perspective issue, the re-use issue, and moving implementation specific information out of core objects. Alternative This extends the (key in my mind) effort to make the core objects of AsyncAPI as close to purely logical as possible. |
Concerns Internally I know a lot of large (=> less agile) enterprises that have coded to the 2.1 spec. To have that code be incompatible with 3.0 would likely decrease AsyncAPI's momentum. And yes, we can maintain separate 2.x and 3.x branches, but we all know the challenges of maintaining two versions of code, even in the medium term. This is especially true when there's a young spec with an imbalance between tooling needs and developers to implement them. Alternative |
Alright. You gave me an idea and I changed the proposal. Now
That's not true. A remote can reference a server object. It was also like this before I made the change described above. Actually, my "microservices" example shows this possibility when the frontend is using a remote that's referencing a WebSocket server definition.
That should be solved by my latest change 👍
We needed this change anyway, even in version 2.
Which one? 🤔
I'm using WebSockets as examples but it has nothing to do with WebSockets only. HTTP servers, HTTP2 SSE, GraphQL subscriptions, and more will have the same problem. It has nothing to do with the protocol but with the architecture design. Some interactions are client/server and some are broker-based. When it's client/server, we need to make it clear if the server is "something we're exposing" or "something we're connecting to". I don't think it's practical nor elegant to have bindings like
I don't think we'd even need to maintain two branches of the spec but instead, we can and have to maintain tooling compatibility with the two major versions. Until v2.x gets deprecated and we don't support it anymore but that can be in 1 or 2 years (or whatever we decide). If someone wants to stay in v2, cool! we'll keep supporting it. If you want to get the most out of the spec and tooling, migrate to v3. I think it's a fair and common thing.
Are we young or stable? 😄 I think we're still young and it's true we're gaining momentum but that shouldn't stop us from evolving without ending up with a Frankenstein specification with tons of band-aids just not to break backward compatibility. The users would love to have something that's clear and beautiful to use. IMHO, it's a problem of tooling vendors (including us) to migrate the tools to give the best experience to the users. Regarding imbalance between tooling needs and developers to implement them, we're working on this. At Postman we're hiring a bunch of people to work exclusively on AsyncAPI. Maybe other companies should follow. Also, the community is growing super fast so that imbalance we'll soon be equilibrated. |
|
I'm still digesting all of this, so this isn't a very considered response from me. One initial thought jumped out at me though: e.g. If I have a Kafka cluster with three brokers I might start with: remotes:
broker1:
url: broker1.myhost.com
protocol: kafka
broker2:
url: broker2.myhost.com
protocol: kafka
broker3:
url: broker3.myhost.com
protocol: kafkaThat gives me three brokers to have to refer to elsewhere/in other specs, which feels clunky. I know we've discussed this idea of groups of servers before, so I think it would be good to resolve this issue as part of a jump to 3.0 It doesn't necessarily have to be over-engineered. Kafka uses bootstrap addresses which are made of combining the broker URLs into a list, so we could do something like that. remotes:
mycluster:
url: broker1.myhost.com,broker2.myhost.com,broker3.myhost.com
protocol: kafkaor remotes:
mycluster:
urls:
- broker1.myhost.com
- broker2.myhost.com
- broker3.myhost.com
protocol: kafkaIt sort of breaks the idea of this really being a "url", but it does simplify things:
|
That improves re-use, but harms the user experience. I can envision explaining the difference between a
Agreed, but I don't think in an optimal way.
I agree that AsyncAPI needs a way to indicate client or server code implementation. But that does not require the introduction of
More accurate wording would be “redefines”. This proposal redefines
I would say its practical and common to separate server and client implementations. For example, many jars don't contain both the client and server for a particular technology. And keeping separation of between logical objects and implementation seems like an elegant, non-hacky concept that allows the spec to be vendor neutral and extensible, and allows end users the ability to switch implementations without altering the logical structure of their architecture. But if a proliferation of bindings is a concern, perhaps a third option would be to add a flag to applicable bindings indicating whether the binding is acting as a client or a server.
I hope that v2 continues to receive support, but that my experience is that shiny new things get all of the attention to the detriment of those who committed to earlier versions of the spec.
"v2.1" projects stability. Radical changes from version to version projects "young". The dissonance between the two needs to be considered. |
@dalelane I think your proposal #465 makes sense and should be included in v3. Actually, everything that's a breaking change should land in v3 and we should be aware of all of them before we release v3, just so we don't release v4 some months later 😄 Let's keep this issue focused on publish/subscribe and its associated problems.
I understand your concern but I think this remote/server difference is way easier to understand than the problem with publish/subscribe. We could argue the same about
I'm sorry but I think introducing verbiage like servers:
wsBackendServer:
url: ...
kind: local # Defaults to "remote" if not specified.
mosquitto:
url: ...My only argument against it was that it was easier to read something like
You may be right but this change is meant to serve as the base ground for the future vision, which does not only focus on async stuff. |
|
It should be possible to define multiple remotes for the same environments. For example, Kafka cluster will consist of number of remote urls:
The above set of remotes (that logically forms a Kafka cluster) will be repeated for different environments (dev, prod). Also we should be able to specify some common remote systems (for example Secret manager) per environment. Do you think a concept of 'environment' should be introduced on the root level? |
|
@ekozynin I think this is an interesting proposal that deserves its own separate discussion. Would you mind opening a new issue so we keep this one focused on solving the publish/subscribe confusion? 🙏 |
|
I am currently developing a websocket communcation system by using AsyncAPI specs. This seems to be the right discussion regarding some issues I see. Remotes.
There should be no Operations and messages When I design an OpenAPI spec, I have combinations of paths and methods that are associated with operationIds, e.g. createEntry, that refers to operations createEntry(). The resource is an Entry. This name createEntry makes complete sense on the server side and on the client side. I then especially generate code for the client that will contain a createEntry(). An AsyncAPI spec should allow the automatic generation of server and client code. A resource in OpenAPI world is losely related to message in AsyncAPI world. E.g. in a websocket connection a client may subscribe and unsubscribe to new entries and will then receive datagrams for new entries. The client therefore will send a subscribe message, an unsubscribe message. Therefore operations createSubscription, deleteSubscription available to clients in generated client code would be appropriate in OpenAPI world and also in AsyncAPI world. These names would be at least OK in the producer/server domain. A handler function createSubscription() in the server, that handles subscription requests coming from future consumers would be OK. That said it seems that an operationId createSubscription that is associated with the message for a subscription would be acceptable. Maybe two operationsId s createSubscription and onCreateSubscription that are both associated with that message would be even better. Channels What is a channel for example in a websocket system? Currently it is one connection. I have defined a channel for entries. This is one websocket connection. In this channel a subscription message is send by the consumer and the entry message is send by the producer. But the unsubscribe message is send over the same channel, too. I don't want to open multiple connections for that. Therefore I want to associate multiple messages with on channel. With the natural association of one (or two) operations per message, it seems that it would be good to have multiple messages associated with one channel (and not via oneOf) and (at least) one operation associated with every single message. In your draft I don't see that option: A channel has one address (i.e. URL, i.e connection with websockets) and still has one message associated. An operation is associated with the channel and not with the message. Proposal I propose to associate multiple messages (array, not one of) with one channel and associate one operation with one message in one channel. On the question of one or two operations per channel/message combination I am torn. It works fine with one for me in OpenApi world, though. |
@fmvilas, that seems reasonable to me. Thanks, @magicmatatjahu ! |
@olamiral this approach makes a lot of sense. This describes the surface area of the service interface without confusing the concepts of client, server, sender, receiver, publisher, subscriber, and also shows how AsyncAPI might incorporate a unified approach for all REST and Event-Driven APIs. What do people think of this aspect of this proposal that is directly relevant to publish/subscribe confusion, independent of the separate request-reply conversation happening in #558 ? |
This perceived problem might be conflating API definition with deployment concerns, leading to API definition divergence and logical duplication. There's a lot to be said for having a single source of truth for the surface area of a service interface, and once we go down the path of having multiple definitions from different perspectives such as What then does it mean to evolve this service API? REST services are naturally relocatable at different origin servers, such as https://example.com/ vs http://localhost:5000/ which makes it easy to do local development or container-based deployment for a service that might be advertised at the http://example.com/ address. REST services are also naturally relocatable at a different root path, such as http://localhost:5000/example/, due to the hierarchical nature of paths in URLs. Perhaps focusing on where AsyncAPI is already good at providing a relocatable service API definition and seeing where it can be improved would help retain the simplicity of a single source of truth, support client code generation and also support server code generation, potentially parameterized at the point of generation or at runtime for relocated deployments? The ideas proposed by @olamiral above would seem to be helpful towards that and perhaps worthy of further discussion. |
|
@jfallows thanks a lot for the feedback. It seems all your comments are really focused on HTTP and REST APIs but these are not the target of AsyncAPI (at least not yet). We're trying to get WebSocket (client-server) and broker-based architectures (Kafka, MQTT, AMQP, etc.) right first. |
|
@fmvilas thanks for responding.
I think you may have misunderstood, my feedback is not specifically focused on HTTP and REST. My feedback is about the general problem of AsyncAPI publish/subscribe confusion, and getting to the root cause, i.e. that specifying Instead, it would seem to be helpful to define the contract of interaction patterns, independent of
The syntactic specifics above are not as important as the principle of describing what, not how, thus addressing the root cause of the A frontend perspective would clearly publish to the A frontend perspective would subscribe to an A frontend perspective would subscribe to a Therefore a single source of truth in AsyncAPI covers both These interaction patterns would seem to apply equally well to WebSocket (client-server) and broker-based architectures (Kafka, MQTT, AMQP, etc.), without creating a There is also the added benefit of fitting nicely with HTTP and REST request-response, but that's for the future so we won't discuss that just yet. 😉 |
|
hi after some thoughts while preparing to work on some tooling for v3 I have now mixed feelings about the new "channels menu" approach As I commented early, I thought this was great as it makes posible documenting integration patterns that span more than one application like SAGAs... not possible with v2 but, on the other side, now with v3 message definitions are "not owned" by the providing application but are defined on a common "channels menu" that is shared on the organization and probably there will be way more friction reggarding write access, concurrent edits, pull requests workflows... etc I believe this could become troublesome very quickly... For an application that just want describe a domain event on a broker topic that probably "it's owned" by this application", to do so on an organization wide shared resource could be like "for publishing a new REST enpoint, need to edit an organization shared openapi.yml" I'm a bit worried about this.. Have we thought of these implications? |
|
@ivangsa Good observation. I myself had a problem with this new approach of describing the messages at the level of channel and not at the level of operation, but I also understand the approach of a channel like a registry, however, there are still quite a few problems here that you described. Since the channel currently does not have any required fields, it would be possible to add the ability to define Another problem I see with the current solution - and I didn't participate in the later discussion and found out after the fact what it looks like - that we don't have the ability to define what message(s) on a given operation we are able to send/receive. In the case of brokers the situation is simple, if the operation is "connected" to the broker under a given channel/eventType then it can receive and send the available messages on a given channel (in theory it should, how it is implemented is another thing), but for example in the case of a client-server, e.g. using websocket I have to (in current solution) create two channels with the same address but with a different set of messages, and depending on whether the operation sends or receives (perform logic from server or front client perspective), it's pointed to a different channel. I think that operations should be able to define the message at their level, but it should be optional, and the message itself should be in the registry of available messages of the given channel. EDIT: I read the previous comments because I didn't remember the discussion, but I still don't understand why we don't have the ability to define message(s) at the operation level. For the case when a channel operates only on one message it is understandable and message at the operation level does not make sense, but if there are 10 or more such messages? Then why do I need operations, if I could describe that the application sends and receives at the level of the channel itself as: channels:
someChannel:
address: ...
actions: [send, receive]
messages:
...Really, please explain it to me because I can see the problem (maybe it doesn't exist) and I see that it cause unnecessary redefinition of channels, but with other messages 😅 cc @fmvilas |
@ivangsa It's not that they are by default not owned by the application. It is that they can* and it all depends on your use case and how you wish to integrate them. It's a pros and cons game as everything else 🙂 You can decide to take that approach and have all messages and channels defined as a menu all applications reference from, or you can have the information owned by the applications (as in version 2, that does not change, only how you define it).
If you go with a menu, It definitely can! In a larger organization, you would need to have processes in place for how global definitions change, but that's governance, right? 🤔 To me this is something you would need in any regard especially if you start to consider versioning, organization-wide design rules, etc. When it's just the application that defines it, individuals get more control than having to change something in a global repository. But then you might lose reusability.
@magicmatatjahu the request/reply example (ignore the Do you find this confusing? And if yes, which parts? From my perspective I quite like this setup, it's clean and clear what messages are in use for which operations. It is quite a change from v2, as you no longer have this all-in-one channel. |
|
@jonaslagoni ahh ok, the reference to the channel menu is just a normal reference, you can split into "channel menu" and "application" or use just one file... (or even $ref from the "channel menu" the channel defined on the "application") thanks for the aclaration... I was thinking that the split as part of the specification.. is just a normal $ref |
|
@jonaslagoni I perfectly understand the current status of v3 spea, but as I specified, these revert channels with the same address but with a different messages list (or a single message) causes, in my opinion, a worse of the UX of spec. Take an example where the
then that spec example you gave could look like (I think that channels:
root:
address: /
messages:
ping:
$ref: '#/components/messages/ping'
pong:
$ref: '#/components/messages/pong'
heartbeat:
$ref: '#/components/messages/heartbeat'
systemStatus:
$ref: '#/components/messages/systemStatus'
subscribe:
$ref: '#/components/messages/subscribe'
unsubscribe:
$ref: '#/components/messages/unsubscribe'
subscriptionStatus:
address: null
messages:
subscriptionStatus:
$ref: '#/components/messages/subscriptionStatus'
operations:
pingPong:
action: send
channel:
$ref: '#/channels/root'
messages:
ping:
$ref: '#/messages/ping'
reply:
channel:
$ref: '#/channels/root'
messages:
pong:
$ref: '#/messages/pong'
heartbeat:
action: receive
channel:
$ref: '#/channels/root'
messages:
heartbeat:
$ref: '#/messages/heartbeat'
systemStatus:
action: receive
channel:
$ref: '#/channels/root'
messages:
systemStatus
$ref: '#/messages/systemStatus'
subscribe:
action: send
channel:
$ref: '#/channels/root'
messages:
subscribe:
$ref: '#/messages/subscribe'
reply:
channel:
$ref: '#/channels/subscriptionStatus'
unsubscribe:
action: send
channel:
$ref: '#/channels/root'
messages:
unsubscribe
$ref: '#/messages/unsubscribe'
reply:
channel:
$ref: '#/channels/subscriptionStatus'
The I have a huge problem with the examples we have because we only use channels:
ping:
address: /{root}
parameters:
root: ...
servers:
- $ref: #/servers/someServer1
- $ref: #/servers/someServer2
bindings:
kafka:
...
messages:
ping:
$ref: '#/components/messages/ping'
pong:
address: /{root}
parameters:
root: ...
servers:
- $ref: #/servers/someServer1
- $ref: #/servers/someServer2
bindings:
kafka:
...
messages:
pong:
$ref: '#/components/messages/pong'
heartbeat:
address: /{root}
parameters:
root: ...
servers:
- $ref: #/servers/someServer1
- $ref: #/servers/someServer2
bindings:
kafka:
...
messages:
heartbeat:
$ref: '#/components/messages/heartbeat'
systemStatus:
address: /{root}
parameters:
root: ...
servers:
- $ref: #/servers/someServer1
- $ref: #/servers/someServer2
bindings:
kafka:
...
messages:
systemStatus:
$ref: '#/components/messages/systemStatus'
subscriptionStatus:
address: null
messages:
subscriptionStatus:
$ref: '#/components/messages/subscriptionStatus'
subscribe:
address: /{root}
parameters:
root: ...
servers:
- $ref: #/servers/someServer1
- $ref: #/servers/someServer2
bindings:
kafka:
...
messages:
subscribe:
$ref: '#/components/messages/subscribe'
unsubscribe:
address: /{root}
parameters:
root: ...
servers:
- $ref: #/servers/someServer1
- $ref: #/servers/someServer2
bindings:
kafka:
...Of course, nothing prevents you from defining all these things in components and making refs everywhere, but in this case with the use of
I like both options very much and they solve the above problem which I described in 2 different ways, but I would prefer to be able to use two solutions than just There are, of course, use cases when defining WDYT? |
|
Please also keep in mind that, regardless of what the spec will say a channel or an operation is, people will have their own understandings of these words (especially if they have not read the spec in detail yet). Going against these common understandings will only result in frequent misunderstandings. |
|
@magicmatatjahu you just gave me an idea and I proposed we apply it (slightly different): #847 (comment). |
|
It's an exciting proposal. I'm looking for something like this because We have this situation:
In this perspective, I have one Queue (Channel) to many Operations (Commands) and believe that the proposal fits the pattern because I can specify many Operations to a single Channel. Thanks @fmvilas |
|
Just an update based on the issue TLDR. If there are additional changes that have been done for this issue that I have missed feel free to add them 🙂
Is currently not being championed for version 3: #689
Completed
Completed
Completed That means overall this issue is done and has been implemented both in the parser, JSON Schema documents, and the spec itself 🎉 |
|
This issue has been automatically marked as stale because it has not had recent activity 😴 It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation. There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model. Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here. Thank you for your patience ❤️ |
|
It's almost there, dear bot. Don't close it. |
|
This issue has been automatically marked as stale because it has not had recent activity 😴 It will be closed in 120 days if no further activity occurs. To unstale this issue, add a comment with a detailed explanation. There can be many reasons why some specific issue has no activity. The most probable cause is lack of time, not lack of interest. AsyncAPI Initiative is a Linux Foundation project not owned by a single for-profit company. It is a community-driven initiative ruled under open governance model. Let us figure out together how to push this issue forward. Connect with us through one of many communication channels we established here. Thank you for your patience ❤️ |
and others
TL;DR: What's new?
kindproperty. By default, all servers areremoteservers (e.g., brokers). However, if thekindproperty is set tolocal, that means the application is the one actually exposing the server (e.g., WebSocket or HTTP Server-Sent Events server).channelsobject is optional. The only two fields that are required areasyncapiandinfo.serversandchannelscan be defined insidecomponentsfor reusability purposes. This way, an AsyncAPI document may only contain acomponentsobject that can serve as an organization-wide menu of servers, channels, messages, etc.Abstract
For some years now, we've been discussing how to solve the publish/subscribe perspective problem. Some were arguing we should be giving priority to broker-based architectures because "they are the most common type of architecture AsyncAPI is used for". Although they're not wrong, by simply changing the meaning of
publishandsubscribekeywords, we'd be leaving many people and use cases out. We have to come up with a way to fix this problem without losing the ability to define client/server interactions too, which are especially useful for public WebSockets and HTTP Server-Sent Events APIs. And more importantly, we need a structure that will allow us to grow well and meet our vision.The latest Thinking Out Loud episodes with @lornajane, @dalelane, @damaru-inc, and @jmenning-solace have been key to better understand the problem. The intent of this proposal is to be a mix of the ideas and feedback I got from the community. I hope I managed to capture them well 😊
Foundational concepts
We gotta start from the beginning: an AsyncAPI file represents an application. This doesn't change. However, if no
channelsoroperationsare provided, the file becomes a "menu", a "library", a collection of reusable items we can later use from other AsyncAPI files.We have been very focused on the meaning of
publishandsubscribebut there is another key aspect of the spec that is confusing to many people:servers. As it is right now, theserverskeyword holds an object with definitions of the servers (brokers) we may have to connect to. However, it also holds information about the server our application is exposing (HTTP server, WebSocket server, etc.) In some tools, we have been incorrectly assuming that if someone specifieswsas the protocol, it means they want to create a WebSocket server instead of a WebSocket client. But what if someone wants to connect to a broker using the WebSocket protocol? This whole thing about the role of our application has been confusing all of us. As @lornajane pointed out on multiple occasions, an application can be both a server and a client, or just a server, or just a client. Therefore,serverscan't be made up of that mix. Exposed server interfaces and remote servers (usually brokers) have to be separated because —even though they look the same— they're semantically different.Remotes vs Local Servers
This proposal introduces the concept of a
remoteorlocalserver. Remote servers are those our application has to connect to. They're usually brokers but can also be other kinds of servers.On the other hand,
localservers are server interfaces our application exposes. Their URL defines where clients can reach them.Example
New
channelsandoperationsobjectsAnother common concern related to the current state of
publishandsubscribeis the channel reusability problem we encounter because these verbs are part of the channel definition. To avoid this problem, we remove the operation verbs from the channel definitions and move them to their own root objectoperations. Let's have a look at an example:There are a few new things here:
address: it's the logical address where you can find this channel. Usually, this is the topic name, the routing key, the URL path, etc.userSignedUpthen? This is the channel identifier. It's an identifier that serves to reference the channel from another part of the document or another document.user/signedupusing JSON Pointer would beuser~1signedupas opposed touser/signeduplike many people would think. That's highly unreadable and error-prone, therefore I'm introducing channel identifiers.channelhints against which channel is this operation performed andactionis the type of operation, i.e., we're "sending" or "receiving".sendUserSignedUpEvent? It's the operation identifier. Yes, the oldoperationIdis now mandatory and it's implicit, i.e., theoperationIdfield doesn't exist anymore.channela$refor JSON Pointer? To keep things simple. If you're referring to a channel here, it must be defined in thechannelsobject. If we allow$refhere, it means it can be dereferenced and therefore the channel ID would be lost. To make things easier and more consistent, this field is simply a string with the name of the channel ID.sendandreceive? No, it's not that I'm hatingpublishandsubscribealready 😅 I'm usingsendandreceivehere to avoid confusion for those thinking that AsyncAPI is only meant to describe pub/sub architectures. I think "send" and "receive" are pretty common verbs that shouldn't be linked to any super-very-special meaning.Organization and reusability at its best
I'm adding two new objects to the
componentsobject:serversandchannels. Some may be already wondering "why? don't we haveserversandchannelsalready in the root of the document?". To understand this decision, I think it's better if I just describe the reusability model I have in mind.Reusability
Some people expressed their interest in having a "menu" of channels, messages, servers, etc. They're not really interested in a specific application. In other words, they want an organization-wide "library". This is the
componentsobject. It's now possible to do something like the following:This is now a valid AsyncAPI file and it can be referenced from other AsyncAPI files:
As you can see, I'm not defining any channel or server in this file but instead, I'm pointing to their definitions in the org-wide document (
common.asyncapi.yaml). And this leads me to the other part of this section: "organization".Organization
The example above shows how we explicitly reference the resources (servers and channels) that our application is using. Having them inside
componentsdoesn't mean they are making any use of it. And that's key. The file is split into two "sections": application-specific and reusable items. Let's see an example:So for those of you who were wondering before "why? don't we have
serversandchannelsalready in the root of the document?": This is the reason, it's no different than it's right now in v2.x but I thought I'd make it more clear this time. Just because we put something incomponentsdoesn't mean the application is making any use of it.Life is better with examples
Microservices
Say we have a social network, a very basic one. We want to have a website, a backend WebSocket server that sends and receives events for the UI to update in real-time, a message broker, and some other services subscribed to some topics in the broker.
We'll define everything that's common to some or all the applications in a file called
common.asyncapi.yaml. Then, each application is going to be defined in its own AsyncAPI file, following the template{app-name}.asyncapi.yaml.The
common.asyncapi.yamlfileThe
backend.asyncapi.yamlfileThe
frontend.asyncapi.yamlfileThe
notifications-service.asyncapi.yamlfileThe
comments-service.asyncapi.yamlfilePublic-facing API
Another common use case for AsyncAPI is to provide a definition of a public-facing API. Examples of this are Slack, Gitter, and Gemini.
This would work differently than it is now. Instead of defining the server as we do with OpenAPI (and AsyncAPI v2), we'd have to define how a client would look like. For instance:
Currently, in v2, the spec lets you describe the server and infer the client from this description (as with OpenAPI). However, this causes a discrepancy with servers, especially in production systems. For instance, my server may be listening on port 5000 because it's behind a proxy. The client should be sending requests/messages to port 80 and they'll be forwarded by the proxy. If we define our port as 80 in the AsyncAPI file and generate a client, everything is ok but if we generate a server, our code will try to listen in port 80, which is not what we want. This means we can't infer clients from server definitions.
The drawback is that we'd have to define 2 files, one for the server and one for the client but the benefit is that we can be 100% accurate with our intents. Tooling can help auto-generating a client definition from a server definition.
Further expansion
Override channel messages with operation-specific messages
The new
operationsobject would help us define additional use cases, like those where we define what message is sent or received to/from a channel at the operation level. Let's see an example:Adding request/reply support
Adding support for the so-demanded request/reply pattern, would be as easy as adding a new verb and a
replykeyword. See example:FAQ
Does it serve as a base ground to meet our vision without a future major version?
I think so. Whenever we're ready to support REST, GraphQL, and RPC APIs, this structure should perfectly serve as a base ground.
Does it enable for better reusability of channels and other elements?
Yes. It completely decouples channels from operations and even clarifies the reusability model of the specification.
Does it allow users to define their whole architecture in a single file?
No, but it allows users to have a "common resources" AsyncAPI file where most of the information can reside.
Is it backward-compatible?
Absolutely not.
Does it remove the publish/subscribe confusion without introducing a new confusing term?
Yes. We keep the basic terms as before with just a bit of reorganization.
Is it easy to define a broker-based microservices architecture?
Yes. I provided an example.
Is it easy to define a public asynchronous API?
Yes. I provided an example.
Does it set the base ground to define an RPC (Remote Procedure Call) system over a message broker?
Yes. I provided a basic non-normative example.
Does it set the base ground to define a point-to-point RPC (Remote Procedure Call) API?
Yes. I provided a basic non-normative example.
The text was updated successfully, but these errors were encountered: