[Proposal] : API Support

[vinodpahuja]

Hi @stoerr,

I am wondering why such a great tool lack API support. So I am proposing an equivalent API layer supporting similar flow.
For now I had just added a placeholder to start with, that I will enhance further based on feedback.

Sample :

• https://grokconstructor.herokuapp.com/api/match?randomize
• https://grokconstructor.herokuapp.com/api/match?example=0

[stoerr]

Thank you for your suggestion! But what would you use the API for? I'm a bit puzzled: GrokConstructor has no database and is open source, and has no really complicated things either, so if you need the algorithms you could just take them into whatever you're doing. Probably reimplementing it would be easier than accessing an API... :-)

[vinodpahuja]

Sure its better to reimplement, but re-using something proven has its own benefit.
Additionally an API layer

• can help to use it easily from some HTTP tool like Postman or httpman
• or an easy integration to some modern UX ( WIP ) like grok101

Please suggest.

[stoerr]

I'm not really convinced. I can see several downsides, even for any external apps using the API. For instance, it makes it dependent on me (and e.g. I haven't been updating the grok patterns for a while). I could suddenly break other apps when I'm changing something, or my site was down. (I'm also paying for the CPU time of https://grokconstructor.appspot.com/ privately. And I wouldn't have time to push something like that forward myself, I have way too little spare time these days. :-( )

Do you see any way https://grokconstructor.appspot.com/ itself could benefit from this?

[vinodpahuja]

I appreciate your concerns and also respect you personal limitations.
I was NOT really suggesting to change the course of existing service, but wanted to proposed it as extension of the tools. That's also not for direct consumption through current deployment but may be as separate branch/release hosted on some other medium.
Actually, this was inspired by some other discussion to include a modernize version of this tool for logstash users.
Please refer : logstash-plugins/logstash-filter-grok#177 (comment)

But I don't think it would be applicable any more, as elastic is proposing their own debugger for their users.
Refer :

• https://www.elastic.co/guide/en/elasticsearch/reference/8.2/grok.html
• Add link to grok debugger logstash-plugins/logstash-filter-grok#124

[vinodpahuja]

Do you see any way https://grokconstructor.appspot.com/ itself could benefit from this?

Other than above, an API (or an AJAX UI) can reduce these kind of crash bug #39, #42, #43
which can be caused due to invalid input pattern only.
Like in these case the problem can be avoided by using NBWORD \w+ instead of WORD \b\w+\b

[stoerr]

Thank you for all those links! Since I had no chance to use the ELK-Stack professionally for a while, I wasn't ware of that grok debugger. That's a good step, though it lacks many features, and I'm rather proud of the incremental construction, though I'm not sure how many use that. 😄

Ouch! Somehow it escaped me that there are that many tickets. So I should find some time to do something about them, since nobody does that for me. 😄 Thanks for reminding me.

About SPA / Ajax: I like it that you want to learn stuff by doing something useful - that's what motivated me to do this, besides that the algorithms were quite interesting and it would have helped me, if there was something like that. And you can take what I say with a couple of grains of salt, anyway, as I'm a sceptical guy. But I don't think it'd help much for grokconstructor. Here's why.

An SPA with Ajax updating only parts of the page is helpful if the page contains loads of stuff that are conceptually individual mini applications, and reloading all that would be lots of work for the browser. In the case of grokconstructor that's not so: there is a simple frame containing mostly fixed text, which very fast to render. As you can see, the matcher renders very fast, with page reloading and all.

There are parts of the application that are slow in some cases (even sometimes get the DeadlineExceededException). That can happen in incremental construction and automatic construction. If that DeadlineExceededException comes, that's because the search takes more than 30 seconds, and that'll happen with SPA/Ajax too. If it does succeed, the output is often very large and much work for the browser to render. If you'd do that with an API: the search takes the same time, the rendering as JSON is probably a tad slower if you do it with a library, and the browser would have to turn the sometimes huge json into HTML and render that, instead of just getting the HTML to render. So I'd be very very surprised if it was faster - I'd rather expect the opposite.

Also, with an SPA you have to be very careful that the browser back button works properly. That is quite valuable for this kind of work.

And lastly, I don't see how Ajax would help with bugs - you just have to fix those (or, in these cases, to turn the exceptions into sensible error messages.)

That said, I could obviously use some help. 😄 So, if you follow this through and have stuff that would benefit https://grokconstructor.appspot.com/ as it is (even if it was just an update of the grok patterns), then, by all means, please make a PR for that. But to have part of an API in there would make my life more complicated if I update it, and I don't quite see what that helps https://grokconstructor.appspot.com/ . The only point I can currently see would be that it could make sure that our codebases don't diverge too far from each other, so that they become too different to benefit from each other.

[vinodpahuja]

Thanks for your detail suggestions.
I see, these changes would not help much to the current state of grokconstructor.
Closing it for now.

[stoerr]

Thanks again for stirring me up on this. 😄 I did a good round of fixing; surprisingly, getting it to deploy again was almost as difficult as the fixing, and the bugs were astonishingly interesting.
Also, I discovered that it was unnoticedly gobbling up more than Google's free quota, and I was paying $30 per month now, without having noticed. 😆 I probably fixed that, too.