General feedback on the issue triage bot

[devoncarew]

This issue is for collecting feedback on the issue triage bot; areas where it could improve, individual issues that it didn't triage well, possible prompt improvements, ...

Please feel free to create new comments here for feedback.

[parlough]

Issue/comment: dart-lang/sdk#56002 (comment)

Feedback: The bot triaged this macro related issue into area-test, likely because the issue discussed failing tests. As this is a common scenario, area-test should be weighted down somehow or have even more explicit prompting around preferring functional area.

[natebosch]

dart-lang/sdk#56028 (comment) would benefit from getting more context than the first comment of the issue.

[devoncarew]

Similarly to the previous comment, for transferred issues (like dart-lang/sdk#56072), we should at least feed in the first and last comment if not all the comments.

We may also consider introducing a slight delay in triaging issues (2 min?). This would allow some time for manual triage.

[devoncarew]

We should auto-close Analyzer Feedback from IntelliJ issues; these almost universally do not contain information (are not customized over the default template that just sends version info).

[devoncarew]

See #274 for a few of the discussed improvements.

[devoncarew]

When an issue comment contains an image, we should pass that image into the gemini API (along with the issue comment's text).

[FMorschel]

Maybe pass the current docs to solve cases like: dart-lang/sdk#56180? I'm not really sure what to suggest but the summary is off.

[devoncarew]

Yeah, I agree that that summary is off; you were asking for an analyzer code assist and it summarized it as a language enhancement request. I don't know if that means we need better prompting here, and the model is already well-trained on the Dart language. Perhaps a better model? General incremental improvements in gemini's capabilities? We're using gemini-1.5-flash to classify now, I could try a dry run w/ just regular gemini 1.5.

[FMorschel]

Maybe a way to ask the bot for a review? Than you could have a little more context into the new review

[devoncarew]

Maybe a way to ask the bot for a review? Than you could have a little more context into the new review

Do you mean to interact with the bot a bit to improve the summary? I think it would be reasonable to re-prompt it based on responses to its summary / classification comment.

[FMorschel]

Exactly. Maybe a keyword (or even pinging it) to ask for it to review the OP and the keyword comment. I believe if that is to become a thing, it should probably always appear at the end of its comments, like: "If you disagree with me, do this to make it so that I review this issue again".

[FMorschel]

Not sure what's going on here but I have the Dart SDK as a fork (intend on helping out soon). Both:

• Switch expressions assist conversion with OR removes comments sdk#56597
• Convert to switch expression/statement assists don't trigger formatting sdk#56602

Have sent me an email with failed jobs and have not been able to place the tags correctly, only commenting.

Maybe there is a bug here? Not sure if this is the right place or if this should have been a new issue.

[FMorschel]

About the above comment, not sure if it has been addressed or what (maybe a tag-specific problem), but I just created a new issue dart-lang/sdk#56631. Got no email errors and the bot could place the tags normally.

[devoncarew]

We had a regression in the triage bot - it was briefly trying to assign too many labels and the label assignment part of the workflow was failing; I suspect that's what you were seeing.

[FMorschel]

I see.

But why does the fork gets the email? I see that I have not received any other issue so only issues I wrote but still don't understand why am I receiving the emails? Is this a bug on GitHub Actions or something?

[devoncarew]

But why does the fork gets the email? I see that I have not received any other issue so only issues I wrote

Good question! I'm not sure why you're being notified. Perhaps it's related to your fork or perhaps related to you being the author of the issue that's triggering the workflow.