Built with ChatGPT: Github bug trends
Utilizing AI to Analyze, Predict, and Navigate Open-Source Software Issues
I work at mobile.dev, an early stage startup focused on improving the mobile development lifecycle. Our most popular product has been Maestro, an open-source UI automation tool that has been growing at a rapid rate.
A big challenge managing such a project is the influx of bug reports and feature requests. The more you grow the harder it becomes to keep track of it all.
Thankfully, with the rise of AI we can start building tools that will give us superpowers and make the situation more manageable.
MVP
Let’s see what we can do with the submitted issues in our repository. There are two use cases that I was interested in:
Detect weekly trends
Identify top 3 most concerning bugs based on severity (more on that later)
First, I did a small manual test by simply copy pasting the text from our repository into ChatGPT UI.
And this is what I asked:
I will provide you with Github issues and you will provide me interesting insights
---
[1.27.0] idb_companion is not able dispatch successful tap events bug Something isn't workingios #1076 opened yesterday by retyui [1.27.0] Not working with iOS 13.7 bug Something isn't workingios #1066 opened 4 days ago by KrisLau [v1.27.0] idb_companion does not start in time bug Something isn't workingios #1063 opened 5 days ago by RomainKB [v1.27.0] inputText failing on secureTextEntry bug Something isn't workingios #1061 opened last week by KrisLau 3 Maestro tests not finding buttons via strings or ids, with flakey behavior. android bug Something isn't workingneeds-informationneeds more information from the issue author #1056 opened last week by stevepopovich 10 Maestro doesn't recognize an emulator even if 'adb devices' does android bugSomething isn't working #1039 opened 2 weeks ago by Barbora827
And the response was…
Excellent!
We might need to fine tune our prompt to get more relevant trends but it understands the task.
The App
Now that we know this works we can create a simple app that queries the data, sends it to ChatGPT and shows the output.
For this task I used the following:
NodeJS to create an app
Github API to fetch repository data
OpenAI API to access ChatGPT (ChatGPT 3.5 + ChatGPT 4)
Discord Webhooks to show the output
Note: I will keep the code to a minimum in favour of showcasing GPT output. Let me know in the comments if I should share more code.
The Data
After fetching all open bug reports from our repository and cleaning up the data, each bug report looked something like this.
To avoid having large text bodies from the bug report, I used GTP-3.5 Turbo model to create a short summary.
The final data output is a list of Github issues for the past 7 days, in JSON format.
Detecting Trends 📈
Now all we have to do is to create a good prompt.
I will provide you with Github issues (bugs only) for a mobile ui automation repository called Maestro.
Provide me with the top 3 most interesting trends.
Make sure to add an emoji at the beginning of each trend, format the text for Discord usage and use hyperlinks.
---
Issues: ${issues}
My intent with this prompt is:
Give ChatGPT some context on what it’s reading
Let it sort through the data and identify any noteworthy trends
I’m keeping it vague but we could be more specific on what we mean with “Trends“
Export it in a nice format to be shown in Discord
After sending the prompt this is our result:
Not bad, I like how it clusters issues and provides links. This is a useful summary of the trending issues for the past 7 days.
Let’s see what else we can do.
Top Issues 🐞
Another interesting use case is to try and identify the top bugs by order of severity.
I will provide you with Github issues (bugs only) for a mobile ui automation repository called Maestro.
Provide me with the top 3 bugs ranked by severity and explain your reasoning.
Make sure to add an emoji at the beginning of each trend, format the text for Discord usage and use hyperlinks.
---
Issues: ${issues}
And the result was…
We could improve this by providing rules for what constitutes “severity“ but overall I’m happy with the output.
Thoughts
With a basic setup and a few lines of code we created a very powerful Insights Tool. Now my team and I can get a quick summary of what’s going on with Github Issues.
This of course only scratches the surface and the possibilities are endless.
Some areas we could improve on:
Provide more data for better context
Create a prompt that better defines “Trends“ and “Severity“.
Would your team benefit from this? Let me know in the comments.