Forum monitoring is essential but tedious: Here’s how AI can help

The problem of forum monitoring

If you build digital products for customers, chances are you’ve explored how AI can help.

As the makers of many popular Atlassian apps like Just Add+ Embed Markdown, Diagrams & Code for Confluence and Notification Assistant for Jira- Email, monitoring Atlassian community forums is vital for our success. These forums serve as a direct line of communication with our users, providing insights into their needs and concerns.

However, the process of monitoring these forums can be tedious. It involves manually reading many posts or reviewing searches tied to multiple keywords, sifting through numerous irrelevant posts, creating Jira tickets for relevant issues, and ensuring that nothing falls through the cracks.

This process is not only time-consuming but also prone to human error, leading to missed opportunities for engagement and support.

I am a semi-technical VP of Product, which roughly translates to: I can write a bit of code and debug it, but no one on my team, including me, would be happy with me shipping code to production.

This had me thinking: What if I could solve the challenge of forum monitoring by leveraging AI?

I decided to face it head-on by putting myself, the least capable technical talent, on the job. Here’s what happened.

Our solution: AI-powered automation

The volume of keyword-relevant forum threads to monitor for Modus Create’s Atlassian apps is quite high.

To streamline this process, we have implemented an AI-powered automation solution that significantly reduces the manual effort involved in forum monitoring.

Here’s how the automation works:

• RSS feed monitoring: We monitor RSS feeds based on product-related keywords to capture relevant discussions.
• AI evaluation of post relevance: Using GitHub Models, we evaluate the relevance of each post to our product.
• Automatic Jira ticket creation: Relevant posts are automatically converted into Jira tickets, ensuring that no important customer issues are missed.
• State tracking: We implement state tracking to avoid duplicate entries, maintaining an organized workflow.

A note on AI-powered development

For this exercise, I used GitHub Copilot as my programming partner. I was able to build this ai-powered automation solution without needing deep programming expertise or having to pull our development team off their core product work. This work was largely completed during my spare time.

While it wasn’t formally tracked, I would guess less than 2-3 hours in total were spent on this project. I say this purely to communicate how much AI code generation helped me accomplish, and how much business value this will drive.

The rest of our technology stack for the solution includes:

• Node.js: The core application is built using Node.js, providing a robust environment for our automation.
• GitHub Actions: We utilize GitHub Actions for scheduled execution, monitoring, and logging for our solution.
• GitHub Models: These models are integral for AI-powered relevance analysis, available through our GitHub Enterprise plan.
• Atlassian API Integration: Seamless integration with the Atlassian API allows for efficient ticket management.
• GitHub Copilot: This tool assists in development, making the coding process smoother and more efficient.

Technical implementation of the AI automation solution

The AI-powered automation was implemented simply, without the requirement of significant expertise. Here’s what we did:

Monitoring RSS feeds

Modus Create maintains a list of keywords we want to monitor, and then we generate relevant search queries for the Atlassian Community forums. It’s nothing elaborate. We gather words we care about, build out a list of them, and then pull them via RSS.

AI-powered relevance analysis

We leverage OpenAI’s GPT-4o from GitHub Models to determine if a post is relevant to our product with the following function:

The important thing to note is that we’re still in the early stages of automation. We will likely want to tune the temperature and other AI settings over time.

In addition, we will explore switching to a different model with a longer input context size, which would allow us to move away from truncating longer messages. We’re also analyzing how the AI performs with human review, which will be discussed later in this post.

Automatic Jira ticket creation

Jira tickets are created via the normal API. We’re not doing anything fancy there, but it does give us a consistent way to analyze this type of work alongside our customer-facing service desk and product development work.

Scheduled execution with GitHub Actions

We utilize GitHub Actions to run our AI-powered solution on a schedule with the following configuration:

While GitHub Actions is not the “normal” way to run work like this, we find its built-in scheduling, logging, and other tooling make it ideal to execute one-off automation projects like this.

We store the state in a JSON file and then commit it right back to the same repo. You may also note that we used a PAT for accessing GitHub Models. This isn’t ideal, but it’s the only way this currently works. To reduce risk, we used a fine-grained token without any permissions granted to it.

Benefits of automated forum monitoring

The implementation of this automated forum monitoring solution has yielded significant results. Initially, we decided not to use AI to block the creation of requests in our backlog, but only add it to the ticket. After a month, we created the following confusion matrix for our data:

This shows that if we start applying only negative filtering, even with our naive implementation, we will reduce the number of items to review by 70%.

We fully acknowledge that these items take the least amount of time to review, so we’re not going to pretend we see a 70% time savings. However, it’s still nice to know we can redirect our time to actually helping customers instead of busywork.

We hope that this improvement to forum monitoring will lead to enhanced response times to customer queries. We’re grateful that our partnership with GitHub made it easy to get this project going, showcasing how AI-powered tools like GitHub Copilot and GitHub Models empower non-traditional developers to create sophisticated solutions.

Why not use AI-powered automation for all customer interactions?

You may be wondering: If we are already using AI-powered automation to score question relevance, why not just use it to respond to customers?

For us at Modus, the quality of response is critical to maintaining our brand reputation and customer trust. At this time, AI responses are not good enough to deploy to customers.

What does the future of AI-powered forum monitoring look like?

Looking ahead, we plan to enhance our AI-powered solution by:

• Fine-tuning the AI prompt for better relevance scoring.
• Evaluating other models for better performance, and the ability to work with GitHub Models' rate and token limits.
• Adding automatic response suggestions so our team has a starting point from which to respond.
• Expanding monitoring capabilities to additional platforms such as LinkedIn/X.
• Implementing analytics to track community engagement trends.

“Make it more” with AI-powered automation

There is a meme in AI circles where you tell an image generation tool to “make it more.” For example, here is the result of a ChatGPT conversation where I kept asking it to “make it more” with the prompt being the true essence of Confluence.

The reason I bring this up isn’t just to jam a silly image into the blog. It’s to remind you to “make it more.”

I asked AI to take our code and convert it into the initial outline of this blog post. I then used Atlassian Intelligence to help write the first draft of this blog in Confluence. Granted, 90% of the copy was rewritten by a human. However, overcoming the blank page helped me draft this blog much faster than my usual writing speed.

Looking ahead: The future is AI

The integration of GitHub’s suite of AI-powered tools, including Models and Copilot, has been instrumental in both the development and execution of our automated forum monitoring solution. This project exemplifies the democratization of software development through AI, enabling teams to innovate and respond to customer needs more effectively than ever before.

As Atlassian, AWS, and GitHub Partners, we believe it’s our responsibility to not only build great tools but also ensure customers feel supported throughout their experience.

This automation helps us deliver on that mission at scale, all while freeing up our team to focus on what really matters: building great digital products.