Retiring the custom AI tool I built: release notes case study

At a Glance

Problem: Release notes were a manual tax I paid every release cycle. Export the tickets from Jira, bugs, stories, spikes, then translate a wall of CSV rows into something a human would actually want to read.

Why I built it: The task repeated the same way every cycle. Same export, same translation, same grind. That repetition was the signal worth acting on.

Approach: I built a custom GPT and trained it on ten “golden” release notes until it reliably matched the tone and format I wanted, then gave it the one thing it couldn’t infer on its own: the exact shape of the Jira export.

Outcome: Drafting time dropped from about three hours to fifteen minutes. Eighteen months later, I retired the tool completely, not because it failed, but because a better version of the same idea showed up inside the tool my team already used every day.

The release notes tax

Every release cycle, someone has to turn a backlog of tickets into a note a customer, a support team, or a stakeholder can actually read. That work is real, but it isn’t hard in the way that matters. It’s repetitive. It’s rules-based. The inputs change, the shape of the output doesn’t.

That combination is exactly what makes a task a good candidate to hand off. Not because it’s low value, the note still has to be accurate and readable, but because the judgment required is narrow and repeatable. I was doing the same translation, cycle after cycle: pull the export, read the rows, decide what mattered, write it in a voice a non-engineer could follow.

The grind itself was the signal. Any task that repeats the same way every time is worth asking whether it needs to keep being a person doing it by hand.

The build

Building the tool was the easy part. Training it was the actual craft.

I didn’t trust the first draft it gave me, or the fifth. The process was a loop: take what the model produced, rewrite it to what I actually wanted, then feed the gap back so it learned the difference between close and correct. I ran that loop until I could hand it ten golden release notes, the exact tone and format I expected, consistently, not occasionally.

Once the pattern was locked, I gave it the one thing it had no way to know on its own: the precise shape of the Jira JQL export. Specific in, useful out. A model can guess at tone. It can’t guess at the exact structure of your data pipeline, and I wasn’t going to let it try.

The custom GPT I built — my own configuration and format.

The rule that mattered

I shared the tool with the whole team as a single URL, with one non-negotiable rule attached to it: proofread every time, and flag anything that looks off.

That rule did more work than the tool itself. Building something useful is one thing. Scaling it across a team without scaling blind trust along with it is the actual product decision. Anyone can hand out a link. Fewer people build in the expectation that the output still gets checked, every time, by a human who knows what right looks like.

That’s the part of this story that isn’t about AI at all. It’s about what you’re willing to let move fast, and what you refuse to let move unchecked.

The outcome

Drafting time went from roughly three hours to about fifteen minutes per cycle. The team used it consistently, and the proofread rule held. I never had a release note go out that hadn’t been checked by a person first.

That’s the number worth remembering, but it’s not the number worth building a whole workflow philosophy on. The philosophy is in what happened next.

The shift

The ground shifted. Rovo now lives inside Jira and drafts release notes right where the work already happens. No export. No standalone GPT. No separate tool to maintain, train, or explain to a new hire.

I retired the ChatGPT version completely. Not because it broke, and not because the ten golden examples stopped working. It’s because a better answer to the same problem showed up inside the tool my team was already using, and holding onto the old version past that point would have been loyalty to a solution, not to the problem it solved.

Where it lives now: a Rovo automation in Jira.

The lesson

Here’s the part that outlasts release notes specifically. The best way to do something with AI has a shelf life. What worked six months ago is already aging, not because it stopped working, but because something built for the problem instead of bolted onto it is probably already out there.

Tracking what changed isn’t a nice-to-have next to the actual product work. For anyone using AI as part of how they build, it is the actual product work. The tool you build today should be built with the expectation that you might be the one who retires it.