Writing

How Our Small Product Team Decides Where AI Is Actually Worth Using

The evolution of AI is getting really exciting right now.

Tools like Claude Code are shifting AI from passive assistants into systems that can actively participate in development workflows.

Instead of simply answering questions, these tools can help navigate codebases, reason through technical problems, and take on operational tasks that often slow down developers and product managers.

But with all this possibility… and potentially a bit of noise… we have to ask ourselves: where does AI actually belong in our workflow?

For a small product team like ours, that decision matters even more. If we’re going to invest time building AI-powered workflows, they need to meaningfully increase our efficiency and help us make better product decisions.

So I designed a workshop using a simple framework to brainstorm 1) where our team was getting slowed down, and 2) where AI could have the biggest impact on our workflows.

The first step was identifying our biggest pain points.

Identify the friction

I created a Miro board, and we spent about ten minutes brainstorming answers to these two questions:

  1. Where are we spending time that doesn’t require deep thinking, creativity, or human judgment?
  2. What tasks feel like a time suck, busy work, or things we simply don’t get to because we lack the time or resources?

And we surfaced dozens of small but persistent pain points, like:

  • Writing user acceptance criteria and release notes.
  • Keeping documentation up to date on workflows and system behavior.
  • Reviewing transcripts in bulk to identify themes and gather insights.
  • Reacting to bug reports, validating issues, and debugging.
  • Using cross-system product data to validate hypotheses and guide product priorities.
  • Updating our omnichannel patient engagement system with conditional logic, localized messaging templates, and multi-step drip campaigns.

None of these tasks are inherently difficult, but together they create a constant stream of manual work and context switching.

Once the stickies were up, we clustered them into themes.

One insight surfaced almost immediately: our biggest problem wasn’t writing enough code. It was context fragmentation.

Because we’ve built nearly everything in-house, information lives across dozens of internal tools and services, and much of our time goes toward reconstructing… and sometimes simply remembering… all the context and where it lives.

That’s time we could be spending on product strategy or system architecture.

Figure out where AI might help reduce the friction

Once the themes were clear, we asked the next question:
How might AI help us solve these problems?

Ideas started flowing quickly.

Some focused on synthesizing large volumes of information… transcripts, product usage data, and logs… using RAG to query data across systems and surface insights faster.

Others targeted operational friction in engineering workflows, such as bug triage and change risk analysis, and explored AI agents to help execute parts of those workflows.

For a few of the themes we identified, we didn’t have time to brainstorm solutions during the workshop. Later, I ran them through AI to explore possible approaches, and it surfaced several ideas we hadn’t considered yet.

Before long, the board was full of possibilities.
So where should we start?

Rate ideas by value, effort, and risk

In a second session, we evaluated every idea using three criteria:

  • Value: How much leverage would this create for the team?
  • Effort: How difficult would it be to implement?
  • Risk: What happens if the AI output is wrong or breaks something?

Some ideas were exciting but required too much effort relative to the value they would create. Others sounded cool but introduced too much operational risk.

A few landed in the sweet spot where the potential upside clearly justified the cost of experimenting.

From there, we landed on three bets.

The bets we prioritized

AI Product Data Chat Layer

The first bet emerged from a recurring frustration for our product manager: we have plenty of data about how our product is used, but turning that data into insights is surprisingly difficult for a non-engineer.

We have usage metrics, operational data, transcripts, and engagement signals… but they all live across different systems.

To make better product decisions, we need to be able to quickly answer questions like:

  • How are different workflows actually being used in practice?
  • Where are the biggest points of friction or drop-off for a patient?
  • How can we measure whether the hypothesis we’re testing is actually correct?

If we had a natural language interface connected to all our product data sources, the team could ask these questions directly.

Instead of writing complex queries, we could simply ask.

This initiative scored highest in value because it improves how the entire company makes decisions. It allows us to evaluate the impact of what we build, prioritize the right improvements, and operate in a much more data-driven way.

It also ranked well because the risk is relatively low. This wouldn’t modify production systems or generate code. It would simply help anyone in the company access and interpret the information we already have in ways they haven’t been able to before.

Plus, there are several out-of-the-box RAG and chat solutions available today, like Vanna or Wren, that make this relatively easy to implement.

Finishing the transition to Linear

The second bet was less about AI itself and more about fixing an operational constraint that prevents AI from being useful.

During the workshop, we realized how fragmented our data is:

  • Bug reports come from observability tools and Microsoft Teams.
  • Grooming and product planning meetings happen in Google Meet (thankfully, with transcripts).
  • Development work is tracked in Jira.
  • The code itself lives in GitHub and GitLab.

We’re currently using Jira and had already begun migrating to Linear, but never fully crossed the finish line.

Jira is powerful, but Linear provides a simpler workflow and a clean API with an MCP server, making it easier to connect tools directly to tickets and development workflows.

The immediate value of finishing the migration is modest, but the real value is that once in place, Linear consolidates development context and supports integrations that allow tools to interact directly with tickets and the codebase. That moves us one step closer to enabling deeper AI integrations over time.

Once this foundation is in place, AI could eventually help with things like:

  • Automatically generating new tickets from grooming discussions, architectural plans, retrospectives, or bug reports.
  • Proposing implementation plans, surfacing trade-offs across different approaches and estimating relative effort.
  • Highlighting downstream impact by identifying other systems or features likely to be affected.
  • Reconstructing investigation context by pulling together bug reports, observability data, past tickets, and relevant code changes.

The effort to finish the transition is relatively low since the migration is already partially complete. It also gives us a chance to clean up the backlog.

The risk of completing the migration itself is also very low. At the end of the day, Linear is simply an issue tracking system used to manage development work. The riskier pieces are the AI capabilities we may layer on top later, which would require separate evaluation and testing.

This bet is about building the operational infrastructure that allows AI to be layered into our workflows over time and opens the door to new possibilities.

Claude Code for marketing adjustments

The third bet came from a smaller but persistent operational pain point.

Our patient engagement system supports many campaign variations across brands, clients, and languages. Updating them can be complex because the messaging infrastructure is largely homegrown, with campaign logic defined in code.

To make the system manageable without constant engineering support, we built an internal admin interface that allows non-engineers to update templates and manage campaigns directly.

But the real friction isn’t writing the updates. It’s finding every place that needs to change and making sure nothing gets missed.

This is where AI really shines.

Instead of manually tracking down every configuration change, a coding assistant like Claude Code could analyze the configuration structure, generate updates across template variations, apply translations, and validate syntax before deployment.

The value here is operational efficiency and a better patient experience. It could save our product manager several hours each week by reducing the manual work required to update campaigns. It would also help prevent outdated templates or confusing messages from being sent unintentionally.

The effort required is relatively low because most of the infrastructure already exists. The team already manages these campaigns through Directus, so this change mainly improves an existing workflow rather than introducing a new system.

The risk level is medium low. The messaging system mostly reads from other parts of the platform rather than modifying them, and all updates are tested in UAT before moving into production.

It also makes the system easier to maintain as campaigns continue to grow.

More broadly, these bets reflect how we’re thinking about AI adoption as a small product team.

None of them are about replacing people with AI.

They’re about reducing the friction that slows teams down and giving us more time to focus on the work that actually requires human judgment.

Discover more from Stephanie Ramsay

Subscribe now to keep reading and get access to the full archive.

Continue reading