How We Built Quickly with Claude AI

The Challenge#

Building an AI assistant that can understand natural language commands and take action across multiple external tools is no small feat. Users don't follow scripts. They say things like:

• "create a ticket for this"
• "make a jira issue from the thread above"
• "log this as a P1 incident and assign it to Sarah"

All of these mean roughly the same thing, but the phrasing varies wildly. We needed an AI that could understand intent, extract parameters, and orchestrate actions reliably.

Why Claude#

We evaluated several LLM providers and ultimately chose Anthropic's Claude for a few key reasons:

1. Excellent at instruction following - Claude reliably follows complex system prompts and tool definitions
2. Strong reasoning - Can understand context and make sensible decisions about ambiguous requests
3. Tool use support - Native support for defining tools and letting the model call them
4. Safety-first approach - Important when the AI is taking real actions on behalf of users

Our Architecture#

Quickly's AI layer works in three stages:

1. Intent Classification#

When a user mentions @Quickly, we first classify what they're trying to do. We define a set of known intents:

• create_issue - Create a new Jira/Linear issue
• update_issue - Update an existing issue
• search_issues - Query for issues matching criteria
• create_doc - Create a Confluence/Notion page
• post_update - Post a message to a channel

Claude analyzes the message and thread context to determine the user's intent.

2. Parameter Extraction#

Once we know the intent, Claude extracts the relevant parameters. For a create_issue intent, this might include:

• Summary (from the thread or explicitly stated)
• Priority (P1, P2, etc.)
• Issue type (Bug, Incident, Task)
• Assignee (if mentioned)
• Project (from channel mapping or explicit)

3. Tool Orchestration#

Finally, Claude decides which tools to call and in what order. For complex requests like "create an incident and draft a postmortem," this might involve:

1. Call Jira API to create the issue
2. Call Confluence API to create the doc, referencing the new issue
3. Call Slack API to post confirmation with links

The Prompt Engineering#

Getting consistent, reliable behavior from an LLM requires careful prompt engineering. Here's what we learned:

Be explicit about the output format. We use structured tool definitions that clearly specify the expected parameters and their types.

Provide examples. Few-shot examples in the system prompt dramatically improve consistency.

Handle edge cases in the prompt. What should happen if the user doesn't specify a project? If the thread is empty? We address these explicitly.

Test extensively. We maintain a test suite of real user messages to ensure changes don't break existing behavior.

Results#

After months of iteration, Quickly now correctly handles ~95% of user requests on the first try. The remaining 5% are typically ambiguous requests where asking for clarification is the right behavior.

What's Next#

We're continuing to improve our AI layer:

• Multi-turn conversations - Remembering context across multiple messages
• Learning from corrections - When users correct Quickly, using that to improve
• Proactive suggestions - Recognizing patterns and offering to help before being asked

If you're interested in the technical details of building AI-powered tools, follow us for more posts like this.