Building Chatbots and Intent Recognition

Summary

This chapter introduces the core concepts and techniques for building conversational agents, focusing on understanding user intentions and extracting relevant information from queries. You will learn about chatbot architectures, dialog systems, intent recognition and classification, entity extraction techniques, and how to build FAQ-based systems. These foundational chatbot concepts prepare you to create intelligent conversational interfaces.

Concepts Covered

This chapter covers the following 15 concepts from the learning graph:

Chatbot
Conversational Agent
Dialog System
Intent Recognition
Intent Modeling
Intent Classification
Entity Extraction
Named Entity Recognition
Entity Type
Entity Linking
FAQ
FAQ Analysis
Question-Answer Pair
User Query
User Intent

Prerequisites

This chapter builds on concepts from:

Introduction to Conversational Interfaces

Every time you ask Siri about the weather, message a customer service bot about your order status, or use ChatGPT to answer a question, you're interacting with a conversational interface. These systems, broadly known as chatbots or conversational agents, have evolved from simple keyword-matching programs to sophisticated AI systems capable of understanding context, extracting information, and maintaining coherent multi-turn dialogues. This chapter explores the foundational concepts behind building these systems, focusing on how they understand what users want and extract the critical information needed to respond appropriately.

At the heart of every effective conversational agent lies the ability to answer two fundamental questions: "What does the user want?" and "What information do I need to fulfill that request?" The first question addresses intent recognition—understanding the user's goal. The second focuses on entity extraction—identifying specific data points like dates, names, locations, or product identifiers. Together, these capabilities transform raw text into structured, actionable information that systems can process and respond to intelligently.

Understanding User Queries

A user query represents any input provided by a user to a conversational system, whether typed into a chat interface, spoken to a voice assistant, or selected from quick-reply options. Unlike structured database queries written in SQL or other formal languages, user queries arrive in natural language—messy, ambiguous, and highly variable. The same intent can be expressed in dozens of ways:

"What's the weather like today?"
"Is it going to rain?"
"Do I need an umbrella?"
"Will it be sunny this afternoon?"

Each query asks fundamentally the same thing (weather information), but uses different vocabulary, structure, and level of specificity. This variability presents both the central challenge and the fundamental requirement for conversational systems: they must map diverse natural language expressions onto a consistent set of system capabilities.

User queries typically contain two types of information. First, they express an intent—the underlying goal or action the user wants to accomplish, such as checking weather, booking a flight, or finding product information. Second, they often include specific details called entities—concrete values like "today," "New York," or "size 10" that parameterize the request. Effective chatbots must identify both components to respond appropriately.

Diagram: User Query Components

Anatomy of a User Query

Type: diagram

Purpose: Illustrate how a natural language user query contains both intent and entity information that must be extracted

Components to show: - User query at top: "Book a flight to San Francisco next Tuesday" - Arrow pointing down to two branches: - Left branch: "Intent: Book Flight" (highlighted in blue) - Right branch: "Entities" containing: - Destination: San Francisco (orange) - Date: next Tuesday (green) - Below intent: "System Action" box showing "Search available flights" - Below entities: "Parameters" box showing structured data

Connections: - Arrows from intent and entities converging at bottom to "Actionable Request" box - Dotted lines showing how entities fill parameter slots in the system action

Style: Flowchart with boxes and arrows, hierarchical layout

Labels: - "Natural Language Input" above user query - "Semantic Understanding" in middle layer - "Structured Output" at bottom

Color scheme: Blue for intent, orange/green for different entity types, gray for system components

Implementation: SVG diagram with clear visual hierarchy

Frequently Asked Questions and Question-Answer Pairs

Many conversational systems begin their lifecycle as FAQ (Frequently Asked Questions) systems. An FAQ system maintains a curated collection of question-answer pairs—explicit mappings from common user questions to predetermined responses. This approach offers several advantages for organizations just starting with conversational AI: it requires no machine learning expertise, leverages existing documentation, and provides predictable, controllable responses.

A question-answer pair consists of two components: a representative question that captures a common user need, and a corresponding answer that addresses that need. For example:

Question	Answer
How do I reset my password?	Click "Forgot Password" on the login page. Enter your email address, and we'll send you a reset link within 5 minutes.
What are your business hours?	We're open Monday through Friday, 9 AM to 6 PM EST. Weekend support is available via email only.
Do you offer student discounts?	Yes! Students receive 20% off with a valid .edu email address. Click here to verify your student status.

The fundamental challenge in FAQ systems lies in matching user queries to the appropriate question-answer pair. Users rarely phrase questions exactly as they appear in the FAQ database. Someone might ask "I can't log in, help!" when the relevant FAQ question is "How do I reset my password?" Effective FAQ systems must handle this variability through synonym expansion, semantic similarity matching, or machine learning-based retrieval techniques covered in earlier chapters.

FAQ analysis involves examining collections of user questions to identify patterns, coverage gaps, and optimization opportunities. By analyzing which questions users ask most frequently, organizations can prioritize high-impact improvements. FAQ analysis also reveals when questions cluster around similar intents but use different phrasings—a signal that intent classification might provide better coverage than simple keyword matching.

Diagram: FAQ System Architecture

FAQ-Based Chatbot Architecture

Type: diagram

Purpose: Show how FAQ systems process user queries through matching, retrieval, and response generation

Components to show: - User interface (top left): Chat window with user query "how do I reset password" - Query processing layer (middle left): - Text normalization box - Synonym expansion box - Embedding generation box - FAQ database (center): Collection of Q&A pairs represented as stacked cards - Matching engine (middle right): - Similarity calculation - Ranking algorithm - Confidence threshold - Response selection (bottom right): Top-ranked answer - Feedback loop (bottom): Thumbs up/down returning to database

Connections: - User query flows through processing pipeline - Processed query connects to matching engine - Matching engine queries FAQ database - Results ranked and filtered by confidence - Selected response returned to user interface - User feedback flows back to database for improvement

Style: Data flow diagram with layered architecture

Labels: - "Input Processing" for normalization layer - "Semantic Matching" for matching engine - "Response Delivery" for output - Confidence scores shown on connection from matching to response (e.g., "0.87")

Color scheme: Purple for user interface, blue for processing, orange for database, green for matching, teal for response

Implementation: Block diagram with directional arrows showing data flow

Chatbots and Conversational Agents

The terms chatbot and conversational agent are often used interchangeably, though subtle distinctions exist. A chatbot typically refers to any software system that engages in text-based conversation with users, regardless of sophistication level. This broad category includes simple rule-based systems that respond to specific keywords, FAQ retrievers, and advanced AI-powered assistants.

A conversational agent implies a higher level of sophistication—a system capable of multi-turn dialogue, context maintenance, and intelligent decision-making. Conversational agents understand conversation flow, remember previous exchanges, and can handle complex, multi-step interactions. While all conversational agents are chatbots, not all chatbots qualify as true conversational agents. A simple FAQ bot that matches keywords to canned responses is a chatbot; an AI assistant that helps you plan a multi-city trip over several conversational turns is a conversational agent.

Modern chatbots exist on a spectrum of capabilities:

Rule-based chatbots: Use pattern matching and decision trees to respond to predefined inputs. Fast and predictable, but brittle when users deviate from expected patterns.
Retrieval-based chatbots: Select responses from a predefined set based on similarity to the user query. More flexible than rule-based systems, but limited to responses in their database.
Generative chatbots: Use language models to generate novel responses dynamically. Highly flexible and capable of handling unexpected inputs, but require careful prompt engineering and safety measures.
Task-oriented agents: Focus on completing specific tasks like booking reservations or answering product questions, often combining retrieval and generation strategies.
Open-domain agents: Engage in general conversation on any topic, prioritizing engagement and coherence over task completion.

The choice of architecture depends on your use case, available data, and tolerance for unpredictable responses. Customer service chatbots often favor retrieval-based or task-oriented approaches to ensure accurate, compliant responses. Entertainment or companion bots may embrace generative models for more engaging, varied interactions.

The following table compares key characteristics across chatbot types:

Characteristic	Rule-Based	Retrieval-Based	Generative	Hybrid
Development complexity	Low	Medium	High	High
Response predictability	Complete	High	Variable	Medium-High
Handling unexpected input	Poor	Moderate	Excellent	Good
Training data required	None	Moderate	Large	Moderate-Large
Response variety	Very low	Medium	Very high	High
Typical accuracy	High (in scope)	Medium-High	Variable	High
Best for	Simple FAQs	Customer support	Open conversation	Enterprise apps

Dialog Systems and Conversation Management

While simple chatbots handle isolated queries independently, dialog systems manage extended conversations with multiple turns, context tracking, and state management. A dialog system maintains awareness of conversation history, understands references to previously mentioned entities, and guides users through multi-step processes toward goal completion.

Consider a conversation with a flight booking system. The dialog unfolds over multiple turns, each building on previous exchanges:

User: "I need to book a flight to Chicago" System: "I can help with that. What date would you like to depart?" User: "Next Monday" System: "Departing Monday, January 22nd. Where will you be flying from?" User: "Boston" System: "Perfect. What time of day do you prefer?" User: "Morning"

Notice how the system doesn't ask for all information at once, but instead guides the user through a structured information-gathering process. It remembers the destination (Chicago) mentioned in the first turn and doesn't ask for it again. When the user says "next Monday," the system resolves the relative date reference to an absolute date. This contextual awareness and conversation management distinguishes dialog systems from simpler single-turn chatbots.

Dialog systems typically implement one of several conversation management strategies:

Finite state machines: Model conversations as a graph of states (e.g., "greeting," "gathering departure info," "confirming booking") with transitions triggered by user inputs. Simple to implement and reason about, but can feel rigid.
Frame-based systems: Define templates (frames) for each task with slots to fill (destination, date, time). The system asks questions to fill empty slots and confirms when complete. Works well for structured tasks with clear information requirements.
Plan-based systems: Model conversation as a planning problem where the system pursues goals while accounting for user intentions and beliefs. More sophisticated but computationally complex.
End-to-end neural systems: Use deep learning models to map conversation history directly to system responses. Flexible and capable of learning from data, but less interpretable and harder to control.

Modern production systems often combine approaches, using structured frameworks for critical transactional flows while employing neural models for handling unexpected inputs or conversational elements outside the main task flow.

Diagram: Dialog System State Machine

Finite State Machine for Flight Booking Dialog

Type: workflow

Purpose: Illustrate how dialog systems manage conversation flow through states and transitions for a flight booking task

Visual style: State diagram with circular nodes for states, arrows for transitions, and labeled conditions

States: 1. Start: "Greeting" Hover text: "System welcomes user and offers to help with flight booking"

State: "Collect Destination" Hover text: "System asks 'Where would you like to fly?' if destination not provided"
State: "Collect Origin" Hover text: "System asks 'Where will you depart from?' if origin not provided"
State: "Collect Date" Hover text: "System asks 'What date?' and resolves relative references like 'next Monday'"
State: "Collect Time Preference" Hover text: "System asks 'What time of day: morning, afternoon, or evening?'"
Decision: "All Slots Filled?" Hover text: "Check if destination, origin, date, and time are all collected"
State: "Display Options" Hover text: "System queries flight database and shows available flights matching criteria"
State: "Confirm Selection" Hover text: "User selects flight; system confirms details before booking"
End: "Booking Complete" Hover text: "System provides confirmation number and sends email receipt"

Transitions: - Greeting → Collect Destination (user expresses flight intent) - Collect Destination → Collect Origin (destination provided) - Collect Destination → Collect Destination (if user provides unclear input) - Collect Origin → Collect Date (origin provided) - Collect Date → Collect Time Preference (date provided and validated) - Collect Time Preference → All Slots Filled? (time preference provided) - All Slots Filled? → Display Options (YES: all required info collected) - All Slots Filled? → [return to missing slot] (NO: redirect to first empty slot) - Display Options → Confirm Selection (user picks a flight) - Display Options → [modify slots] (user wants to change criteria) - Confirm Selection → Booking Complete (user confirms)

Color coding: - Green: Start state - Blue: Information gathering states - Yellow: Decision point - Orange: Transaction states - Purple: End state

Edge labels: - Show user intents that trigger transitions (e.g., "provides destination", "changes mind", "confirms")

Swimlanes: Single flow representing system perspective

Implementation: Mermaid state diagram or interactive SVG with clickable states

Understanding User Intent

While user queries vary greatly in phrasing, they typically express a limited set of underlying intentions. User intent represents the goal a user wants to accomplish—the action they expect the system to take or the information they seek. Understanding intent is fundamental to conversational AI because it allows systems to map diverse surface forms onto consistent behaviors.

In a banking chatbot, user queries like "What's my balance?", "How much money do I have?", "Check my account," and "Show my funds" all express the same intent: check_balance. Similarly, "I lost my card," "My credit card was stolen," and "I need to freeze my card" all map to report_lost_card. By identifying the intent category rather than processing each unique phrasing separately, systems can provide consistent responses and scale to handle variation.

Intent recognition is the task of automatically identifying which intent category a user query belongs to. This classification problem typically uses machine learning models trained on labeled examples. Given a new user query, the model predicts the most likely intent from a predefined set of possibilities.

Intent modeling refers to the process of designing your intent taxonomy—deciding what intents your system should recognize and how granular they should be. Good intent modeling balances specificity and coverage:

Too few intents (e.g., just "question" and "command"): System can't differentiate between different user needs and provide appropriate responses
Too many intents (e.g., separate intents for "check savings balance" and "check checking balance"): System becomes brittle, requires more training data, and may fragment related queries

Effective intent modeling follows several principles:

Mutual exclusivity: Each user query should map to exactly one intent; overlapping intents create classification ambiguity
Actionable distinction: Different intents should trigger different system responses; if two intents lead to the same action, they should probably merge
Balanced frequency: Avoid creating highly specific intents for rare queries while lumping common queries into catch-all categories
User-centric naming: Define intents based on user goals, not system implementation details

Here's an example intent taxonomy for a restaurant reservation chatbot:

make_reservation: User wants to book a table
modify_reservation: User wants to change an existing booking
cancel_reservation: User wants to cancel
check_availability: User asks if tables are available (without committing to book)
ask_location: User wants to know where the restaurant is located
ask_hours: User asks about opening hours or specific date availability
ask_menu: User wants to see the menu or asks about specific dishes
ask_dietary: User has questions about allergies, vegetarian options, etc.
chitchat: General conversation not related to specific booking tasks

Intent classification is the machine learning task that implements intent recognition. Modern intent classifiers typically use one of several approaches:

Traditional ML with engineered features: Extract features like n-grams, TF-IDF vectors, or part-of-speech patterns, then train classifiers like logistic regression, SVM, or random forests. Interpretable and works well with limited data, but requires feature engineering expertise.
Deep learning with word embeddings: Encode queries using pre-trained word embeddings (Word2Vec, GloVe), then pass through neural networks (CNNs, LSTMs) for classification. Better handles semantic similarity without manual feature engineering.
Transformer-based models: Fine-tune pre-trained language models (BERT, RoBERTa, DistilBERT) on labeled intent data. Currently achieves state-of-the-art performance, especially with limited training examples, due to transfer learning from large-scale pre-training.
Large language model prompting: Use LLMs like GPT-4 with few-shot examples in the prompt to classify intents. No training required, highly flexible, but slower and more expensive per query than fine-tuned models.

The choice depends on your available labeled data, latency requirements, and accuracy needs. Many production systems use a hybrid approach: fast, fine-tuned classifiers for common intents with LLM fallback for edge cases or confidence scores below a threshold.

Diagram: Intent Classification Pipeline

Intent Classification Architecture

Type: diagram

Purpose: Show the complete pipeline from user query to predicted intent, including preprocessing, feature extraction, classification, and confidence scoring

Components to show: - Input layer (top): User query text box: "I need to change my reservation for tomorrow" - Preprocessing layer: - Text normalization box (lowercasing, punctuation removal) - Tokenization box - Stopword filtering (optional, shown with dotted border) - Feature extraction layer: - Option A: TF-IDF vectorization (shown on left branch) - Option B: BERT encoding (shown on right branch, highlighted as preferred) - Model layer (center): - Intent classifier neural network - Input dimension matching feature vectors - Output layer with softmax activation - Output layer (bottom): - Intent probabilities table showing: - modify_reservation: 0.87 (highlighted in green) - cancel_reservation: 0.08 - make_reservation: 0.03 - ask_hours: 0.02 - Confidence threshold line at 0.70 - Final prediction: "modify_reservation" with confidence 0.87

Connections: - User query → Preprocessing (solid arrow) - Preprocessing → Feature extraction (splits into two paths) - Both feature extraction paths → Model (merge) - Model → Output probabilities - Threshold check → Final prediction

Annotations: - Badge on BERT encoding: "Recommended: Better generalization" - Badge on output: "High confidence - proceed with action" - Note near threshold: "Queries below 0.70 escalate to human"

Style: Flowchart with layered architecture, showing parallel paths for different approaches

Labels: - "Text Processing" for preprocessing layer - "Semantic Encoding" for feature extraction - "Classification" for model layer - "Prediction & Confidence" for output

Color scheme: - Blue for preprocessing - Purple for feature extraction - Orange for model - Green for high-confidence predictions - Yellow for medium confidence - Red for below-threshold (not shown in this example)

Implementation: SVG diagram with clear information flow and decision points

Entity Extraction and Recognition

While intent recognition identifies what users want, entity extraction identifies the specific details within their queries. Entities are the concrete values—dates, names, locations, product IDs, monetary amounts—that parameterize user requests. A query like "Book a table for 4 people tomorrow at 7 PM" expresses the intent make_reservation, but also contains critical entities:

Party size: 4 people
Date: tomorrow
Time: 7 PM

Without extracting these entities, the system knows the user wants a reservation but lacks the information needed to fulfill it. Entity extraction transforms unstructured text into structured data that systems can act upon.

Entity types categorize the kinds of information your system needs to extract. Common entity types include:

Temporal: dates, times, durations (e.g., "tomorrow," "3:30 PM," "two weeks")
Numeric: quantities, amounts, measurements (e.g., "4 people," "$50," "2 miles")
Geographic: locations, addresses, regions (e.g., "Boston," "123 Main St," "New England")
Personal: names, titles, contact information
Categorical: options from predefined sets (e.g., "vegetarian," "window seat," "economy class")
Custom: domain-specific entities like product IDs, account numbers, or reservation codes

Named Entity Recognition (NER) is the task of identifying and classifying named entities—specific named references to people, organizations, locations, and other proper nouns. Traditional NER focuses on a standard set of entity types (Person, Organization, Location, Date, etc.), while custom entity extraction extends this to domain-specific categories relevant to your application.

Modern entity extraction systems use several approaches:

Rule-based extraction: Use regular expressions and pattern matching to find entities with predictable formats (dates, phone numbers, email addresses). Fast and accurate for well-formatted inputs, but brittle with variation.
Dictionary-based lookup: Maintain lists of known entities (city names, product names, etc.) and match query text against these dictionaries. Works well for closed-domain entities but requires maintenance and misses variations.
Sequence labeling models: Treat entity extraction as a token-level classification problem where each word receives a label (B-PERSON, I-PERSON, O for outside entity, etc.). CRF (Conditional Random Fields) and BiLSTM-CRF models were standard; now transformer-based models like BERT for token classification achieve state-of-the-art results.
LLM-based extraction: Prompt large language models to extract entities from text, either through few-shot examples or by fine-tuning on labeled data. Highly flexible and can adapt to new entity types without retraining specialized models.

Many production systems combine approaches: use rules for simple, high-confidence entities like dates and phone numbers; employ trained models for complex entities; leverage LLMs for rare or newly introduced entity types.

The following table shows example entity extractions from user queries:

User Query	Intent	Entities Extracted
"Book a flight to NYC next Friday"	book_flight	destination: "NYC", date: "next Friday"
"Table for 2 at 8 PM tonight"	make_reservation	party_size: 2, time: "8 PM", date: "tonight"
"Cancel my order #12345"	cancel_order	order_id: "12345"
"What's the weather in Boston tomorrow?"	check_weather	location: "Boston", date: "tomorrow"
"Send $50 to John Smith"	transfer_money	amount: "$50", recipient: "John Smith"

Entity linking takes entity extraction one step further by connecting recognized entities to entries in a knowledge base or database. For example, when a user mentions "Apple," entity linking disambiguates whether they mean the fruit, the technology company, or Apple Records. The system links the recognized entity to a specific identifier in a knowledge graph, enabling richer semantic understanding and integration with structured data sources.

Entity linking typically involves:

Candidate generation: Identify possible knowledge base entries the mention could refer to (e.g., "Apple" might link to Apple Inc., Apple (fruit), or Apple Corps)
Disambiguation: Use context to determine which candidate is most likely (e.g., a query about "iPhone and Apple" clearly refers to the company)
Linking: Connect the entity mention to the canonical knowledge base identifier

This process enables more sophisticated reasoning. A travel chatbot that links "Paris" to a knowledge graph can access related information like country, population, time zone, and major attractions without explicitly storing all connections in the chat system.

Diagram: Entity Extraction Architecture

Multi-Strategy Entity Extraction System

Type: diagram

Purpose: Show how modern entity extraction systems combine rule-based, model-based, and LLM approaches for comprehensive coverage

Components to show: - Input (top): User query: "Book 2 tickets to Boston on March 15th for John Smith" - Parallel extraction strategies (three branches):

Branch 1 - Rules (left): - Regex patterns box - Date parser (extracts "March 15th") - Number extractor (extracts "2") - Email/phone patterns

Branch 2 - ML Model (center): - BERT-based NER model - Token classification layer - Outputs: Person ("John Smith"), Location ("Boston") - Confidence scores shown: 0.94, 0.89

Branch 3 - LLM (right): - GPT-4 few-shot prompt - Custom entity extraction - Fallback for ambiguous cases - Shown with dotted border (used when others have low confidence)

Merging layer (middle):
Conflict resolution logic
Priority: Rules > ML > LLM for known patterns
Confidence aggregation
Entity linking layer (bottom middle):
Knowledge base lookup
"Boston" → Boston, MA (city ID: BST-MA-US)
"John Smith" → Account #7834 (from customer database)

Output (bottom): Structured entity dictionary:

{
  "quantity": 2,
  "destination": "Boston, MA",
  "destination_id": "BST-MA-US",
  "date": "2024-03-15",
  "passenger": "John Smith",
  "passenger_id": "7834"
}

Connections: - Query flows into all three extraction branches simultaneously - Extracted entities from each branch flow to merging layer - Merged entities flow to entity linking - Linked entities produce final structured output

Annotations: - "Fast, high precision" label on Rules branch - "Balanced accuracy & coverage" on ML branch - "Flexible fallback" on LLM branch - "Canonicalization" label on linking layer

Style: Parallel pipeline architecture with merge point

Color scheme: - Green for rules (deterministic) - Blue for ML model - Purple for LLM - Orange for merging logic - Teal for entity linking - Gray for output structure

Implementation: Block diagram with parallel data flows converging to single output

Diagram: Named Entity Recognition with BIO Tagging

NER Sequence Labeling Visualization

Type: microsim

Learning objective: Demonstrate how NER models label each token in a sequence with BIO tags to identify entity boundaries and types

Canvas layout (900x500px): - Top section (900x100): Input sentence display - Middle section (900x300): Interactive token labeling visualization - Bottom section (900x100): Control panel and legend

Visual elements: - Input sentence: "John Smith works at Apple in San Francisco" - Tokens displayed in boxes, each showing: - Token text (large) - BIO tag (small, below token) - Entity type (color-coded background)

Token breakdown: - "John": B-PERSON (light blue background) - "Smith": I-PERSON (light blue background) - "works": O (white background) - "at": O (white background) - "Apple": B-ORG (light orange background) - "in": O (white background) - "San": B-LOC (light green background) - "Francisco": I-LOC (light green background)

Interactive controls: - Dropdown: Select example sentence (5 pre-loaded examples) - Radio buttons: Show/hide BIO tags, Show/hide entity types - Button: "Add Custom Sentence" (allows user to type their own) - Checkbox: "Highlight entities only" (grays out O tokens)

Additional visualization: - Arrows connecting I-tags to their B-tag start - Brackets grouping multi-token entities - Color legend showing entity types: - Light blue: PERSON - Light orange: ORG - Light green: LOC - Light yellow: DATE - Light purple: MISC - White: O (outside entity)

Example sentences in dropdown: 1. "John Smith works at Apple in San Francisco" 2. "The meeting is scheduled for January 15th in New York" 3. "Dr. Emily Johnson published research at MIT last year" 4. "Amazon launched new products in Europe and Asia" 5. "The conference will be held on March 3rd, 2024"

Default parameters: - Selected sentence: Example 1 - Show BIO tags: true - Show entity types: true - Highlight entities only: false

Behavior: - When user selects different sentence, tokens update with new labels - When user toggles "Highlight entities only", O tokens fade to 50% opacity - When user hovers over a token, show full annotation details in tooltip - When user clicks "Add Custom Sentence", show text input and run simple rule-based NER

Implementation notes: - Use p5.js for rendering tokens and interactions - Implement simple regex-based NER for custom sentences (capital words = potential entities) - Store pre-labeled examples with correct BIO tags - Use color coding for clear visual distinction between entity types

Building Your First Intent-Based Chatbot

With an understanding of intents and entities, you're ready to build a practical intent-based chatbot. This architecture combines the intent classification and entity extraction techniques covered in this chapter to create a system that understands structured user requests and responds appropriately.

The basic architecture follows these steps:

Receive user input: Capture the user's message from a chat interface, API, or voice input transcription.
Preprocess text: Normalize the input by lowercasing, removing extra whitespace, and handling special characters. Optionally apply spelling correction for robustness.
Classify intent: Pass the preprocessed text through your intent classifier to determine which action the user wants to perform. If confidence is below your threshold (typically 0.6-0.8), route to a fallback handler.
Extract entities: Run entity extraction to identify specific values referenced in the query. Combine rule-based extraction for common patterns with ML models for more complex entities.
Validate completeness: Check whether all required entities for the identified intent have been extracted. If information is missing, generate a follow-up question to fill the gaps.
Execute action: With intent and entities identified, trigger the appropriate system action—query a database, call an API, or retrieve a response from your knowledge base.
Generate response: Format the results into a natural language response appropriate for the identified intent. Include error handling for failed actions.
Collect feedback: Provide thumbs up/down or other feedback mechanisms to capture user satisfaction and improve your models over time.

Let's walk through a concrete example. A user asks: "What's the weather like in Seattle tomorrow?"

Step 1: Input received: "What's the weather like in Seattle tomorrow?"

Step 2: Preprocessed: "what's the weather like in seattle tomorrow"

Step 3: Intent classification: - Intent: check_weather (confidence: 0.92)

Step 4: Entity extraction: - Location: "Seattle" (type: CITY) - Date: "tomorrow" (normalized to: 2024-01-16)

Step 5: Validation: - Required entities present: location ✓, date ✓ - Proceed to action

Step 6: Execute action: - Call weather API: getWeather(location="Seattle", date="2024-01-16") - Result: {temp: 52°F, conditions: "partly cloudy", precipitation: 20%}

Step 7: Generate response: - "The weather in Seattle tomorrow will be partly cloudy with a high of 52°F and a 20% chance of rain."

Step 8: Display with feedback buttons for continuous improvement.

This straightforward pipeline handles the majority of user queries in task-oriented chatbots. More sophisticated systems add context tracking to handle multi-turn conversations, personalization based on user history, and graceful degradation when components fail.

Here's a comparison of different chatbot architectures and when to use each:

Architecture	Best For	Advantages	Limitations
Rule-based pattern matching	Simple FAQs, very small domain	Fast, predictable, no training needed	Brittle, doesn't scale
Intent + Entity extraction	Task-oriented chatbots with clear actions	Structured, interpretable, efficient	Requires training data, limited to predefined intents
Retrieval-based (RAG)	Knowledge-intensive Q&A	Grounded responses, cites sources	Can't perform actions, needs good retrieval
Generative (LLM-based)	Open-domain conversation, creative tasks	Flexible, handles unexpected inputs	Unpredictable, hallucination risk, expensive
Hybrid (Intent + LLM)	Enterprise chatbots needing both structure and flexibility	Combines reliability and adaptability	More complex to build and maintain

For most business applications—customer support, internal IT help desks, booking systems—the intent + entity extraction architecture offers the best balance of accuracy, control, and cost-effectiveness. You can always add generative components for specific use cases while maintaining structured handling for critical transactions.

Advanced Topics: Context and Multi-Turn Dialogue

Real conversations rarely consist of isolated single-turn exchanges. Users make references to previous statements, ask follow-up questions, and change topics mid-conversation. Handling this conversational context separates basic chatbots from sophisticated dialog systems.

Consider this multi-turn exchange:

User: "What's the weather in Boston?" System: "Currently 45°F and cloudy in Boston." User: "What about tomorrow?" System: "Tomorrow in Boston will be sunny with a high of 52°F." User: "And New York?" System: "Tomorrow in New York will be partly cloudy with a high of 48°F."

Notice how the system maintains context across turns. The second query "What about tomorrow?" omits the location, but the system understands it still refers to Boston from the first query. The third query "And New York?" changes the location but maintains the temporal context (tomorrow). This contextual resolution requires the system to track conversational state.

Modern dialog systems implement context tracking through several mechanisms:

Conversation history buffer: Store the last N turns of the conversation, feeding them as context to the intent classifier and entity extractor. This helps models understand references and pronouns.
Entity memory: Maintain a dictionary of entities mentioned in the conversation, updating it as new information arrives. When entities are missing from the current query, check the memory before asking the user.
Dialog state tracking: Model the conversation as a structured state object tracking the current task, filled slots, and next expected information. Common in task-oriented systems like booking or troubleshooting bots.
Attention mechanisms: Use transformer models that can attend to relevant parts of conversation history when processing new inputs, automatically learning which context matters for each turn.

The complexity of context tracking should match your use case. Simple FAQ bots may need no context at all. Task-oriented bots benefit from slot-filling frameworks. Open-domain conversational agents require sophisticated neural approaches to maintain coherence over long conversations.

FAQ Analysis for Continuous Improvement

Building a chatbot is not a one-time effort—effective conversational systems evolve based on real user interactions. FAQ analysis provides systematic methods for identifying gaps, measuring performance, and prioritizing improvements.

Key metrics to track for FAQ and intent-based systems:

Coverage rate: Percentage of user queries that match to a known intent or FAQ above your confidence threshold
Accuracy: For queries with user feedback, percentage marked as helpful/correct
Response time: Latency from query submission to response delivery
Escalation rate: Percentage of conversations that transfer to human agents
Intent distribution: How frequently each intent appears in real traffic
Unhandled query patterns: Clusters of low-confidence queries that might represent missing intents

Regular FAQ analysis sessions should examine logs to find:

Common question variations: Multiple users asking the same thing in different ways suggests you need better training examples or synonym handling for that intent.
Coverage gaps: Frequent low-confidence queries about topics not in your current intent set indicate missing capabilities.
Ambiguous intents: Queries that oscillate between multiple intents or show low confidence across the board may indicate overlapping intent definitions needing refinement.
Entity extraction failures: Queries where the intent was correctly identified but entity extraction missed critical information require better entity training data or additional extraction rules.
Temporal patterns: Usage spikes for certain intents during specific times (e.g., "reset password" on Monday mornings, "check order status" after promotional emails) can inform staffing and proactive messaging.

By analyzing these patterns monthly or quarterly, you can systematically improve your chatbot's capabilities. Start by focusing on high-frequency, low-accuracy queries—small improvements here deliver large impact. Build out coverage for newly discovered intents. Refine ambiguous intent boundaries to reduce classification errors.

The most successful chatbot teams implement continuous learning loops where user feedback directly updates training data, models retrain weekly or monthly, and performance dashboards make improvement trends visible to stakeholders.

Summary and Key Takeaways

This chapter introduced the foundational concepts for building conversational interfaces that understand user intentions and extract relevant information from natural language queries. You learned how chatbots and conversational agents differ in sophistication, how dialog systems manage multi-turn conversations, and how intent classification and entity extraction transform unstructured text into actionable structured data.

Key concepts to remember:

User queries are natural language inputs that express intents and contain entities needing extraction
FAQ systems map user questions to predefined answers, forming the simplest conversational interface
Chatbots range from simple rule-based systems to sophisticated conversational agents with context tracking
Dialog systems manage multi-turn conversations with state tracking and context awareness
Intent recognition identifies what users want; entity extraction identifies the specific details they're referencing
Intent modeling requires careful design to balance granularity, coverage, and actionability
Named Entity Recognition (NER) identifies people, places, organizations, and other proper nouns
Entity linking connects recognized entities to knowledge base entries for deeper semantic understanding
Context tracking enables multi-turn conversations by maintaining entity memory and conversation history
FAQ analysis drives continuous improvement by identifying coverage gaps and accuracy issues

These concepts form the foundation for more advanced conversational AI architectures. In later chapters, you'll see how Retrieval Augmented Generation (RAG) extends beyond FAQ matching with semantic search, how knowledge graphs enable entity linking and reasoning, and how modern LLMs can handle both intent classification and entity extraction through prompting rather than training specialized models.

The intent + entity architecture remains fundamental even as models grow more sophisticated—understanding what users want and what information they're providing applies whether you're using regex patterns, fine-tuned BERT models, or few-shot prompting with GPT-4. Master these concepts, and you'll be prepared to build conversational interfaces across the full spectrum of modern AI approaches.