RSK World - Complete Documentation - BERT Chatbot | BERT Chatbot | Natural Language Processing | Conversational AI | Transformers | PyTorch | RSK World

Project Description

This project implements a BERT-based chatbot system fine-tuned for conversational AI. BERT (Bidirectional Encoder Representations from Transformers) uses bidirectional context to understand language, making it excellent for chatbots that need to understand conversation context. The system includes fine-tuning strategies, response generation, context management, and advanced features like semantic similarity search, intent classification, and sentiment analysis.

BERT uses transformer architecture with bidirectional encoding to understand context from both directions. The implementation provides complete PyTorch support with Transformers (Hugging Face), comprehensive training pipeline, REST API server, evaluation metrics, and deployment tools for conversational AI applications.

Project Screenshots

1 / 4

Core Features

BERT Architecture

Bidirectional context encoding
Transformer encoder layers
Pre-trained BERT base model
Fine-tuning for conversations
Natural language understanding

Context Management

Multi-turn conversation support
Session-based context
Sliding window context
Context-aware responses
Conversation history tracking

Semantic Similarity

BERT embedding search
Cosine similarity matching
Top-K response retrieval
Response database indexing
Similarity-based ranking

Intent Classification

10 intent categories
Keyword-based detection
Confidence scoring
Intent-aware responses
Real-time classification

Sentiment Analysis

Positive/Negative/Neutral
Sentiment scoring
Sentiment-aware responses
Real-time analysis
Multi-class classification

REST API Server

Flask-based API
Multiple endpoints
CORS enabled
Request logging
Production-ready

Advanced Features

Evaluation Metrics

BLEU scores (1-4 grams)
METEOR score
ROUGE-L score
Perplexity calculation
Batch evaluation support

Data Augmentation

Synonym replacement
Paraphrasing techniques
Noise injection
Automated dataset expansion

Response Ranking

Multi-factor ranking system
Similarity-based (50%)
Intent confidence (20%)
Sentiment matching (30%)

Conversation Analytics

Dataset statistics
Word frequency analysis
Topic distribution
Conversation length analysis
Export capabilities

REST API Endpoints

Endpoint	Method	Description	Request Body	Response
/chat	POST	Main chat interface	{"message": "text", "session_id": "optional"}	Response with intent & sentiment
/intent	POST	Intent classification	{"message": "text"}	Intent & confidence score
/sentiment	POST	Sentiment analysis	{"message": "text"}	Sentiment scores
/similar	POST	Find similar responses	{"message": "text", "top_k": 5}	List of similar responses
/evaluate	POST	Evaluate response quality	{"reference": "text", "candidate": "text"}	BLEU, METEOR, ROUGE scores
/context/<id>	GET	Get conversation context	N/A	Conversation history
/context/<id>	DELETE	Clear conversation context	N/A	Success message

Technologies Used

Python 3.8+ PyTorch 2.0+ Transformers 4.30+ Hugging Face BERT NLTK 3.8+ spaCy 3.6+ Jupyter Notebook Flask 2.3+ Evaluation Metrics

Installation & Usage

Installation

Install all required dependencies for the BERT chatbot project:

# Install all requirements
pip install -r requirements.txt

# Download NLTK data
python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords')"

# Download spaCy model (optional)
python -m spacy download en_core_web_sm

Quick Setup

Set up the project structure and verify installation:

# Run quick setup script
python quick_start.py setup

# Test imports
python quick_start.py test-imports

# Run quick test
python quick_start.py quick-test

Training the Model

Fine-tune the BERT model on your conversational dataset:

# Basic training
python train.py --data_path data/conversations.json --output_dir models/bert-chatbot

# Training with custom parameters
python train.py --data_path data/conversations.json \
                --output_dir models/bert-chatbot \
                --epochs 10 \
                --batch_size 16 \
                --learning_rate 2e-5 \
                --max_length 512

Interactive Chatbot

Run the chatbot in interactive mode for conversations:

# Run chatbot with default model
python chatbot.py

# Run with trained model
python chatbot.py --model_path models/bert-chatbot/checkpoint.pt

# Run with custom session ID
python chatbot.py --session_id user123

# Commands in chat:
# - Type 'quit', 'exit', or 'bye' to end
# - Type 'clear' to clear context
# - Type 'intent' to see intent classification
# - Type 'sentiment' to see sentiment analysis

REST API Server

Start the Flask API server for web integration:

# Start API server (default port 5000)
python api_server.py

# API will be available at http://localhost:5000

# Example API calls:
# POST /chat - {"message": "Hello", "session_id": "user123"}
# POST /intent - {"message": "What is BERT?"}
# POST /sentiment - {"message": "I love this chatbot!"}
# POST /similar - {"message": "Hello", "top_k": 5}
# GET /context/user123
# DELETE /context/user123

Data Augmentation

Augment your training data to improve model performance:

# Run data augmentation
python data_augmentation.py

# This will create augmented versions of your conversations
# using synonym replacement, paraphrasing, and noise injection

Evaluation Metrics

Evaluate chatbot responses using comprehensive metrics:

from evaluation import ChatbotEvaluator

evaluator = ChatbotEvaluator()

# Evaluate single response
scores = evaluator.evaluate_response(
    reference="Hello, how can I help you?",
    candidate="Hi there! How may I assist you?"
)

# Returns dictionary with:
# - bleu_1, bleu_2, bleu_3, bleu_4: BLEU scores (1-4 grams)
# - meteor: METEOR score (semantic similarity)
# - rouge_l: ROUGE-L score (longest common subsequence)
# - perplexity: Language model perplexity

# Batch evaluation
references = ["Ref 1", "Ref 2", ...]
candidates = ["Cand 1", "Cand 2", ...]
scores = evaluator.evaluate_batch(references, candidates)

# Print detailed scores
for metric, value in scores.items():
    print(f"{metric}: {value:.4f}")

Metric Descriptions:

BLEU (1-4): Measures n-gram precision between reference and candidate
METEOR: Considers synonyms and word order, more semantic than BLEU
ROUGE-L: Measures longest common subsequence, good for longer responses
Perplexity: Measures how well the model predicts the next word

Conversation Analytics

Analyze your conversation dataset:

# Generate conversation statistics
python conversation_stats.py

# This will display:
# - Total conversations
# - Word frequency analysis
# - Topic distribution
# - Conversation length statistics

Export Conversations

Export conversations to various formats:

# Export conversations
python export_conversations.py

# Exports to:
# - CSV format
# - Plain text format
# - Training format
# - Statistics export

Jupyter Notebook

Open the interactive Jupyter notebook for demonstrations:

# Start Jupyter notebook
jupyter notebook notebooks/BERT_Chatbot_Demo.ipynb

# Or use JupyterLab
jupyter lab notebooks/BERT_Chatbot_Demo.ipynb

Project Structure

                bert-chatbot/

                ├── README.md                          # Main documentation

                ├── requirements.txt                   # Python dependencies

                ├── config.py                          # Configuration settings

                ├── setup.py                           # Package setup script

                ├── LICENSE                            # MIT License

                ├── PROJECT_INFO.md                    # Project overview

                │

                ├── Core Modules

                │   ├── model.py                       # BERT model architecture

                │   ├── train.py                       # Training script

                │   ├── chatbot.py                     # Interactive chatbot

                │   ├── context_manager.py             # Context management

                │   └── utils.py                       # Utility functions

                │

                ├── Advanced Features

                │   ├── advanced_features.py           # Semantic search, intent, sentiment

                │   ├── evaluation.py                  # Evaluation metrics

                │   ├── data_augmentation.py           # Data augmentation

                │   └── conversation_stats.py          # Analytics

                │

                ├── API & Services

                │   ├── api_server.py                  # REST API server

                │   └── logger_config.py               # Logging system

                │

                ├── Utilities

                │   ├── export_conversations.py       # Export tools

                │   └── quick_start.py                # Setup script

                │

                ├── Data

                │   └── conversations.json            # Training data (18 conversations)

                │

                ├── Models

                │   └── (trained model checkpoints)

                │

                ├── Notebooks

                │   └── BERT_Chatbot_Demo.ipynb       # Jupyter notebook demo

                │

                └── Tests

                    └── test_chatbot.py               # Unit tests

Configuration Options

Model Configuration (config.py)

Customize model and training parameters in config.py:

# Model Configuration
MODEL_NAME = "bert-base-uncased"  # BERT model variant
MAX_LENGTH = 512                  # Maximum sequence length
BATCH_SIZE = 16                   # Training batch size
LEARNING_RATE = 2e-5              # Learning rate
NUM_EPOCHS = 3                    # Number of training epochs
WARMUP_STEPS = 100                # Warmup steps for LR scheduler

# Chatbot Configuration
CONTEXT_WINDOW = 5                # Previous messages in context
MAX_RESPONSE_LENGTH = 100         # Maximum response length
TEMPERATURE = 0.7                 # Generation temperature
TOP_P = 0.9                       # Nucleus sampling
TOP_K = 50                        # Top-K sampling

# Data Split
TRAIN_SPLIT = 0.8                 # Training data percentage
VAL_SPLIT = 0.1                   # Validation data percentage
TEST_SPLIT = 0.1                  # Test data percentage

Advanced Features Usage

Semantic Similarity Search

Find semantically similar responses using BERT embeddings:

from advanced_features import SemanticSimilaritySearch
from model import BERTChatbot, BERTTokenizerWrapper
import torch

# Initialize
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = BERTChatbot().to(device)
tokenizer = BERTTokenizerWrapper()
similarity_search = SemanticSimilaritySearch(model, tokenizer, device)

# Add responses to database
responses = ["Hello, how can I help?", "What is BERT?", ...]
similarity_search.add_responses(responses)

# Find similar responses
similar = similarity_search.find_similar("Hi there", top_k=5)
# Returns: [("Hello, how can I help?", 0.95), ...]

Intent Classification

Classify user intent into 10 categories:

from advanced_features import IntentClassifier

intent_classifier = IntentClassifier(model, tokenizer, device)

# Classify intent
intent, confidence = intent_classifier.classify_with_keywords("Hello, how are you?")
# Returns: ("greeting", 0.95)

# Available intents:
# - greeting, question, request, complaint, compliment
# - goodbye, clarification, confirmation, negation, other

Sentiment Analysis

Analyze sentiment with positive, negative, and neutral scores:

from advanced_features import SentimentAnalyzer

sentiment_analyzer = SentimentAnalyzer(model, tokenizer, device)

# Analyze sentiment
sentiment = sentiment_analyzer.analyze("I love this chatbot!")
# Returns: {'positive': 0.85, 'negative': 0.05, 'neutral': 0.10}

# Use in response selection
if sentiment['positive'] > 0.7:
    # Select positive response
    pass

Response Ranking

Rank responses using multi-factor scoring system:

from advanced_features import ResponseRanker

response_ranker = ResponseRanker(
    similarity_search, 
    intent_classifier, 
    sentiment_analyzer
)

# Rank candidate responses
candidates = ["Response 1", "Response 2", "Response 3"]
ranked = response_ranker.rank_responses(
    query="What is machine learning?",
    candidates=candidates,
    top_k=3
)

# Ranking factors:
# - Semantic similarity: 50% weight
# - Intent confidence: 20% weight
# - Sentiment match: 30% weight

Data Augmentation

Augment training data programmatically:

from data_augmentation import DataAugmenter

augmenter = DataAugmenter()

# Augment single conversation
conversation = {
    "messages": [
        {"role": "user", "content": "What is BERT?"},
        {"role": "bot", "content": "BERT is a transformer model."}
    ]
}

augmented = augmenter.augment_conversation(
    conversation,
    methods=['synonym', 'paraphrase', 'noise']
)

# Methods available:
# - 'synonym': Replace words with synonyms
# - 'paraphrase': Paraphrase sentences
# - 'noise': Add random noise

API Usage Examples

Chat Endpoint (cURL)

Send a chat message and get response with intent and sentiment:

curl -X POST http://localhost:5000/chat \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Hello, what is BERT?",
    "session_id": "user123"
  }'

# Response:
# {
#   "response": "BERT is a transformer model...",
#   "session_id": "user123",
#   "intent": "question",
#   "intent_confidence": 0.92,
#   "sentiment": {"positive": 0.6, "negative": 0.1, "neutral": 0.3},
#   "timestamp": "2025-01-15T10:30:00"
# }

Intent Classification (cURL)

Classify user intent:

curl -X POST http://localhost:5000/intent \
  -H "Content-Type: application/json" \
  -d '{"message": "Can you help me?"}'

# Response:
# {
#   "intent": "request",
#   "confidence": 0.88,
#   "message": "Can you help me?"
# }

Sentiment Analysis (cURL)

Analyze message sentiment:

curl -X POST http://localhost:5000/sentiment \
  -H "Content-Type: application/json" \
  -d '{"message": "This is amazing!"}'

# Response:
# {
#   "sentiment": {
#     "positive": 0.92,
#     "negative": 0.03,
#     "neutral": 0.05
#   },
#   "message": "This is amazing!"
# }

Similarity Search (cURL)

Find similar responses:

curl -X POST http://localhost:5000/similar \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is machine learning?",
    "top_k": 5
  }'

# Response:
# {
#   "similar_responses": [
#     {"response": "...", "similarity": 0.95},
#     {"response": "...", "similarity": 0.88}
#   ]
# }

Python Requests Example

Use the API with Python requests library:

import requests

# Chat endpoint
response = requests.post(
    'http://localhost:5000/chat',
    json={
        'message': 'Hello, how are you?',
        'session_id': 'user123'
    }
)
data = response.json()
print(f"Bot: {data['response']}")
print(f"Intent: {data['intent']}")
print(f"Sentiment: {data['sentiment']}")

# Get context
context = requests.get('http://localhost:5000/context/user123')
print(context.json())

# Clear context
requests.delete('http://localhost:5000/context/user123')

Intent Categories

Intent	Description	Example Keywords	Example Phrases
greeting	Initial greetings and salutations	hello, hi, hey, greetings	"Hello", "Hi there", "Good morning"
question	Asking for information	what, how, why, when, where	"What is BERT?", "How does it work?"
request	Requesting assistance or action	can, could, please, help	"Can you help?", "Please explain"
complaint	Expressing dissatisfaction	bad, wrong, error, problem, issue	"This doesn't work", "There's an error"
compliment	Expressing appreciation	good, great, excellent, amazing, love	"Great job!", "This is amazing!"
goodbye	Farewell and closing	bye, goodbye, see you, farewell	"Goodbye", "See you later"
clarification	Asking for clarification	what do you mean, explain, clarify	"What do you mean?", "Can you clarify?"
confirmation	Confirming understanding	yes, correct, right, exactly, sure	"Yes, that's right", "Exactly"
negation	Expressing disagreement or denial	no, not, wrong, incorrect, disagree	"No, that's wrong", "I disagree"
other	Other intents not categorized	N/A	Any other input

Dataset Information

Conversation Dataset

The project includes 18 conversations covering diverse topics:

Machine Learning & AI
Deep Learning & Neural Networks
BERT & Transformers
Python Programming
Natural Language Processing
Tokenization & Preprocessing
Fine-tuning Techniques
Attention Mechanisms
GPT & Language Models
Chatbot Evaluation
Transfer Learning
And more technical topics...

Data Format

Conversations are stored in JSON format:

[
  {
    "id": "conv_1",
    "topic": "Machine Learning",
    "messages": [
      {
        "role": "user",
        "content": "What is machine learning?"
      },
      {
        "role": "bot",
        "content": "Machine learning is a subset of AI..."
      }
    ]
  }
]

Adding Custom Data

Add your own conversations to the dataset:

from utils import load_conversations, save_conversations

# Load existing conversations
conversations = load_conversations('data/conversations.json')

# Add new conversation
new_conv = {
    "id": f"conv_{len(conversations) + 1}",
    "topic": "Your Topic",
    "messages": [
        {"role": "user", "content": "User message"},
        {"role": "bot", "content": "Bot response"}
    ]
}
conversations.append(new_conv)

# Save updated conversations
save_conversations(conversations, 'data/conversations.json')

Troubleshooting & Best Practices

Common Issues

CUDA Out of Memory: Reduce batch_size in config.py or use CPU mode
Model Not Found: Ensure BERT model downloads correctly on first run
NLTK Data Missing: Run python -c "import nltk; nltk.download('punkt'); nltk.download('stopwords')"
API Connection Error: Check if api_server.py is running on port 5000
Import Errors: Verify all dependencies installed: pip install -r requirements.txt
Context Too Long: Reduce CONTEXT_WINDOW in config.py or MAX_LENGTH

Best Practices

Training Data: Use diverse, high-quality conversations for better results
Context Management: Keep CONTEXT_WINDOW between 3-7 for optimal performance
Batch Size: Use smaller batches (8-16) for limited GPU memory
Learning Rate: Start with 2e-5 and adjust based on training loss
Evaluation: Regularly evaluate on validation set during training
Session Management: Use unique session_ids for different users
API Rate Limiting: Implement rate limiting for production deployments
Logging: Monitor logs in models/bert-chatbot/logs/ for debugging

Performance Optimization

GPU Usage: Set CUDA_VISIBLE_DEVICES for multi-GPU systems
Model Quantization: Consider model quantization for faster inference
Batch Processing: Process multiple requests in batches when possible
Caching: Cache frequent responses to reduce computation
Context Pruning: Remove old context beyond CONTEXT_WINDOW

Contact Information

Get in Touch

Developer: Molla Samser
Designer & Tester: Rima Khatun

rskworld.in

help@rskworld.in support@rskworld.in

+91 93305 39277

License

This project is for educational purposes only. See LICENSE file for more details.

Theme Settings

Color Scheme

Display Options

Font Size

BERT Chatbot

Project Description

Project Screenshots

Core Features

BERT Architecture

Context Management

Semantic Similarity

Intent Classification

Sentiment Analysis

REST API Server

Advanced Features

Evaluation Metrics

Data Augmentation

Response Ranking

Conversation Analytics

REST API Endpoints

Technologies Used

Installation & Usage

Installation

Quick Setup

Training the Model

Interactive Chatbot

REST API Server

Data Augmentation

Evaluation Metrics

Conversation Analytics

Export Conversations

Jupyter Notebook

Project Structure

Configuration Options

Model Configuration (config.py)

Advanced Features Usage

Semantic Similarity Search

Intent Classification

Sentiment Analysis

Response Ranking

Data Augmentation

API Usage Examples

Chat Endpoint (cURL)

Intent Classification (cURL)

Sentiment Analysis (cURL)

Similarity Search (cURL)

Python Requests Example

Intent Categories

Dataset Information

Conversation Dataset

Data Format

Adding Custom Data

Troubleshooting & Best Practices

Common Issues

Best Practices

Performance Optimization

Contact Information

Get in Touch

License