README.md

Product Assistant Backend

Product Assistant is an intelligent backend service that supports product search and recommendation using a Large Language Model (LLM), integrates with Telegram for authorization and notifications, uses Kafka for messaging (price change notifications), and collects traces via OpenTelemetry. The service is containerized using Docker and deployed with Docker Compose (or Kubernetes via separate pipelines).

---

Table of Contents

• Overview
• Architecture
• Components
  • Authorization via Telegram
  • LLM Integration for Product Querying
  • Product Management
  • Wishlist Functionality
  • Kafka-based Notifications & Telegram Notifications
  • Tracing with OpenTelemetry
• Installation & Setup
  • Environment Variables
  • Local Development with Docker Compose
• CI/CD Pipeline
• Usage
  • API Endpoints
  • Swagger UI
• Testing & Quality Assurance
• Developer Documentation
• Future Enhancements
• Contributing
• License

---

Overview

Product Assistant Backend is designed to help users search for and compare technical products. Its main features include:

• Authorization via Telegram: Users log in using Telegram, with JWT tokens generated for secure API access.
• LLM-powered product search: The system uses a Large Language Model to analyze user queries, combined with vector search (embeddings) to return a list of relevant products.
• Product Management: The database stores abstract products (aggregated product data) and parsed products (specific entries from various marketplaces) along with product images.
• Wishlist: Users can add products to their wishlist to receive notifications on price changes.
• Notifications via Kafka and Telegram: When prices change, the parser triggers Kafka events that are consumed by a notification service which sends alerts via Telegram.
• Tracing: OpenTelemetry collects and exports trace data to help monitor the application.
• Containerized Deployment: The entire stack is packaged with Docker and can be deployed via Docker Compose (or Kubernetes via separate CI/CD pipelines).

---

Architecture

The system is composed of several interacting components:

flowchart LR
    A[User Frontend]
    B[API Backend]
    C[Postgres/TimescaleDB]
    D[Parser (Node.js)]
    E[LLM Service]
    F[Kafka]
    G[Notification Service]
    H[Telegram Bot]
    I[OpenTelemetry Collector]
    J[Jaeger UI]
    
    A -->|HTTP API| B
    B -->|SQL| C
    D -->|Parses & Updates| C
    D -->|Price change event| F
    F -->|Message| G
    G -->|Telegram message| H
    B -->|Traces| I
    I --> J

Components

Authorization via Telegram

Flow:

1. The frontend uses the Telegram Login Widget to authenticate users.

2. Telegram returns user data (ID, first name, last name, username, photo URL, auth date, and hash).

3. The backend validates the hash, registers the user (if new) or updates their record, and generates a JWT token for further API requests.

Key Components:

• TelegramAuthService: Validates Telegram data and generates JWT tokens.

• JWT middleware ensures that protected endpoints (e.g., wishlist) are accessed only with valid tokens.

LLM Integration for Product Querying

Flow:

1. User submits a search query via the frontend.

2. The backend retrieves a list of products (using embeddings) from the database.

3. It builds a prompt that includes user instructions, the search query, and product context.

4. The LLM returns a JSON response with recommended product IDs and a comment, which is parsed and sent to the frontend.

Key Components:

• LLMService: Constructs the prompt and handles responses.

• LangchangoClient: Interacts with the LLM using a defined message history.

Product Management

• Abstract Products: Aggregated product data stored in abstract_products (includes name, JSON characteristics, and timestamps).

• Parsed Products: Specific entries for each abstract product from different marketplaces (includes price, URL, market name, etc.) stored in parsed_products.

Wishlist Functionality

Flow:

• Users can add or remove abstract products to/from their wishlist. The backend stores these in wish_list_items.

Key Components:

• FavouriteService: Implements adding, deleting, and listing wishlist items.

• JWT middleware ensures that wishlist actions are tied to the authenticated user.

Kafka-based Notifications & Telegram Notifications

Flow:

1. When a parsed product’s price changes, the parser triggers an update that calls PublishPriceChange.

2. This method publishes a message to a Kafka topic (e.g., "price_changes").

3. A Kafka consumer in the Notification Service receives the event, retrieves the list of users following the product, and sends a notification via Telegram.

Key Components:

• NotificationService: Publishes and consumes Kafka messages, sends Telegram notifications.

• Kafka Producer/Consumer: Handles messaging.

Tracing with OpenTelemetry

Flow:

1. All API endpoints are wrapped in OpenTelemetry middleware (otelhttp.NewHandler).

2. Internal operations may also create manual spans.

3. The collector exports traces to Jaeger, where they can be visualized.

Key Components:

• diagnostics.NewOTel: Configures and initializes the tracer provider.

• Integration via otelhttp and manual span creation in critical sections.

Docker Compose Deployment

• Local Deployment: The entire stack (DB, Kafka, OTEL Collector, Jaeger, Parser, Ollama, and API Backend) is defined in a docker-compose.yml file, enabling local testing and development.

• Key Components in Docker Compose:

• Containers for TimescaleDB, Kafka, Zookeeper, OTEL Collector, Jaeger, Ollama, and your API server.

• Volumes for persistent data.

• Environment variables to configure connectivity between services.

Installation & Setup

Prerequisites

• Docker & Docker Compose: Ensure Docker is installed and Docker Compose is available.

• Git: For cloning the repository.

• GitLab Runner: For CI/CD integration (if deploying via GitLab).

Clone Repository

git clone <repository-url>
cd <repository-directory>

Environment Variables

Create a .env file (or set environment variables via your deployment system) with the following keys:

# Server settings
SERVER_AppName=ProductAssistant
SERVER_Port=8080
SERVER_Secret=secret

# Database settings
DATABASE_Hostname=timescaledb
DATABASE_Port=5432
DATABASE_User=postgres
DATABASE_Password=secret
DATABASE_Dbname=products_db

# LLM settings
LLM_Embedder=bge-m3
LLM_MODEL_NAME=mistral-large-latest
LLM_API_KEY=your-api-key
LLM_Endpoint=http://ollama:11434

# Kafka settings
KAFKA_BOOTSTRAP_SERVERS=kafka:9092
KAFKA_PRODUCER_TOPIC=price_changes
KAFKA_CONSUMER_TOPIC=price_changes
KAFKA_CONSUMER_GROUP_ID=notifications_group

# OTEL settings
OTEL_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_OTLP_INSECURE=true
OTEL_OTLP_TIMEOUT=10s
OTEL_OTLP_HTTP=false
OTEL_RETRY_ENABLED=true
OTEL_RETRY_INITIAL_INTERVAL=5s
OTEL_RETRY_MAX_INTERVAL=30s
OTEL_RETRY_MAX_ELAPSED_TIME=1m

# Telegram settings
TELEGRAM_BOT_TOKEN=7665408380:AAFKbUgbdQb3iX6X-29H3spv4KGG1FXs_IM
TELEGRAM_JWT_SECRET=secret

Local Deployment with Docker Compose

In the deployment/ directory, you have a docker-compose.yml file. Run file scripts/run.cmd

This will spin up all necessary services. You can verify that the API is running by accessing:

• API: http://localhost:8080
• Swagger UI: http://localhost:8080/* swagger/
• Jaeger UI: http://localhost:16686

Usage

API Endpoints

The backend exposes endpoints for:

• Authorization via Telegram: /auth/telegram

• Product Search (LLM-based): /llm-request

• Product Details: /get-abstract-product and /get-parsed-products

• Wishlist Management: /favourites/...

Check the Swagger UI for full API documentation.

Swagger UI

Visit http://localhost:8080/swagger/ (or your deployment URL) to explore and test API endpoints.