diff --git a/README.md b/README.md index bc65bd84e49f08018dc65303be590046399a66d1..5848f7305fc08eb18b2d93842bae497802e62498 100644 --- a/README.md +++ b/README.md @@ -1,95 +1,248 @@ -# Model +# Product Assistant Backend -Проект представляет СЃРѕР±РѕР№ сервис, который взаимодействует СЃ базой данных для получения информации Рѕ продуктах, РёС… фотографиях Рё ценах. Сервис также включает функциональность для обработки запросов СЃ использованием ChatAssistant Рё предоставляет API для взаимодействия СЃ клиентами. +**Product Assistant** is an intelligent backend service that supports product search and recommendation using a Large Language Model (LLM), integrates with Telegram for authorization and notifications, uses Kafka for messaging (price change notifications), and collects traces via OpenTelemetry. The service is containerized using Docker and deployed with Docker Compose (or Kubernetes via separate pipelines). -### Компоненты проекта +--- -1. **cmd/assistant/main.go**: РћСЃРЅРѕРІРЅРѕР№ файл, который инициализирует конфигурацию, создает сервер РЅР° базе Gin Рё определяет маршруты для обработки запросов. -2. **internal/services/store/Store.go**: Содержит методы для взаимодействия СЃ базой данных, такие как получение продуктов РїРѕ идентификаторам, получение фотографий Рё цен РїРѕ продукту. -3. **internal/services/chatassistant**: Пакет, отвечающий Р·Р° обработку запросов СЃ использованием ChatAssistant. -4. **internal/config**: Пакет для инициализации конфигурации проекта. +## Table of Contents -### Запуск проекта +- [Overview](#overview) +- [Architecture](#architecture) +- [Components](#components) + - [Authorization via Telegram](#authorization-via-telegram) + - [LLM Integration for Product Querying](#llm-integration-for-product-querying) + - [Product Management](#product-management) + - [Wishlist Functionality](#wishlist-functionality) + - [Kafka-based Notifications & Telegram Notifications](#kafka-based-notifications--telegram-notifications) + - [Tracing with OpenTelemetry](#tracing-with-opentelemetry) +- [Installation & Setup](#installation--setup) + - [Environment Variables](#environment-variables) + - [Local Development with Docker Compose](#local-development-with-docker-compose) +- [CI/CD Pipeline](#cicd-pipeline) +- [Usage](#usage) + - [API Endpoints](#api-endpoints) + - [Swagger UI](#swagger-ui) +- [Testing & Quality Assurance](#testing--quality-assurance) +- [Developer Documentation](#developer-documentation) +- [Future Enhancements](#future-enhancements) +- [Contributing](#contributing) +- [License](#license) -Для запуска проекта необходимо выполнить следующие шаги: +--- -1. **РЎР±РѕСЂРєР° Docker-образа**: - Создайте Dockerfile СЃ содержимым: +## Overview - ```dockerfile - # Рспользуем базовый образ СЃ Go - FROM golang:1.23.2 +Product Assistant Backend is designed to help users search for and compare technical products. Its main features include: - # Устанавливаем рабочую директорию - WORKDIR /app +- **Authorization via Telegram:** Users log in using Telegram, with JWT tokens generated for secure API access. +- **LLM-powered product search:** The system uses a Large Language Model to analyze user queries, combined with vector search (embeddings) to return a list of relevant products. +- **Product Management:** The database stores abstract products (aggregated product data) and parsed products (specific entries from various marketplaces) along with product images. +- **Wishlist:** Users can add products to their wishlist to receive notifications on price changes. +- **Notifications via Kafka and Telegram:** When prices change, the parser triggers Kafka events that are consumed by a notification service which sends alerts via Telegram. +- **Tracing:** OpenTelemetry collects and exports trace data to help monitor the application. +- **Containerized Deployment:** The entire stack is packaged with Docker and can be deployed via Docker Compose (or Kubernetes via separate CI/CD pipelines). - # Копируем файлы проекта - COPY go.mod go.sum ./ - RUN go mod download - COPY . . +--- - # РЎР±РѕСЂРєР° приложения - RUN go build -o main ./cmd/assistant +## Architecture - # Указываем РїРѕСЂС‚ - EXPOSE 8080 +The system is composed of several interacting components: - # Запуск приложения - CMD ["./main"] - ``` +```mermaid +flowchart LR + A[User Frontend] + B[API Backend] + C[Postgres/TimescaleDB] + D[Parser (Node.js)] + E[LLM Service] + F[Kafka] + G[Notification Service] + H[Telegram Bot] + I[OpenTelemetry Collector] + J[Jaeger UI] + + A -->|HTTP API| B + B -->|SQL| C + D -->|Parses & Updates| C + D -->|Price change event| F + F -->|Message| G + G -->|Telegram message| H + B -->|Traces| I + I --> J +``` -2. **РЎР±РѕСЂРєР° Рё запуск контейнера**: - Выполните команды: +## Components +### Authorization via Telegram +Flow: - ```sh - docker build -t myapp . - docker run -p 8080:8080 myapp - ``` +1. The frontend uses the Telegram Login Widget to authenticate users. -3. **Запуск сервера**: - Сервер будет доступен РїРѕ адресу `http://localhost:8080`. +2. Telegram returns user data (ID, first name, last name, username, photo URL, auth date, and hash). -### API Маршруты +3. The backend validates the hash, registers the user (if new) or updates their record, and generates a JWT token for further API requests. -- **POST /llm-request**: Обрабатывает запросы СЃ использованием ChatAssistant. -- **GET /get-product**: Возвращает информацию Рѕ продукте РїРѕ его идентификатору. -- **GET /get-matched-ids**: Возвращает СЃРїРёСЃРѕРє продуктов, соответствующих запросу пользователя. +Key Components: -### Примеры запросов +* TelegramAuthService: Validates Telegram data and generates JWT tokens. -- **POST /llm-request**: - ```json - { - "query": "example query" - } - ``` +* JWT middleware ensures that protected endpoints (e.g., wishlist) are accessed only with valid tokens. +### LLM Integration for Product Querying +Flow: -- **GET /get-product**: - ```sh - curl "http://localhost:8080/get-product?Id=123" - ``` +1. User submits a search query via the frontend. -- **GET /get-matched-ids**: - ```sh - curl "http://localhost:8080/get-matched-ids?query=example&limit=10" - ``` +2. The backend retrieves a list of products (using embeddings) from the database. -### Зависимости +3. It builds a prompt that includes user instructions, the search query, and product context. -Проект использует следующие зависимости: +4. The LLM returns a JSON response with recommended product IDs and a comment, which is parsed and sent to the frontend. -- **Gin**: Веб-фреймворк для создания HTTP-сервера. -- **GORM**: ORM-библиотека для работы СЃ базой данных. -- **Go**: Язык программирования для написания серверной части. +Key Components: -### Авторы Рё благодарности +* LLMService: Constructs the prompt and handles responses. -Покажите СЃРІРѕСЋ признательность тем, кто внес вклад РІ проект. +* LangchangoClient: Interacts with the LLM using a defined message history. -### Лицензия +### Product Management +* Abstract Products: +Aggregated product data stored in abstract_products (includes name, JSON characteristics, and timestamps). -Укажите, РїРѕРґ какой лицензией распространяется проект. +* Parsed Products: +Specific entries for each abstract product from different marketplaces (includes price, URL, market name, etc.) stored in parsed_products. -### Статус проекта +### Wishlist Functionality +Flow: +* Users can add or remove abstract products to/from their wishlist. The backend stores these in wish_list_items. + +Key Components: + +* FavouriteService: Implements adding, deleting, and listing wishlist items. + +* JWT middleware ensures that wishlist actions are tied to the authenticated user. + +### Kafka-based Notifications & Telegram Notifications +Flow: + +1. When a parsed product’s price changes, the parser triggers an update that calls PublishPriceChange. + +2. This method publishes a message to a Kafka topic (e.g., "price_changes"). + +3. A Kafka consumer in the Notification Service receives the event, retrieves the list of users following the product, and sends a notification via Telegram. + +Key Components: + +* NotificationService: Publishes and consumes Kafka messages, sends Telegram notifications. + +* Kafka Producer/Consumer: Handles messaging. + +### Tracing with OpenTelemetry +Flow: + +1. All API endpoints are wrapped in OpenTelemetry middleware (otelhttp.NewHandler). + +2. Internal operations may also create manual spans. + +3. The collector exports traces to Jaeger, where they can be visualized. + +Key Components: + +* diagnostics.NewOTel: Configures and initializes the tracer provider. + +* Integration via otelhttp and manual span creation in critical sections. + +### Docker Compose Deployment +* Local Deployment: +The entire stack (DB, Kafka, OTEL Collector, Jaeger, Parser, Ollama, and API Backend) is defined in a docker-compose.yml file, enabling local testing and development. + +* Key Components in Docker Compose: + +* Containers for TimescaleDB, Kafka, Zookeeper, OTEL Collector, Jaeger, Ollama, and your API server. + +* Volumes for persistent data. + +* Environment variables to configure connectivity between services. + +## Installation & Setup + +### Prerequisites +* Docker & Docker Compose: +Ensure Docker is installed and Docker Compose is available. + +* Git: +For cloning the repository. + +* GitLab Runner: +For CI/CD integration (if deploying via GitLab). + +### Clone Repository +```bash +git clone <repository-url> +cd <repository-directory> +``` +### Environment Variables +Create a .env file (or set environment variables via your deployment system) with the following keys: +```env +# Server settings +SERVER_AppName=ProductAssistant +SERVER_Port=8080 +SERVER_Secret=secret + +# Database settings +DATABASE_Hostname=timescaledb +DATABASE_Port=5432 +DATABASE_User=postgres +DATABASE_Password=secret +DATABASE_Dbname=products_db + +# LLM settings +LLM_Embedder=bge-m3 +LLM_MODEL_NAME=mistral-large-latest +LLM_API_KEY=your-api-key +LLM_Endpoint=http://ollama:11434 + +# Kafka settings +KAFKA_BOOTSTRAP_SERVERS=kafka:9092 +KAFKA_PRODUCER_TOPIC=price_changes +KAFKA_CONSUMER_TOPIC=price_changes +KAFKA_CONSUMER_GROUP_ID=notifications_group + +# OTEL settings +OTEL_OTLP_ENDPOINT=http://otel-collector:4317 +OTEL_OTLP_INSECURE=true +OTEL_OTLP_TIMEOUT=10s +OTEL_OTLP_HTTP=false +OTEL_RETRY_ENABLED=true +OTEL_RETRY_INITIAL_INTERVAL=5s +OTEL_RETRY_MAX_INTERVAL=30s +OTEL_RETRY_MAX_ELAPSED_TIME=1m + +# Telegram settings +TELEGRAM_BOT_TOKEN=7665408380:AAFKbUgbdQb3iX6X-29H3spv4KGG1FXs_IM +TELEGRAM_JWT_SECRET=secret +``` +### Local Deployment with Docker Compose +In the deployment/ directory, you have a docker-compose.yml file. Run file scripts/run.cmd + +This will spin up all necessary services. You can verify that the API is running by accessing: + +* API: http://localhost:8080 +* Swagger UI: http://localhost:8080/* swagger/ +* Jaeger UI: http://localhost:16686 + +## Usage +### API Endpoints +The backend exposes endpoints for: + +* Authorization via Telegram: /auth/telegram + +* Product Search (LLM-based): /llm-request + +* Product Details: /get-abstract-product and /get-parsed-products + +* Wishlist Management: /favourites/... + +Check the Swagger UI for full API documentation. + +### Swagger UI +Visit http://localhost:8080/swagger/ (or your deployment URL) to explore and test API endpoints. -Если разработка проекта замедлилась или остановилась, добавьте соответствующее уведомление РІ начало README. \ No newline at end of file