An advanced, production-ready data engineering and AI ecosystem designed to scrape, clean, store, and analyze classical and modern Arabic poetry. This project showcases a complete lifecycle—from raw web scraping to analytical dashboards and LLM-powered Retrieval-Augmented Generation (RAG).
The platform is designed to process data through five distinct architectural phases:
- Data Ingestion (Scraping): High-performance asynchronous scraping of poets, poems, and metadata from target literature archives using
scrapling. - Data Transformation & Cleaning: Structural normalization, deduplication, and handling text bidirectional issues using
pandas. - Data Warehousing (DW): Designing and loading structured data into a dedicated Data Warehouse schema optimized for analytical queries.
- AI & LLM Integration (RAG): Embedding text chunks and building a specialized semantic search and Q&A engine over Arabic poetic heritage.
- Business Intelligence (BI): Interactive Power BI dashboards tracking poetic eras, rhyme distribution, and geographical insights.
The project leverages modern Python libraries pinned to specific versions to guarantee environment stability and workflow reproducibility:
- Scraping Engine:
scrapling[fetchers] (v0.4.9)— Configured with advanced fetchers to handle dynamic web content and bypass standard anti-scraping blockers efficiently. - Data Processing:
pandas (v2.2.0)— Utilized for structuring, inspecting, and manipulating extracted datasets into unified structures. - Arabic Text Processing:
arabic-reshaper (v3.0.1)— To reshape disconnected Arabic lettering for rendering engines.python-bidi (v0.6.10)— Implements the Unicode Bidirectional Algorithm to properly display Right-to-Left (RTL) Arabic text.
- Pipeline Initialization: Successfully initialized git configuration, localized environments, and set up strict dependency management (
requirements.txt). - Scraping Pipeline (Phase 1): Built a scalable script using
scraplingto fetch detailed data for poets and poems.
- Data Cleaning Scripts: Standardizing naming conventions, filtering noise out of raw text, and dealing with diacritics (تشكيل).
- Data Warehouse (DW) Design: Setting up the star/snowflake schema for relational analytical queries.
- RAG Implementation: Chunking poetry texts and hooking them up with an LLM for conversational AI context.
- Power BI Dashboard: Building visualization reports for literary metadata analytics.
To reproduce the project environment locally, clone the repository and install the verified dependencies:
# Clone the repository
git clone [https://github.com/AsmaaEssamSultan/end-to-end-arabic-poetry-platform.git](https://github.com/AsmaaEssamSultan/end-to-end-arabic-poetry-platform.git)
# Navigate to the project directory
cd "end-to-end-arabic-poetry-platform"
# Install pinned requirements
pip install -r requirements.txt