mirror of
https://github.com/DS4SD/docling.git
synced 2025-07-31 14:34:40 +00:00
Add Streamlit Docling concierge readme file and script
This commit is contained in:
Sari Kassar
parent
b1e79cadc7
commit
6556557f50
107
streamlit-docling-concierge-readme-file.md
Normal file
107
streamlit-docling-concierge-readme-file.md
Normal file
@ -0,0 +1,107 @@
|
|||||||
|
# Docling Concierge - Streamlit Web Interface
|
||||||
|
|
||||||
|
A user-friendly web application for Docling document processing built with Streamlit. This app allows both technical and non-technical users to process documents using natural language instructions or guided selections through an intuitive interface.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Simple User Interface**: Designed specifically for non-technical users
|
||||||
|
- **Visual Controls**: Checkboxes, toggles, and dropdowns for all options
|
||||||
|
- **Flexible Input**: Upload files or provide URLs
|
||||||
|
- **Rich Output Formats**: Markdown, HTML, JSON, Text, Doctags
|
||||||
|
- **Interactive Previews**: See processed documents before downloading
|
||||||
|
- **Batch Processing**: Handle multiple documents at once
|
||||||
|
- **Smart Suggestions**: Get guidance on best options for your documents
|
||||||
|
- **Visual Progress Tracking**: Monitor document processing with progress bars
|
||||||
|
|
||||||
|
## Compatibility
|
||||||
|
|
||||||
|
This version of Docling Concierge is optimized for Docling 2.25.1 and similar versions, with the following adjustments:
|
||||||
|
- No dependencies on deprecated `document_type` attributes
|
||||||
|
- No references to Docling backend classes
|
||||||
|
- Proper language code mappings for different OCR engines
|
||||||
|
- Enhanced error handling and progress visualization
|
||||||
|
|
||||||
|
## Installation
|
||||||
|
|
||||||
|
1. Make sure you have Python 3.8+ installed on your system
|
||||||
|
2. Install Docling (if not already installed):
|
||||||
|
```bash
|
||||||
|
pip install -U docling==2.25.1
|
||||||
|
```
|
||||||
|
3. Install Streamlit and other dependencies:
|
||||||
|
```bash
|
||||||
|
pip install streamlit pandas
|
||||||
|
```
|
||||||
|
4. Download the `docling_concierge_streamlit.py` file from this repository
|
||||||
|
5. Run the Streamlit app:
|
||||||
|
```bash
|
||||||
|
streamlit run docling_concierge_streamlit.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
1. **Upload Documents**:
|
||||||
|
- Use the "Upload Files" tab to select documents from your computer
|
||||||
|
- Or use the "Enter URLs" tab to process documents from the web
|
||||||
|
|
||||||
|
2. **Describe Your Task**:
|
||||||
|
- Type what you want to do in natural language
|
||||||
|
- Example: "Convert this PDF to markdown with tables"
|
||||||
|
- The app will automatically detect intent and pre-select appropriate options
|
||||||
|
|
||||||
|
3. **Adjust Processing Options** (Optional):
|
||||||
|
- **Basic Options**: Output format, image handling
|
||||||
|
- **OCR Settings**: Enable text recognition, select language
|
||||||
|
- **Advanced Features**: Table extraction, code detection, formula processing
|
||||||
|
|
||||||
|
4. **Process Documents**:
|
||||||
|
- Click the "Process Documents" button
|
||||||
|
- Monitor progress through the status indicator and progress bar
|
||||||
|
|
||||||
|
5. **Work with Results**:
|
||||||
|
- Preview processed documents directly in the app
|
||||||
|
- Download individual results or all files as a ZIP
|
||||||
|
|
||||||
|
## Language Support for OCR
|
||||||
|
|
||||||
|
The app includes smart language mapping that automatically translates user-friendly language names to the correct codes for each OCR engine:
|
||||||
|
|
||||||
|
- **EasyOCR**: Uses codes like "en", "es", "fr", etc.
|
||||||
|
- **Tesseract**: Uses codes like "eng", "spa", "fra", etc.
|
||||||
|
- **RapidOCR**: Uses appropriate codes for this engine
|
||||||
|
|
||||||
|
Users can simply check the languages they need without knowing the specific codes required by each engine.
|
||||||
|
|
||||||
|
## Common Processing Tasks
|
||||||
|
|
||||||
|
- **Document Conversion**: PDF to Markdown, DOCX to HTML, etc.
|
||||||
|
- **Table Extraction**: Pull tables from documents into structured formats (with fast or accurate options)
|
||||||
|
- **OCR Processing**: Extract text from scanned documents and images
|
||||||
|
- **Content Analysis**: Detect and enhance code blocks, formulas, images
|
||||||
|
- **Image Handling**: Choose between embedded, referenced, or placeholder modes
|
||||||
|
|
||||||
|
## Tips for Non-Technical Users
|
||||||
|
|
||||||
|
- Use the examples in the sidebar for quick starts
|
||||||
|
- Start with default options for most documents
|
||||||
|
- Enable OCR for scanned documents or images
|
||||||
|
- Use "Markdown" format for most readable output
|
||||||
|
- Choose "HTML" for web-ready content with formatting
|
||||||
|
- Use the "Reset All Settings" button in the sidebar to start fresh
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Python 3.8+
|
||||||
|
- Docling 2.25.1 (or compatible version)
|
||||||
|
- Streamlit
|
||||||
|
- Pandas
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
This project is licensed under the same terms as Docling.
|
||||||
|
|
||||||
|
## Acknowledgements
|
||||||
|
|
||||||
|
Based on the original Docling Concierge command-line tool.
|
||||||
833
streamlit-docling-concierge.py
Normal file
833
streamlit-docling-concierge.py
Normal file
@ -0,0 +1,833 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Docling Concierge - Streamlit Web Interface (Docling 2.25.1 or similar)
|
||||||
|
Updated to remove references to `document_type`, which no longer exists.
|
||||||
|
|
||||||
|
Features:
|
||||||
|
- No docling.backends usage
|
||||||
|
- Spinner + progress bar for processing
|
||||||
|
- OCR language code mapping for Tesseract/EasyOCR
|
||||||
|
- Displays `res.input.format` instead of `document_type` in results
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional, List, Dict
|
||||||
|
|
||||||
|
import streamlit as st
|
||||||
|
import pandas as pd
|
||||||
|
|
||||||
|
# --------------------------------
|
||||||
|
# Attempt to import Docling modules
|
||||||
|
# --------------------------------
|
||||||
|
missing_modules = []
|
||||||
|
try:
|
||||||
|
import docling
|
||||||
|
except ImportError as e:
|
||||||
|
missing_modules.append(f"docling: {str(e)}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
from docling.document_converter import DocumentConverter, PdfFormatOption
|
||||||
|
except ImportError as e:
|
||||||
|
missing_modules.append(f"docling.document_converter: {str(e)}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
from docling.datamodel.base_models import InputFormat, OutputFormat
|
||||||
|
except ImportError as e:
|
||||||
|
missing_modules.append(f"docling.datamodel.base_models: {str(e)}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
from docling.datamodel.document import ConversionResult
|
||||||
|
except ImportError as e:
|
||||||
|
missing_modules.append(f"docling.datamodel.document: {str(e)}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
from docling.datamodel.pipeline_options import (
|
||||||
|
PdfPipelineOptions, OcrEngine, TableFormerMode,
|
||||||
|
EasyOcrOptions, TesseractCliOcrOptions, TesseractOcrOptions,
|
||||||
|
OcrMacOptions, RapidOcrOptions, OcrOptions,
|
||||||
|
)
|
||||||
|
except ImportError as e:
|
||||||
|
missing_modules.append(f"docling.datamodel.pipeline_options: {str(e)}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
from docling_core.types.doc import ImageRefMode
|
||||||
|
except ImportError as e:
|
||||||
|
missing_modules.append(f"docling_core.types.doc: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
class DoclingConciergeStreamlit:
|
||||||
|
"""
|
||||||
|
Streamlit interface to Docling functionality, tailored to Docling 2.25.1.
|
||||||
|
No references to `document_type` (use input.format instead).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.converter = DocumentConverter()
|
||||||
|
self.temp_dir = Path(tempfile.mkdtemp(prefix="docling_"))
|
||||||
|
|
||||||
|
# Default pipeline options
|
||||||
|
self.pipeline_options = PdfPipelineOptions(
|
||||||
|
do_ocr=True,
|
||||||
|
do_table_structure=True,
|
||||||
|
do_code_enrichment=False,
|
||||||
|
do_formula_enrichment=False,
|
||||||
|
do_picture_classification=False,
|
||||||
|
do_picture_description=False,
|
||||||
|
enable_remote_services=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Simple mappings for input detection
|
||||||
|
self.format_aliases = {
|
||||||
|
"pdf": InputFormat.PDF,
|
||||||
|
"image": InputFormat.IMAGE,
|
||||||
|
"docx": InputFormat.DOCX,
|
||||||
|
"word": InputFormat.DOCX,
|
||||||
|
"excel": InputFormat.XLSX,
|
||||||
|
"xlsx": InputFormat.XLSX,
|
||||||
|
"pptx": InputFormat.PPTX,
|
||||||
|
"powerpoint": InputFormat.PPTX,
|
||||||
|
"html": InputFormat.HTML,
|
||||||
|
"markdown": InputFormat.MD,
|
||||||
|
"md": InputFormat.MD,
|
||||||
|
"asciidoc": InputFormat.ASCIIDOC,
|
||||||
|
"csv": InputFormat.CSV,
|
||||||
|
"xml": InputFormat.XML_JATS,
|
||||||
|
"json": InputFormat.JSON_DOCLING,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Mapping output format strings to docling's OutputFormat
|
||||||
|
self.output_format_aliases = {
|
||||||
|
"markdown": OutputFormat.MARKDOWN,
|
||||||
|
"md": OutputFormat.MARKDOWN,
|
||||||
|
"html": OutputFormat.HTML,
|
||||||
|
"json": OutputFormat.JSON,
|
||||||
|
"text": OutputFormat.TEXT,
|
||||||
|
"txt": OutputFormat.TEXT,
|
||||||
|
"doctags": OutputFormat.DOCTAGS,
|
||||||
|
}
|
||||||
|
|
||||||
|
# OCR engine choices for the UI
|
||||||
|
self.ocr_engines = {
|
||||||
|
"EasyOCR (recommended)": OcrEngine.EASYOCR,
|
||||||
|
"Tesseract": OcrEngine.TESSERACT,
|
||||||
|
"Tesseract CLI": OcrEngine.TESSERACT_CLI,
|
||||||
|
"macOS OCR (Apple devices only)": OcrEngine.OCRMAC,
|
||||||
|
"RapidOCR": OcrEngine.RAPIDOCR,
|
||||||
|
}
|
||||||
|
|
||||||
|
# ImageRefMode choices
|
||||||
|
self.image_modes = {
|
||||||
|
"Embedded (include images in file)": ImageRefMode.EMBEDDED,
|
||||||
|
"Referenced (link to separate image files)": ImageRefMode.REFERENCED,
|
||||||
|
"Placeholder (text only, no images)": ImageRefMode.PLACEHOLDER,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Common language list for user-friendly display
|
||||||
|
self.common_languages = [
|
||||||
|
"English", "Spanish", "French", "German", "Chinese",
|
||||||
|
"Japanese", "Korean", "Russian", "Arabic", "Hindi",
|
||||||
|
"Portuguese", "Italian"
|
||||||
|
]
|
||||||
|
|
||||||
|
# -----------
|
||||||
|
# Define OCR language code mappings
|
||||||
|
# -----------
|
||||||
|
self.lang_code_map_tesseract = {
|
||||||
|
"English": "eng",
|
||||||
|
"Spanish": "spa",
|
||||||
|
"French": "fra",
|
||||||
|
"German": "deu",
|
||||||
|
"Chinese": "chi_sim", # Tesseract can use chi_tra for traditional
|
||||||
|
"Japanese": "jpn",
|
||||||
|
"Korean": "kor",
|
||||||
|
"Russian": "rus",
|
||||||
|
"Arabic": "ara",
|
||||||
|
"Hindi": "hin",
|
||||||
|
"Portuguese": "por",
|
||||||
|
"Italian": "ita",
|
||||||
|
}
|
||||||
|
|
||||||
|
# EasyOCR language codes
|
||||||
|
self.lang_code_map_easyocr = {
|
||||||
|
"English": "en",
|
||||||
|
"Spanish": "es",
|
||||||
|
"French": "fr",
|
||||||
|
"German": "de",
|
||||||
|
"Chinese": "ch_tra", # or ch_sim
|
||||||
|
"Japanese": "ja",
|
||||||
|
"Korean": "ko",
|
||||||
|
"Russian": "ru",
|
||||||
|
"Arabic": "ar",
|
||||||
|
"Hindi": "hi",
|
||||||
|
"Portuguese": "pt",
|
||||||
|
"Italian": "it",
|
||||||
|
}
|
||||||
|
|
||||||
|
# RapidOCR might follow a similar convention to EasyOCR
|
||||||
|
self.lang_code_map_rapidocr = {
|
||||||
|
"English": "en",
|
||||||
|
"Spanish": "es",
|
||||||
|
"French": "fr",
|
||||||
|
"German": "de",
|
||||||
|
"Chinese": "ch",
|
||||||
|
"Japanese": "ja",
|
||||||
|
"Korean": "ko",
|
||||||
|
"Russian": "ru",
|
||||||
|
"Arabic": "ar",
|
||||||
|
"Hindi": "hi",
|
||||||
|
"Portuguese": "pt",
|
||||||
|
"Italian": "it",
|
||||||
|
}
|
||||||
|
|
||||||
|
def configure_streamlit_page(self):
|
||||||
|
"""Set up the Streamlit page layout and style."""
|
||||||
|
st.set_page_config(
|
||||||
|
page_title="Docling Concierge",
|
||||||
|
page_icon="📄",
|
||||||
|
layout="wide",
|
||||||
|
initial_sidebar_state="expanded",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Custom CSS for styling
|
||||||
|
st.markdown("""
|
||||||
|
<style>
|
||||||
|
.main-header {
|
||||||
|
font-size: 2.5rem;
|
||||||
|
color: #1E88E5;
|
||||||
|
}
|
||||||
|
.sub-header {
|
||||||
|
font-size: 1.5rem;
|
||||||
|
color: #424242;
|
||||||
|
}
|
||||||
|
.card {
|
||||||
|
border-radius: 5px;
|
||||||
|
padding: 20px;
|
||||||
|
margin-bottom: 20px;
|
||||||
|
background-color: #f8f9fa;
|
||||||
|
border-left: 4px solid #1E88E5;
|
||||||
|
}
|
||||||
|
.feature-title {
|
||||||
|
font-weight: bold;
|
||||||
|
color: #1E88E5;
|
||||||
|
}
|
||||||
|
.hint {
|
||||||
|
font-size: 0.9rem;
|
||||||
|
color: #757575;
|
||||||
|
font-style: italic;
|
||||||
|
}
|
||||||
|
.success {
|
||||||
|
padding: 10px;
|
||||||
|
border-radius: 5px;
|
||||||
|
background-color: #E8F5E9;
|
||||||
|
border-left: 4px solid #4CAF50;
|
||||||
|
}
|
||||||
|
.error {
|
||||||
|
padding: 10px;
|
||||||
|
border-radius: 5px;
|
||||||
|
background-color: #FFEBEE;
|
||||||
|
border-left: 4px solid #F44336;
|
||||||
|
}
|
||||||
|
.stButton>button {
|
||||||
|
width: 100%;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
|
""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
def show_welcome(self):
|
||||||
|
"""Welcome header and intro text."""
|
||||||
|
st.markdown('<p class="main-header">📄 Docling Concierge</p>', unsafe_allow_html=True)
|
||||||
|
st.markdown('<p class="sub-header">Transform and extract content from your documents with ease</p>',
|
||||||
|
unsafe_allow_html=True)
|
||||||
|
|
||||||
|
with st.expander("ℹ️ About this app", expanded=False):
|
||||||
|
st.markdown("""
|
||||||
|
**Docling Concierge** helps you process documents in various formats. You can:
|
||||||
|
|
||||||
|
- **Convert between formats**: PDF to Markdown, Word to HTML, and more
|
||||||
|
- **Extract content**: Tables, images, code snippets, and formulas
|
||||||
|
- **Process text**: Apply OCR to scanned documents
|
||||||
|
- **Analyze content**: Enhance with image descriptions and classifications
|
||||||
|
|
||||||
|
Simply upload your documents or provide URLs, select the options you need,
|
||||||
|
and let Docling handle the rest.
|
||||||
|
""")
|
||||||
|
|
||||||
|
def show_sidebar(self):
|
||||||
|
"""Sidebar with quick examples and info."""
|
||||||
|
with st.sidebar:
|
||||||
|
st.markdown("### 📚 Quick Examples")
|
||||||
|
st.markdown("Try these examples (click to fill):")
|
||||||
|
|
||||||
|
example_queries = [
|
||||||
|
"Convert PDF to Markdown with tables",
|
||||||
|
"Extract tables from document and save as HTML",
|
||||||
|
"Process scanned PDF with OCR",
|
||||||
|
"Convert document to text only (no images)",
|
||||||
|
"Extract code blocks from programming PDF"
|
||||||
|
]
|
||||||
|
|
||||||
|
for query in example_queries:
|
||||||
|
if st.button(query, key=f"example_{query}"):
|
||||||
|
st.session_state.query = query
|
||||||
|
|
||||||
|
st.markdown("---")
|
||||||
|
st.markdown("### 🔍 Document Types")
|
||||||
|
st.markdown("""
|
||||||
|
- 📄 **Documents**: PDF, DOCX, PPTX
|
||||||
|
- 📊 **Data**: XLSX, CSV
|
||||||
|
- 🌐 **Web**: HTML
|
||||||
|
- 📝 **Text**: Markdown, Text
|
||||||
|
- 🖼️ **Images**: JPG, PNG
|
||||||
|
""")
|
||||||
|
|
||||||
|
st.markdown("---")
|
||||||
|
st.markdown("### 🎯 Output Formats")
|
||||||
|
st.markdown("""
|
||||||
|
- **Markdown**: Perfect for documentation
|
||||||
|
- **HTML**: For web publishing
|
||||||
|
- **JSON**: For data processing
|
||||||
|
- **Text**: Plain text extraction
|
||||||
|
- **Doctags**: Docling token format
|
||||||
|
""")
|
||||||
|
|
||||||
|
st.markdown("---")
|
||||||
|
st.markdown("### 💡 Tips")
|
||||||
|
st.markdown("""
|
||||||
|
- Use OCR for scanned documents
|
||||||
|
- Enable table extraction for spreadsheet-like content
|
||||||
|
- Choose image mode based on your needs
|
||||||
|
- Batch process multiple files at once
|
||||||
|
""")
|
||||||
|
|
||||||
|
st.markdown("---")
|
||||||
|
st.markdown("Made with ❤️ using Docling")
|
||||||
|
|
||||||
|
# Reset button if you want to clear all state
|
||||||
|
if st.button("🔄 Reset All Settings"):
|
||||||
|
for key in list(st.session_state.keys()):
|
||||||
|
if key not in ["authenticated", "username"]:
|
||||||
|
del st.session_state[key]
|
||||||
|
st.rerun()
|
||||||
|
|
||||||
|
def show_input_section(self):
|
||||||
|
"""Section to upload files or enter URLs."""
|
||||||
|
st.markdown('<p class="feature-title">📥 Input Documents</p>', unsafe_allow_html=True)
|
||||||
|
|
||||||
|
input_tab1, input_tab2 = st.tabs(["Upload Files", "Enter URLs"])
|
||||||
|
|
||||||
|
with input_tab1:
|
||||||
|
st.markdown("""<p class="hint">
|
||||||
|
Upload one or more documents to process.
|
||||||
|
Supports PDF, Word, Excel, PowerPoint, Images, and more.
|
||||||
|
</p>""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
uploaded_files = st.file_uploader(
|
||||||
|
"Upload documents",
|
||||||
|
accept_multiple_files=True,
|
||||||
|
type=[
|
||||||
|
"pdf", "docx", "xlsx", "pptx", "jpg", "jpeg", "png",
|
||||||
|
"html", "md", "csv", "xml", "json", "txt"
|
||||||
|
],
|
||||||
|
help="Select one or more files from your computer"
|
||||||
|
)
|
||||||
|
|
||||||
|
if uploaded_files:
|
||||||
|
st.session_state.file_paths = []
|
||||||
|
for uploaded_file in uploaded_files:
|
||||||
|
temp_file_path = self.temp_dir / uploaded_file.name
|
||||||
|
with open(temp_file_path, "wb") as f:
|
||||||
|
f.write(uploaded_file.getbuffer())
|
||||||
|
st.session_state.file_paths.append(temp_file_path)
|
||||||
|
|
||||||
|
# Show info about uploaded files
|
||||||
|
files_df = pd.DataFrame({
|
||||||
|
"File Name": [file.name for file in uploaded_files],
|
||||||
|
"Size": [f"{file.size / 1024:.1f} KB" for file in uploaded_files],
|
||||||
|
"Type": [file.type for file in uploaded_files]
|
||||||
|
})
|
||||||
|
st.dataframe(files_df, use_container_width=True)
|
||||||
|
|
||||||
|
with input_tab2:
|
||||||
|
st.markdown("""<p class="hint">
|
||||||
|
Enter one or more URLs to process (one per line).
|
||||||
|
</p>""", unsafe_allow_html=True)
|
||||||
|
|
||||||
|
url_input = st.text_area(
|
||||||
|
"Enter document URLs",
|
||||||
|
placeholder="https://example.com/document.pdf\nhttps://another-site.org/paper.pdf",
|
||||||
|
)
|
||||||
|
|
||||||
|
if url_input:
|
||||||
|
urls = [url.strip() for url in url_input.split("\n") if url.strip().startswith("http")]
|
||||||
|
if urls:
|
||||||
|
st.session_state.urls = urls
|
||||||
|
# Show data frame of URLs
|
||||||
|
urls_df = pd.DataFrame({
|
||||||
|
"URL": urls,
|
||||||
|
"Type": ["Web URL" for _ in urls]
|
||||||
|
})
|
||||||
|
st.dataframe(urls_df, use_container_width=True)
|
||||||
|
else:
|
||||||
|
st.warning("No valid URLs found. URLs must start with http:// or https://")
|
||||||
|
|
||||||
|
def show_natural_language_query(self):
|
||||||
|
"""User can type a plain-English request for what to do."""
|
||||||
|
st.markdown('<p class="feature-title">🔍 What do you want to do?</p>', unsafe_allow_html=True)
|
||||||
|
|
||||||
|
query = st.text_input(
|
||||||
|
"Describe what you want to do with your documents",
|
||||||
|
value=st.session_state.get("query", ""),
|
||||||
|
placeholder="Example: Convert this PDF to markdown with tables"
|
||||||
|
)
|
||||||
|
|
||||||
|
if query:
|
||||||
|
st.session_state.query = query
|
||||||
|
# Try to guess an output format from the user's text
|
||||||
|
output_format = self._extract_output_format_from_query(query)
|
||||||
|
if output_format:
|
||||||
|
for alias, format_value in self.output_format_aliases.items():
|
||||||
|
if format_value == output_format:
|
||||||
|
st.session_state.output_format = alias.capitalize()
|
||||||
|
break
|
||||||
|
|
||||||
|
def show_visual_options(self):
|
||||||
|
"""Checkboxes and advanced features for the user."""
|
||||||
|
st.markdown('<p class="feature-title">⚙️ Processing Options</p>', unsafe_allow_html=True)
|
||||||
|
|
||||||
|
tab1, tab2, tab3 = st.tabs(["Basic Options", "OCR Settings", "Advanced Features"])
|
||||||
|
|
||||||
|
# ----------
|
||||||
|
# Basic Options
|
||||||
|
# ----------
|
||||||
|
with tab1:
|
||||||
|
output_formats = ["Markdown", "HTML", "JSON", "Text", "Doctags"]
|
||||||
|
selected_format = st.selectbox(
|
||||||
|
"Output Format",
|
||||||
|
options=output_formats,
|
||||||
|
index=output_formats.index(st.session_state.get("output_format", "Markdown")),
|
||||||
|
help="Select the format you want to convert your documents to"
|
||||||
|
)
|
||||||
|
st.session_state.output_format = selected_format
|
||||||
|
|
||||||
|
image_mode_options = list(self.image_modes.keys())
|
||||||
|
selected_image_mode = st.selectbox(
|
||||||
|
"Image Handling",
|
||||||
|
options=image_mode_options,
|
||||||
|
index=0,
|
||||||
|
help="Choose how images should be handled in the output"
|
||||||
|
)
|
||||||
|
st.session_state.image_mode = self.image_modes[selected_image_mode]
|
||||||
|
|
||||||
|
# ----------
|
||||||
|
# OCR Settings
|
||||||
|
# ----------
|
||||||
|
with tab2:
|
||||||
|
ocr_enabled = st.checkbox(
|
||||||
|
"Enable OCR (Text Recognition)",
|
||||||
|
value=True,
|
||||||
|
help="Apply OCR for scanned documents or images"
|
||||||
|
)
|
||||||
|
st.session_state.ocr_enabled = ocr_enabled
|
||||||
|
|
||||||
|
if ocr_enabled:
|
||||||
|
ocr_engine_options = list(self.ocr_engines.keys())
|
||||||
|
selected_ocr_engine = st.selectbox(
|
||||||
|
"OCR Engine",
|
||||||
|
options=ocr_engine_options,
|
||||||
|
index=0,
|
||||||
|
help="Select the OCR engine to use"
|
||||||
|
)
|
||||||
|
st.session_state.ocr_engine = self.ocr_engines[selected_ocr_engine]
|
||||||
|
|
||||||
|
force_ocr = st.checkbox(
|
||||||
|
"Force Full-Page OCR",
|
||||||
|
value=False,
|
||||||
|
help="OCR entire pages even if they appear to have machine-readable text"
|
||||||
|
)
|
||||||
|
st.session_state.force_ocr = force_ocr
|
||||||
|
|
||||||
|
# Expandable area for choosing languages
|
||||||
|
with st.expander("OCR Languages", expanded=False):
|
||||||
|
selected_languages = []
|
||||||
|
|
||||||
|
# Determine which language map we should use
|
||||||
|
if st.session_state.ocr_engine == OcrEngine.EASYOCR:
|
||||||
|
lang_map = self.lang_code_map_easyocr
|
||||||
|
elif st.session_state.ocr_engine == OcrEngine.RAPIDOCR:
|
||||||
|
lang_map = self.lang_code_map_rapidocr
|
||||||
|
else:
|
||||||
|
# Tesseract, Tesseract CLI, OCRMAC fallback
|
||||||
|
lang_map = self.lang_code_map_tesseract
|
||||||
|
|
||||||
|
col1, col2 = st.columns(2)
|
||||||
|
half = len(self.common_languages) // 2
|
||||||
|
|
||||||
|
with col1:
|
||||||
|
for lang in self.common_languages[:half]:
|
||||||
|
# If user checks the box, we map e.g. "English" to "eng" or "en"
|
||||||
|
if st.checkbox(lang, value=(lang == "English")):
|
||||||
|
code = lang_map.get(lang, "eng") # fallback to "eng"
|
||||||
|
selected_languages.append(code)
|
||||||
|
|
||||||
|
with col2:
|
||||||
|
for lang in self.common_languages[half:]:
|
||||||
|
if st.checkbox(lang):
|
||||||
|
code = lang_map.get(lang, "eng")
|
||||||
|
selected_languages.append(code)
|
||||||
|
|
||||||
|
custom_lang = st.text_input(
|
||||||
|
"Add other language codes (comma-separated)",
|
||||||
|
placeholder="e.g., eng, spa, fra"
|
||||||
|
)
|
||||||
|
if custom_lang:
|
||||||
|
custom_langs = [x.strip().lower() for x in custom_lang.split(",")]
|
||||||
|
selected_languages.extend(custom_langs)
|
||||||
|
|
||||||
|
if not selected_languages:
|
||||||
|
# fallback if user didn't check anything
|
||||||
|
selected_languages = ["eng"]
|
||||||
|
|
||||||
|
st.session_state.ocr_langs = selected_languages
|
||||||
|
|
||||||
|
# ----------
|
||||||
|
# Advanced Features
|
||||||
|
# ----------
|
||||||
|
with tab3:
|
||||||
|
st.markdown("#### Content Extraction")
|
||||||
|
col1, col2 = st.columns(2)
|
||||||
|
|
||||||
|
with col1:
|
||||||
|
table_extraction = st.checkbox(
|
||||||
|
"Extract Tables",
|
||||||
|
value=True,
|
||||||
|
help="Recognize and extract tables from documents"
|
||||||
|
)
|
||||||
|
st.session_state.do_table_structure = table_extraction
|
||||||
|
if table_extraction:
|
||||||
|
table_mode = st.radio(
|
||||||
|
"Table Extraction Mode",
|
||||||
|
options=["Fast", "Accurate"],
|
||||||
|
index=0,
|
||||||
|
horizontal=True,
|
||||||
|
help="Fast is quick but less precise, Accurate is slower but more reliable"
|
||||||
|
)
|
||||||
|
st.session_state.table_mode = (
|
||||||
|
TableFormerMode.FAST if table_mode == "Fast" else TableFormerMode.ACCURATE
|
||||||
|
)
|
||||||
|
|
||||||
|
code_extraction = st.checkbox(
|
||||||
|
"Extract Code Blocks",
|
||||||
|
value=False,
|
||||||
|
help="Identify and extract programming code blocks"
|
||||||
|
)
|
||||||
|
st.session_state.do_code_enrichment = code_extraction
|
||||||
|
|
||||||
|
with col2:
|
||||||
|
formula_extraction = st.checkbox(
|
||||||
|
"Extract Mathematical Formulas",
|
||||||
|
value=False,
|
||||||
|
help="Detect mathematical equations and formulas"
|
||||||
|
)
|
||||||
|
st.session_state.do_formula_enrichment = formula_extraction
|
||||||
|
|
||||||
|
image_classification = st.checkbox(
|
||||||
|
"Classify Images",
|
||||||
|
value=False,
|
||||||
|
help="Add classifications to images in the document"
|
||||||
|
)
|
||||||
|
st.session_state.do_picture_classification = image_classification
|
||||||
|
|
||||||
|
image_description = st.checkbox(
|
||||||
|
"Describe Images",
|
||||||
|
value=False,
|
||||||
|
help="Generate descriptive text for images (may require remote services)"
|
||||||
|
)
|
||||||
|
st.session_state.do_picture_description = image_description
|
||||||
|
|
||||||
|
if image_description:
|
||||||
|
st.session_state.enable_remote_services = True
|
||||||
|
|
||||||
|
def process_documents(self):
|
||||||
|
"""Process documents based on user settings, with progress bar."""
|
||||||
|
has_files = hasattr(st.session_state, 'file_paths') and st.session_state.file_paths
|
||||||
|
has_urls = hasattr(st.session_state, 'urls') and st.session_state.urls
|
||||||
|
|
||||||
|
if not (has_files or has_urls):
|
||||||
|
st.warning("Please upload at least one document or provide a URL.")
|
||||||
|
return
|
||||||
|
|
||||||
|
# Gather sources
|
||||||
|
sources = []
|
||||||
|
if has_files:
|
||||||
|
sources.extend(st.session_state.file_paths)
|
||||||
|
if has_urls:
|
||||||
|
sources.extend(st.session_state.urls)
|
||||||
|
|
||||||
|
# Determine output format
|
||||||
|
output_format_str = st.session_state.get("output_format", "Markdown").lower()
|
||||||
|
output_format = self.output_format_aliases.get(output_format_str, OutputFormat.MARKDOWN)
|
||||||
|
|
||||||
|
# OCR settings
|
||||||
|
ocr_enabled = st.session_state.get("ocr_enabled", True)
|
||||||
|
ocr_engine = st.session_state.get("ocr_engine", OcrEngine.EASYOCR)
|
||||||
|
force_ocr = st.session_state.get("force_ocr", False)
|
||||||
|
ocr_langs = st.session_state.get("ocr_langs", ["eng"])
|
||||||
|
|
||||||
|
# Image handling
|
||||||
|
image_mode = st.session_state.get("image_mode", ImageRefMode.EMBEDDED)
|
||||||
|
|
||||||
|
# Configure pipeline options
|
||||||
|
self.pipeline_options.do_ocr = ocr_enabled
|
||||||
|
self.pipeline_options.do_table_structure = st.session_state.get("do_table_structure", True)
|
||||||
|
self.pipeline_options.do_code_enrichment = st.session_state.get("do_code_enrichment", False)
|
||||||
|
self.pipeline_options.do_formula_enrichment = st.session_state.get("do_formula_enrichment", False)
|
||||||
|
self.pipeline_options.do_picture_classification = st.session_state.get("do_picture_classification", False)
|
||||||
|
self.pipeline_options.do_picture_description = st.session_state.get("do_picture_description", False)
|
||||||
|
self.pipeline_options.enable_remote_services = st.session_state.get("enable_remote_services", False)
|
||||||
|
|
||||||
|
# If table extraction is on, set the mode
|
||||||
|
if st.session_state.get("do_table_structure", True):
|
||||||
|
mode = st.session_state.get("table_mode", TableFormerMode.FAST)
|
||||||
|
self.pipeline_options.table_structure_options.mode = mode
|
||||||
|
|
||||||
|
# Create the OCR options object
|
||||||
|
if ocr_enabled:
|
||||||
|
if ocr_engine == OcrEngine.EASYOCR:
|
||||||
|
ocr_options = EasyOcrOptions(force_full_page_ocr=force_ocr)
|
||||||
|
elif ocr_engine == OcrEngine.TESSERACT_CLI:
|
||||||
|
ocr_options = TesseractCliOcrOptions(force_full_page_ocr=force_ocr)
|
||||||
|
elif ocr_engine == OcrEngine.TESSERACT:
|
||||||
|
ocr_options = TesseractOcrOptions(force_full_page_ocr=force_ocr)
|
||||||
|
elif ocr_engine == OcrEngine.OCRMAC:
|
||||||
|
ocr_options = OcrMacOptions(force_full_page_ocr=force_ocr)
|
||||||
|
elif ocr_engine == OcrEngine.RAPIDOCR:
|
||||||
|
ocr_options = RapidOcrOptions(force_full_page_ocr=force_ocr)
|
||||||
|
else:
|
||||||
|
ocr_options = EasyOcrOptions(force_full_page_ocr=force_ocr)
|
||||||
|
|
||||||
|
# Assign the languages (e.g. ["eng", "spa"])
|
||||||
|
ocr_options.lang = ocr_langs
|
||||||
|
self.pipeline_options.ocr_options = ocr_options
|
||||||
|
|
||||||
|
# Create PdfFormatOption with no custom backend
|
||||||
|
pdf_format_option = PdfFormatOption(
|
||||||
|
pipeline_options=self.pipeline_options
|
||||||
|
)
|
||||||
|
|
||||||
|
format_options = {
|
||||||
|
InputFormat.PDF: pdf_format_option,
|
||||||
|
InputFormat.IMAGE: pdf_format_option,
|
||||||
|
}
|
||||||
|
|
||||||
|
converter = DocumentConverter(format_options=format_options)
|
||||||
|
|
||||||
|
results = []
|
||||||
|
output_files = []
|
||||||
|
|
||||||
|
# Show a spinner + progress bar for each file
|
||||||
|
with st.spinner("Processing documents..."):
|
||||||
|
progress_bar = st.progress(0)
|
||||||
|
total_sources = len(sources)
|
||||||
|
|
||||||
|
for i, source in enumerate(sources):
|
||||||
|
short_name = os.path.basename(str(source))
|
||||||
|
st.info(f"Processing {i+1}/{total_sources}: {short_name}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = converter.convert(source)
|
||||||
|
results.append(result)
|
||||||
|
|
||||||
|
if result.status.value == "success":
|
||||||
|
base_filename = os.path.basename(str(result.input.file))
|
||||||
|
base_name = os.path.splitext(base_filename)[0]
|
||||||
|
|
||||||
|
# Save based on the chosen output format
|
||||||
|
if output_format == OutputFormat.MARKDOWN:
|
||||||
|
out_file = self.temp_dir / f"{base_name}.md"
|
||||||
|
result.document.save_as_markdown(out_file, image_mode=image_mode)
|
||||||
|
|
||||||
|
elif output_format == OutputFormat.HTML:
|
||||||
|
out_file = self.temp_dir / f"{base_name}.html"
|
||||||
|
result.document.save_as_html(out_file, image_mode=image_mode)
|
||||||
|
|
||||||
|
elif output_format == OutputFormat.JSON:
|
||||||
|
out_file = self.temp_dir / f"{base_name}.json"
|
||||||
|
result.document.save_as_json(out_file, image_mode=image_mode)
|
||||||
|
|
||||||
|
elif output_format == OutputFormat.TEXT:
|
||||||
|
out_file = self.temp_dir / f"{base_name}.txt"
|
||||||
|
result.document.save_as_markdown(
|
||||||
|
out_file, strict_text=True, image_mode=ImageRefMode.PLACEHOLDER
|
||||||
|
)
|
||||||
|
|
||||||
|
elif output_format == OutputFormat.DOCTAGS:
|
||||||
|
out_file = self.temp_dir / f"{base_name}.doctags"
|
||||||
|
result.document.save_as_document_tokens(out_file)
|
||||||
|
|
||||||
|
output_files.append(out_file)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
st.error(f"Error processing {short_name}: {str(e)}")
|
||||||
|
|
||||||
|
# Update the progress bar
|
||||||
|
progress_percent = int((i + 1) / total_sources * 100)
|
||||||
|
progress_bar.progress(progress_percent)
|
||||||
|
|
||||||
|
st.success("Processing complete!")
|
||||||
|
|
||||||
|
st.session_state.results = results
|
||||||
|
st.session_state.output_files = output_files
|
||||||
|
return results, output_files
|
||||||
|
|
||||||
|
def show_results(self, results, output_files):
|
||||||
|
"""Display the final results. Replaced `document_type` with `res.input.format`."""
|
||||||
|
if not results:
|
||||||
|
return
|
||||||
|
|
||||||
|
st.markdown('<p class="feature-title">📝 Results</p>', unsafe_allow_html=True)
|
||||||
|
|
||||||
|
col1, col2 = st.columns([2, 1])
|
||||||
|
|
||||||
|
with col1:
|
||||||
|
success_count = sum(r.status.value == "success" for r in results)
|
||||||
|
fail_count = len(results) - success_count
|
||||||
|
|
||||||
|
if success_count > 0:
|
||||||
|
st.markdown(f"<div class='success'>✅ Successfully processed {success_count} document(s)</div>",
|
||||||
|
unsafe_allow_html=True)
|
||||||
|
if fail_count > 0:
|
||||||
|
st.markdown(f"<div class='error'>❌ Failed to process {fail_count} document(s)</div>",
|
||||||
|
unsafe_allow_html=True)
|
||||||
|
|
||||||
|
with col2:
|
||||||
|
if output_files:
|
||||||
|
import zipfile
|
||||||
|
import io
|
||||||
|
|
||||||
|
zip_buffer = io.BytesIO()
|
||||||
|
with zipfile.ZipFile(zip_buffer, "w", zipfile.ZIP_DEFLATED) as zf:
|
||||||
|
for fp in output_files:
|
||||||
|
zf.write(fp, arcname=fp.name)
|
||||||
|
|
||||||
|
zip_buffer.seek(0)
|
||||||
|
|
||||||
|
st.download_button(
|
||||||
|
"📦 Download All Results",
|
||||||
|
data=zip_buffer,
|
||||||
|
file_name="docling_results.zip",
|
||||||
|
mime="application/zip",
|
||||||
|
use_container_width=True
|
||||||
|
)
|
||||||
|
|
||||||
|
if output_files:
|
||||||
|
st.markdown("### Processed Documents")
|
||||||
|
for i, (res, out_fp) in enumerate(zip(results, output_files), start=1):
|
||||||
|
if res.status.value != "success":
|
||||||
|
continue
|
||||||
|
|
||||||
|
with st.expander(f"Document {i}: {out_fp.name}"):
|
||||||
|
st.markdown(f"**Source**: {res.input.file}")
|
||||||
|
|
||||||
|
# Replaced: st.markdown(f"**Type**: {res.document.document_type}")
|
||||||
|
# With: show the format from res.input
|
||||||
|
st.markdown(f"**Format**: {res.input.format}")
|
||||||
|
|
||||||
|
# If pages are present
|
||||||
|
st.markdown(f"**Pages**: {len(res.document.pages)}")
|
||||||
|
|
||||||
|
preview_tab, download_tab = st.tabs(["Preview", "Download"])
|
||||||
|
|
||||||
|
with preview_tab:
|
||||||
|
try:
|
||||||
|
with open(out_fp, "r", encoding="utf-8") as f:
|
||||||
|
content = f.read()
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
content = f"<Cannot preview file {out_fp.suffix}>"
|
||||||
|
|
||||||
|
ext = out_fp.suffix.lower()
|
||||||
|
if ext == ".md":
|
||||||
|
st.markdown(content)
|
||||||
|
elif ext == ".html":
|
||||||
|
st.components.v1.html(content, height=400)
|
||||||
|
elif ext == ".json":
|
||||||
|
st.json(content)
|
||||||
|
else:
|
||||||
|
st.text(content)
|
||||||
|
|
||||||
|
with download_tab:
|
||||||
|
with open(out_fp, "rb") as f:
|
||||||
|
file_data = f.read()
|
||||||
|
st.download_button(
|
||||||
|
f"Download {out_fp.name}",
|
||||||
|
data=file_data,
|
||||||
|
file_name=out_fp.name,
|
||||||
|
mime="application/octet-stream",
|
||||||
|
use_container_width=True
|
||||||
|
)
|
||||||
|
|
||||||
|
def _extract_output_format_from_query(self, query: str) -> Optional[OutputFormat]:
|
||||||
|
"""Try to guess an OutputFormat from user text like 'Convert to Markdown'."""
|
||||||
|
to_format_match = re.search(r'to\s+(\w+)', query, re.IGNORECASE)
|
||||||
|
if to_format_match:
|
||||||
|
fmt_str = to_format_match.group(1).lower()
|
||||||
|
if fmt_str in self.output_format_aliases:
|
||||||
|
return self.output_format_aliases[fmt_str]
|
||||||
|
|
||||||
|
# Check any direct mention (like 'markdown', 'html', etc.)
|
||||||
|
for format_name, format_value in self.output_format_aliases.items():
|
||||||
|
if format_name in query.lower():
|
||||||
|
return format_value
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
"""Run the entire Streamlit app."""
|
||||||
|
if missing_modules:
|
||||||
|
st.error("Error: Some Docling modules could not be imported:")
|
||||||
|
for module in missing_modules:
|
||||||
|
st.write(f"- {module}")
|
||||||
|
|
||||||
|
st.info("""
|
||||||
|
Possible solutions:
|
||||||
|
1. Make sure you have the correct version of Docling installed:
|
||||||
|
```
|
||||||
|
pip install -U docling
|
||||||
|
```
|
||||||
|
2. If you installed from source or development version, make sure all dependencies are installed:
|
||||||
|
```
|
||||||
|
pip install -e .[dev]
|
||||||
|
```
|
||||||
|
3. Check your PYTHONPATH if using a development environment
|
||||||
|
""")
|
||||||
|
return
|
||||||
|
|
||||||
|
self.configure_streamlit_page()
|
||||||
|
self.show_welcome()
|
||||||
|
self.show_sidebar()
|
||||||
|
self.show_input_section()
|
||||||
|
self.show_natural_language_query()
|
||||||
|
self.show_visual_options()
|
||||||
|
|
||||||
|
if st.button("🔄 Process Documents", type="primary", use_container_width=True):
|
||||||
|
with st.spinner("Processing documents..."):
|
||||||
|
results, output_files = self.process_documents()
|
||||||
|
if results:
|
||||||
|
self.show_results(results, output_files)
|
||||||
|
|
||||||
|
# If results already exist in session (e.g. user pressed 'Process' before),
|
||||||
|
# show them again
|
||||||
|
if hasattr(st.session_state, 'results') and hasattr(st.session_state, 'output_files'):
|
||||||
|
self.show_results(st.session_state.results, st.session_state.output_files)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Entry point for Streamlit."""
|
||||||
|
app = DoclingConciergeStreamlit()
|
||||||
|
app.run()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Loading…
Reference in New Issue
Block a user