Creating Documents#
This guide explains how to create and configure Document instances to process textual and visual content for analysis.
Documents serve as the container for the content from which information (aspects and concepts) can be extracted.
⚙️ Configuration Parameters#
The minimum configuration for a document requires either raw_text, paragraphs, or images:
from pathlib import Path
from contextgem import Document, Paragraph, create_image
# Create a document with raw text content
contract_document = Document(
raw_text=(
"...This agreement is effective as of January 1, 2025.\n\n"
"All parties must comply with the terms outlined herein. The terms include "
"monthly reporting requirements and quarterly performance reviews.\n\n"
"Failure to adhere to these terms may result in termination of the agreement. "
"Additionally, any breach of confidentiality will be subject to penalties as "
"described in this agreement.\n\n"
"This agreement shall remain in force for a period of three (3) years unless "
"otherwise terminated according to the provisions stated above..."
),
paragraph_segmentation_mode="newlines", # Default mode, splits on newlines
)
# Create a document with more advanced paragraph segmentation using a SaT model
report_document = Document(
raw_text=(
"Executive Summary "
"This report outlines our quarterly performance. "
"Revenue increased by [15%] compared to the previous quarter.\n\n"
"Customer satisfaction metrics show positive trends across all regions..."
),
paragraph_segmentation_mode="sat", # Use SaT model for intelligent paragraph segmentation
sat_model_id="sat-3l-sm", # Specify which SaT model to use
)
# Create a document with predefined paragraphs, e.g. when you use a custom
# paragraph segmentation tool
document_from_paragraphs = Document(
paragraphs=[
Paragraph(raw_text="This is the first paragraph."),
Paragraph(raw_text="This is the second paragraph with more content."),
Paragraph(raw_text="Final paragraph concluding the document."),
# ...
]
)
# Create document with images
# Path is adapted for doc tests
current_file = Path(__file__).resolve()
root_path = current_file.parents[4]
image_path = root_path / "tests" / "images" / "invoices" / "invoice.png"
# Create a document with only images (no text)
image_document = Document(
images=[
create_image(image_path), # contextgem.Image instance
# ...
]
)
# Create a document with both text and images
mixed_document = Document(
raw_text="This document contains both text and visual elements.",
images=[
create_image(image_path), # contextgem.Image instance
# ...
],
)
The Document class accepts the following parameters:
Parameter |
Type |
Default Value |
Description |
|---|---|---|---|
|
|
|
The main text of the document as a single string. |
|
|
|
List of |
|
|
|
List of |
|
|
|
List of |
|
|
|
List of |
|
|
|
Mode for paragraph segmentation. |
|
|
|
SaT model ID for paragraph/sentence segmentation or a local path to a SaT model. See wtpsplit models for available options. |
|
|
|
Whether to pre-segment sentences during Document initialization. When |
🎯 Adding Aspects and Concepts for Extraction#
Before extracting information from a document with an LLM, you must define and add aspects and concepts to your document instance. These components serve as the foundation for targeted analysis and structured information extraction.
Aspects define the text segments (sections, topics, themes) to be extracted from the document. They can be combined with concepts for comprehensive analysis.
Concepts define specific data points to be extracted or inferred from the document content: entities, insights, structured objects, classifications, numerical calculations, dates, ratings, and assessments.
For detailed guidance on creating and configuring these components, see:
Aspect Extraction - Complete guide to defining and using aspects
Supported Concepts - All available concept types and how to use them
📍 Locating Paragraphs and Sentences#
Extraction results reference the document’s own Paragraph and Sentence objects (see e.g. Aspect Extraction for details on references). To find the position of such an object in the document, use get_paragraph_index() and get_sentence_index():
# 0-based position of a paragraph in document.paragraphs
para_index = document.get_paragraph_index(paragraph)
# (paragraph_index, sentence_index) position of a sentence,
# where sentence_index is 0-based within the paragraph's sentences
para_index, sent_index = document.get_sentence_index(sentence)
Lookups are keyed by each object’s unique ID rather than text equality, so paragraphs or sentences with identical text (e.g. duplicate clauses in a contract) resolve to their specific occurrences. Unique IDs are preserved by serialization, so lookups also work across to_dict()/from_dict() round-trips.
This makes the methods suitable for working with extraction references, e.g. citing where in the document extracted information was found, ordering references combined from multiple extracted items, or retrieving surrounding context:
# Human-readable citation for a reference paragraph
ref_para = concept.extracted_items[0].reference_paragraphs[0]
citation = f"Found in paragraph {document.get_paragraph_index(ref_para) + 1}"
# Order references combined from multiple extracted items
# (references within a single extracted item are already in document order)
combined_refs = [
para
for item in concept.extracted_items
for para in item.reference_paragraphs
]
ordered_refs = sorted(combined_refs, key=document.get_paragraph_index)
# Surrounding context of a reference paragraph
ref_index = document.get_paragraph_index(ref_para)
context_window = document.paragraphs[max(0, ref_index - 2) : ref_index + 3]