Choosing the Right LLM(s)

Choosing the Right LLM(s)#

🧭 General Guidance#

Your choice of LLM directly affects the accuracy, speed, and cost of your extraction pipeline. ContextGem integrates with various LLM providers (via LiteLLM), enabling you to select models that best fit your needs.

Since ContextGem specializes in deep single-document analysis, models with large context windows are recommended. While each use case has unique requirements, our experience suggests the following practical guidelines. However, please note that for sensitive applications (e.g., contract review) where accuracy is paramount and speed/cost are secondary concerns, using the most capable model available for all tasks is often the safest approach.

Choosing LLMs - General Guidance#

Aspect Extraction

Concept Extraction

A smaller/distilled non-reasoning model capable of identifying relevant document sections (e.g., gpt-4o-mini). This extraction resembles multi-label classification. Complex aspects may occasionally require larger or reasoning models.

For basic concepts (e.g., titles, payment amounts, dates), the same smaller/distilled non-reasoning model is often sufficient (e.g., gpt-4o-mini). For complex concepts requiring nuanced understanding within specific aspects or the entire document, consider a larger non-reasoning model (e.g., gpt-4o). For concepts requiring advanced understanding or complex reasoning (e.g., logical deductions, evaluation), a reasoning model like o3-mini may be appropriate.

🏷️ LLM Roles#

Each LLM serves a specific role in your pipeline, defined by the role parameter. This allows different models to handle different tasks. For instance, when an aspect or concept has llm_role="extractor_text", it will be processed by an LLM with that matching role.

You can assign any pre-defined role to an LLM regardless of whether it’s actually a “reasoning” model (like o3-mini) or not (like gpt-4o). This abstraction helps organize your pipeline based on your assessment of each model’s capabilities and task complexity. For simpler use cases, you can omit role assignments, which will default to "extractor_text".

Example of selecting different LLMs for different tasks#
# Example of selecting different LLMs for different tasks

import os

from contextgem import Aspect, Document, DocumentLLM, DocumentLLMGroup, StringConcept

# Define LLMs
base_llm = DocumentLLM(
    model="openai/gpt-4o-mini",
    api_key=os.environ.get("CONTEXTGEM_OPENAI_API_KEY"),
    role="extractor_text",  # default
)

# Optional - attach a fallback LLM
base_llm_fallback = DocumentLLM(
    model="openai/gpt-3-5-turbo",
    api_key=os.environ.get("CONTEXTGEM_OPENAI_API_KEY"),
    role="extractor_text",  # must have the same role as the parent LLM
    is_fallback=True,
)
base_llm.fallback_llm = base_llm_fallback

advanced_llm = DocumentLLM(
    model="openai/gpt-4o",  # can be a larger model (reasoning or non-reasoning)
    api_key=os.environ.get("CONTEXTGEM_OPENAI_API_KEY"),
    role="reasoner_text",
)

# You can organize LLMs in a group to use them in a pipeline
llm_group = DocumentLLMGroup(
    llms=[base_llm, advanced_llm],
)

# Assign the existing LLMs to aspects/concepts
document = Document(
    raw_text="document_text",
    aspects=[
        Aspect(
            name="aspect_name",
            description="aspect_description",
            llm_role="extractor_text",
            concepts=[
                StringConcept(
                    name="concept_name",
                    description="concept_description",
                    llm_role="reasoner_text",
                )
            ],
        )
    ],
)

# Then use the LLM group to extract all information from the document
# This will use different LLMs for different aspects/concepts under the hood
# document = llm_group.extract_all(document)
Contents