> For the complete documentation index, see [llms.txt](https://documentation.astera.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://documentation.astera.com/astera-intelligence/use-cases/multipage-document-extraction.md).
# Multipage Document Extraction
## Overview
This guide explains how to configure a *Dataflow* to extract structured data from multi-page PDF documents using the *Text Converter* and *LLM Generate* transformations. You will configure a *Text Converter* to split a PDF page-by-page, pass each page to an LLM for extraction, and use the *Union JSON Results* feature in *LLM Generate* to merge the per-page outputs into a single consolidated record.
## Prerequisites
* An LLM shared connection configured, such as *OpenAi* with a valid API key
* An OCR shared connection configured, such as *GoogleOCR*
* A PDF source file to process
* A destination path for the structured extracted output
## How multipage processing works
LLM context windows cannot reliably hold an entire multi-page document.
Below is an example of an invoice document. The table shown in the document spans several pages.
The recommended approach is to split the PDF by page in the *Text Converter*, send each page as a separate record to *LLM Generate*, and enable *Union JSON Results* to consolidate the per-page extractions into one output record.
When *Split Output* is enabled on the *Text Converter*, each page becomes its own output record. *LLM Generate* then fires once per record, and *Union JSON Results* merges all responses at the output node of *LLM Generate*.
The dataflow follows this structure:
*Text Converter* → *LLM Generate* → *JSON Parser* → *Destination*
## Step 1: Set up a Variables object
A *Variables* object lets you parameterize file paths and the output schema so you can reuse the dataflow without editing it each time.
1. In a new *Dataflow*, drag a *Variables* object onto the canvas.
2. Add the following parameters:
* **schema** *(optional):* A JSON string defining the fields you want extracted. For example:
```json
{
"invoice_number": "string",
"invoice_date": "string",
"supplier_name": "string",
"line_items": [
{
"description": "string",
"quantity": "number",
"unit_price": "number",
"amount": "number"
}
],
"total": "number"
}
```
* **SourceFilePath:** Full path to the source PDF.
* **DestinationPath:** Full path to the output file.
* **DocumentType** *(optional):* A label for the document category, such as `invoice`. This is useful for referencing in the prompt via `{Input.DocumentType}` when the same dataflow handles different document types.
{% hint style="info" %}
**Note:** Although `schema` is optional, providing one significantly improves reliability. Without it, the LLM generates its own structure per page — field names can vary between pages and runs. In inferred mode, Union JSON Results merges by property name, so `invoice_no` and `invoice_number` become two separate fields in the output rather than one consolidated value.
Keep the schema as flat as possible. Deeply nested structures increase the chance of the LLM omitting or misplacing fields on partial pages.
For arrays like `line_items`, define just one example element in the schema. The LLM repeats the pattern for as many items as it finds on the page.
{% endhint %}
## Step 2: Add and configure the Text Converter
The *Text Converter* reads the PDF and converts it to plain text. The *Split Output* setting controls whether it produces one record per page or one record for the entire document.
1. Drag a *Text Converter* onto the canvas.
2. In its properties, under *PDF Converter Options*, set:
* *Text Converter Model:* Select your configured OCR engine.
* *Shared Connection:* Select the shared connection configured for your selected OCR engine.
* *Split Output:* Enable this to output one text record per page. This is required for page-by-page LLM processing. Disable it only if the entire document fits within your LLM's context limit and a single extraction call is sufficient.
* *Pages To Read:* Leave blank to process all pages. To limit processing, specify individual pages or ranges, such as `1,3,5-7`.
3. Map the file path input:
## Step 3: Add and configure LLM Generate
*LLM Generate* calls your LLM once per incoming record. With *Split Output* enabled, each record represents one page.
1. Drag an *LLM Generate* transformation onto the canvas.
2. Define the *Input Layout* with at least these fields, and any other fields relevant to the use case:
* **DocumentText** (String) — receives the page text from *Text Converter*.
* **Schema** (String) — this field can be passed to the prompt as a template variable when your flow may handle multiple schemas. It can also be defined directly in the prompt if it will not change.
3. Open the *LLM Template* editor and write the *System Prompt*:
Write a minimal *User Prompt*, such as `Extract as per instructions.`
{% hint style="info" %}
**Note:** The system prompt is where the LLM receives its behavioral rules. LLMs follow format constraints, null-handling rules, and schema compliance more reliably when they are set in the system prompt rather than the user prompt. For extraction tasks, it is better to keep the user prompt minimal and place all instructions in the system prompt.
{% endhint %}
4. On the next screen, under *General*, configure the connection and union settings:
* *AI Provider:* Select your provider, such as `OpenAi`.
* *Shared Connection:* Select the shared connection where you configured your selected AI provider's authentication.
* *Model Type:* `UseBaseModels`
* *Base Model:* Select the model to use, such as `Gpt_5_Mini`.
* *Union JSON Results:* Enable this to merge all per-page JSON responses into one output record. See [How union merging works](#how-union-merging-works) for how the merge mode is determined.
* *Union JSON Key Field:* Select the input field whose distinct values determine grouping. Leave as `` to merge all records into a single output. For flows where a single source file produces records that should be merged separately — for example, pages belonging to different document types — select the field that identifies the group. See [Union grouping](#union-grouping) for examples.
5. Under *Ai Sdk Options*, set:
* *Max Tokens:* Controls the maximum number of output tokens the LLM can generate per response. The default is 3000, which is too low for multi-field extraction — increase it based on your model's limit. For GPT-5-mini (max 64,000), `60000` is a good setting. For GPT-4o-mini (max 16,384), use `15000`. Increase if responses are being truncated.
* *Temperature:* `1` — this is the default temperature for GPT-5-mini. For other OpenAI models, such as GPT-4-mini, use `0` for deterministic results.
6. Under *Loop Options*, set:
* *Run Items In Parallel:* Enable this for faster processing on multi-page documents.
* *Degree Of Parallelism:* `10` is a good default. Lower it if you hit API rate limits on LLM calls visible in the *Job Monitor*.
* *Preserve Order:* This option is enabled by default. Keep it enabled if you require pages to be merged in sequence even when they complete out of order.
## How union merging works
*Union JSON Results* and *Union JSON Key Field* are configured in the *LLM Generate* *Properties* panel under *General*, alongside the connection and model settings. *Union JSON Results* supports two merge modes, controlled by whether an input field named `schema`, case-insensitive, exists and is receiving a value.
### Mapped schema
When the input layout contains a field named exactly `schema`, or `Schema`, and that field is mapped to a value, the output JSON is built according to that schema. *LLM Generate* uses it to guide merging:
* **Arrays** such as `line_items` are concatenated across pages. Items from page 2 are appended after items from page 1.
* **Scalar fields** such as `invoice_number` and `total` take the first non-empty value found across pages. Values treated as empty, case-insensitive, are `""`, `"N/A"`, and `"null"`.
This is the most predictable mode for documents like multi-page invoices where header fields appear on page 1 and line items span all pages.
**Example input layout for mapped schema mode:**
### Inferred schema
If the input layout does not contain a field named `schema`, or that field has no value, LLM Generate infers the output structure from the JSON responses themselves. It examines all per-page outputs to determine which properties are present, whether each property is an array or a scalar, and then merges them using the same rules: arrays are concatenated, scalars take the first non-empty value.
### Union grouping
*Union JSON Key Field* groups records by the distinct values of the selected input field. You get one merged output record per unique value. For example, selecting a `FilePath` field in a multi-file flow produces one output per file. Set it to `` to merge everything into a single output regardless of source.
Another use case is when a single file contains multiple document types, such as a package that includes a purchase order, an invoice, and a receipt. If your flow identifies the document type per page and populates a `DocumentType` field, you can set *Union JSON Key Field* to `DocumentType`. *LLM Generate* then produces three separate output records. One contains all PO pages merged together. One contains invoice pages. One contains receipt pages. Each is unioned using the schema relevant to that type.
## Step 4: Add a JSON Parser and destination
After *LLM Generate* merges the per-page results, the output is a single JSON string available in the Output node. Parse it into structured fields using a *JSON Parser*, then write to any structured destination.
1. Drag a *JSON Parser* onto the canvas after *LLM Generate*.
2. Map `LLMGenerate.Output.Prompt.Result` → `JSONParser.Input`.
3. To build the *JSON Parser* output layout, right-click the *JSON Parser* on the canvas and select *Run Flow To Generate Layout*. This runs the dataflow up to that point and generates the output layout from the actual LLM response. Alternatively, open the *JSON Parser* properties and provide the expected JSON schema directly to build the layout manually.
4. Connect the *JSON Parser* to any structured destination, such as a database table, an Excel file, or an XML destination.
> [**Smart Source**](/dataflows/sources/smart-document-source.md)**:** If the output schema may vary between documents, for example when different document types produce different JSON shapes, use *Smart Source* instead of *JSON Parser*. *Smart Source* parses the incoming JSON at runtime and maps it to a fixed output layout, so it can handle varying input structures without breaking the downstream mapping.
## Step 5: Run the dataflow
1. Save the dataflow.
2. Click *Run*.
3. Monitor the *Job Trace*. Each page input to *LLM Generate* appears as a separate trace entry.
If a page's JSON is invalid or truncated, an error is logged. Increase *Max Tokens* and rerun. Change the model if the current model does not allow enough tokens.
When the run completes, verify the output. For a 5-page invoice, the merged JSON will have a single `line_items` array containing entries from every page, and scalar fields such as `invoice_number` will contain the value from the first page where they appear.
## Troubleshooting
* **Empty or null fields in output:** The LLM did not find that data on the page. Preview the *Text Converter* output to verify OCR quality. If text is garbled, try enabling *Force OCR* or adjusting *Deskew*.
* **Invalid or truncated JSON error in job log:** The LLM returned text outside the JSON structure, or the response was cut off. Add `Return valid JSON only — no markdown, no explanation` to the system prompt, and increase *Max Tokens*.
* **Rate limit errors from the LLM provider:** Reduce *Degree Of Parallelism* or add retry logic using a *Workflow* wrapper around the dataflow.
* **Pages out of order in merged output:** Ensure *Preserve Order* is enabled in *LLM Generate*.
* **Context limit exceeded:** If *Split Output* is disabled and the full document exceeds the model's context window, enable *Split Output* in *Text Converter* to process page by page instead.
* **All records merged into one instead of one per file:** In multi-file flows, set *Union JSON Key Field* to the file path field. If it is left as ``, all pages from all files merge into a single output.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://documentation.astera.com/astera-intelligence/use-cases/multipage-document-extraction.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
