AI SDK Options

Overview

Astera’s LLM Generate object allows integration with LLM providers such as OpenAI, Llama, or custom LLM APIs. As part of its configuration, the object includes Ai SDK Options that provide fine-tuning controls over the LLM model’s behavior and output quality.

These settings allow users to:

• Manage token limits

• Control randomness and creativity of the output

• Access confidence metrics like LogProbs and Perplexity

This document covers all AI SDK options available within the LLM Generate object.

AI SDK Options

Expanding the AI SDK Options group box reveals the following options:

• Max Tokens: Limits the output tokens. This limit can be adjusted according to the maximum tokens we want in the output.

• Temperature: Controls randomness in model predictions. At temperature 0 (default), the model's output is deterministic and predictable, at higher temperatures randomness and creativity increases in the output.

• Top P: Also called nucleus sampling. Controls the diversity of the generated output by selecting tokens from a subset of the most likely options. Top P 0.1 (default) means the model will only consider the smallest set of tokens whose cumulative probability is at least 10%. This significantly narrows down the possible token choices, making the output more focused and less random. Increasing the Top P results in less constrained and more creative responses.

• Evaluation Metrics: Enabling this option introduces three additional fields in the output:

  • Output Tokens: Total number of tokens in the generated result.

  • LogProbs: Log-probabilities associated with each generated token. These represent the likelihood (on a logarithmic scale) of a specific token being generated, based on the model's understanding of the input and context. They indicate the model’s confidence in generating each token.

  • PerplexityScore: Measure of how well the language model predicts a sequence of tokens. Lower perplexity (closer to 1) indicates better predictive performance, while higher perplexity suggests greater uncertainty in predicting the next token.

Use Case

To understand the use of AI SDK Options, let’s look at a use case. Building on the ticket classification use case we worked on in the last document, LLM Generate, we now want to record the confidence score of the classification as well. To achieve this, we will use the AI SDK Options.

Using AI SDK Options in LLM Generate

The evaluation metrics is an array of metrics for each output token. We want to limit the output tokens to 1 so that we only need to work with a single metrics record.

1. To do this, we have updated the categories slightly, so the result is no more than 1 token.

1. On the next screen, under the AI SDK Options group box, we will specify the output token limit as 1 and enable the evaluation metrics.

1. Notice that additional output fields are visible under the Prompt node. We can use these fields in downstream processing.

2. Let’s say we want to include the confidence score of the AI-generated category in the delimited destination.

Log Probs

In its raw form, log probs is a JSON structure containing the metrics of each token.

To use it for evaluating the confidence score, we can parse the LogProbs and calculate its exponential. A value closer to 1 means higher confidence of the AI-generated category.

We can parse it using a JSON Parser.

1. Drag and drop a JSON Parser from the toolbox and map LogProbs to JSON Parser’s Text input node.

1. To generate the output structure of the JSON Parser, copy the LogProbs from the Data Preview window, go to Properties of the JSON Parser object, click Generate Layout by Providing Sample Text, paste the copied text and click Generate to generate the layout.

1. Next, we take the exponential of LogProb to obtain a value between 0 and 1, with a value closer to 1 indicating a higher confidence of the result.

2. Drag and drop an Expression object to the designer and map LogProb to it.

1. Go to the Properties of the Expression object, open the Expression Builder by clicking on under the Expression column for the mapped field, LogProb and type the expression for calculating the exponential.

1. The output of the Expression object is a confidence score on a scale of 0 to 1 and can be used in downstream processing.

2. Let’s say we want to map the confidence score to the destination. To do this, remove the inbound maps to the Delimited Destination object, add all the required fields to the JSON Parser, followed by mapping them to the Expression object and finally to the Delimited Destination object.

1. Preview the output of Delimited Destination object to verify the results.