Video

To make an API call in Astera, an API Client object, along with its API Connection, needs to be configured.

First, drag-and-drop an API Connection object from the Toolbox and configure it in the dataflow. Alternatively, you can use an API Connection object in a shared action file within the scope of the project you are working with.

The API Connection object contains the Base URL, authentication details, and shared parameters for the API endpoint.

You can learn all about the configuration and usage of the API Connection object here.

Next, let’s configure the API Client object.

1. First, drag-and-drop an API Client object from the Toolbox onto the dataflow.

2. Right-click on the API Client object’s header and select Properties.

The API Client screen will now open. Here you will have to specify the following,

Shared Connection: Establish your API Client’s connection from this drop-down that lists all shared connections from within the flow as well as from the project.

HTTP Method: The HTTP request verb defines the operation you want to make on the API resource.

Resource: the resource of the API from which you want to make a request. This will be appended after the Base URL from the selected shared connection to form the complete endpoint. Any URI or path parameters must be included in the resource text enclosed in curly brackets, {}.

Input Content Type: This is the content-type header for the request payload which is default to application/JSON type. The actual request payload layout can be defined in the input layout screen.

Output Content Type: This is the content type of the response payload which is default to application/JSON type. The actual response payload layout can be defined in the output layout screen.

3. Click Next. A Parameters screen will appear.

Here you will have to specify the following:

Override Inherited Parameter: Check this to override any parameters previously defined and inherited from the shared connection.

Name: The name of your parameter.

Parameter Key: Since the Name column does not allow any special characters, the parameter key can be used to define an alternate name including any special characters to replace the name in the API request.

Parameter Location: The parameter type such as Query, URI, and Header.

Data Type: Specify the data type of your parameter.

Format: Define the datatype format of the parameter’s value sent in the API request.

Plaintext: Check this box to disable URL encoding the parameters when the request is sent. The parameters will be sent in plaintext format, or you could optionally encode your parameter values manually using the URLEncode function from the toolbox.

Default Value: The parameter’s value for which you want to make a request.

4. Click Next. An API Client Output Layout screen will now open.

Here, we will select the Generate Layout by Running Request to build the response layout. Alternatively, you can build the layout manually or use a sample text.

Next, click OK.

5. Once done, click Next, and you will be led to the Pagination Options screen.

Here, you can select the type of pagination that has been specified by the API providers. Astera offers the following pagination types.

6. When done, click Next, and you will be taken to the Service Options screen.

Request Options:

• Request Delay: Delay time (in milliseconds) before sending a request.

• Retry Count: Number of retry attempts to be made in case of a time-out error.

• Retry Delay: The duration (in milliseconds) between each consecutive retry attempt.

• Continue on Retry Failure: Check to succeed the flow even after all retries have failed.

• Use Parallelism: Check this option to send requests in parallel. Check this to send requests in parallel. Number of requests to be sent in parallel (max limit of 10).

• Follow Redirect: Check to allow forwarding a 3xx response to the redirected URL.

• Include Authentication: Check to include authentication in the redirected API call.

• Redirect Limit: Number of allowed redirect calls from a request. -1 indicates no limit.

• Keep Connection Alive: Check to keep the TCP connection open to reuse for all subsequent requests to the same server.

• Enable E-Tags: To learn about E-Tags, click here.

• Retrieval: Check this to enable e-tags to request caching for GET requests.

• Updates: Check this to enable request concurrency control using etags for PUT, PATCH or DELETE requests.

Response Options:

• Ignore HTTP Status Codes: Selecting this option will show and allow processing responses other than 2xx in the flow, which are otherwise considered an error.

• Include Content as String: Adds a field for serialized response content string in the Response-Info output node.

• Include Response Headers: Adds all response headers as a collection in the Response-Info output node.

• Include Raw Bytes: Adds a field for response content in the form of raw bytes in the Response-Info output node.

7. Click Next, and the Config Parameters screen will appear.

Config Parameters can enable the deployment of flows by eliminating hardcoded values and provide a dynamic way of changing multiple configurations with a simple value change.

8. Click OK, and the API Client object will be configured.

Now, right-click on the API Client object’s header, and select Preview Output.

Your request has been executed successfully, as you can see that the HTTP status code is 200 which means that the API Client has successfully carried out the GET request for the provided status.

This concludes our discussion on making API calls with the API Client object in Astera Data Stack.

JSON/XML File – For this type of API source you only need to provide the Open API Specification File Path in JSON or XML file formats.

1. Specify the File Path and click OK.

This API will be populated in the API Browser panel, where you can simply expand the nodes and drag-and-drop methods onto the designer window.

JSON/XML URL – For this type of API source, you will need to provide the URL in JSON or XML format.

1. Specify the URL and click OK.

This API will be populated in the API Browser panel.

What is an API?

API (Application Programming Interface) is defined as an interface or medium through which one software communicates with another. In other words, it is a set of contracts that allows different software systems to share information with each other. The greatest advantage of an API is that different programs and devices can communicate with each other in a secure manner, without interference.

APIs are messengers that conform to the technical contract between two parties. They are language and platform-independent, which means C# can talk to Java, and Unix can communicate with Mac without any difficulty. An API is not the same as a remote server. In fact, it is part of a remote server that receives requests and sends responses. More precisely, an API is a structured request and response.

API Browser in Astera

The API Browser in Astera has narrowed down the steps to make HTTP calls using just one-step authentication. Once you have imported an API in Astera, all endpoint operations in that API are populated at once. API definition describes what requests are available and what the responses will look like.

So, once you load an API definition, all supported methods are populated in the API Browser unlike Legacy, where all supported methods must be configured separately in each object.

There are two methods of configuring APIs in Astera. For open APIs, you only need to provide the API Import Source and File Path or Base URL to configure the connection with a specific API. Once this standardized information is provided, any API that you have imported will populate in the API Browser, along with their methods, for example, GET, PUT, POST, PATCH, and DELETE, and they will remain accessible until their authentication period expires. From the API Browser in Astera, you can simply drag and drop operations, and use them in your flows.

It is important to note that a project must be created before importing APIs to work with the API Browser. However, you can access the API without a project when it’s an API Connection contained in the flow.

The API Browser, along with all its features and functionalities, works only within the scope of a project. Otherwise, it will give you the following error,

When a user imports an API, a shared connection file is created within the project automatically. The shared action file contains the Base URL of the imported API.

HTTP Request Methods

Astera supports the following HTTP request methods:

1. PUT: To update data to a specified resource to be processed on an API.

2. GET: To retrieve data from a specified resource on an API.

3. POST: To create or update an existing record on an API.

4. DELETE: To delete a specified resource on an API.

5. PATCH: To apply partial modifications to an existing resource.

Creating a Project for API Browser

To work with the API Browser in Astera, you must first create an API Client Project.

Follow the steps below to create an API Client Project in Astera,

1. Go to Menu Bar > Project > New > API Client Project.

Provide a name to the API Client Project and point the path to the location and directory where you want to save it.

2. Now, open the API Browser panel on your Astera client from Menu Bar > View > Data Service > API Browser.

3. Once selected, an API Browser panel will open on the left side of your Astera client window.

Here, you can see a couple of icons in the toolbar of the API Browser:

• Import API: By clicking this option, you can import different APIs with various available options.

• Remove API from Browser: This option removes the selected API from the API Browser.

• Refresh API Tree: This option allows you to redraw the browser tree after you have deleted some operations.

• Expand/Collapse all: These options show/hide all the requests in the CAPI file.

• Add Request: This option allows you to add a new HTTP request to the CAPI file by specifying the request name, resource, and HTTP method.

• Edit Properties: You can use this option to change the shared connection or the API name of the CAPI.

• Open API Connection: This option allows you to directly open the shared API Connection from the project for the API opened in the API Browser.

• Save CAPI file: Any changes made to the CAPI file are saved when you click on this option.

How to Import APIs in Astera

To import an API in Astera, click the Import API icon. An Import API screen will open.

Here, first, you need to select the API Import Source type from the drop-down menu. Astera offers three ways to import APIs.

Users can create and maintain custom API collections in case the API provider does not offer existing documentation for its APIs.

1. From the API Browser, open the import wizard and select Custom API as the API Import Source.

2. Next, provide a Name for your custom API and the Base URL of the API provider. Upon import, a new API shared connection (.sact) and a Custom-API (.capi) file will be created in the project.

Alternatively, the custom API can also point to an existing pre-configured connection from the project.

You can configure the API connection object in the shared connection file by providing valid authentication and defining parameters, if needed.

Once you are done configuring the connection object, the CAPI file will open in the API browser.

To add API requests to your custom CAPI file, click the Add Request icon from the top toolbar menu of the API browser.

Here, define the following request properties:

Request Name: This is used as the request name and description.

Resources: The unique request resource path including the URI or path parameters which appends after the Server Base URL.

HTTP Method: Select the standard HTTP method to be used for this request.

The request will be added to the CAPI file in the API Browser. Repeat this process to add all the required requests in your CAPI file.

Once you have populated the requests in your CAPI file, it may look something like this in the API browser.

To configure the parameters, input/output layout, or pagination options for any request, right-click on it and choose the Edit Request option.

You can also configure and save the request properties by dragging and dropping.

• Drag the request from the API browser to a flow designer.

• Right-click on the API Client and select Properties. Make changes to the properties of the API client object.

• To save the changes, simply drag and drop the client object back to the API browser from the flow designer.

Once you are done populating your CAPI file by configuring all request properties and authentication, click on the Save CAPI file icon on the top of the API browser to save your changes.

This will save all the configurations you have made including parameters, input/output body, and pagination settings to the request.

Sharing and adding the CAPI file to a new project

• Fully configured CAPI files act as a connector for your API provider. If you want to add the CAPI file to another project, right-click on the CAPI file from the project explorer and click on copy full path.

• Then open the other project, right-click on the folder you want to add the CAPI file to, and click on Add Existing Items.

• A box will open. Paste the file path in the box next to the File name and click Open.

The CAPI file will be added to the project along with its corresponding .Sact file.

This concludes the basic concepts of working with the API Browser in Astera.

Video

To make an API call, an API Connection object needs to be configured first. This object stores all the common information that can be shared across multiple API requests.

Configuring The API Connection Object

1. Drag-and-drop the API Connection object from the Toolbox onto a dataflow.

2. Right-click on the API Connection object and select Properties from the context menu.

A configuration window will appear on your screen with the following options:

• Base URL: Here, you can specify the base URL of the API which will prepend as a common path to all API endpoints sharing this connection. A Base URL usually consists of the scheme, hostname, and port of the API web address.

• Timeout (msec): Specify the duration, in milliseconds, to wait for the API server to respond before giving a timeout error.

• Include Client SSL Certificate: Check this box to include an imported client certificate for the specified base URL.

• Enable Authentication Logs: Select this checkbox to enable authentication logging for APIs.

• Authentication – Security Type: Specify the authentication type for the API.

Astera supports the following authentication types.

Types Of Authentications:

Identification and verification of a user is an important aspect of authentication. Authentication allows an application to determine whether a user identity is valid/authorized; based on the outcome, a user is provided access control to the application.

For APIs, authentication plays a key role in authorizing requests to the API platform’s resources. The following authentication types are available within the API Connection object.

1. No Authentication

2. OAuth 2

3. API Key

4. Basic Authentication

5. Bearer Token

6. AWS Signature

7. NTLM

No Authentication

With this security type, the user can send API requests without including any authentication parameters.

OAuth 2

This type is used when an unrelated application login is used to acquire permission to access data on your behalf for another application. Instead of giving away your password to the application, OAuth 2 enables delegated authorization through a third-party Authorization Server.

In response to a valid authorization, the Auth Server issues an Access Token with a restricted scope and validity to authenticate the user with permissions. When the Access Token expires, its Refresh Token is used to obtain another valid Access Token.

Configure an OAuth 2 request to generate Access and Refresh tokens. The tokens will be implicitly added to the request and auto-refreshed if expired.

The OAuth 2 authentication supports different flows for various scenarios. You can select any of the following Grant Types:

1. Implicit

2. Authorization Code

3. Authorization Code (with PKCE)

4. Password

5. Client Credentials

Implicit

In this Grant Type, you only need to provide an Authentication URL and Client ID to request a token without an intermediate code exchange. It was built for apps such as native Java script clients, and mobile or browser-based applications where client secrets cannot be exposed.

Hence, this flow promptly gets the token directly exposed in the URL and is considered less secure for web applications.

• Authentication URL: This is the login page, where the API user authorizes itself to the Authentication Server.

• Client ID: This is the public identifier for accessing the registered API Server application.

Authorization Code

This flow type is popular for mobile and web server-side applications.

In this Grant Type, you need to provide an Authentication URL, Access Token URL, Client ID, and, optionally, a Client Secret to authorize.

The flow first requests a one-time authorization code from the authorization server. The authorized request is redirected to the API Server along with its client secret which then authenticates the user for its resources by exchanging the code for an Access Token.

• Authentication URL: This is the login page, where the API user authorizes itself to the Auth Server.

• Access Token URL: This URL is provided to generate an Access Token for authentication after the user has been authorized successfully.

• Client ID: The public identifier for accessing the registered API Server application.

• Client Secret: It is provided alongside the Client ID, as a secret credential to access the registered application from the Auth Server.

After providing the authentication details, click on the Request Token option to sign in and fetch the token(s).

Authorization Code with PKCE

The Proof Key for Code Exchange flow has replaced implicit authentication flow by being more secure to be used in single-page native, mobile, and browser-based apps. As such apps existing on the browser cannot store client secrets, this Authorization Code flow keeps the client secret hidden.

Instead, the client sends a dynamically generated string generated using a code_verifier hashed to a code_challange to the Auth Server. The Auth Server stores this for verifying the client during the OAuth2 exchange.

The Client app then makes an authorization request and receives the Auth Code as a result. It then requests an Access Token by sending the Auth Code together with the code_verifier that is hashed by the Authorization server and compared to its saved copy for verification.

In this Grant Type, you need to provide an Authentication URL, Access Token URL, and the Client ID to authorize.

Password

In this Grant Type, you need an Access Token URL, Username, Password, Client ID, and Client Secret to authorize. It is considered for internal services and not recommended for third-party applications as it authenticates the given credentials in a single step.

Since user credentials are exposed to the client application, this flow type outlaws the OAuth2 principles and is now deprecated.

• Access Token URL: The URL through which the Access token is going to be generated for authentication.

• Username: The application login name of the user for authentication.

• Password: The application user password is provided for authentication.

• Client ID: The public identifier for accessing the registered API Server Application.

• Client Secret: It is provided alongside the Client ID, as a secret credential to access the registered application from the Auth Server.

After providing the authentication details, click on Request Token to fetch the token(s).

Client Credentials

In this Grant Type, you need the Access Token URL, Client ID, and Client Secret to authorize. This is used with the client application. It self-authenticates access to its resources without a user context.

• Access Token URL: This URL is provided to generate an access token for authentication.

• Client ID: The public identifier for accessing the registered API Server application.

• Client Secret: It is provided alongside the Client ID, as a secret credential to access the registered application from the Auth Server.

After providing the authentication details, click on Request Token to fetch the token(s).

Additional OAuth 2 Info

An OAuth 2 authentication flow requires some additional parameters to specify resources and scope permissions associated with the given Access Token.

To provide additional information required by an API provider for an OAuth2 request, click on the Additional Info button.

• Resource: Use this to identify the URL of the web API intended for user access.

• Scope: Use this to specify what the authenticating application can do on behalf of a user by imposing a limit on which resources it can access and with what rights.

• State: This parameter is useful to protect against XSRF as the client generates and sends a random string while the Auth Server returns it back again on authenticating as a verification.

• Response Type: This parameter is used to specify the expected type to be received from the authorization server on valid authorization. The most common inputs are “code” and “token”. Code is used for the Authorization Code grant type where it is exchanged in the follow-up request for the token. A token is used for implicit grant type where the Access Token is returned directly.

• Callback URL: Redirected URL after the authentication request at which the token/code will be returned. For Astera, use “http://localhost:8050/” or “https://localhost:8050/”

• Include SSL Certificate: To include the client certificate in the OAuth2 token generation request.

• Ignore Certificate Errors: Check to ignore any certificate errors while authenticating.

• Additional Parameters: Any additional parameters apart from the above list that are required to be sent in the authentication request can be added here as key-value pairs, separated by a comma.

Token Caching and Auto-Refresh

Following the security policy of authenticating an API call, clients are required to obtain Access/Refresh tokens for authenticating an API request. These tokens may have a defined validity and need to be invoked again to generate a new token.

Once authentication details are fully configured, users need to manually Request Token in the API Connection.

Handling token expiry and Automation

For the OAuth2 grant flow which requires users to authenticate when requesting a token, the refresh token can be used to obtain a new access token. While other grant flows directly make the call to request an access token, Astera can automatically obtain a new token in the background so your flows can be automated.

You can make use of the auto-generation and caching of these tokens which enables you to automate API requests ensuring new tokens are generated for use without needing to manually update the tokens each time.

Using ‘Client Credentials’ or ‘Password’ OAuth2 Grant Types

These grant flows work by making a single call requesting an Access Token along with the provided client application credentials. Since the flow is not dependent on any user input for authentication, it can be automated for the regeneration of a new token when the existing token expires.

Here, we have a pre-configured authentication with an expired token. Let’s see what happens when this flow is executed with an expired token.

The job trace shows that an expired token was found, and a new token has been generated for this connection and saved to the server cache for future reuse.

On the next run, the server is bound to check the cache for a valid token before opting to generate a new one. The cache stores a token for each unique connection used across all jobs running on the server.

Use of a Refresh Token

For other OAuth2 grant flows that require the user to authenticate first, the refresh token is used to regenerate the access token automatically.

Using Default User Browser for User Authentication

Some API Providers restrict using an embedded browser for authenticating using the OAuth2 code exchange. An alternate option is to request token through a more secure browser-based OAuth authentication.

In this article, we’ll discuss how to run an OAuth2.0 flow for Google Calendar API using the user’s default browser. Users will first need to create an oauth2 application on the Google Developers’ account and obtain the client ID and secret.

Authenticating the Client Application

For this example, we will be authenticating Google APIs which do not allow the use of an Embedded Browser for an OAuth2 exchange.

1. Open the API Connection to configure authentication information.

As Google Calendar API works with OAuth2.0 security with Authorization Code grant type, we can select and configure it accordingly.

We must enter parameters such as Authentication URL, Access Token URL, Client ID, Client Secret, and Additional Information according to the authentication and authorization information provided by Google. Now, let’s click on the Request Token button to generate the access and refresh tokens.

This opens the Embedded Browser of the Astera Client which will result in an error as Google does not allow authentication via an embedded browser. For such platforms, it is necessary to use a more secure user-default browser for OAuth2 authentication exchange.

Close the embedded browser window. Now, check the option to Use System’s Default Browser and click on the Request Token button again.

This opens the user system’s default browser for authentication, and this allows us to successfully retrieve the access token on logging in. In our case, the default Microsoft Edge web browser has opened.

Click on Continue.

The generated Access Token along with the Refresh Token (if supported by the API provider) are displayed on the REST Connection window with their respective expiry date and time.

Tested System Browsers

The following browsers have been successfully tested for the Astera Client,

• Google Chrome

• Microsoft Edge

• Firebox

API Key

An API Key is a key-value pair that a client provides when it makes an API request. They can be sent in the Query string or as a request Header.

It requires two parameters for authentication:

1. Key

2. Value

API Key as a Query

API Key as a Header

Basic Authentication

Basic Authentication is structured according to the HTTP protocol to provide a Username and Password when making an API request.

In basic HTTP authentication, a request header parameter is included in the form of “Authentication: Basic”, where the encoded string is the Base64 encoded.

Bearer Token

Bearer Token is an HTTP-based authentication. The access token generated by the server in response to a login request is in turn included in the request header.

To generate a Bearer Token, you need:

1. User Name

2. Password

3. Token URL

SSL Certificate Authentication

API clients can enable the use of a private signed certificate to authenticate themselves when accessing APIs through mutual TLS. You can configure APIs to use a .pem or a .pkt certificate paired with a certificate key or password.

A Client certificate contains information used to identify the client including a digital signature and it is imported for a specific domain. All HTTPS - SSL-enabled requests matching the domain URL will authenticate using the installed client certificate.

All certificates used in authenticating API requests from the client will be imported to Astera’s Server and are included as authentication when an API request is sent. To import a client certificate for authenticating API requests,

1. Navigate to the Server tab on the main menu bar.

2. Right-click on the cluster node and select Client Certificates.

This opens the wizard to manage client SSL certificates.

3. Click on the import icon at the top left to add a certificate authenticating to a domain.

Importing a .pem certificate

• Define the requested domain which will include this certificate.

• Browse the .pem client certificate file obtained as a counterpart to the authenticating server certificate present on the API provider.

• Provide the matching key file for the given client certificate.

Click Import.

Now, this certificate can be used with SSL-enabled authentication for API requests sent to the given domain.

Importing a .pfx certificate

• Define the requested domain which will include this certificate.

• Browser the .pfx client certificate file obtained as a counterpart to the authenticating server certificate present on the API provider.

• Enter the password for the certificate.

Click Import.

Now this certificate can be used with SSL-enabled authentication for API requests sent to the given domain.

Enabling SSL Certificate Authentication

Once the certificate has been imported for the respective domain, let’s see how to make an API request with SSL enabled.

You need to enable SSL verification to include the certificate when making an API call. To enable SSL, open the API Connection object which has the Base URL domain, and the authentication configured. To include the SSL certificate, check the option to Include Client SSL Certificate.

Click OK and preview the API Client to send a request.

This request now includes the certificate to validate the client on the mutual TLS authentication.

Shared Parameters

This is where you can define query or header parameters to be shared across all clients using the same connection.

• Name: The name of a Query or Header parameter can be defined here.

• Parameter Location: This option defines whether the parameter has a Query location or a Header location.

• Data Type: This option defines the data type of the parameter from a list of options.

The parameter values defined here will be inherited by all API clients using this connection unless overridden individually.

4. Once done, click Next and you will be led to the Config Parameters screen.

Here, config parameter values can be changed according to your application. Parameters not changed will use their default values.

5. Click Next, and you will be led to the General Options screen.

Here, you can add any Comments that you wish to add. The rest of the options for this object have been disabled.

6. Click OK to close the window.

You have successfully configured the API Connection object.

Using the API Connection Object

In a Dataflow

1. Click on File in the main toolbar, hover over New, and select Dataflow from the drop-down menu.

2. Once the Dataflow is open, drag-and-drop the API Connection and API Client objects from the Toolbox onto the dataflow.

3. Configure the API Connection object for the Base URL and Authentication.

Right-click on the API Client object and select Properties from the context menu.

A new API Client Properties window will open.

The Shared Connection dropdown list shows us the API Connection object present in the same dataflow.

You can now use this API Client object to make API calls within Astera.

In a Project

1. Navigate to the main toolbar, click Project, hover over New, and select a project type.

2. Locate the Project Explorer on the right, right-click on the project or one of its folders and select Add New Item from the context menu.

This will open a new window where a new SharedAction can be added to the project.

3. Within the SharedAction file, drag-and-drop the API Connection object from the Toolbox.

4. Configure the API Connection object with Base URL, Authentication, and Shared Parameters and save the SharedAction file

This API Connection can be used in any flow document contained in the same project.

5. Next, open a new dataflow within the project.

6. Drag-and-drop the API Client object onto the dataflow, right-click on it, and select Properties from the context menu.

A new window will open.

Here, you can see the name of the Shared Connection within the drop-down menu of the Properties option.

• If shared connections with duplicate names exist in the project, only one will be shown and used.

• If duplicate connections exist in the flow and the project, the flow connection will be given preference.

This concludes our discussion on the configuration and use of the API Connection object in Astera Data Stack.

What is eTag?

An eTag also called an entity tag is an HTTP response header field that includes an identifier for the specific version or the state of the resource at the time the request was sent. This identifier helps to differentiate between the different versions of the resource and to check if the caches at the client side hold the updated representation of the state of the resource.

How do they work?

Let’s try to understand what is meant by eTags and how these options work.

If the client wants to check if the caches of a resource are usable or fresh, it can send the eTag in the If-None-Match header field in the request to the server. The server will match the client’s eTag with the one that it has for the current version of the resource. If the ETags match, the server will not send any representation of the state of the resource in the response implying that the client’s caches are fresh and usable.

Two eTag Use-Cases

There are two major uses of eTags in API requests:

• Data Caching

• Concurrency Control.

Let’s investigate these uses one by one. For now, let’s see how this Data Caching works with a use case.

If None Match eTag

So, we will make an API call to one of the endpoint operations of Box APIs. Here, we have a dataflow in which we are making an API call to fetch file details from one of the files on our Box account.

We will send a GET request to the /file/{fileid} resource with the help of the API client and API connection object. We have configured the API connection object in a shared action file. From the API documentation site, we can see that Box supports OAuth 2 authentication and the grant type of authorization code. Hence, we have already generated our access code after providing the client credentials from our Box app.

Coming back to our flow, let’s open the properties of the API Client object. Here, we are using the shared action API connection object that is providing the base URL. We have specified the HTTP method of GET and provided the name of the resource. Here, the curly brackets specify that the path parameter of file id will be passed along the request to fetch the information on the file related to that file id. In our flow, we are providing the file id through the constant object.

From the API documentation of Box APIs, we can see that the if none match header is supported for the endpoint at which we are making a call in our dataflow.

Now, if we go to the Service options screen of the API Client object, we can see that we have a checkbox to enable eTags which further gives us two checkboxes of,

• Retrieve if None Match Header

• Update Using If-Match Header.

We need to enable the eTag and the If-None-Match header checkbox.

When the request is sent to the server to fetch the file information for the first time via the GET API request, the response will be returned with an eTag value. This eTag value along with the response will be stored in the response caches at the client side.

• The field of “Is cached response” in the response info node will be returned True because we are making the call for the first time and receiving the response from the server that will be cached.

• In the future, if the client makes an API call again to fetch the file information, the eTag in response caches will be first compared with the latest eTag from the Server. So, for the consecutive API calls having the same cached eTag, we will see the “Is Cached Response” field as true.

• In case, the file has not changed or updated, and the caches are reusable. The server will send a No modification response as we can see in our job trace. This means that the server does not have to send the requested information again in the response instead the client can use its response caches.

So, this is how eTags help to prevent unnecessary download and retrievable of information in turn saving the server’s bandwidth and request processing time.

If-Match eTag

Let’s look at another use case of eTag related to concurrency control.

It is possible that more than one client is sending requests to update the resources of the server. Then, to prevent loss of changes and to detect simultaneous updates, the client can send an eTag in the If-Match header field in the request to the server, if another client updates the resource in between or the file is modified, the server can compare the client’s eTag with its own current one and if they don’t match, the server can prevent clients from overwriting the changes or it will ensure that the latest version of the resource gets updated.

Let’s try to understand it with the help of a use case.

• We will make two update API calls to change the name of a file uploaded at our Box account. But, in between the two calls we will make some changes to the file at the server side and see how the eTags play their part in ensuring consistency.

• Here, we have a dataflow in which we are making a PUT request to update the name of the file associated with the file ID of “983656708053” on our Box account. From the API documentation, we can see that the if-match header field is supported in the PUT /file/{fileid} endpoint of Box APIs so in the API client service options, we will enable the If-Match Header checkbox.

• The API client first checks if an eTag and response corresponding to this endpoint URL already exists in the cache. In the job trace, we can see that there is no eTag in the response caches because we are making the update request for this file for the first time.

• Behind the scenes, the client first makes a Get call to the same endpoint URL and stores the eTag and response in its cache. Next, a PUT request is sent to the same endpoint URL including the eTag received earlier as the value of the If-Match header.

• The server processes the update request and the eTag and the response returned is cached as the eTag received matches to the most recent version of the resource at the server.

This concludes our discussion on the eTag request service options and how they help with response caching and maintaining concurrency control in Astera.

The API Browser provides a convenient option to import pre-built and pre-tested CAPI connectors directly from Astera's GitHub repository.

These connectors are carefully curated and include a comprehensive list of endpoints that have been thoroughly tested and configured for seamless consumer use. This option allows users to easily access and integrate these connectors into their projects, ensuring reliable and efficient connectivity with the associated APIs.

Additionally, Astera offers users an option to build their own custom API connectors. Please visit the documentation here.

1. To start, open the API Browser from the View -> Data Service -> API Browser.

2. Select Import API.

This will open a new window.

3. From the API Import Source dropdown menu, select Custom Connectors.

Selecting this option will bring up a new interface on the same screen.

4. If we open the Connector drop-down menu, we can see a list of available CAPI Connectors.

5. For our use case, we will select the AgileCRM connector.

Astera will automatically create a shared action file and CAPI file in the project, as well as populate the API Browser with all the possible endpoints.

Once the connection is authorized, the endpoints can be used in various flows in accordance with the application. Just drag-and-drop any endpoint to a Dataflow and map any required inputs to use it.

This concludes the working of Custom CAPI Connectors in Astera.

Let’s see what steps are required to import a Postman Collection to the API Browser.

1. Open an Integration Project.

2. Open the API Browser through View > Data Service > API Browser.

3. Click on the Import API option on the API Browser.

This will open the Import API window.

4. Select Postman collection from the API Import Source drop-down.

5. Browse and provide the path to the Postman Collection and click OK.

If there is already a Shared Connection available, then we can re-utilize it, instead of auto-generating a new one, by checking the Use Existing Connection checkbox.

Once the Postman Collection is successfully imported, it will populate the API Browser with the available endpoints.

The Centerprise API file (.capi) and Shared Connection files will automatically generate and be saved in their respective folders.

6. Now, drag-and-drop any endpoint onto a logic designing artefact i.e., a Dataflow to consume.

Postman Collection Format

Variables Convention

To import a Postman Collection to the API Browser successfully, we must follow certain conventions:

The Postman collection must include a variable, namely baseUrl. (This variable is not case sensitive)

All other variables, except for the baseUrl, will be discarded.

During the import, the baseUrl variable defined in all the endpoints will be replaced with the Base Url text box value in the Shared Connection.

This means that the Shared Connection’s Base Url will be populated with the baseUrl variable’s Current Value that is defined under the Variable section in the collection.

Preservation of Authentication Information

All valid Postman Collections will be imported with pre-configured Shared Connections. These Shared Connections will have the same Authentication Type selected as in the collections i.e., API Key, Auth Code, Client Credentials, etc.

Example of an API Key Security Type

Example of an OAuth 2.0 Security Type

Preservation of Endpoint’s Configuration

Upon importing a Postman Collection, each endpoint’s configuration i.e., methods, resources, parameters, and request/response payloads will also be preserved.

HTTP Method and Resource

Parameter

All parameters with their respective default values are populated in the API Client’s Parameter window.

Payload

The input and output layouts/payload are structured in the respective Input and Output Layout windows. Additionally, the sample text bodies used to generate the layouts are preserved in the Sample JSON Text window.

Pagination refers to managing the traffic of records coming from a source. It divides the records into a discrete number of pages so that they are comprehensible for a user.

Pagination is not supported by all APIs. For those that do support it, Astera offers four types of paginations.

Offset

This type of pagination requires two parameters: A Limit and an Offset value to be specified by the user. A Limit specifies the number of records that you want to fetch in a one-page request, and an Offset simply tells the number of records to be skipped before selecting records.

• Offset Parameter: Select the offset parameter of the API that you are working with, as specified on the Parameters screen.

• Initial Offset: The record index from which you want to start your pagination.

• Limit Parameter: Select the limit parameter of the API that you are working with, as specified on the Parameters screen.

• Limit: Number of records on a one-page request.

• Number of Pages: The number of pages indicates the number of request iterations which you want to be processed. Each iterative request incrementally adds the respective offset and limit values for the next set of records page.

• Read Till End: Check this option if you want to fetch all the records. Selecting this will disable the ‘Number of pages’ option and all the records will be returned as requests are sent in a loop till no more data is found.

• Repeating Item: This option is only enabled when you check the Read till end box. You can choose a repeating item or the collection node of the data from the output layout of the API client object. The repeating item helps the API client recognize the end of records, as whenever an empty response node is returned, the client stops sending further requests, and pagination ends.

Cursor

This type of pagination generates a token to indicate a pointer for the next page of records. You can set a limit to the number of pages you want to process.

• Cursor Field: Here, you can specify the field from the output layout which contains the cursor from the server response.

• Cursor Parameter: Here, you can select the parameter to be used to send the cursor value received in the previous request of the API that you are working with, as specified on the Parameters screen. Alternatively, you can choose to send the cursor as an input body layout field by selecting the ‘Use Input Body Parameters’ checkbox.

• Number of Pages: Here, you can specify the number of pages or the number of requests to be made iterating over the data set. Additionally, you can simply check the Read till End option if you want to fetch all records without specifying the number of pages.

Next URL

This type is the same as Cursor pagination, except that it generates a URL instead of a token for every subsequent page.

• Next URL Field: Here, you can specify the field from the response layout that contains the URl to fetch the next set of records.

• Number of Pages: Here, you can specify the number of pages or requests you want to fetch, or you can simply check the Read Till End option if you want to fetch all records without specifying a page number limit.

Page Number

In this type of pagination, you can specify the number of pages you would like to fetch in one go.

• Page Number Parameter: Here, you can specify the page number parameter of the API that you are working with, as specified on the Parameters screen.

• Start Page Number: The page number from where you want to start fetching your output, or the lower limit.

• End Page Number: The page number where you want to end.

• Read till end: Check this option if you want to fetch all the available records. Selecting this will disable the End page number option and make requests till no data is returned.

• Repeating item: This option is only enabled when you check the Read till end box. You will be required to choose a repeating item, which can be one of the collection nodes from the output layout of the API client object. The repeating item helps the API client recognize the end of records, as whenever an empty response node is returned, the client stops reading the response and the pagination ends.

This concludes our discussion of pagination for APIs in Astera.

What is an HTTP redirection?

HTTP redirection, also known as URL forwarding, allows an API to provide more than one URL location to the resource in the response. HTTP redirects usually happen due to temporal or permanent unavailability of the application, website, or pages. For example, unavailability due to server maintenance or re-organization of the URL links.

Redirect responses from the server have a 3xx series HTTP status code along with a Location header parameter that provides the URL to the resource’s new address.

Use Case

In this use case, we have a GET API resource account that returns account details based on the provided ID Query parameter.

Upon previewing the API Client, a request is sent to fetch the account for the given Id. In the response returned the API request is redirected returning a 302 Found status code response indicating that the resource is temporarily unavailable.

It may be due to server maintenance or any other unforeseeable reason. We can see that the Location header parameter is received with the response too. The Value of this header is the address to the alternative resource that must be accessed to retrieve the required account details.

Let’s see how we can configure the API client properties to automatically follow any redirect responses to the new URL Location.

Enable Auto-Redirect Calls

Right-click on the API Client and select properties. Next, navigate to the Service Options window. Here, there are multiple options available to configure the redirect call(s).

• Follow Redirect Calls From 3xx Code Responses – This option allows auto-redirecting a 3xx HTTP response to the redirected location URL.

• Redirect Authentication Information – This option allows forwarding all the authentication details along with the redirected call.

• Redirect Limit – This option allows us to specify a limit to the number of redirected calls followed.

Let’s enable the redirect and authentication options while keeping the redirect limit as 1.

Now, upon previewing the output we can see that a 200 OK status response is received instead of a 302 Found. The request URL field shows that the request was successfully auto-redirected to the redirecting URL.

Now, let’s execute the Dataflow.

Here, we can see the job traces show all steps of the redirected calls including how the authentication information was forwarded along with the request, what was the redirect limit, where the request was redirected to, and if the job executed successfully.

Scenario 1 - No Authentication Information Redirected

1. Let’s consider a scenario where the redirected API requires authentication, and we don’t send the authentication information along with the redirect call by unchecking the Redirect Authentication Information option from the Service Options window.

2. Upon executing the job, we can see that the request is redirected without the authentication information, and as a result, the server sends back a 401 Unauthorized error response.

Scenario 2 (Multiple Redirect Calls)

Now, let’s consider a scenario where an API request hops through more than one redirected call.

1. The first redirect request returns a 3xx series response. We can see in the Job Trace that on redirecting the request we received a 307-status response indicating that the service is temporarily unavailable. As the redirect count was set to 1 so, only one redirect call was sent by the API Client.

2. In such a situation, we need to follow all the redirect requests until a 200 OK response is received. For that, we can increase the Redirect Limit count.

For example, we will set the limit to 2 and send the request.

In the Job Progress window, we can see that two redirect calls have been exhausted, but we still received a 307-status response.

Let’s increase the limit to 3 and send the request.

Finally, a 200 OK response is received on the third redirect call.

This concludes how HTTP redirect calls are automated by the API Client in Astera.

In this article, we will be discussing various HTTP methods. We will see how HTTP requests can be made through the API Client object in Astera.

For our use cases, we have made use of the Petstore Open-API definition. We can import the API to the API Browser using its import URL.

Once done, it automatically establishes various pre-defined endpoints as API Client objects. They can then be dragged and dropped onto a dataflow for further configurations and transformations.

To learn more about importing a URL to the API Browser, click here.

Making a GET Request

1. First, drag-and-drop the Get a file’s metadata or Content by ID endpoint from the browser onto the Dataflow.

In this scenario, we want to get metadata for a file with fileID,

“184Gi7q9iPQyiR6lkG3bdSi5z3-9eeT-d”.

For this, we will pass the relevant fileID using a ConstantValue object.

2. To explore the API Client object for this method, right-click on the object’s header and select Properties.

This will open the API Client screen where the connection info of your API is defined.

The Shared Connection, Method, and Resource here are already configured. Notice that Resource consists of ‘files’ along with the ‘fileid’ URL parameter.

3. Click Next.

Here, the ‘field’ URL parameter follows from the defined resource.

For our use case, we will use this parameter to get details for a pet.

Click Next to proceed to the Output Layout screen, where you can view the Response Layout of your API. There are two ways in which you can generate the output layout if required.

• The first one is by providing sample text by clicking the Generate Layout by providing Sample Text option.

• The other way to do this is by running a request by clicking the Generate Layout by running Request option.

4. Click Next to proceed to the Pagination Options screen.

For our use case, we have selected None.

5. Click OK.

6. You can preview the data by right-clicking on the object and selecting Preview Output from the context menu.

As seen below, the GET request that was made, has fetched data according to the user application.

Making a POST Request

Now, let’s try creating a new file.

1. Drag-and-drop the POST method as an API Client object and open its properties.

We will pass the required parameters to the POST request object using a Variables object.

2. Now, right-click on the API Client object and select Preview Output.

You can see that the HTTPStatusCode is “200”, which means that the API has successfully carried out the action requested by the client.

Let’s verify it by making a GET request for the same FileId that we had posted earlier.

You can see that a GET request for FileID has returned the same information that we had posted.

Making a DELETE Request

Now, let’s try making a DELETE request.

1. For this, we will first make a GET request to check whether that file exists in the records before we try to delete this record.

We will pass a fileId using a ConstantValue object.

2. Right-click on the API Client object and select Preview Output.

It has fetched the details of the file with the fileId and the status shows that the field is available.

To delete this file record, we will drag and drop another DELETE API Client object onto the flow and configure its Properties according to the DELETE method.

3. Pass the fileId to the DELETE request object using a ConstantValue object.

4. Right-click on the API Client object and select Preview Output. You can see that it has returned HTTPStatusCode, “204”, which indicates successful execution.

Let’s verify it by making a GET request again, and check if the fileId, has been deleted.

5. Right-click on the API Client object and select Preview Output.

You can see that Astera has returned error 404 which means that there is no pet found with PetId, “5”, and the pet record has been successfully deleted from petstore API.

You can see that Astera has returned error 404 which means that there is no fileId found, and the file record has been successfully deleted from Google Drive API.

Making a PUT Request

Let us now look at the PUT HTTP Method.

1. Drag-and-drop the GET endpoint from the API Browser onto the Dataflow.

2. Right-click on the object and select Properties from the context menu.

3. Click Next, and the Parameters screen will appear.

For this use case, we will update the file with a fileId. Let’s define this ID in the Default Value field.

4. Click OK and preview the output by right-clicking on the object and selecting Preview Output

As you can see in the preview screen below, the GET method has retrieved the file Metadata by ID.

5. Now, drag-and-drop the relevant endpoint from the API Browser onto the Dataflow.

For our use case, we will be using this Patch object for the PUT method so we can update the ID.

Right-click on the object and select Properties from the context menu.

Our Shared Connection has already been defined. The HTTP Method is Put, and the Resource to update is a file.

6. Click Next, and you will be led to the Output Layout screen

We have defined the fileId here that we wish the resource to be updated to,

If required, an output can be generated by running a request using the available option.

7. Click OK, right-click on the object, and select Preview Output.

As you can see here, the fileId has been updated,

8. We will now preview the output of the GET object we have configured to verify if the pet status has been updated.

As you can see, the value has been updated.

Making a PATCH Request

Let’s make a GET request to see what information is there in the File ID where we want to update something.

1. To make a GET request, drag-and-drop the GET API Client object onto the Dataflow.

2. Pass userID ‘1pGLAWbY7zu1nYFjMFB5GmTjVK2kXGHP1’ to the id under the Parameters node in the API Client object using the Variable transformation object.

3. Right-click the API Client object’s header and select Preview Output.

Here is what the output looks like:

4. Drag-and-drop the Update a file’s metadata API Client object to use the PATCH method.

5. Pass fileId’ 1pGLAWbY7zu1nYFjMFB5GmTjVK2kXGHP1’, and name, “Astera”, using a Variables resource object.

6. Right-click on the object’s header, and select Preview Output.

You can see that the HTTPStatusCode is 200, which means that the API has successfully carried out the PATCH request. Let’s verify it by making a GET request for the same fileId which we altered.

7. Right-click the APIClient object’s header and select Preview Output.

As you can see, the request has been successfully carried out and the email address has been updated.

This concludes our discussion on the HTTP method operations in Astera.

Raw Preview Request/Response

The raw request and response preview features allow API developers to view the exact request and response payloads being exchanged between clients and servers in their APIs.

This feature provides a detailed look at the headers, body, parameters, and metadata of the HTTP request and response, which can help API developers debug issues, test APIs, and optimize performance. By using raw preview request and response capabilities, API developers can gain a deeper understanding of how their APIs are being used and troubleshoot issues quickly and efficiently.

Raw Preview in Astera

Astera lets the user preview the Raw request and Raw response both from the API Client object.

1. Drag-and-drop an API Client object and configure it.

For our use case, we have used an API Client making a GET Call to a resource.

2. Right-Click on the object and select Preview Raw Request.

This will show the raw request in the Raw Data Preview window.

As you can see, it has shown the HTTP method as well as the resource, host server details, and the Content-Type of the Request.

It even shows us tabs on the Request, Parameters, and Body.

3. To preview the raw response, right-click on the API Client object and select Preview Raw Response from the context menu.

This will generate a raw response in the Raw Data Preview window.

As you can see above, the raw response has been generated, which shows us the entire HTTP response in raw form. It even has tabs that show us the Parameters, body, and response info.

CURL Command

Curl is a command-line utility that can be used to send HTTP requests to APIs and retrieve the respective responses.

It allows API developers and testers to easily interact with APIs and perform tasks such as testing, debugging, and troubleshooting. Curl supports various HTTP methods such as GET, POST, PUT, and DELETE, and can handle HTTP headers, cookies, and authentication.

It is a simple, yet powerful tool that is widely used in API development and management.

Copy CURL in Astera

Astera lets the user copy and view the CURL command from the Raw Data Preview window to help in comparing and debugging results from any external clients such as Windows command prompt or Postman.

This concludes Raw Preview and Copy CURL in Astera.

The API Client now supports the text/xml content type, allowing for seamless integration of XML-based data into API requests and responses. This enhancement enables users to effectively interact with APIs that require XML-formatted data or supply XML responses.

To use the text/xml content type in the Astera Client, you can now send and retrieve data in XML format. This content type allows you to exchange data using XML-based representations.

XML, being a widely adopted format for hierarchical data structuring, supplies a standardized approach for representing and exchanging information between various systems. With the text-xml content type in the API Client, users can make SOAP API requests in integration flows.

Use Case

To start, let’s make an API request using text/xml content type in the API Client, which allows you to send and retrieve data in XML format.

1. To start, drag-and-drop the API Connection and API Client objects onto the Dataflow.

2. Configure the API Connection with the Base URL and Authentication.

We are calling a SAP Success Factor API as an example.

3. Next, open the properties of the API Client object and select the configured shared connection from the drop-down.

4. Choose the POST method and define the resource to be appended after the base URL.

5. To define an xml input and output, select the input and output content types as text/xml.

6. Proceed to the next screen to define any parameters.

7. Click Next to navigate to the Input Layout screen.

On the Input Layout screen click on Generate Layout by Providing Sample Text and supply the raw xml content to be sent in the request.

Click Generate to automatically create the layout for the request, allowing you to supply any default values or supply mappings from the preceding flow.

8. Click Next and you will be led to the Output Layout screen.

Here, there are two options to create a layout, by supplying sample text or running a request to generate the layout.

9. Right-click on the API Client object and select Preview Output.

In the Data Preview window, we can see the following result:

This concludes seamlessly integrating XML SOAP API requests and responses, enabling smooth data exchange and integration with APIs.

API logging is the process of keeping track of how an application programming interface (API) is being used.

It helps in understanding how often the API is being used, how long each request takes, and any errors that occur. API logging can be used for troubleshooting, monitoring performance, and identifying security threats.

API Logging in Astera

Astera allows the user to enable API Logging. There are three types of logging that are offered in the tool:

• Authentication Logs

• Incoming Logs

• Outgoing Logs

1. To view API Logs in Astera, right-click on the Server node in the Server Explorer.

2. Select API Logs.

This will open a new window.

On the left-hand side of the window, a list of the API calls that have been logged will appear.

Logs can be filtered based on a date range as well as type.

There are three types of logs that can be filtered.

Authentication:

These are logs that are created during authentication.

This option can be enabled in the API Connection object by selecting the Enable Authentication Logs checkbox.

Incoming:

These are the API calls that are made to all APIs deployed on the Astera Server from either a third-party application such as Postman or the Astera Client itself.

Outgoing:

These are the API calls that have been made from an API Client object present in the flows.

1. To enable Outgoing API call logging, open the Service Options: Request Options screen in the API Client object.

2. Select the checkbox at the bottom of the screen that says Enable API Logging.

It is unchecked by default.

Selecting this option is going to enable all outgoing API call logging.

We can see all outgoing API calls in the log, alongside whether they have ended in success or failure.

On the right-hand side of the screen, two tabs can be seen.

The Overview tab gives a list of details regarding the selected API in the log.

It gives various details including Client IP, Remote IP, Content-Type, Status, and much more of both the Request and the Response.

The Inspectors tab gives the raw information regarding the Request and Response of the selected API call within the log.

The purge frequency of API logging can be set through the Cluster Settings through the Server Explorer.

This concludes API Logging in Astera Data Stack.