Request configuration

type

Type of the request. Defines if the request should be performed as a synchronous request or use an asynchronous request flow.

Possible values

• sync

• async

Default value: sync

method

Defines the request method to use for the request.

Available methods:

• GET

• POST

• PUT

• DELETE

url

Defines to which URL the request is made. For example, the endpoint from which data is fetched, like https://dummyjson.com/products

Note: You can use placeholders within the URL to apply dynamic values to the request URL. For example, https://exampleapi.com/{{account.id}}/products

headers

Under headers, you can define an array of headers to be sent as part of the request. Headers are defined as key-value pairs, like below.

You can configure multiple headers, and header values can be configured as static values, like the agent below, or you can use placeholders within headers.

"headers": [
                            {
                            "name": "Authorization",
                            "value": "{{user.access_token}}"
                            },
                            {
                            "name": "agent",
                            "value": "supermetrics"
                            ]

parameters

Similar to headers, under parameters, you can define an array of parameters to be sent as part of the request. Request parameters are added to the request url in the format {key}={value}.

Parameters are defined as key-value pairs, like below.

You can configure multiple parameters, and parameter values can be configured as static values, like the country code below, or you can use placeholders within headers.

"parameters": [
                            {
                            "name": "starting_from",
                            "value": "{{query.start_date.date}}"
                            },
                            {
                            "name": "country_code",
                            "value": "US"
                            }
                            ]

body

The body defines data that is sent in the body of the request. Under the body object, you can define further settings for the response body:

• type: The content type of the body defines what kind of data the body contains. The type can be none, text, json, xml, form, or urlencoded_form.

• formData: an array of key-value pairs which contain the form data. Structured in the same way as headers and parameter arrays.

response

The response object defines how the response received from the API is handled.

• What format is the response in

• Where in the response is the relevant data

• Whether the response is manipulated with data adapters

• How the responses are paginated

Please see the additional documentation on response handling below.

type

Type of the response. Defines the format in which we expect the API response to be.

Possible values

• JSON

• JSONL

• XML

• CSV

• CSVZipped: Use when the response is a zip file containing a CSV file

• JSONArray

• JSONGzipped: Use when the response is a gzip file containing JSON-formatted data

• BigQuery: Response handler tailored specifically for the data format used by the BigQuery API.

Default value: JSON

dataRows

Data rows define where to read data from. Usually, this points to an array of elements.

For example, a response might look like:

{
                            "data": [
                            { /* row 1 */ },
                            { /* row 2 */ },
                            ...
                            ]
                            }

To be able to fetch data from the rows, dataRows would be defined as $.data.*

pagination

Defines how to paginate through the pages when the response contains multiple pages of data.

Please see pagination below for further documentation.

Cursor

Cursor pagination fetches next page based on a cursor (also called a token) provided by the previous page.

• type: Defines the pagination type used

• maxResultsPerPage: Defines the expected max results and is used to detect the last page

• nextPageParameterName: The next request will have this parameter

• nextPagePath: Where to find the value of the cursor for the next page in the response

Example configuration:

"pagination": {
                            "type": "cursor"
                            "maxResultsPerPage": 1000,
                            "nextPageParameterName": "next_page_token",
                            "nextPagePath":
                            "id": "",
                            "source": "jsonPath",
                            "value": "$.data.next_page_token"
                            },
                            },
                        

In URL cursor

Same as Cursor but cursor is always sent as request URL parameter regardless of request type (GET or POST).

Example configuration:

"pagination": {
                            "type": "in_url_cursor"
                            "maxResultsPerPage": 1000,
                            "nextPageParameterName": "next_page_token",
                            "nextPagePath":
                            "id": "",
                            "source": "jsonPath",
                            "value": "$.data.next_page_token"
                            },
                            },

Row count

Row count pagination fetches new pages until one of four stop clauses is reached:

1. The page response returns fewer rows than what’s configured in maxResultsPerPage

2. The page response returns a greater-than-or-equal number of rows than the total rows value

3. The total rows value is less than or equal to the calculated current page * maxResultsPerPage

4. We go over the configured maxPages of the pagination config.

Example configuration:

{
                            ...
                            "pagination": {
                            "type": "row_count",
                            "nextPagePath": "page"
                            "maxResultsPerPage": 1000,
                            "maxPages": 100,
                            "rowCountPath":
                            "source": "jsonPath",
                            "value": "$.total_rows"
                            }
                            }
                            }

Offset

Offset pagination counts all the result rows, and then adds the total number of fetched rows to a variable, which is sent to the data source with the next request.

Example configuration:

{
                            ...
                            "pagination": {
                            "type": "offset",
                            "nextPageParameterName": "offset",
                            "maxResultsPerPage": 5,
                            "maxPages": 10
                            }
                            ...
                            }