Pagination

Many of the exposed collections in the Connect API return a significant amount of data. This can cause timeouts during requests because of the time to retrieve and serialize the data, as well as the amount of data that needs to be transmitted.

The way pagination is exposed in the Connect API is consistent among all types of collections. This makes the API more intuitive to use. The order in which the data is returned might differ depending on the nature of the data. Historical data has less relevancy than current active data. This must be considered in the implementation of each collection, so that the implementation serves the needs of clients.

Implementation Proposal

The Connect API interface is implemented by several applications, using different technologies and approaches. Proposing a rigid solution that applies to all of them is a daunting task. A better approach is to provide an interface that clients can adapt to their needs and provide guidelines to implement that interface.

In the following section, we will look at the proposed interface and describe the intention.

Pagination Interface

The following proposal applies to all the collection endpoints in the Connect API and explains the inner workings of pagination within the Connect API.

ℹ️ The examples use a generic entityto avoid specific representations that already exist in the Connect API.

The Connect API exposes several endpoints that return a collection of entities. The structure of these collections is similar. Assuming a collection of entities a client will retrieve them with the request GET https://core.envaseconnect.cloud/entties. The response will have the following structure:

{
    "self": "/entities",
    "items": [
        {
            "id": "1",
            "self": "/entities/1"
        },
        {
            "id": "2",
            "self": "/entities/2"
        },
        {
            "id": "3",
            "self": "/entities/3"
        }
    ]
}

Collection responses always have a self property pointing to the location of the collection, and an items property that contains the entities of the collection. The items folder currently returns all the entities, and the pagination interface will modify that behavior.

The pagination interface changes the response to provide feedback to clients that additional pages are available. Let's take a look at the response for the default request.

Default Request

When a client sends the default request to a collection endpoint (GET /entities), a default item limit is used, and the first page will be returned:

{
    "self": "/entities?limit=100",
    "next": "/entities?offset=100&limit=100",
    "items": [
        {
            "id": "1",
            "self": "/entities/1"
        },
        {
            "id": "2",
            "self": "/entities/2"
        },
        {
            "id": "3",
            "self": "/entities/3"
        }
        ...        
    ]
}

In the example, the implementation chose to use a limit of 100 entities in the response, so the self property value indicates this. The response also provides the query to retrieve the next page as the next property. The response will include a maximum of limit items in the array.

Specifying A Limit

A client might decide to specify a different limit than the default used by the implementation. This can be done by specifying the limit query parameter in the request (GET /entities?limit=200). The implementation must honor the parameter and adjust the response accordingly:

{
    "self": "/entities?limit=200",
    "next": "/entities?offset=200&limit=200",
    "items": [
        {
            "id": "1",
            "self": "/entities/1"
        },
        {
            "id": "2",
            "self": "/entities/2"
        },
        {
            "id": "3",
            "self": "/entities/3"
        }
        ...        
    ]
}

The self and next properties are now adjusted to the request, and the array will contain as many as 200 items if they exist.

Paginating Using Offset

The offset query parameter allows a client to move through the data pages. The offset indicates the number of records from where the first record should be returned. When it is not specified, it would be assumed to have a value of 0.

⚠️ The offset is not based on the ID of an entity. IDs can be numeric but they are not numbers.

Once a client has specified a limit (or used the default), the next page will be retrieved by setting the offset to the previous offset plus the limit size (offset = offset + limit). A client will likely perform the following request to paginate through data:

GET /entities?limit=300 // Initial call sets the limit to 300, offset is assumed to be 0
GET /entities?limit=300&offset=300 // Next page is at offset 300 (0 + 300)
GET /entities?limit=300&offset=600 // Next page is at offset 600 (300 + 300)
GET /entities?limit=300&offset=900 // and so on...

⚠️ Requesting an offset or limit that goes beyond the number of records, or an offset + limit that goes beyond the number of records is not an error, and the response should include the partial data that can be retrieved. If there's no data in the requested range, an empty array should be returned.

Stopping Pagination

The response includes the property next which points to the URL and query to retrieve the next page. A client can assume that if the next property is empty, there is no more data. If data is queried beyond that point, the implementation should not return an error, and the response should include an empty next property and an empty items array.

The following example shows the response of the last page:

{
    "self": "/entities?offset=3400&limit=200",
    "next": "",
    "items": [
        {
            "id": "3401",
            "self": "/entities/3401"
        },
        {
            "id": "3402",
            "self": "/entities/3402"
        },
        {
            "id": "3403",
            "self": "/entities/3403"
        }
        ...        
    ]
}

API Implementations

The implementation details for pagination are left to the individual TMS applications provided they follow the specified interface. This means that details like page sizes can change from one implementation to another. Still, it's recommended that applications follow the recommendations below in their implementations.

General Collections Implementation

Most collections in the API will implement pagination by providing the data in a consistent order. This allows the same call to return the same data every time. The recommendation is that the data is sorted by id in the cases where this value makes sense, or better yet in creation order. Since id values do not change and neither does the creation date/time, this guarantees that a call to the same pages will return the same data, unless the data is allowed to be deleted.

Shipments API

As TMS systems are used, the number of shipments increases. Shipments with a lower id or creation date/time become less relevant and are kept for historical purposes.

Because of this, most clients will want to retrieve the latest shipments first, meaning those with a higher id or creation date/time. For that reason, the Shipments API should return data in reverse order, and pagination should be implemented so that the latest data is returned within the first page.