How to implement Pagination and Page Size with Azure Cosmos DB

Introduction

This article describes how to implement pagination with Azure Cosmos DB REST service using max item count and continuation token to handle cases where the query result spans multiple pages.

The query engine can use the continuation token to continue the query without having to repeat previous executions and retrieve the next page according to the given page size. 

Scope

This article proposes a possible pagination implementation example with Azure Cosmos DB and focuses only on pagination-related aspects, excluding other types of configuration or explanations.

Problem Statement

At the current state Azure Cosmos DB Connector 1.0 does not support pagination handling externally (outside its connectors). According to the connector’s documentation, connectors such as List Documents and Query Documents, obtain a list of existing documents based on the specified query, with pagination support handled internally by the connector.

As a result, the connectors will always return the entire result without the possibility of determining the number of pages to retrieve or fetch the next pages.

Solution

To achieve a successful pagination implementation it is necessary to use HTTP Request Connector to call the Azure Cosmos DB REST service instead of using Azure Cosmos DB Connectors for operations where pagination is available, namely List Documents and Query Documents.

When calling Azure Cosmos DB REST service using HTTP Request Connector it is possible to pass relevant headers for the pagination implementation. Table 1 shows the minimal required headers necessary to query Azure Cosmos DB REST API as well as pagination-related headers such as “x-ms-continuation” and “x-ms-max-item-count”.

More information about request headers to query the REST API can be found in the official documentation of Azure Cosmos DB.

Header	Required	Type	Description
Authorization	Required	String	The authorization token for the request.
Content-Type	Required (on PUT, PATCH, and POST)	String	For POST on query operations, it must be application/query+json.  For PATCH operations, it must be application/json_patch+json.
x-ms-continuation	Optional	String	Token to retrieve the next page of results by resubmitting the request with this token.
x-ms-max-item-count	Optional	Number	An integer indicating the maximum number of items to be returned per page.
x-ms-date	Required	Date	The date of the request per RFC 1123 date format expressed in UTC time.
x-ms-documentdb-query-enablecrosspartition	Optional	Boolean	Required for operations against documents and attachments when the collection definition includes a partition key definition.
x-ms-version	Required	String	The version of the Cosmos DB REST service.

Table 1: Small List of Azure Cosmos DB REST Request Headers 

Implementation

This section describes one way to implement pagination and page size with Azure Cosmos DB.

Figure 1 depicts an interface example composed of an HTTP Listener, two Set Variable Components, a Transform Message, an HTTP Request, and a Logger. 

Figure 1: Basic Interface Example

 

On the interface, we preemptively allocate a variable to store the continuation token used to retrieve the next page of a given request. The continuation token is present in Cosmos DB REST API’s response after the first call if the result spans multiple pages. In that case, the continuation token present in Cosmos DB REST API’s response can be retrieved as follow:

 

attributes.headers.'x-ms-continuation'

 

Note: To retrieve the next page using the continuation token it is important to resent the same request including the previously retrieved continuation token which means the query and parameter must be identical.

 

attributes.queryParams.pageSize.

 

The header name to tell the page size to Azure Cosmos DB is named "x-ms-max-item-count".

 

 

 Figure 2: Transform Message with Header definitions

 

 

Our implementation example contains a “Transform Message” component in charge of setting the required headers as well as pagination-related headers. The headers’ names must comply with those described above in Table 1. All the headers are stored in a target variable, namely “request_headers” and must be provided in the “header” section in the HTTP Request Connector as shown in Figure 2.

Note: That skipNullOn=”everywhere” declaration is used in case no continuation token is provided in the incoming request. In that manner, the field for the continuation token will be excluded and this header will not be present on the request.

 

Figure 3: HTTP Request with Custom Headers