How to use one Object Store v2 with multiple TTLs (time to live) in Anypoint Studio

This article describes how to implement a Token Expiration Strategy in Anypoint Studio that handles multiple TTLs (time to live) based on key-value entries using one single Object Store Connector, for example, storing multiple access tokens inside a single Object Store and enforcing the expiration strategy for each token according to their assigned TTL (time to live).

Introduction

The Object Store Connector is a Mule component that allows the storage of key-value and it can be used for many different purposes. One of the uses of the Object Store is to store access tokens. 

The main idea behind access token storage is to promote the reuse of the token to reduce unnecessary roundtrips to the authentication server and avoid overloading the authentication server with consecutive calls to retrieve the token.

Rather than requesting a new token, use the stored token during future calls until it expires. 

However, it is essential to implement a Token Expiration/Revocation Strategy to retrieve a new valid token from the authentication server.

This article describes how to implement a Token Expiration Strategy in Anypoint Studio that handles multiple TTLs (time to live) based on key-value entries using one single Object Store Connector, for example, storing multiple access tokens inside a single Object Store and enforcing the expiration strategy for each token according to their assigned TLL (time to live).

Problem statement

Object Store v2 does not offer the option to specify a TTL (time to live) for each key-value entry stored in the Object Store. To resume, each key-value pair stored inside the Object share the same TTL. Consequently, constraining the use of multiple Object Stores to handle multiple TTLs.

It is noteworthy to mention that some operations on the Object Store such as storing a new key-value pair or updating an existing key-value entry, reset the TTL for all key-value pairs inside the Object Store. Due to the reset of the TTL, the Object Store won’t be able to distinguish between expired and valid tokens. 

Solution

The “Token Expiration Strategy” is a mule flow based on logical and conditional checks to verify for relevant access tokens if their validity time has not been exceeded. This flow is triggered whenever the access tokens are being requested.

The flow in charge of enforcing the Token Expiration Strategy follows the following conditional schema depicted in Figure 1.

Figure 1: Token Expiration Strategy’s Conditional Schema

First of all, it is necessary to check if the token is already stored in the Object Store otherwise a new token must be requested and stored.

In case the token already exists, then its validity time is calculated and compared to the desired TTL. In other words, if it is valid we can reuse the access token straight away otherwise we must request a new token and store it inside the Object Store.

This solution makes use of custom properties file that stores key-value pairs such as <“tokenName”: “custom TTL”>. This properties file is read in the Token Expiration Strategy flow to retrieve the TTL for a specific access token.

Implementation

This section will describe every step to implement the “Token Expiration Strategy” in Anypoint Studio.

Creation of a properties file (.yaml)

As I already mentioned before, the creation of a properties file (.yaml) is necessary to define key-value pairs that use the access token’s name as the Key and the access token’s TTL as its value. This properties file is read in the Token Expiration Strategy flow to retrieve the TTL for a specific access token. Figure 2 depicts an example of the configuration file.

Figure 2: YAML Configuration file 

Note: This solution uses Object Store Connector’s Store operation that will store a given Key and value in the Object Store. This Key will be named according to the access token’s names defined in our properties file.

Creation of an Object Store

When creating the Object Store several values should be set as shown below:

• Max Entries: Represents max no. of entries allowed in the object store.

• Entry TTL: Represents Entry timeout (after what time an entry times out).

• Entry TTL Unit: Entry time to live unit (Seconds/Minutes/Hours/Days).

• Expiration Interval: The frequency at the expiration thread should run.

• Expiration Interval unit: The expiration interval.

• Persistent: When checked the data is stored in the disk, otherwise in-memory (data lost when the app restarts or crashes).

When setting the values several aspects must be taken into account, such as, how to choose the Entry TTL to handle multiple TTLs. The value for this particular field must be equal to or greater than the highest Token’s TTL defined in the previously created properties file.

This requirement must be fulfilled to avoid the scenario where the Object Store Connector reaches the established Entry TTL e.g. “3500 seconds” and marks all tokens as expired whereas some tokens might have a higher TTL, such as “7000”, as defined in the properties file. 

Rolling or Static TTL

Let’s create a mule flow as shown in the below image, composed of an HTTP Listener, a Logger that prints the start of the flow in the logs, two Set Variable Connectors, a Flow Reference to the “Token Expiration Strategy Flow” and another Logger that prints the end of the flow in the logs. 

Figure 4: Simple Interface

The first “Set Variable” component is in charge of storing the token’s name, in our case, it is sent as a query parameter.

tokenName: attributes.queryParams.tokenName

The second  “Set Variable” component stores the TTL for a specific access token based on the access token name stored previously. The TTL is stored in our custom properties file. Using Dataweave’s lookup function p() it is possible to read values from our custom properties using our previously defined property placeholder.

In this solution, our property placeholders correspond to the concatenation of the root node of our custom property placeholders, namely token_tll, and the access token’s name, as shown below.

ttl: p('token_tll.' ++ vars.tokenName

Object Structure to store

This section strives to describe the object as well as the required information that is needed to ensure the success of the Token Expiration Strategy.

The object that we want to store and keep inside our Object Store is the token and the object’s creation DateTime which can be created using Dataweave’s now() function. An example of this object is shown below.

Figure 5: Object to Store Containing Token and DateTime 

On entering the values for the Key and value that we want to store, our Store Operation will look like Figure 6. Where the Key is specified by the access token name and the value takes our token object. 

Figure 6: Object Store’s Store Connector 

Generate Token Expiration Strategy Flow 

This section shows how to generate the main flow responsible for the Expiration Strategy which must handle the following use cases:

• The key does not exist

• The key exists but the access token is expired

• The key exists and the access token is still valid

 Figure 7: Token Expiration Strategy Flow

Figure 8: Get New token Flow

 

The key-expiration-strategy_flow starts by retrieving the token from the Object store using the “tokenName” as the Key. Note that we use an empty String ““ as the default value in case the key is not present in the Object store. Thus, avoiding the use of the Object Store v2  “Contains” Connector to verify if the key exists inside the Object store as well as avoiding a “Key Not Found” error being thrown. 

 

Figure 9: Object Store’s Retrieve Connector

After retrieving the token, we check using a Choice Router if the token has been retrieved or the default value “”. A new token must be requested and stored if the result of the Retrieve Connector is an empty String “”.

If the token exists in the Object Store then its validity interval must be processed and verified. According to the code snippet below, vars.token_object.date corresponds to the token's insertion time in the Object store after being requested.

(now() - (vars.token_object.date as DateTime)).seconds > ((vars.ttl as Number) - 200)

To verify if the token is still valid we need to subtract the time which is stored with the token inside the Object Store from the current time, namely now() function. The result is then converted in seconds and compared if it is greater than the defined TTL for this token. 

Note that during the process we deduct 200 seconds from the initial TTL. The main idea behind this operation is to take into consideration delays that can occur until the actual call, such as the time from requesting a new token and storing it.

In this manner, we make sure that the token is still valid when the request is sent.

It is important to point out that the “Token Expiration Strategy” does not remove any key from the Object Store. Instead, it re-uses the store operation to update/overwrite any pre-existing value associated with a key.