Storing data

The Finsemble storage client interfaces with the storage service to provide access to a central location for persistent storage and retrieval of application data.

The storage client delivers one form of cross-origin resource sharing (CORS), specifically cross-origin storage. However, by using our storage client, different data stores can be transparently used:

• Multiple users can be supported
• Future storage enhancements can be inherited (e.g., storage backup, migration, permissions)
• You can directly access HTML5's localStorage, IndexedDB, and other APIs from your component window

The Storage Client API supports a key-value schema. Each call to get and save must include a key.

All data is keyed off of a username, which is specified by the Authentication Service invoking the [StorageClient.setUser method]{@link StorageClient#setUser} during startup. Because the username is always set automatically, you never need to do it. If authentication is disabled, the underlying Authentication Service sets the username to defaultUser.

The high-level storage architecture looks something like this:

Topics

Key-value is a simple concept. However, sometimes you need a little more flexibility than key-value provides. To allow further partitioning of data, we allow you to pass in a topic. Topics act like a namespace and help prevent key conflicts. All of Finsemble's core storage uses one of the three topics described below. You can use any topic you want (such as an application or company name).

Topics can be mapped to specific storage adapters. For example, you could have a component that has a MySQL backend and another with a Redis backend. It's up to you.

By default, Finsemble saves to three topics which you can extend via config:

• finsemble.workspace is storage for the workspace and is only accessed when a user initializes workspaces or explicitly saves them.
• finsemble.workspace.cache is temporary storage for high-frequency updates (e.g., window moves, resizes, opens, closes).
• Finsemble is a general topic that contains everything outside of the workspace. User preferences fall under this topic.

Configuration

In your Finsemble config, you can specify storage adapters in the "servicesConfig.storage" setting. You can (and should) define your own topics for storage. The topics you define can be mapped to any adapter by inserting a corresponding entry under the topicToDataStoreAdapters property inside config.json. The property name (e.g., finsemble.workspace) is the topic, and the value is the adapter (e.g., IndexedDB). You can specify the default adapter using the defaultStorage setting:

"servicesConfig": {
    "storage": {
        "defaultStorage": "IndexedDBAdapter",
        "topicToDataStoreAdapters" : {
            "Finsemble" : "IndexedDBAdapter",
            "finsemble.workspace" : "IndexedDBAdapter",
            "finsemble.workspace.cache" : "IndexedDBAdapter"
        },
        "dataStoreAdapters": {
            "LocalStorageAdapter": "$applicationRoot/adapters/localStorageAdapter.js",
            "IndexedDBAdapter": "$applicationRoot/adapters/indexedDBAdapter.js"
        }
    }
    .....
}

Storage config is processed during the initialization of the Storage service, so all related config settings take place before any storage clients are initialized. If a topic doesn't have a storage adapter in the config, it defaults to the "defaultStorage" value. If no "defaultStorage" value exists, it then it defaults to localStorage.

---

Custom storage adapters

The storage service receives requests from the storage client to save data under a particular topic and key. The storage service receives the request, determines which adapter should handle the request, and passes on the information. When the adapter has completed the operation, the storage service returns a message that everything is done.

If you want to use an adapter that uses a remote database (e.g., MongoDB, Oracle's relational database, and so on), you simply need to build an adapter that converts key-value requests into something that your backend can handle.

In general:

1. To build your adapter in the seed project, add its code to the folder /src/adapters. You can copy the localStorageAdapter from /src/adapters/localStorageAdapters.js or use the sample code below.
2. Next, you must configure Finsemble to use your custom adapter in /configs/application/config.json under the finsemble.servicesConfig.storage section.
3. Finally, you must create an entry for the build process in the file /build/webpack/adapters.entries.json.

note

The localStorageAdapter and IndexedDB adapter use the BaseStorage adapter to generate keys to store data. The BaseStorage adapter combines several variables available to it and can inherit the set methods for a couple of those variables, namely: setBaseName() and setUser(). The compound key generation function is simply:

// return full underlying key (based baseName + userName + topic + key) this.getCombinedKey = function (self, params) {     return self.baseName + ":" + self.userName + ":" + params.topic + ":" + params.key; };

We provide 2 storage adapters that you can use in your own implementation, or you can modify to fit your needs.

• MongoDB storage adapter uses a REST API and sends JSON payloads
• AWS storage adapter is similar but focused on AWS.

Creating a sample storage adapter

To illustrate how easy it is to create your own adapter, we're going to create an InMemoryStorageAdapter. Basically, the adapter will save data to memory. When you quit the app, nothing will be saved. This has limited practical purpose, but it neatly illustrates the ideas and code required to build a functional (and useful) storage adapter.

We don't want to break our example project, so we're going to write data to a custom topic. All data saved on that topic will go to our InMemoryStorageAdapter.

Create the file

Create a file called InMemoryStorageAdapter.js in the folder src/adapters.

Here's some sample code to get started:

/*
* The baseStorage model provides several utility functions, such as `getCombinedKey`, which will produce a compound key string (for use with a simple key:value store) incorporating the username, topic, and key. For example: For the default user, Finsemble topic and activeWorkspace key: `Finsemble:defaultUser:finsemble:activeWorkspace.`
.
*/
var BaseStorage = require("@finsemble/finsemble-core").models.baseStorage;
var {
    Clients: { Logger },
} = require("@finsemble/finsemble-core");
//Because calls to this storage adapter will likely come from many different windows, we will log successes and failures in the central logger.
Logger.start();

const InMemoryStorageAdapter = function () {
    // #region Initializes a new instance of the InMemoryStorageAdapter.
    BaseStorage.call(this, arguments);

    this.myStorage = {};

    /**
     * Save method.
     * @param {object} params
     * @param {string} params.topic A topic under which the data should be stored.
     * @param {string} params.key The key whose value is being set.
     * @param {any} params.value The value being saved.
     * @param {function} cb callback to be invoked upon save completion
     */
    this.save = (params, cb) => {
        // Implement here
        return cb(null, { status: "success" });
    };

    /**
     * Get method.
     * @param {object} params
     * @param {string} params.topic A topic under which the data should be stored.
     * @param {string} params.key The key whose value is being set.
     * @param {function} cb callback to be invoked upon completion
     */
    this.get = (params, cb) => {
        // Implement here
        return cb(null, null);
    };

    /**
     * Returns all keys that we're saving data for.
     * @param {*} params
     * @param {*} cb
     */
    this.keys = (params, cb) => {
        // Implement here
        return cb([]);
    };

    /**
     * Delete method.
     * @param {object} params
     * @param {string} params.topic A topic under which the data should be stored.
     * @param {string} params.key The key whose value is being deleted.
     * @param {function} cb callback to be invoked upon completion
     */
    this.delete = (params, cb) => {
        // Implement here
        cb();
    };

    /**
     * This method should be used very, very judiciously. It's essentially a method designed to wipe the database for a particular user.
     */
    this.clearCache = (params, cb) => {
        // Implement here
        cb(null, { status: "success" });
    };

    /**
     * Wipes the storage container.
     * @param {function} cb
     */
    this.empty = (cb) => {
        // Implement here
        cb();
    };
};

InMemoryStorageAdapter.prototype = new BaseStorage();
new InMemoryStorageAdapter("InMemoryStorageAdapter");
module.exports = InMemoryStorageAdapter;

note

You only need to implement the save, get, keys, and delete functions. The functions clearCache and empty are optional, but can be useful for clearing out data during testing; they should probably be disabled on the server side in production.

Config

Right now this file is doing nothing. To make it useful, you need to modify some code. First, all requests for the topic IntraSession should go to the new adapter. Second, you need to modify where the actual adapter sits (under dataStoreAdapters). Lastly, you need to put the file in the build.

To do this, go to config.json and change servicesConfig.storage as below.

    "storage": {
        "topicToDataStoreAdapters" : {
            "Finsemble" : "InMemoryStorageAdapter",
            "finsemble.workspace" : "InMemoryStorageAdapter",
            "finsemble.workspace.cache" : "IndexedDBAdapter",
            "IntraSession": "InMemoryStorageAdapter"
        },
        "dataStoreAdapters": {
            "LocalStorageAdapter": "$applicationRoot/adapters/localStorageAdapter.js",
            "IndexedDBAdapter": "$applicationRoot/adapters/indexedDBAdapter.js",
            "InMemoryStorageAdapter": "$applicationRoot/adapters/InMemoryStorageAdapter.js"
        }
}

note

In practice, you should configure your new adapter for use for the finsemble and finsemble.workspace topic. When the user does a formal save operation, the data is copied from the finsemble.workspace.cache topic to the finsemble.workspace topic, copying it to remote storage. For the purposes of this example, the InMemoryStorageAdapter will not persist data.

Build

Next, if you are using the built-in Webpack/gulp build system provided by Finsemble, add the following definition to your build/webpack/webpack.adapters.entries.json file:

    "InMemoryStorageAdapter": {
        "output": "adapters/InMemoryStorageAdapter",
        "entry":"./src/adapters/InMemoryStorageAdapter.js"
    }

Implement adapter functions

To start, you have a member variable on our class called myStorage. It's a simple object that does nothing interesting. To make the adapter have simple functionality, give it the ability to save and retrieve data. Add in the following code in the appropriate places. Take a second to read through what's going on here before implementing these two methods in your src/adapters/InMemoryStorageAdapter.js.

this.save = (params, cb) => {
    //Retrieves a key that looks like this:
    //applicationUUID:userName:topic:key
    const combinedKey = this.getCombinedKey(this, params);

    //Assign the value to the key on our storage object.
    this.myStorage[combinedKey] = params.value;

    return cb(null, { status: "success" });
};

this.get = (params, cb) => {
    const combinedKey = this.getCombinedKey(this, params);
    const data = this.myStorage[combinedKey];
    return cb(null, data);
};

this.keys = (params, cb) => {
    const keyPreface = this.getKeyPreface(params);
    const allKeys = Object.keys(this.myStorage);
    const keys = allKeys.filter((key) => key.startsWith(keyPreface)).map((key) => key.replace(keyPreface, ""));
    return cb(keys);
};

this.delete = (params, cb) => {
    const combinedKey = this.getCombinedKey(this, params);
    delete this.myStorage[combinedKey];
    cb();
};

this.clearCache = (params, cb) => {
    const userPreface = this.getUserPreface(this);
    Object.keys(this.myStorage).forEach((key) => {
        if (key.startsWith(userPreface)) {
            delete this.myStorage(key);
        }
    });

    cb(null, { status: "success" });
};

this.empty = (cb) => {
    this.myStorage = {};
    cb();
};

/**
 * Get the prefix used to filter keys for particular topics and key prefixes.
 *
 * @param {object} params
 * @param {string} params.topic The topic
 * @param {string} params.keyPrefix The key prefix (optional).
 * @private
 */
this.getKeyPreface = (params) => {
    const keyPrefix = "keyPrefix" in params ? params.keyPrefix : "";
    const preface = `${this.getUserPreface()}:${params.topic}:${keyPrefix}`;

    return preface;
};

/**
 * Get prefix for all the user's stored data.
 * @private
 */
this.getUserPreface = () => {
    const preface = `${this.baseName}:${this.userName}`;
    return preface;
};

Testing the adapter

Make sure this thing is working. Generate a sample component using the Finsemble CLI. At your command prompt, use >finsemble-cli add component adapterTest. Put the following code into the JavaScript file created in src/components/adapterTest:

function setData(key, value) {
    FSBL.Clients.StorageClient.save({ key: key, value: value, topic: "IntraSession" }, (err, data) => {
        console.log("StorageClient.set callback invoked");
        if (err) {
            console.error("StorageClient.save error for key", key, err);
        } else {
            console.log("Data saved successfully!");
        }
    });
}

function getData(key) {
    FSBL.Clients.StorageClient.get({ key: key, topic: "IntraSession" }, (err, data) => {
        console.log("StorageClient.get callback invoked");
        if (err) {
            console.error("StorageClient.get error for key", key, err);
        } else {
            console.log("Data found for", key, "data:", data);
        }
    });
}
//The component gets Webpacked. If you don't explictily make the function global, you won't be able to interact with it in the console later on.
window.getData = getData;
window.setData = setData;

Then, start Finsemble and spawn the sample adapterTest component. When adapterTest is loaded, click it and type CTRL+SHIFT+I. This will open the Chrome developer console so that you can test your adapter. Let's see what it does with a couple types of data:

setData("Chicken_Names", ["Edna", "Peepy", "Taylor", "Violet", "McKenna", "Harold"]);
setData("Sporks", { seemUseful: true, areUnderUtilized: true });

When you see the callback logged for both sets, try to retrieve the data. First, reload the component by pressing CTRL+R (this clears the window object). This demonstrates that you didn't save this data locally. Once your component reloads, see what comes back when you try to retrieve the list of chicken names that we saved earlier. Enter the code below into your console.

getData("Chicken_Names");

Edna and her friends have flown all the way from the storage service back to a component. Without our storage adapter, this wouldn't be possible. While this is a silly example, it tackles the basics.

Fine-tuning the timeouts

The schema setting finsemble.servicesConfig.storage.adapterOperationTimeout allows you to specify how long to wait until the operation times out if it doesn't complete.

Finsemble uses the value of 10 seconds as the default. This value is usually more than enough. Most operations, such as reaching a remote storage service via a custom adapter, take a lot less than one second.

We set the default to 10 seconds because occasionally there might be an increased demand on a service. For example, if everyone starts working at 9 AM, they will be logging in to the same service at the same time. As another example, an organization might have scheduled every desktop to restart at the same time. In such situations, the server gets overloaded with traffic so that requests can take longer than usual.

Even with such increased loads, for the majority of cases the default setting of 10 seconds is sufficient. But if in your case requests regularly time out at the peak load, you need to make the timeout interval longer. Deciding on the exact value depends on the service and might take some fine-tuning. You need to tweak this value until your peak storage load is handled correctly while keeping the timeout value as low as is reasonable. We recommend that you increase the timeout by small amounts each time.

Don’t set the timeout value too high. Generally, it's better to fail an operation than to wait too long because chances are the operation will fail anyway. In other words, if a server didn’t respond in 60 seconds (1 minute), chances are something is wrong and the server won’t respond at all. Waiting 600 seconds (10 minutes) won’t help.

caution

Setting the timeout value too high might make you think you have a problem that isn't actually there. You might see that after increasing the timeout, some other, unrelated feature is not working. For example, the Toolbar is not rendering in a reasonable amount of time. Before you extended the timeout, the Toolbar was fine. It isn't rendering now because it's waiting for a storage operation to complete. This operation was timing out in the default time of 10 seconds before, but now it's taking much longer. It looks like there is a Toolbar problem, when in fact the problem is still with the storage and the Toolbar is only a side effect.

See also

Storage Client documentation.

Read the Router tutorial to better understand how Finsemble sends and receives messages.