101. What is QIX and Why Should You Care?

Learning goal:


The QIX Engine

The QIX Engine is Qlik’s technology that underpins their data discovery tools like QlikView and Qlik Sense. It is a powerful, in-memory data engine that enables us to rapidly explore relationships in various sources of data, without having to write queries. This experience leverages QIX’s Associative Model. If you’ve ever used a Qlik dashboard before, you are familiar with the Associative Model. Try this example dashboard to get a better feel for it. Click on any data point to filter it and watch how the rest of the data in the application updates in response. This is the true power of data association: exploring the data in a quick and unbounded way.

While Qlik’s core products use this under the hood, we can leverage the power of QIX in any type of solution thanks to Qlik’s APIs. These APIs give us direct access to the QIX Engine, enabling us to incorporate the Associative Model into other technologies. Before we jump into the Engine and the APIs though, let’s get a better understanding of what the Associative Model is and why we might use it over traditional data solutions.

The Associative Model: An Overview

In the Associative Model, all data points are associated. These associations exist even across tables, allowing us to model complex relationships between various data sources and tables. With these associations in place, the model can provide numerous benefits:

  • searching: the model can search on any set of search terms and rapidly return related items from across the model

  • querying: the model performs fast calculations across the data set, even working across tables to produce results without requiring any joins

  • filtering: because all data is associated, the model can seamlessly apply filters on any piece of data in the model and update the rest of the model to reflect the filter

  • speed: all of these operations are performed fast thanks to the associative model and its underlying technology

Contrast these attributes with your typical database. In order to query across tables, you need to write complicated joins to get your results in the format you need. If you want to filter across multiple sets of data, you need to update all of your queries that yield those data sets to keep them in sync with your filters. Highlighting relationships or searching across entire models is typically not even possible with these tools. In general, these solutions require you to build and manage functionality like dynamic calculation and filtering from the application side; in contrast, Qlik’s Associative Model handles these functions for you in the Engine, making it quick and easy for a user to explore data.

For more high level details on the Associative Model and it’s advantages over query-based tools, check out this white paper.

Let’s explore some of these unique attributes of the Associative Model in more depth.

The Associative Model: Breakdown

Understanding QIX as a State Machine

The Associative Model can load in various data tables and link those sets of data together via associations. When a model is loaded into memory, these sets of data and their associations are modeled into a context, or state, of data. Say I’ve loaded in the following table of data to a QIX application:

Upon opening the app with this data, the state of the model I open will encompass all 4 records loaded in:

Having a state of data is only useful if that state can change somehow. This is where Qlik’s associations come in handy. Qlik can rapidly filter by any data point in our model and produce a new state of data.

Let’s say we wanted to filter to the "Electronics" department. We can instruct the QIX Engine to do so, and it will modify the state of our data into a new state based on that applied filter:


In the QIX world, we refer to these filters as “selections”. We can apply these selections from any piece of data, and the QIX Engine will provide us a current state based on the results of applying those selections to our data model.


In this way, we can really think of the QIX Engine and it's associative model as a state machine for data sets. It keeps track of the context of our current view of the data and makes it easy for us to change that view from any direction. While the example above is very simplistic with a single table, this state machine becomes extremely powerful when we start to model in complex relationships in our data and perform dynamic calculations across the model. Let's explore those concepts next.

Modeling Associations

Imagine you load in the following table to the QIX Engine:

QIX will look at each column and model it as a field in our data model. A field represents a distinct list of values. So for items, QIX will take each distinct value and create an "Item" field from it:

The same will be done with the Department column. Notice how we have 4 records in our table for Department, but only 3 values in our Department field. That is because our field only captures the distinct Department values, of which there are three: Furniture, Clothing, and Electronics.

Our data model now has two sets of data: Items and Departments. But it’s missing the association between the two. Which items correspond with which departments? Our table records model this relationship. For example, on the first row of our table we see that the item "Chair" comes from the department "Furniture". QIX will use these table records to define the associations between our Items and Departments.

Exploring the Associations with Selections

Now that we’ve modeled these associations, we can explore them via selections. Selections are like filters in our data model. We can pick any field in our model and select values that we want to filter. When we apply a selection, QIX will propagate that filter across the data model based on defined associations.

Let’s try applying a filter in our Department field. Let’s select “Clothing” as our Department.

With "Clothing" selected in our model, we can see that the state of the model has changed. The 4 records we had previously in our table have now been reduced to the 2 records that are in the "Clothing" department. You’ll also notice that the colors of our various field values within our Department and Item field changed. Why is that?

In this example, the colors are indicative of the state of the field value. Any field value can have 5 possible states. To keep things simple at first, let’s talk about 3 main states. Later we will expand on this knowledge and talk about all 5. At a high level, we can consider the following states:

  • Selected: if the data point has been actively applied as a filter, it is a selected value

  • Possible: if a data point has not been applied as a filter but is still part of the current state of the data model, it is a possible value

  • Excluded: if a data point is not part of the current state of the data model due to selections that have been applied, it is an excluded value

In our diagram, we are representing selected values in green with a checkmark, possible values in white, and excluded values in grey. Let’s walk through our model field by field and review the states.

In our Department field, we applied a selection to the "Clothing" field value. This field value is selected, and is therefore highlighted in green. The rest of the model will update based on this selection.

Because we’ve selected "Clothing" in this field, the other color options are now excluded. We won’t see any records in the model for "Furniture" or "Electronics", so these are highlighted in grey.

Now let’s consider our Item field. Our "Clothing" selection in the Department field has changed the states of our Item values via the associations defined in our table. With the "Clothing" selection applied, there are only two possible items that correspond with that color: T-shirt and Pants. These two values are highlighted in white in our diagram: they are still a part of the data set but aren’t directly selected, so they are possible values.

On the other hand, we have two items that are now excluded from the data set. Chair and Camera are not associated with the department "Clothing", so they are highlighted in grey. These values are not present in the current state of our data model.

With QIX, we can filter any data point to update the data model state. So we can continue our data exploration process and apply an additional selection.

Let’s select the item "T-Shirt".

We’ve now applied two selections in different fields. Our data model will filter to the result of combining those two filters, leaving just 1 possible record in our table: the record for T-Shirt.

Notice how the model applied both selections and only left records that met both sets of selections. This is an important feature of the Associative Model. Our data model state will represent the combination of selections. Our records have to be associated with the item "T-Shirt" AND the department "Clothing", as opposed to an being associated with the item "T-Shirt" OR the department "Clothing", which would have left more records.

Deeper into the Selection Model

Fields can have multiple values selected. Let’s start from the data model with no selections, where all values are possible so we see all records.

Let’s filter to two items: T-Shirt and Camera.

Our records now reflect any rows that are either T-Shirts or Cameras. In my Department field, I see two possible values: "Clothing" and "Electronics".

What if I select "Clothing"?

By selecting Clothing, I end up excluding "Cameras" from my data set, even though it is part of my "Item" selection. This is a state we did not cover before known as selected excluded. The selected excluded state refers to a field value that has been selected, but is still excluded due to selections in other fields in the model. In this case, we’ve selected "Camera", but we’ve subsequently selected "Clothing". QIX gives priority to the latest selection applied and thus will exclude "Camera" but keep "T-Shirt" in view. We’ve updated our diagram to indicate selected excluded values with a checkmark within the grey.

If we were to remove the "Clothing" selection that we just added, QIX will update the state so that both "Camera" and "T-Shirt" are still selected.

We’ve introduced 4 states now. Let’s add the 5th one and redefine all of them.

Initially we referred to any data point that was not possible in the data set as “Excluded”. However, we just saw that there is a variant on “Excluded” known as “Selected excluded”. There is actually a 2nd variant as well, known as “Alternative”. “Alternative” values are field values that excluded from the data set due to a selection in their own field, but not excluded due to selections from other fields. Therefore, they are an “alternative” to the selected values within their field; they would be part of the model otherwise.

Take our current state for example, where we’ve selected "T-Shirt" and "Camera" in the Item field.In this state, our Department field has 2 possible values (Clothing and Electronics) and 1 excluded value (Furniture). This excluded value is truly excluded because it does not associate with selections in other places in the model. If we were to select it, QIX will have to remove contradicting selections in the model. QIX will not allow you to apply combinations of selections that yield no associations whatsoever, so we can’t possibly select the department "Furniture" and the items "T-Shirt" and "Camera".

By selecting the department "Furniture", QIX will remove the "T-Shirt" and "Camera" filter:

Let’s go back to our previous state and look at our other field, Item.

Item has two selected values: "T-Shirt" and "Camera". It also has two excluded values: "Pants" and "Chair". However, these values are not excluded due to selections in other fields in the model; they are purely excluded due to the selections in their own field, Item. If we were to select "Chair", QIX would not have to remove any other field selections to update the state. These types of excluded values are therefore referred to as “Alternative”.

Let’s update the diagram to highlight these Alternative values in a lighter shade of grey

Going forward, we will use this lighter shade of grey for any “Alternative” values in our fields.

Revisiting our field value state list from before, we can now define all 5 as so:

  • Selected – the field value has been directly selected and is part of the current data set

  • Possible – the field value has not been selected but is still part of the data set

  • Alternative – the field value has been excluded due to a selection in it’s field, but not due to other selections in the model

  • Excluded – the field value has been excluded due to selections in other fields in the model

  • Selected Excluded – the field value is selected, but has subsequently been excluded by a selection in another field

For more on these states, see

Try playing with the diagram yourself to see how selections work! Click on the various field values to select and deselect them, and see the results in the model.

Expanding the Model with Multiple Data Sets

The Associative Model is pretty cool, but with a single table it’s not adding much more than what you can accomplish with traditional querying. Where it really shines is building larger models that associate multiple data sets. Let’s add in a second data set and see how the associations are handled.

Say we have a data set about our item sales on a daily basis:

Let’s load this into our model. QIX identifies fields by the field names. If we add this table that has a field called "Item", it will assume it is the same field from our first table. Our two tables will therefore be associated by the "Item" field.

With this new table added, we’ve introduced two new fields: Day and Sales. We can see these are associated with Items, which are also associated with Department via our first table.

With these new associations in place, we can start to explore the data like before with selections. First, let’s make some selections in Fruit. We’ll select "T-Shirt" and "Camera".

Notice how the selection propagates through the model via the associations. The Item selection updates our Department field thanks to the associations through the first table; it updates our Day and Sales fields through associations through the second table. The state of our data model now only covers department and sales data for the items "T-Shirt" and "Camera".

We aren’t limited to filtering on the common linkage of "Item" however. In the Associative Model, we can select any data point that has been loaded in. For example, what if I’m curious about sales for just items from the clothing department?

Let’s clear our selections and select "Clothing" in our Color field.

Notice what happened. The selection of "Clothing" filtered the possible Item values down to "T-Shirt" and "Pants". These items then filtered the possible Day and Sales values down based on the corresponding item. For example, the only possible days associated with item that are from the "Clothing" department are Tuesday and Wednesday.

We can add as many tables to this structure as we like, enabling rapid exploration of complex data structures without any need to normalize data with joins.

Go ahead and play with this model for yourself! Click field values to add or remove selections and watch how the data model updates its state in response.

Dynamic Calculations Across the Model

Being able to explore associations in our data is great, but what if we want to analyze that data? The Associative Model has us covered. It can perform dynamic calculations on the data. These calculations are extremely fast, allowing us to make selections and see the results instantly. Let’s explore this feature with a simple calculation.

We can calculate our total sales in the data set by summing up the values in the Sales field. This value was calculated by summing all of our data in the sales field. Using the Associative Model, we can quickly filter the data by applying selections. As we apply selections, this calculation will be re-executed on the fly for us, keeping in sync with the current state of the data model.

Let’s select "Clothing" in the Department field as an example.

As you can see, the model applied the selection, updated the state, and then re-calculated our Sales total.

Try this for yourself! Make selections in different fields and watch how the Sales number always updates in responses to the changing data model.

Another great feature of the Associative Model is that it can do calculations across the model. For example, what if I wanted to calculate the sum of Sales, but broken out by the Department of the Item? The Sales numbers and the Department values come from completely different tables. However, QIX will use the associations to automatically figure out how to join these values together into a single calculated result.

This calculated set of data will rapidly update just like our dynamic calculation from before. Go ahead and make selections in the data model; notice how our dynamically calculated data of sales by department updates with the data model.

Leveraging the Associative Model in Our Own Solutions

As you can see, the Associative Model is a great tool for exploring and analyzing data. Thanks to Qlik’s open APIs, we can leverage the QIX Engine that powers the associative experience in any technology that we like. In the rest of this guide, we will walk through how the QIX Engine API works and build an example dashboard for analyzing data along the way.

This guide will focus primarily on the Engine API and how to use it on the web. It won’t cover advanced usage of the Associative Model itself, such as formula writing. This topic has been covered extensively by the Qlik community. We will recommend external resources along the way for those subjects.

Fin

Recap

  • Qlik’s Associative Model establishes relationships in our data that can be explored and analyzed

  • This model is powered by the QIX Engine, which has an API that we can use to work with it directly

  • The QIX Engine is like a state machine for dynamic data

  • We filter values in the model through “selections”

  • Calculations can be performed across the model

Resources