Data Collections and expressions

[Q&A] Components, Datasets and Plug-ins
2

Hi JesseFizor,
You are asking about how Data Collections work and how to use it to consolidate data.

Data collections, implemented by component M5159, are sets of calculated, consolidated data. It is useful to make periodic snapshots of data, which can then be used to display calculation-extensive data in tables, charts or exports.

Data collections can also be integrated with Application Logging using M7606.

Configuring data collections

When installed, Data Collections can be found in the Management subsystem.

Here, you can define a Data collection. It has various options, this example is about the use case you asked.

image
image1490×538 22.2 KB

The choses settings here are:

  1. The type is “Interval” and allows you to select an interval. With this interval, you an create calendar date-based data collections with a specified interval:

    1. Daily
    2. Weekly
    3. Monthly
    4. Quarterly
    5. Yearly
      When you select ‘snapshot’, it calculates data every moment you run the process. Combined with the Task Scheduler, it allows for smaller interval like hourly.

  2. With Cardinality (single or multiple), you identify if you want one single measurement per Data collection, or multiple. As you want data per day, per product, you choose multiple.

  3. Then, you select the record type you want to check per day, and the filter expression to define the products. In this example, I filtered on the products that start with code ‘11’, to make the list smaller.

  4. At value expression, you define the calculation for the value. The built-in fields ‘interval_from’ and ‘interval_to_exclusive()’ are date fields that you can use for filtering.

    1. When you choose daily Data collections, these fields are the same
    2. When you choose weekly or longer Data collections, these fields store the first and last date.
    3. Be aware that the interval_to_inclusive() is a date-field; when you look at Datetime-fields you need to add 24 hours

Running and reviewing data collections

Below the configuration, you can see the data collections. It starts generating from the ‘start date’ and from there, up until today’s date. Be careful, especially initially, not to set a start date long in the past to prevent creating large amounts of data.

image
image1478×179 7.09 KB

When you click ‘generate’, it generates records for all Date intervals between the Start date and today.
When you click ‘generate and process’, it also calculates all values.

When setting up and testing, we recommend to first Generate, and then Process one first to validate data. The process button is on the Data collection page.

image

Scheduling data collections

Now, when your dataset is Created and you want to generate Data automatically, you can use the Process generate_sets_for_collectioncontrollercomponent_xxxxx in the Scheduler, to run the collections in the interval you need.

Typically, you use low use hours of the application, like after business hours, to calculate data.

2 Likes