While the data Datazoom collects is built on standards and common interaction models, customers often need to transform the standard set of data to better suit their needs to avoid post-processing. Transformations conform data while in transit before it lands in a connector.
Save money by eliminating extra data before it is sent to the Connector data store, Save time by having the data in the desired schema as soon as it arrives in the Connector.
The Data Transformation service is built on a rules engine that can trigger a set of actions, altering a data message in realtime before the data leaves Datazoom. The rules engine allows for complex rule definitions using multiple criteria and actions. The goal is to enable any possible manipulation of an event message.
Each connector instance can have a ruleset attached – the rules will be applied to all data sent to that connector regardless of how many data pipes to which the connector is attached – to ensure data consistency in the final destination.
A ruleset consists of criteria, actions and results:
The criteria defines which messages should be affected by a rule. The criteria could match with all messages or only very specific messages. Criteria can be data across multiple fields using a variety of operators to limit the event stream down to only the events that need to be transformed.
The actions define what types of transformations that should be applied to each message that passed the criteria stage. See example use cases below.
The results are either simple value replacements or complex functions that can be executed to generate the result.
Data point renaming
Customers may not want to use the Datazoom Data Dictionary as we have defined it. Transformations allow for field names or hierarchical locations to be updated in real time.
Example: key renaming
As a simple example, client_ip can become ip_address.
Example: node location change
Change the node location of specific key. For example, move from user_details.content_session_id to video.content_session_id to better fit data analysts’ needs.
Rename data points and values
Example: value renaming
Conform values Datazoom determined for its dictionary to your custom defined value list
Rule example: if device_type = “ott device”, then set device_type to “set-top box”
Example: Modify field value based on the value of another field
Examine all of the data in a message and use it to change the value of a specific field. For example, Datazoom’s Data Dictionary field player_viewable is a boolean value that is set to true when the player_viewable_percent field is greater than or equal to 50%. What if a customer considers something viewable when only 25% of the video is viewable? In this case the customer could build their own rule to transform the originally collected value in player_viewable.
If player_viewable_percent >= 25, then set event.metrics.player_viewable to TRUE
If device_type = ott device and OS = Roku, then set device_type to “roku”
Example: new metadata field
Generate new data to add to a message, create a new field using logic based on values in another field.
Rule Example: if device_type = “ott device” and OS = “Roku”, then create a new field device.platform and set the value to “Roku”
Example: new event types
Change the event name from Milestone to Quartile to comply with Connector expectations or to clarify for users of the data that only Quartile data will be available in this data set.
If milestone_percent = 25, then set event.type to first_quartile
If milestone_percent = 50, then set event.type to midpoint
If milestone_percent = 76, then set event.type to third_quartile
Example: math operations
Change the unit of measurement for a numeric field.
Rule example: for all playhead_position values, multiply by 1000 to convert seconds to milliseconds
Example: hashing for PII handling
Handle PII fields with built in anonymization logic.
For all client_ip values, apply hash algorithm to anonymize value with a one-way hash
For latitude & longitude fields, reduce precision to anonymize value by reducing number of decimals
Get started with transformations
Contact your Datazoom account representative to create an implementation plan and build, test and verify the results of rulesets.