Difference between revisions of "Using the Datonis Analytics Feature (Notebooks)"

Revision as of 12:02, 13 June 2018

Datonis Documentation Home > Using the Datonis Analytics Feature (Notebooks)

Introduction

The Datonis Platform provides a python notebook feature for performing interactive and exploratory data analysis and running machine learning algorithms on your sensor data. This feature is generally useful when you have collected a sizeable amount of data (for a few days/months).

A notebook provides an integrated containerized big data computing environment. It lets the platform user run plain python programs or distributed programs using the pyspark framework which can be made scalable by provisioning a separate hadoop cluster based on the processing needs. The main advantage of the notebook feature is the ability to perform data analysis close to where the thing/sensor data is stored in the platform, thus eliminating the need for expensive and time consuming ETL in order to get the data to an environment where analytics can be performed. This is a premium feature and only available in the enterprise accounts.

Getting Started

A notebook features an interactive UI editor and the code is organized into paragraphs. A notebook can have multiple paragraphs for logically separating steps to be performed in a program. Paragraphs can be run individually or all together in an order. Each paragraph can optionally print some output or even display graphical elements like charts etc, when performing interactive analysis of data.

Datonis Provider

The DatonisProvider is a python library that provides APIs related to the Datonis environment. A datonis provider instance can be created as follows

%pyspark
from datonis import DatonisProvider
dp = DatonisProvider(sc)

This provides the 'dp' object that will be used to call the API methods

Here 'sc' is a special variable automatically created at the start of the notebook program depicting the Spark Execution Context. This is typical of all spark/pyspark programs.

The Datonis provider has 3 main APIs:

• get_thing_data: Loading This API lets you load data of your things/sensors from the timeseries store. Its returns a distributed collection of data points grouped into named columns called as a dataframe (See Dataframes). This API takes in the following inputs:
  • time range: start time and end time between which to fetch the sensor data. These should be python datetime objects.
  • thing_template_key: A string representing the thing_template key for which to fetch data.
  • thing_keys: An array of strings represeting the thing keys for which to fetch data.
  • metrics/properties: An array of properties of the things you are interested in.
  • timegroup (optional): The aggregation/grouping level to be applied based on the time domain before returning data.
    • raw: This is the default option. This returns all data for the thing/sensor without applying any aggregation or grouping.
    • minute: Data is aggregated on a minute level. Hence you get upto 60 data points if you query for a hour of data.
    • n_minute: Data is aggregated on a 'n' minute level. E.g. a value of '5_minute' will get upto 12 data points if you query for a hour of data.
    • hour: Data is aggregated on a hourly level.
    • month: Data is aggregated on a monthly level
    • day: Data is aggregated on a day level.
  • timezone: The timezone to use to return data. By default this is UTC.

A few examples of how to call the API are given below:

#Getting Raw Data

Saving and publishing a Machine Learning Model: This function lets you