Difference between revisions of "Using the Datonis Analytics Feature (Notebooks)"
From Datonis
m (Tag: Visual edit) |
m (Tag: Visual edit) |
||
| (4 intermediate revisions by the same user not shown) | |||
| Line 19: | Line 19: | ||
The Datonis provider has 3 main APIs: | The Datonis provider has 3 main APIs: | ||
| − | * '''''get_thing_data''''': | + | * '''''get_thing_data''''': This API lets you load data of your things/sensors from the timeseries store. Its returns a distributed collection of data points grouped into named columns called as a dataframe (See [http://spark.apache.org/docs/2.1.1/api/python/pyspark.sql.html#pyspark.sql.DataFrame Dataframes]). This API takes in the following inputs: |
** time range: start time and end time between which to fetch the sensor data. These should be python datetime objects. | ** time range: start time and end time between which to fetch the sensor data. These should be python datetime objects. | ||
** thing_template_key: A string representing the thing_template key for which to fetch data. | ** thing_template_key: A string representing the thing_template key for which to fetch data. | ||
| Line 33: | Line 33: | ||
** timezone: The timezone to use to return data. By default this is UTC. | ** timezone: The timezone to use to return data. By default this is UTC. | ||
A few examples of how to call the API are given below: | A few examples of how to call the API are given below: | ||
| − | #Getting | + | #Getting Raw Data |
| − | + | thing_template_key = "972adct84d" # Thing_template_key of things | |
| + | thing_keys = ["t7f7b9a3ea","4bc27857f1","c241314e1a"] # Thing keys whose data we wish to aggregate | ||
| + | metrics = ["pressure", "forging_temp", "job.value"] # List of metrics that we want to aggregate | ||
| + | thing_data_frames = dp.get_thing_data(start_time, end_time, thing_template_key, thing_keys, metrics) | ||
| + | thing_data_frames.show() | ||
| + | +--------------------+----------+----------------+--------+------------+-----+ | ||
| + | | ts| thing_key| thing_name|pressure|forging_temp|value| | ||
| + | +--------------------+----------+----------------+--------+------------+-----+ | ||
| + | |2018-05-16 01:38:...|c241314e1a|Forging Hammer-1| 28.0| 1196.0| 2.0| | ||
| + | |2018-05-16 01:38:...|t7f7b9a3ea|Forging Hammer-2| 14.0| 1023.0| 3.0| | ||
| + | |2018-05-16 01:38:...|4bc27857f1|Forging Hammer-3| 21.0| 815.0| 4.0| | ||
| + | |2018-05-16 01:39:...|c241314e1a|Forging Hammer-1| 11.0| 1065.0| 2.0| | ||
| + | |2018-05-16 01:39:...|t7f7b9a3ea|Forging Hammer-2| 12.0| 870.0| 2.0| | ||
| + | |2018-05-16 01:39:...|4bc27857f1|Forging Hammer-3| 22.0| 786.0| 2.0| | ||
| + | |2018-05-16 01:39:...|c241314e1a|Forging Hammer-1| 7.0| 1133.0| 2.0| | ||
| + | |2018-05-16 01:39:...|t7f7b9a3ea|Forging Hammer-2| 8.0| 940.0| 4.0| | ||
| + | |20 | ||
| + | |||
| + | #Getting Minute level Data | ||
| + | metrics = ["pressure", "forging_temp"] | ||
| + | thing_data_frames = dp.get_thing_data(start_time, end_time, thing_template_key, thing_keys, metrics, timegroup="minute") | ||
| + | thing_data_frames.show() | ||
| + | +--------------------+----------+----------------+-----------------+-------------------+-----------------+-----------------+-----------------+-------------+---------------+-------------+-------------+-------------+ | ||
| + | | ts| thing_key| thing_name|forging_temp::sum|forging_temp::count|forging_temp::max|forging_temp::min|forging_temp::avg|pressure::sum|pressure::count|pressure::max|pressure::min|pressure::avg| | ||
| + | +--------------------+----------+----------------+-----------------+-------------------+-----------------+-----------------+-----------------+-------------+---------------+-------------+-------------+-------------+ | ||
| + | |2018-05-16 01:38:...|t7f7b9a3ea|Forging Hammer-2| 1023.0| 1.0| 1023.0| 1023.0| 1023.0| 14.0| 1.0| 14.0| 14.0| 14.0| | ||
| + | |2018-05-16 01:38:...|c241314e1a|Forging Hammer-1| 1196.0| 1.0| 1196.0| 1196.0| 1196.0| 28.0| 1.0| 28.0| 28.0| 28.0| | ||
| + | |2018-05-16 01:38:...|4bc27857f1|Forging Hammer-3| 815.0| 1.0| 815.0| 815.0| 815.0| 21.0| 1.0| 21.0| 21.0| 21.0| | ||
| + | |2018-05-16 01:39:...|t7f7b9a3ea|Forging Hammer-2| 1810.0| 2.0| 940.0| 870.0| 905.0| 20.0| 2.0| 12.0| 8.0| 10.0| | ||
| + | |2018-05-16 01:39:...|4bc27857f1|Forging Hammer-3| 786.0| 1.0| 786.0| 786.0| 786.0| 22.0| 1.0| 22.0| 22.0| 22.0| | ||
| + | |2018-05-16 01:39:...|c241314e1a|Forging Hammer-1| 2198.0| 2.0| 1133.0| 1065.0| 1099.0| 18.0| 2.0| 11.0| 7.0| 9.0| | ||
| + | |2018-05-16 01:40:...|c241314e1a|Forging Hammer-1| 1148.0| 1.0| 1148.0| 1148.0| 1148.0| 25.0| 1.0| 25.0| 25.0| 25.0| | ||
| + | |2018-05-16 01:40:...|4bc27857f1|Forging Hammer-3| 1695.0| 2.0| 874.0| 821.0| 847.5| 31.0| 2.0| 19.0| 12.0| 15.5| | ||
| + | |||
| + | |||
| + | DataFrames can also be accessed as SQL tables by performing the following operation | ||
| + | %pyspark | ||
| + | table_name = "workcenter_hierarchy" | ||
| + | dataframe.createOrReplaceTempView(table_name) | ||
| + | %sql | ||
| + | select * from workcenter_hierarchy | ||
| + | * '''''save_and_publish_ml_model:''''' This API lets you save a trained spark machine learning model on the file system for later use. This also publishes this model as a first class model in Datonis ML Models repository. The API takes the following parameters: | ||
| + | ** model: The python object representing the machine learning model. | ||
| + | ** name: Name given to the model. The model is referred to by this name in Datonis. | ||
| + | ** description: A description of what the model represents. | ||
| + | ** input_fields: An array of independent input variables used to create the model. | ||
| + | ** output_fields: An array of depedent output variables. | ||
| + | An example of saving a trained model. | ||
| + | %pyspark | ||
| + | dp.save_and_publish_aml_model(model, "solar-panel-dc-current-predictor", "Predicts Solar Panel's Ideal DC Output", ["AmbientTemperature", "DcVoltage", "Irradiation", "ModuleTemperature"], ["DcCurrent"]) | ||
| + | * load_ml_model_by_name(name): This API lets you load a previously trained & saved model from the file system into the pyspark execution context. A loaded model can further be used for predicting the values of the output variables based on the values of the input variables that are fed to it. The API takes in the following parameters as input: | ||
| + | ** Name of the model: The name given to the model when it was saved and published. | ||
| + | An example of loading a previous saved model. | ||
| + | %pyspark | ||
| + | model = dp.load_ml_model_by_name("solar-panel-dc-current-predictor") | ||
Latest revision as of 13:00, 13 June 2018
Datonis Documentation Home > Using the Datonis Analytics Feature (Notebooks)
Introduction
The Datonis Platform provides a python notebook feature for performing interactive and exploratory data analysis and running machine learning algorithms on your sensor data. This feature is generally useful when you have collected a sizeable amount of data (for a few days/months).
A notebook provides an integrated containerized big data computing environment. It lets the platform user run plain python programs or distributed programs using the pyspark framework which can be made scalable by provisioning a separate hadoop cluster based on the processing needs. The main advantage of the notebook feature is the ability to perform data analysis close to where the thing/sensor data is stored in the platform, thus eliminating the need for expensive and time consuming ETL in order to get the data to an environment where analytics can be performed. This is a premium feature and only available in the enterprise accounts.
Getting Started
A notebook features an interactive UI editor and the code is organized into paragraphs. A notebook can have multiple paragraphs for logically separating steps to be performed in a program. Paragraphs can be run individually or all together in an order. Each paragraph can optionally print some output or even display graphical elements like charts etc, when performing interactive analysis of data.
Datonis Provider
The DatonisProvider is a python library that provides APIs related to the Datonis environment. A datonis provider instance can be created as follows
%pyspark from datonis import DatonisProvider dp = DatonisProvider(sc)
This provides the 'dp' object that will be used to call the API methods
Here 'sc' is a special variable automatically created at the start of the notebook program depicting the Spark Execution Context. This is typical of all spark/pyspark programs.
The Datonis provider has 3 main APIs:
- get_thing_data: This API lets you load data of your things/sensors from the timeseries store. Its returns a distributed collection of data points grouped into named columns called as a dataframe (See Dataframes). This API takes in the following inputs:
- time range: start time and end time between which to fetch the sensor data. These should be python datetime objects.
- thing_template_key: A string representing the thing_template key for which to fetch data.
- thing_keys: An array of strings represeting the thing keys for which to fetch data.
- metrics/properties: An array of properties of the things you are interested in.
- timegroup (optional): The aggregation/grouping level to be applied based on the time domain before returning data.
- raw: This is the default option. This returns all data for the thing/sensor without applying any aggregation or grouping.
- minute: Data is aggregated on a minute level. Hence you get upto 60 data points if you query for a hour of data.
- n_minute: Data is aggregated on a 'n' minute level. E.g. a value of '5_minute' will get upto 12 data points if you query for a hour of data.
- hour: Data is aggregated on a hourly level.
- month: Data is aggregated on a monthly level
- day: Data is aggregated on a day level.
- timezone: The timezone to use to return data. By default this is UTC.
A few examples of how to call the API are given below:
#Getting Raw Data thing_template_key = "972adct84d" # Thing_template_key of things thing_keys = ["t7f7b9a3ea","4bc27857f1","c241314e1a"] # Thing keys whose data we wish to aggregate metrics = ["pressure", "forging_temp", "job.value"] # List of metrics that we want to aggregate thing_data_frames = dp.get_thing_data(start_time, end_time, thing_template_key, thing_keys, metrics) thing_data_frames.show() +--------------------+----------+----------------+--------+------------+-----+ | ts| thing_key| thing_name|pressure|forging_temp|value| +--------------------+----------+----------------+--------+------------+-----+ |2018-05-16 01:38:...|c241314e1a|Forging Hammer-1| 28.0| 1196.0| 2.0| |2018-05-16 01:38:...|t7f7b9a3ea|Forging Hammer-2| 14.0| 1023.0| 3.0| |2018-05-16 01:38:...|4bc27857f1|Forging Hammer-3| 21.0| 815.0| 4.0| |2018-05-16 01:39:...|c241314e1a|Forging Hammer-1| 11.0| 1065.0| 2.0| |2018-05-16 01:39:...|t7f7b9a3ea|Forging Hammer-2| 12.0| 870.0| 2.0| |2018-05-16 01:39:...|4bc27857f1|Forging Hammer-3| 22.0| 786.0| 2.0| |2018-05-16 01:39:...|c241314e1a|Forging Hammer-1| 7.0| 1133.0| 2.0| |2018-05-16 01:39:...|t7f7b9a3ea|Forging Hammer-2| 8.0| 940.0| 4.0| |20
#Getting Minute level Data metrics = ["pressure", "forging_temp"] thing_data_frames = dp.get_thing_data(start_time, end_time, thing_template_key, thing_keys, metrics, timegroup="minute") thing_data_frames.show() +--------------------+----------+----------------+-----------------+-------------------+-----------------+-----------------+-----------------+-------------+---------------+-------------+-------------+-------------+ | ts| thing_key| thing_name|forging_temp::sum|forging_temp::count|forging_temp::max|forging_temp::min|forging_temp::avg|pressure::sum|pressure::count|pressure::max|pressure::min|pressure::avg| +--------------------+----------+----------------+-----------------+-------------------+-----------------+-----------------+-----------------+-------------+---------------+-------------+-------------+-------------+ |2018-05-16 01:38:...|t7f7b9a3ea|Forging Hammer-2| 1023.0| 1.0| 1023.0| 1023.0| 1023.0| 14.0| 1.0| 14.0| 14.0| 14.0| |2018-05-16 01:38:...|c241314e1a|Forging Hammer-1| 1196.0| 1.0| 1196.0| 1196.0| 1196.0| 28.0| 1.0| 28.0| 28.0| 28.0| |2018-05-16 01:38:...|4bc27857f1|Forging Hammer-3| 815.0| 1.0| 815.0| 815.0| 815.0| 21.0| 1.0| 21.0| 21.0| 21.0| |2018-05-16 01:39:...|t7f7b9a3ea|Forging Hammer-2| 1810.0| 2.0| 940.0| 870.0| 905.0| 20.0| 2.0| 12.0| 8.0| 10.0| |2018-05-16 01:39:...|4bc27857f1|Forging Hammer-3| 786.0| 1.0| 786.0| 786.0| 786.0| 22.0| 1.0| 22.0| 22.0| 22.0| |2018-05-16 01:39:...|c241314e1a|Forging Hammer-1| 2198.0| 2.0| 1133.0| 1065.0| 1099.0| 18.0| 2.0| 11.0| 7.0| 9.0| |2018-05-16 01:40:...|c241314e1a|Forging Hammer-1| 1148.0| 1.0| 1148.0| 1148.0| 1148.0| 25.0| 1.0| 25.0| 25.0| 25.0| |2018-05-16 01:40:...|4bc27857f1|Forging Hammer-3| 1695.0| 2.0| 874.0| 821.0| 847.5| 31.0| 2.0| 19.0| 12.0| 15.5|
DataFrames can also be accessed as SQL tables by performing the following operation
%pyspark table_name = "workcenter_hierarchy" dataframe.createOrReplaceTempView(table_name) %sql select * from workcenter_hierarchy
- save_and_publish_ml_model: This API lets you save a trained spark machine learning model on the file system for later use. This also publishes this model as a first class model in Datonis ML Models repository. The API takes the following parameters:
- model: The python object representing the machine learning model.
- name: Name given to the model. The model is referred to by this name in Datonis.
- description: A description of what the model represents.
- input_fields: An array of independent input variables used to create the model.
- output_fields: An array of depedent output variables.
An example of saving a trained model.
%pyspark dp.save_and_publish_aml_model(model, "solar-panel-dc-current-predictor", "Predicts Solar Panel's Ideal DC Output", ["AmbientTemperature", "DcVoltage", "Irradiation", "ModuleTemperature"], ["DcCurrent"])
- load_ml_model_by_name(name): This API lets you load a previously trained & saved model from the file system into the pyspark execution context. A loaded model can further be used for predicting the values of the output variables based on the values of the input variables that are fed to it. The API takes in the following parameters as input:
- Name of the model: The name given to the model when it was saved and published.
An example of loading a previous saved model.
%pyspark
model = dp.load_ml_model_by_name("solar-panel-dc-current-predictor")
