https://docs.datonis.io/mediawiki/api.php?action=feedcontributions&user=Sayali&feedformat=atomDatonis - User contributions [en]2024-03-29T10:44:37ZUser contributionsMediaWiki 1.28.2https://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1360Edge++: Rails Framework2020-06-12T12:49:35Z<p>Sayali: /* Pages and functions for operator screens: */</p>
<hr />
<div>== Factory Server ==<br />
This is the main environment in which our operator screens will be developed and served to users. This by default runs on port no 3001. <br />
* In order to cache master data, one needs to schedule '''master_data_cache''' function with appropriate frequency. This function by default will cache models configured in generic '''edge_master_data_config'''. One can create account level function with the same name to override '''edge_master_data_config''' that suites one's use case. <br />
<br />
* Functions and pages (generic/account level) in which edge++ flag is true on production will be recurrently cached onto this local setup. One can always develop function/page on local setup directly keeping in mind the '''name''' should be '''uniquely different''' from any function/page present on production as we recurrently sync our database with production. <br />
* While developing functions for data processing one should update Redis keys as per use case which you can later refer in the operator screen. <br />
* To configure session time on local rails server we need to update '''ttl''' in edge.yml in edge_plus_plus repository <br />
<br />
== Branching Details: ==<br />
We have created a long living branch '''edge++''' in github '''factory''' repository. This branch is forked from rel_4.8_dev and have kept to its bare minimum so to have only UI components and Faas Infrastructure. On committing to this branch github action will create new docker container image on docker hub which is altizon/factory. <br />
<br />
== Developer Workflow ==<br />
Developers should login using email and password. On successful login you'll see only single account for which license_key is configured in .env file.<br />
<br />
==== Event processing function: ====<br />
* This function will be executed by factory rake task for every event. This event will be available in args[:event]. <br />
* While performing business logic one should update contextual variables in Redis as per the use case.<br />
* This function should return processed event in the end which one wants to send to datonis<br />
* In case if we don't want to push anything in output queue, we should return nil in the end of this function. <br />
On saving your event processing function, its name will be cached in Redis by background worker and will be available in Edge to be configured for related Adapter.<br />
<br />
You need to click sync_edge___functions to get new function available in settings tab.<br />
<br />
Following is the sample function screenshot:<br />
<br />
[[File:Sample event processing function.png|border|800x800px]]<br />
<br />
==== Pages and functions for operator screens: ====<br />
These can be developed on similar lines as done for production environment. All the UI components available in production factory environment are available here as well.<br />
<br />
The pages and functions developed locally here should be committed by developers for version management. <br />
<br />
== Redis ==<br />
Some well known keys that should not be directly touched by functions in Factory server are as follows:<br />
{| class="wikitable"<br />
!Redis Key<br />
!Significance<br />
|-<br />
|__access_key__<br />
|This should be account admin access key on Datonis. It is updated automatically once Edge is configured with access key.<br />
|-<br />
|__number_of_input_queues__<br />
|Degree of parallelism for processing event stream. This is internally updated while booting.<br />
|-<br />
|__faas_functions__<br />
|Set of names of faas_functions available in local MongoDB<br />
|-<br />
|__input_queue_i__<br />
|These are input queue names by default. Lets say if you configure number of input queues in .env file to be 5, these queue names will be __input_queue_1__ to __input_queue_5__<br />
|-<br />
|__output_queue__<br />
|The messages to be sent to Edge are pushed to Redis output-queue<br />
|-<br />
|__thing-key_alert_queue__<br />
|For each thing-key, 15 latest alerts are pushed to Redis. For example: __d9a3cb4f52_alert_queue__<br />
|}</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1359Edge++: Rails Framework2020-06-12T12:46:49Z<p>Sayali: </p>
<hr />
<div>== Factory Server ==<br />
This is the main environment in which our operator screens will be developed and served to users. This by default runs on port no 3001. <br />
* In order to cache master data, one needs to schedule '''master_data_cache''' function with appropriate frequency. This function by default will cache models configured in generic '''edge_master_data_config'''. One can create account level function with the same name to override '''edge_master_data_config''' that suites one's use case. <br />
<br />
* Functions and pages (generic/account level) in which edge++ flag is true on production will be recurrently cached onto this local setup. One can always develop function/page on local setup directly keeping in mind the '''name''' should be '''uniquely different''' from any function/page present on production as we recurrently sync our database with production. <br />
* While developing functions for data processing one should update Redis keys as per use case which you can later refer in the operator screen. <br />
* To configure session time on local rails server we need to update '''ttl''' in edge.yml in edge_plus_plus repository <br />
<br />
== Branching Details: ==<br />
We have created a long living branch '''edge++''' in github '''factory''' repository. This branch is forked from rel_4.8_dev and have kept to its bare minimum so to have only UI components and Faas Infrastructure. On committing to this branch github action will create new docker container image on docker hub which is altizon/factory. <br />
<br />
== Developer Workflow ==<br />
Developers should login using email and password. On successful login you'll see only single account for which license_key is configured in .env file.<br />
<br />
==== Event processing function: ====<br />
* This function will be executed by factory rake task for every event. This event will be available in args[:event]. <br />
* While performing business logic one should update contextual variables in Redis as per the use case.<br />
* This function should return processed event in the end which one wants to send to datonis<br />
* In case if we don't want to push anything in output queue, we should return nil in the end of this function. <br />
On saving your event processing function, its name will be cached in Redis by background worker and will be available in Edge to be configured for related Adapter.<br />
<br />
You need to click sync_edge___functions to get new function available in settings tab.<br />
<br />
Following is the sample function screenshot:<br />
<br />
[[File:Sample event processing function.png|border|800x800px]]<br />
<br />
==== Pages and functions for operator screens: ====<br />
These can be developed on similar lines as done for production environment. All the UI components available in production factory environment are available here as well.<br />
<br />
== Redis ==<br />
Some well known keys that should not be directly touched by functions in Factory server are as follows:<br />
{| class="wikitable"<br />
!Redis Key<br />
!Significance<br />
|-<br />
|__access_key__<br />
|This should be account admin access key on Datonis. It is updated automatically once Edge is configured with access key.<br />
|-<br />
|__number_of_input_queues__<br />
|Degree of parallelism for processing event stream. This is internally updated while booting.<br />
|-<br />
|__faas_functions__<br />
|Set of names of faas_functions available in local MongoDB<br />
|-<br />
|__input_queue_i__<br />
|These are input queue names by default. Lets say if you configure number of input queues in .env file to be 5, these queue names will be __input_queue_1__ to __input_queue_5__<br />
|-<br />
|__output_queue__<br />
|The messages to be sent to Edge are pushed to Redis output-queue<br />
|-<br />
|__thing-key_alert_queue__<br />
|For each thing-key, 15 latest alerts are pushed to Redis. For example: __d9a3cb4f52_alert_queue__<br />
|}</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1356Edge++: Rails Framework2020-06-12T12:41:09Z<p>Sayali: </p>
<hr />
<div>== Factory Server ==<br />
This is the main environment in which our operator screens will be developed and served to users. This by default runs on port no 3001. <br />
* In order to cache master data, one needs to schedule '''master_data_cache''' function with appropriate frequency. This function by default will cache models configured in generic '''edge_master_data_config'''. One can create account level function with the same name to override '''edge_master_data_config''' that suites one's use case. <br />
<br />
* Functions and pages (generic/account level) in which edge++ flag is true on production will be recurrently cached onto this local setup. One can always develop function/page on local setup directly keeping in mind the '''name''' should be '''uniquely different''' from any function/page present on production as we recurrently sync our database with production. <br />
* While developing functions for data processing one should update Redis keys as per use case which you can later refer in the operator screen. <br />
* To configure session time on local rails server we need to update '''ttl''' in edge.yml in edge_plus_plus repository <br />
<br />
== Branching Details: ==<br />
We have created a long living branch '''edge++''' in github '''factory''' repository. This branch is forked from rel_4.8_dev and have kept to its bare minimum so to have only UI components and Faas Infrastructure. On committing to this branch github action will create new docker container image on docker hub which is altizon/factory. <br />
<br />
== Developer Workflow ==<br />
Developers should login using email and password. On successful login you'll see only single account for which license_key is configured in .env file.<br />
<br />
==== Event processing function: ====<br />
* This function will be executed by factory rake task for every event. This event will be available in args[:event]. <br />
* While performing business logic one should update contextual variables in Redis as per the use case.<br />
* This function should return processed event in the end which one wants to send to datonis<br />
* In case if we don't want to push anything in output queue, we should return nil in the end of this function. <br />
On saving your event processing function, its name will be cached in Redis by background worker and will be available in Edge to be configured for related Adapter.<br />
<br />
You need to click sync_edge___functions to get new function available in settings tab.<br />
<br />
Following is the sample function screenshot:<br />
<br />
[[File:Sample event processing function.png|border|800x800px]]<br />
<br />
== Redis ==<br />
Some well known keys that should not be directly touched by functions in Factory server are as follows:<br />
{| class="wikitable"<br />
!Redis Key<br />
!Significance<br />
|-<br />
|__access_key__<br />
|This should be account admin access key on Datonis. It is updated automatically once Edge is configured with access key.<br />
|-<br />
|__number_of_input_queues__<br />
|Degree of parallelism for processing event stream. This is internally updated while booting.<br />
|-<br />
|__faas_functions__<br />
|Set of names of faas_functions available in local MongoDB<br />
|-<br />
|__input_queue_i__<br />
|These are input queue names by default. Lets say if you configure number of input queues in .env file to be 5, these queue names will be __input_queue_1__ to __input_queue_5__<br />
|}</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=File:Sample_event_processing_function.png&diff=1355File:Sample event processing function.png2020-06-12T12:40:31Z<p>Sayali: </p>
<hr />
<div>Event can be found in args[:event]</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1346Edge++: Rails Framework2020-06-12T12:18:34Z<p>Sayali: </p>
<hr />
<div>== Factory Server ==<br />
This is the main environment in which our operator screens will be developed and served to users. This by default runs on port no 3001. <br />
* In order to cache master data, one needs to schedule '''master_data_cache''' function with appropriate frequency. This function by default will cache models configured in generic '''edge_master_data_config'''. One can create account level function with the same name to override '''edge_master_data_config''' that suites one's use case. <br />
<br />
* Functions and pages (generic/account level) in which edge++ flag is true on production will be recurrently cached onto this local setup. One can always develop function/page on local setup directly keeping in mind the '''name''' should be '''uniquely different''' from any function/page present on production as we recurrently sync our database with production. <br />
* While developing functions for data processing one should update Redis keys as per use case which you can later refer in the operator screen. <br />
* To configure session time on local rails server we need to update '''ttl''' in edge.yml in edge_plus_plus repository <br />
<br />
== Branching Details: ==<br />
We have created a long living branch '''edge++''' in github '''factory''' repository. This branch is forked from rel_4.8_dev and have kept to its bare minimum so to have only UI components and Faas Infrastructure. On committing to this branch github action will create new docker container image on docker hub which is altizon/factory. <br />
<br />
== Developer Workflow ==<br />
Developers should login using email and password. On successful login you'll see only single account for which license_key is configured in .env file.<br />
<br />
==== Event processing function: ====<br />
* This function will be executed by factory rake task for every event. This event will be available in args[:event]. <br />
* While performing business logic one should update contextual variables in Redis as per the use case.<br />
* This function should return processed event in the end which one wants to send to datonis<br />
* In case if we don't want to push anything in output queue, we should return nil in the end of this function. <br />
On saving your event processing function, its name will be cached in Redis by background worker and will be available in Edge to be configured for related Adapter.<br />
<br />
You need to click sync_edge___functions to get new function available in settings tab.<br />
<br />
== Redis ==<br />
Some well known keys that should not be directly touched by functions in Factory server are as follows:<br />
{| class="wikitable"<br />
!Redis Key<br />
!Significance<br />
|-<br />
|__access_key__<br />
|This should be account admin access key on Datonis. It is updated automatically once Edge is configured with access key.<br />
|-<br />
|__number_of_input_queues__<br />
|Degree of parallelism for processing event stream. This is internally updated while booting.<br />
|-<br />
|__faas_functions__<br />
|Set of names of faas_functions available in local MongoDB<br />
|-<br />
|__input_queue_i__<br />
|These are input queue names by default. Lets say if you configure number of input queues in .env file to be 5, these queue names will be __input_queue_1__ to __input_queue_5__<br />
|}</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1341Edge++: Rails Framework2020-06-12T12:03:29Z<p>Sayali: </p>
<hr />
<div>== Factory Server ==<br />
This is the main environment in which our operator screens will be developed and served to users. This by default runs on port no 3001. <br />
* In order to cache master data, one needs to schedule '''master_data_cache''' function with appropriate frequency. This function by default will cache models configured in generic '''edge_master_data_config'''. One can create account level function with the same name to override '''edge_master_data_config''' that suites one's use case. <br />
<br />
* Functions and pages (generic/account level) in which edge++ flag is true on production will be recurrently cached onto this local setup. One can always develop function/page on local setup directly keeping in mind the '''name''' should be '''uniquely different''' from any function/page present on production as we recurrently sync our database with production. <br />
* While developing functions for data processing one should update Redis keys as per use case which you can later refer in the operator screen. <br />
* To configure session time on local rails server we need to update '''ttl''' in edge.yml in edge_plus_plus repository <br />
<br />
== Branching Details: ==<br />
We have created a long living branch '''edge++''' in github '''factory''' repository. This branch is forked from rel_4.8_dev and have kept to its bare minimum so to have only UI components and Faas Infrastructure. On committing to this branch github action will create new docker container image on docker hub. <br />
<br />
== Developer Workflow ==<br />
Developers should login using email and password. On successful login you'll see only single account for which license_key is configured in .env file.<br />
<br />
==== Event processing function: ====<br />
* This function will be executed by factory rake task for every event. This event will be available in args[:event]. <br />
* While performing business logic one should update contextual variables in Redis as per the use case.<br />
* This function should return processed event in the end which one wants to send to datonis<br />
* In case if we don't want to push anything in output queue, we should return nil in the end of this function. <br />
<br />
== Redis ==<br />
Some well known keys that should not be directly touched by functions in Factory server are as follows:<br />
{| class="wikitable"<br />
!Redis Key<br />
!Significance<br />
|-<br />
|__access_key__<br />
|This should be account admin access key on Datonis. It is updated automatically once Edge is configured with access key.<br />
|-<br />
|__number_of_input_queues__<br />
|Degree of parallelism for processing event stream. This is internally updated while booting.<br />
|-<br />
|__faas_functions__<br />
|Set of names of faas_functions available in local MongoDB<br />
|-<br />
|__input_queue_i__<br />
|These are input queue names by default. Lets say if you configure number of input queues in .env file to be 5, these queue names will be __input_queue_1__ to __input_queue_5__<br />
|}</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Datonis_Edge&diff=1337Edge++: Datonis Edge2020-06-12T11:47:51Z<p>Sayali: /* Basic Flow: */</p>
<hr />
<div><br />
== Edge++ Configuration on Datonis Edge: ==<br />
=== Edge++ Service: ===<br />
Edge++ Service offers support of Redis Data-store which is used for exchanging data between Edge and the Faas Framework. Edge++ Service lets us configure the settings related to the Redis store.<br />
* '''Send-to-Datonis''':<br />
** This flag is true if the events in the output-queue are to be transmitted to Datonis.<br />
** Make this flag false if the data should not be transmitted to Datonis.<br />
<br />
=== EdgeCloudService: ===<br />
Once the access-key and secret-key are configured, set '''enable Edge++ setup''' flag to true. This will update the Datonis access-key in the Redis Store on save.<br />
<br />
This will be used by the Rails framework to set up its components.<br />
<br />
=== Thing Settings: ===<br />
In Settings Tab for in any adapter configuration of a particular Thing, you can find the following Edge++ related settings.<br />
# '''Publish Mode - Send/scan Edge++''': Sends the events (scan/send data, instruction, alerts) to the Redis input queue, and the FaasFunction will process and push the processed event into the Output queue.<br />
# '''Sync Process Functions:''' This button populates the Data Process Function and the Instruction Process Function dropdown which are configured on the rails framework.<br />
# '''Data Process Function:''' Select the function which you want to use to process the thing’s data event.<br />
# '''Instruction Process Function''': Select the function which you want to use to process the thing’s instruction event. You can select “--Not-Selected--” option which does not send instruction event to Redis.<br />
<br />
== JSON Structures: ==<br />
<br />
JSON formats used for pushing data into Redis output-queue.<br />
<br />
=== Data JSON: ===<br />
* data: should be a map of metrics (Configured on Datonis) <String, value> You can add new datonis metrics not defined on Edge.<br />
* data_faas_function : Helper faas function name for Edge++.<br />
* instruction_faas_function : Helper faas function name for Edge++<br />
* type : “data”<br />
{<br />
"thing_key":"d9a3cb4f52",<br />
"timestamp":1590067159361,<br />
"data":{<br />
"T":1,<br />
"H":2<br />
},<br />
"data_faas_function":"data_faas_function#12",<br />
"instruction_faas_function":"instruction_faas_function#11",<br />
"type":"data"<br />
} <br />
<br />
=== Instruction JSON: ===<br />
You can insert instructions in the output queue using this format.<br />
{<br />
"thing_key":"d9a3cb4f52",<br />
"timestamp":1591308083000,<br />
"alert_key":"877c5e3cc4",<br />
"instruction_wrapper":{<br />
"instruction":{<br />
"T":"1"<br />
}<br />
},<br />
"type":"instruction",<br />
"instruction_faas_function":"instruction_faas_function#11"<br />
} <br />
<br />
=== Alert JSON: ===<br />
Alert_type:<br />
<br />
INFO - 0, WARNING - 1, ERROR - 2, CRITICAL - 3<br />
{<br />
"alert":{<br />
"thing_key":"d9a3cb4f52",<br />
"timestamp":1589294688742,<br />
"alert_key":"a28cb7bt36",<br />
"alert_type":0,<br />
"message":"[]",<br />
"data":{<br />
"execution_status":"success"<br />
}<br />
}<br />
}</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1336Edge++: Rails Framework2020-06-12T11:45:04Z<p>Sayali: /* Factory Server */</p>
<hr />
<div>== Factory Server ==<br />
This is the main environment in which our operator screens will be developed and served to users. This by default runs on port no 3001. <br />
* In order to cache master data, one needs to schedule '''master_data_cache''' function with appropriate frequency. This function by default will cache models configured in generic '''edge_master_data_config'''. One can create account level function with the same name to override '''edge_master_data_config''' that suites one's use case. <br />
<br />
* Functions and pages (generic/account level) in which edge++ flag is true on production will be recurrently cached onto this local setup. One can always develop function/page on local setup directly keeping in mind the '''name''' should be '''uniquely different''' from any function/page present on production as we recurrently sync our database with production. <br />
* While developing functions for data processing one should update Redis keys as per use case which you can later refer in the operator screen. <br />
<br />
==== Github Branch: ====<br />
We have created a long living branch '''edge++''' in github '''factory''' repository. This branch is forked from rel_4.8_dev and have kept to its bare minimum so to have only UI components and Faas Infrastructure. On committing to this branch github action will create new docker container image on docker hub. <br />
<br />
== Redis ==<br />
Some well known keys that should not be directly touched by functions in Factory server are as follows:<br />
{| class="wikitable"<br />
!Redis Key<br />
!Significance<br />
|-<br />
|__access_key__<br />
|This should be account admin access key on Datonis. It is updated automatically once Edge is configured with access key.<br />
|-<br />
|__number_of_input_queues__<br />
|Degree of parallelism for processing event stream. This is internally updated while booting.<br />
|-<br />
|__faas_functions__<br />
|Set of names of faas_functions available in local MongoDB<br />
|-<br />
|__input_queue_i__<br />
|These are input queue names by default. Lets say if you configure number of input queues in .env file to be 5, these queue names will be __input_queue_1__ to __input_queue_5__<br />
|}</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Deployment&diff=1333Edge++: Deployment2020-06-12T11:21:39Z<p>Sayali: /* Edge++ Deployment on Rails setup */</p>
<hr />
<div>== Edge++ Deployment on Rails setup ==<br />
The configuration Github repository can be found here [https://github.com/Altizon/edge_plus_plus <nowiki>[1]</nowiki>] This repository is a bunch of shell scripts that will help you orchestrate dockerized environment. This environment is served using docker-compose. <br />
<br />
For booting setup we have to do ./boot.sh and to bring it down ./shutdown.sh <br />
<br />
==== Configuration: ====<br />
Couple of fields to be set up in .env file.<br />
* LICENSE_KEY: License key of account on boarded on Datonis<br />
* NUMBER_OF_INPUT_QUEUES: Degree of parallelism at which we want to process our event stream.<br />
Basic building blocks of the architecture in terms of dockers are as follows:<br />
{| class="wikitable"<br />
!Name<br />
!Docker Image<br />
!Purpose<br />
|-<br />
|redis<br />
|redis:6.0.3-buster<br />
|Redis server<br />
|-<br />
|mongo<br />
|mongo:4.2.7-bionic<br />
|MongoDB for master data caching<br />
|-<br />
|factory-server<br />
|altizon/factory<br />
|Factory Server to provide operator screens using FaasInfra<br />
|-<br />
|sidekiq<br />
|altizon/factory<br />
|Sidekiq running asynchronous tasks<br />
|-<br />
|factory-rake<br />
|altizon/factory<br />
|There will be N no of instances of this kind listening to their respective input queues.<br />
|-<br />
|factory-function-cache<br />
|altizon/factory<br />
|Function Task schedular to start master data caching like account and user preferences to local MongoDB from production<br />
|-<br />
|update_number_of_queues<br />
|altizon/factory<br />
|Updating __number_of_input_queues__ in Redis on boot<br />
|-<br />
|wget_call<br />
|altizon/factory<br />
|Calling first API request to server initiating asset compilation<br />
|}<br />
<br />
=== Data Persistance ===<br />
* The data being saved in MongoDB is a volume mounted on docker. Hence, it is safe to assume that your data will persist in MongoDB across boots of Edge++.<br />
* For Redis we use Redis Persistance to ensure Redis maintains its state across boots. We use .aof files for saving our state.<br />
<br />
== FAQs for runnning Edge++ on Ubuntu VM using Oracle VirtualBox ==<br />
<br />
=== Oracle VirtualBox related FAQS ===<br />
* For starting virtualBox with SecureBoot hardware (Only if you have SecureBoot enabled in your machine)<br />
** [https://askubuntu.com/questions/900118/vboxdrv-sh-failed-modprobe-vboxdrv-failed-please-use-dmesg-to-find-out-why vboxdrv.sh: failed: modprobe vboxdrv failed. Please use 'dmesg' to find out why]<br />
* Password Based Encryption:<br />
** [https://book.cyberyozh.com/encryption-of-virtual-data-on-virtual-machine-virtualbox/ Encryption of virtual data on virtual machine VirtualBox]<br />
* VirtualBox Port forwarding from VM to the Host machine:<br />
** [https://stackoverflow.com/questions/9537751/virtualbox-port-forward-from-guest-to-host Virtualbox "port forward" from Guest to Host]<br />
* Setting up redis container inside vm and accessing it via host.<br />
** [https://medium.com/better-programming/dockerizing-and-pythonizing-redis-41b1340979de Configure and Run a Docker Container for Redis and Use it for Python]<br />
** Network type of vm - NAT<br />
** Redis config - bind 0.0.0.0:port instead of 127.0.0.1<br />
** Start the container and run Redis<br />
** Port forwarding of vm- host: <redis-port> - guest: <redis-port><br />
** This will make redis available on localhost:port<br />
** Ref - [https://askubuntu.com/questions/645026/cannot-get-or-set-to-redis-on-vagrant-guest-from-host-due-to-error-connection Cannot get or set to Redis on Vagrant guest from host due to "Error: Connection reset by peer"]</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=File:Edge%2B%2B_Instruction_Flow.jpg&diff=1331File:Edge++ Instruction Flow.jpg2020-06-12T11:17:44Z<p>Sayali: </p>
<hr />
<div>Instruction Execution</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Deployment&diff=1330Edge++: Deployment2020-06-12T11:13:11Z<p>Sayali: </p>
<hr />
<div>== Edge++ Deployment on Rails setup ==<br />
The configuration Github repository can be found here [https://github.com/Altizon/edge_plus_plus <nowiki>[1]</nowiki>] This repository is a bunch of shell scripts that will help you orchestrate dockerized environment. This environment is served using docker-compose. <br />
<br />
==== Configuration: ====<br />
Couple of fields to be set up in .env file.<br />
* LICENSE_KEY: License key of account on boarded on Datonis<br />
* NUMBER_OF_INPUT_QUEUES: Degree of parallelism at which we want to process our event stream.<br />
Basic building blocks of the architecture in terms of dockers are as follows:<br />
{| class="wikitable"<br />
!Name<br />
!Docker Image<br />
!Purpose<br />
|-<br />
|redis<br />
|redis:6.0.3-buster<br />
|Redis server<br />
|-<br />
|mongo<br />
|mongo:4.2.7-bionic<br />
|MongoDB for master data caching<br />
|-<br />
|factory-server<br />
|altizon/factory<br />
|Factory Server to provide operator screens using FaasInfra<br />
|-<br />
|sidekiq<br />
|altizon/factory<br />
|Sidekiq running asynchronous tasks<br />
|-<br />
|factory-rake<br />
|altizon/factory<br />
|There will be N no of instances of this kind listening to their respective input queues.<br />
|-<br />
|factory-function-cache<br />
|altizon/factory<br />
|Function Task schedular to start master data caching like account and user preferences to local MongoDB from production<br />
|-<br />
|update_number_of_queues<br />
|altizon/factory<br />
|Updating __number_of_input_queues__ in Redis on boot<br />
|-<br />
|wget_call<br />
|altizon/factory<br />
|Calling first API request to server initiating asset compilation<br />
|}<br />
<br />
=== Data Persistance ===<br />
* The data being saved in MongoDB is a volume mounted on docker. Hence, it is safe to assume that your data will persist in MongoDB across boots of Edge++.<br />
* For Redis we use Redis Persistance to ensure Redis maintains its state across boots. We use .aof files for saving our state.<br />
<br />
== FAQs for runnning Edge++ on Ubuntu VM using Oracle VirtualBox ==<br />
<br />
=== Oracle VirtualBox related FAQS ===<br />
* For starting virtualBox with SecureBoot hardware (Only if you have SecureBoot enabled in your machine)<br />
** [https://askubuntu.com/questions/900118/vboxdrv-sh-failed-modprobe-vboxdrv-failed-please-use-dmesg-to-find-out-why vboxdrv.sh: failed: modprobe vboxdrv failed. Please use 'dmesg' to find out why]<br />
* Password Based Encryption:<br />
** [https://book.cyberyozh.com/encryption-of-virtual-data-on-virtual-machine-virtualbox/ Encryption of virtual data on virtual machine VirtualBox]<br />
* VirtualBox Port forwarding from VM to the Host machine:<br />
** [https://stackoverflow.com/questions/9537751/virtualbox-port-forward-from-guest-to-host Virtualbox "port forward" from Guest to Host]<br />
* Setting up redis container inside vm and accessing it via host.<br />
** [https://medium.com/better-programming/dockerizing-and-pythonizing-redis-41b1340979de Configure and Run a Docker Container for Redis and Use it for Python]<br />
** Network type of vm - NAT<br />
** Redis config - bind 0.0.0.0:port instead of 127.0.0.1<br />
** Start the container and run Redis<br />
** Port forwarding of vm- host: <redis-port> - guest: <redis-port><br />
** This will make redis available on localhost:port<br />
** Ref - [https://askubuntu.com/questions/645026/cannot-get-or-set-to-redis-on-vagrant-guest-from-host-due-to-error-connection Cannot get or set to Redis on Vagrant guest from host due to "Error: Connection reset by peer"]</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Deployment&diff=1329Edge++: Deployment2020-06-12T11:10:30Z<p>Sayali: </p>
<hr />
<div>== Edge++ Deployment on Rails setup: ==<br />
The configuration Github repository can be found here [https://github.com/Altizon/edge_plus_plus <nowiki>[1]</nowiki>] This repository is a bunch of shell scripts that will help you orchestrate dockerized environment. This environment is served using docker-compose. <br />
<br />
Basic building blocks of the architecture in terms of dockers are as follows:<br />
{| class="wikitable"<br />
!Name<br />
!Docker Image<br />
!Purpose<br />
|-<br />
|redis<br />
|redis:6.0.3-buster<br />
|Redis server<br />
|-<br />
|mongo<br />
|mongo:4.2.7-bionic<br />
|MongoDB for master data caching<br />
|-<br />
|factory-server<br />
|altizon/factory<br />
|Factory Server to provide operator screens using FaasInfra<br />
|-<br />
|sidekiq<br />
|altizon/factory<br />
|Sidekiq running asynchronous tasks<br />
|-<br />
|factory-rake<br />
|altizon/factory<br />
|There will be N no of instances of this kind listening to their respective input queues.<br />
|-<br />
|factory-function-cache<br />
|altizon/factory<br />
|Function Task schedular to start master data caching like account and user preferences to local MongoDB from production<br />
|-<br />
|update_number_of_queues<br />
|altizon/factory<br />
|Updating __number_of_input_queues__ in Redis on boot<br />
|-<br />
|wget_call<br />
|altizon/factory<br />
|Calling first API request to server initiating asset compilation<br />
|}<br />
<br />
=== Data Persistance: ===<br />
* The data being saved in MongoDB is a volume mounted on docker. Hence, it is safe to assume that your data will persist in MongoDB across boots of Edge++.<br />
* For Redis we use Redis Persistance to ensure Redis maintains its state across boots. We use .aof files for saving our state.<br />
<br />
== FAQs for runnning Edge++ on Ubuntu VM using Oracle VirtualBox ==<br />
<br />
=== Oracle VirtualBox related FAQS ===<br />
* For starting virtualBox with SecureBoot hardware (Only if you have SecureBoot enabled in your machine)<br />
** [https://askubuntu.com/questions/900118/vboxdrv-sh-failed-modprobe-vboxdrv-failed-please-use-dmesg-to-find-out-why vboxdrv.sh: failed: modprobe vboxdrv failed. Please use 'dmesg' to find out why]<br />
* Password Based Encryption:<br />
** [https://book.cyberyozh.com/encryption-of-virtual-data-on-virtual-machine-virtualbox/ Encryption of virtual data on virtual machine VirtualBox]<br />
* VirtualBox Port forwarding from VM to the Host machine:<br />
** [https://stackoverflow.com/questions/9537751/virtualbox-port-forward-from-guest-to-host Virtualbox "port forward" from Guest to Host]<br />
* Setting up redis container inside vm and accessing it via host.<br />
** [https://medium.com/better-programming/dockerizing-and-pythonizing-redis-41b1340979de Configure and Run a Docker Container for Redis and Use it for Python]<br />
** Network type of vm - NAT<br />
** Redis config - bind 0.0.0.0:port instead of 127.0.0.1<br />
** Start the container and run Redis<br />
** Port forwarding of vm- host: <redis-port> - guest: <redis-port><br />
** This will make redis available on localhost:port<br />
** Ref - [https://askubuntu.com/questions/645026/cannot-get-or-set-to-redis-on-vagrant-guest-from-host-due-to-error-connection Cannot get or set to Redis on Vagrant guest from host due to "Error: Connection reset by peer"]</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1328Edge++: Rails Framework2020-06-12T10:56:30Z<p>Sayali: </p>
<hr />
<div>== Factory Server ==<br />
This is the main environment in which our operator screens will be developed and served to users. This by default runs on port no 3001. <br />
* In order to cache master data, one needs to schedule '''master_data_cache''' function with appropriate frequency. This function by default will cache models configured in generic '''edge_master_data_config'''. One can create account level function with the same name to override '''edge_master_data_config''' that suites one's use case. <br />
<br />
* Functions and pages (generic/account level) in which edge++ flag is true on production will be recurrently cached onto this local setup. One can always develop function/page on local setup directly keeping in mind the '''name''' should be '''uniquely different''' from any function/page present on production as we recurrently sync our database with production. <br />
* While developing functions for data processing one should update Redis keys as per use case which you can later refer in the operator screen. <br />
<br />
== Redis ==<br />
Some well known keys that should not be directly touched by functions in Factory server are as follows:<br />
{| class="wikitable"<br />
!Redis Key<br />
!Significance<br />
|-<br />
|__access_key__<br />
|This should be account admin access key on Datonis. It is updated automatically once Edge is configured with access key.<br />
|-<br />
|__number_of_input_queues__<br />
|Degree of parallelism for processing event stream. This is internally updated while booting.<br />
|-<br />
|__faas_functions__<br />
|Set of names of faas_functions available in local MongoDB<br />
|-<br />
|__input_queue_i__<br />
|These are input queue names by default. Lets say if you configure number of input queues in .env file to be 5, these queue names will be __input_queue_1__ to __input_queue_5__<br />
|}</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Deployment&diff=1327Edge++: Deployment2020-06-12T10:38:32Z<p>Sayali: </p>
<hr />
<div>== Edge++ Deployment on Rails setup: ==<br />
The configuration Github repository can be found here [https://github.com/Altizon/edge_plus_plus <nowiki>[1]</nowiki>] This repository is a bunch of shell scripts that will help you orchestrate dockerized environment. This environment is served using docker-compose. <br />
<br />
Basic building blocks of the architecture in terms of dockers are as follows:<br />
{| class="wikitable"<br />
!Docker Image<br />
!Purpose<br />
|-<br />
|redis:6.0.3-buster<br />
|Redis server<br />
|-<br />
|mongo:4.2.7-bionic<br />
|MongoDB for master data caching<br />
|-<br />
|altizon/factory<br />
|Factory Server to provide operator screens using FaasInfra<br />
|-<br />
|altizon/factory<br />
|Sidekiq running asynchronous tasks<br />
|-<br />
|<br />
|<br />
|}<br />
<br />
=== Data Persistance: ===<br />
* The data being saved in MongoDB is a volume mounted on docker. Hence, it is safe to assume that your data will persist in MongoDB across boots of Edge++.<br />
* For Redis we use Redis Persistance to ensure Redis maintains its state across boots. We use .aof files for saving our state.<br />
<br />
== FAQs for runnning Edge++ on Ubuntu VM using Oracle VirtualBox ==<br />
<br />
=== Oracle VirtualBox related FAQS ===<br />
* For starting virtualBox with SecureBoot hardware (Only if you have SecureBoot enabled in your machine)<br />
** [https://askubuntu.com/questions/900118/vboxdrv-sh-failed-modprobe-vboxdrv-failed-please-use-dmesg-to-find-out-why vboxdrv.sh: failed: modprobe vboxdrv failed. Please use 'dmesg' to find out why]<br />
* Password Based Encryption:<br />
** [https://book.cyberyozh.com/encryption-of-virtual-data-on-virtual-machine-virtualbox/ Encryption of virtual data on virtual machine VirtualBox]<br />
* VirtualBox Port forwarding from VM to the Host machine:<br />
** [https://stackoverflow.com/questions/9537751/virtualbox-port-forward-from-guest-to-host Virtualbox "port forward" from Guest to Host]<br />
* Setting up redis container inside vm and accessing it via host.<br />
** [https://medium.com/better-programming/dockerizing-and-pythonizing-redis-41b1340979de Configure and Run a Docker Container for Redis and Use it for Python]<br />
** Network type of vm - NAT<br />
** Redis config - bind 0.0.0.0:port instead of 127.0.0.1<br />
** Start the container and run Redis<br />
** Port forwarding of vm- host: <redis-port> - guest: <redis-port><br />
** This will make redis available on localhost:port<br />
** Ref - [https://askubuntu.com/questions/645026/cannot-get-or-set-to-redis-on-vagrant-guest-from-host-due-to-error-connection Cannot get or set to Redis on Vagrant guest from host due to "Error: Connection reset by peer"]</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1326Edge++: Rails Framework2020-06-12T10:38:22Z<p>Sayali: /* Edge++ Configuration on Rails setup: */</p>
<hr />
<div>== Factory Server: ==<br />
This is the main environment in which our operator screens will be developed and served to users. This by default runs on port no 3001. <br />
<br />
In order to cache master data, one needs to schedule '''master_data_cache''' function with appropriate frequency. This function by default will cache models configured in generic '''edge_master_data_config'''. One can create account level function with the same name to override '''edge_master_data_config''' that suites one's usecase.</div>Sayalihttps://docs.datonis.io/mediawiki/index.php?title=Edge%2B%2B:_Rails_Framework&diff=1324Edge++: Rails Framework2020-06-12T10:04:08Z<p>Sayali: Created page with " == Edge++ Configuration on Rails setup: =="</p>
<hr />
<div><br />
== Edge++ Configuration on Rails setup: ==</div>Sayali