Enterprise Gateway
Enterprise Gateway allows you to access on-premise databases and applications that are normally inaccessible due to security restrictions and firewalls. This gateway establishes a secure connection (or a safe tunnel) between Built.io Flow and your server behind the firewall. As a result, you can easily consume data from your on-premise databases, such as DB2, MySQL, MongoDB or any other on-premise applications, and execute workflows using Built.io Flow.
While using Enterprise Gateway, you do not have to provide any information about your server, such as hostname, port, or any other connection details. Enterprise Gateway uses a unique gateway key to register itself and create a secure connection to Built.io Flow.
Using Enterprise Gateway
Perequisites
- Gateway key (Enterprise Gateway should be enabled for your account. If this is not enabled, contact support.)
- Enterprise Gateway installed on your local machine (Linux or Mac)
- Node.js (4 or above)
- Redis
Overview of steps
To use Enterprise Gateway to interact with your enterprise services, you need to perform the following steps:
Step 1. Create Enterprise Gateway agent key
Step 2. install Enterprise Gateway
Step 3. Update Enterprise Gateway config file
Step 4. Start Enterprise Gateway
Detailed steps
Step 1. Create Enterprise Gateway key
To connect Built.io Flow with the gateway on your server, you will need to first create an agent key in the Built.io Flow UI. Here’s how to get the key.
1. Log in to your Built.io Flow account
2. Click your profile name located on the top right-hand side corner. From the drop down options, select ‘Globals’.
3. If Enterprise Gateway is enabled for your account, you will see an option (‘Enterprise Gateway’) on the left-side menu of the Settings panel. Click this option.
4. Click the ‘+’ icon to create a new enterprise connection agent. Provide a suitable name for this agent and ‘Save’.
5. Once created, hover on the agent name, and click the ellipses or more options icon (three vertical dots), and select ‘View.
6. This will display the gateway key for the enterprise agent that you just created. Note down this key as it will be used in the config.json file (in step 2).
Step 2. Install Enterprise Gateway
Enterprise gateway requires Redis as its dependency. Read more about Redis here. Download and install Redis.
Next, get the Enterprise Gateway Zip file by sending a request to our support team at support-flow@built.io.
Once you have unzipped the Enterprise Gateway file, run the following command inside the Enterprise Gateway folder.
npm install
Your Enterprise Gateway package comes with some default plugins provided by Built.io Flow. Plugins are ways to connect to your on-prem systems and communicate with them by using custom actions. Each plugin further has triggers—which execute the workflow when a specific event occurs—and actions—which are tasks (such as create new row)—that you wish to perform in the defined on-prem system.
You can use these default plugins or can create custom plugins that suit your specific business requirements, to use Enterprise Gateway.
Given below is the list of the default plugins provided by Built.io Flow:
1. aws
2. file
3. http
4. mongodb
5. msaccess
6. mssql
7. mysql
8. oracle
9. saphanadb
10. spreadsheet
11. unix
You can install the required dependencies of the default or custom plugin modules using the following command:
npm run install <module>
Here, <module> is the name of the default/custom plugin. For example, npm run install mongodb
Step 3. Update the Enterprise Gateway config file (./config.json)
Open the ‘config’ file that is located under at the root level of your Enterprise Gateway.
First, specify the Enterprise Gateway key that you created in step 1.
"gateway_key": "PASTE YOUR EG KEY HERE",Under ‘application’, adding ‘workers’ would add Enterprise Gateway instances by the corresponding number. The more the number of workers, the better is the performance of your Enterprise Gateway.
"application": {
"workers": 4
},
Specify the Enterprise Gateway key that you created in step 1.
Then, under ‘Redis’, specify the port information of the locally installed Redis.
"redis": {
"sockets": [
{
"host": "127.0.0.1",
"port": 26379
}
]
Specify the triggers that should be active. ‘Timeout’ should be in milliseconds, and the ‘interval’ is the time interval (in milliseconds) at which the triggers would run.
"triggers": {
"active": false,
"timeout": 120000,
"interval": 30000,
"events": [
{
"plugin": "mongodb",
"event": "new_row",
"connection": "c2",
"webhook_url": "PASTE FLOW WEBHOOK URL HERE",
"input": {
"collection": "test"
}
}
]
}
Under events, specify the details of the trigger event. Connection should be the ID of the connection that you have created (see below). Webhook URL is the URL of the workflow that you want to trigger when the trigger event happens. Lastly, you can provide any input data for the workflow.
You can create multiple connections for multiple systems. You need to provide an ID for each connection (which will be used in triggers as given in the above step), and other connection details, such as username, password, port, hostname, and so on. These details vary depending on the system that you are connecting.
Step 4. Start Enterprise Gateway
To start Enterprise Gateway, first run the Redis server:
$ redis-server
Then, use the following command to start Enterprise Gateway:
npm run start
This will start Enterprise Gateway on your machine.
Here are some other helpful commands for your Enterprise Gateway application:
npm run restart // To restart Enterprise Gateway npm run stop // To stop Enterprise Gateway npm run delete // To delete the instances of older running Enterprise Gateway npm run list // To see the status of the instances of Enterprise Gateway npm run actionlogs // To start monitoring action logs npm run applogs // To start monitoring master app logs npm run triggerlogs // To start monitoring trigger logs
Creating custom plugins
Plugins are ways to connect to your on-prem systems and communicate with them by using custom actions. The Enterprise Connector action houses all the plugins that you add in Enterprise Gateway. Each plugin represents and connects to an on-prem system (such as MySQL). This plugin further has actions, which are tasks (such as create new row) that you wish to perform in the defined on-prem system.
So, once Enterprise Gateway is installed and enabled for your account, you can create new plugins, and relevant actions and triggers within each plugin. Let us understand how to do this with the help of an example.
Let's say you want to connect to MySQL. To do this, we will first create a new ‘MySQL’ plugin in Enterprise Gateway.
Custom plugins
To create a custom plugin, follow the steps given below:
1. Navigate to the Enterprise Gateway folder.
2. Create a new folder inside the ‘/plugins’ directory. The folder name should be the name of the service you are adding the plugin for. For e.g., /plugins/mysql
Now that we have created the plugin, we can add actions and triggers in it.
Actions
An action, within a plugin, is a task that you wish to perform in your on-premise app using Enterprise Gateway. For example, create object in MySQL.
To do this, follow the steps given below:
1. Under the new plugin folder you just created, add a folder called ‘actions’. For example, /plugins/mysql/actions
2. In the 'actions' folder, create a new index.js file, and copy the code given below in it. This file contains all the actual working of the action. You can, therefore, change the code as per your requirement.
Example code for /plugins/mysql/actions/index.js file:
module.exports = {
find : find
}
function find () {
//specify Json input schema to draw input form in Enterprise connector action to get input from user
this.input = {
'type': 'object',
'properties': {
}
}
// specify Json schema to define the structure of output this specific action will return to flow
this.output = {
'type': 'object',
'properties': {
}
}
this.execute = function (input, output) {
// input contains all information filled by user
//var connection = this.getConnectionById(‘c2’)
// you can get connection data from config.json using getConnectionById helper function.
// your action's code goes here
// output is callback function which accepts arguments like this
// output(error, response)
output(null, { success : true})
}
}
Once you add an action, restart your Enterprise Gateway by using the ‘npm run restart’ command. Then, login to your Built.io Enterprise account and configure a new workflow. Drag and drop the Enterprise Connector action (read more about it here) onto the canvas. Double click on the Enterprise Connector action and select the plugin you just configured, listed under the 'Plugins' field. You will now see the actions associated with that plugin under the 'Action' drop down field. Select the required action and use it like any other action of your workflow to perform the task within your on-prem system.
Triggers
A trigger is some event that fires off a workflow. Within Enterprise Gateway, you can create custom triggers that fires off a workflow when something happens in your local server or on-premise app, such as new row added in your MongoDB database. Within the trigger, you can specify which workflow to be executed in case the trigger event happens.
This helps you listen to the events of your on-prem system, and accordingly perform tasks automatically in any web application or other on-prem system.
To use a trigger, you need to first create a trigger and then define the list of active triggers in the ./config.json file (as explained in step 3 of the'Detailed Steps' section).
To do this, follow the steps given below:
1. Under the new plugin folder you just created, add a folder called ‘triggers’. For example, /plugins/mysql/triggers
2. In the 'triggers' folder, create a new index.js file, and copy the code given below in it. This file contains all the actual working of the trigger. You can, therefore, change the code as per your requirement.
Example code for /plugins/mysql/triggers/index.js file:
module.exports = {
new_row : newRow
}
function newRow () {
this.input = {
'type': 'object',
'properties': {
}
}
this.output = {
'type': 'object',
'properties': {
}
}
this.activate = function(input, output){
var that = this
var connection = this.getConnectionById(‘c2’)
// you can get connection data from Enterprise Gateway config using getConnectionById helper function.
app.logger.info('Trigger activate function’)
mongo.connect(connection, function(err, db){
if(err){
return output(err)
}
var collection = db.collection(data.collection);
collection.find({}, {'created_at': 1}).toArray(function(err, docs) {
if (docs && docs.length) {
that.setMeta({'last_sync_token': {'created_at': _.last(docs).created_at}})
}else{
that.setMeta({'last_sync_token': null })
}
db.close()
output()
});
})
}
this.execute = function (input, output) {
var that = this
//app.logger.info('execute function called')
var connection = this.getConnectionById(‘c2’)
// you can get connection data from Enterprise Gateway config using getConnectionById helper function.
mongo.connect(connection, function(err, db){
//app.logger.info('error came',err)
if(err){
return output(err)
}
var query = null
if(that.meta && that.meta.last_sync_token && that.meta.last_sync_token.created_at){
var lastSyncToken = that.meta.last_sync_token
query = {'created_at': {'$gt': new Date(lastSyncToken.created_at)}}
}else{
query = { }
}
var collection = db.collection(data.collection);
//app.logger.info('query is', query)
collection.find(query, {'created_at': 1}).toArray(function(err, docs) {
//app.logger.info('pk lets c', err,docs)
if (docs && docs.length) {
that.setMeta({'last_sync_token': {'created_at': _.last(docs).created_at}})
}
output(err, docs)
db.close()
});
})
}
}
Once you add a trigger, you need to specify the list of active triggers in the ./config.json file. To do this, edit the ./config.json file, and copy the snippet given below. Make changes to the code as per your preference.
Set ‘active’ to ‘true’ to activate the trigger. Specify trigger ‘timeout’ in milliseconds. ‘Interval’ is the time interval (in milliseconds) at which the event keeps triggering. Under events, specify the ‘plugin’ and ‘event’ name, and the ‘connection’ it should use to run. Lastly, enter the webhook URL of the workflow that should be executed when the event is triggered.
"triggers": {
"active": active,
"timeout": 120000,
"interval": 30000,
"events": [
{
"plugin": "mongodb",
"event": "new_row",
"connection": "c2",
"webhook_url": "PASTE FLOW WEBHOOK URL HERE",
"input": {
"collection": "test"
}
}
]
},
</p>
After making changes to the ./config.json file, you need to restart the Enterprise Gateway. Now, whenever the defined event happens in the defined on-prem system, the trigger will fire off the specified workflow through its webhook URL.
VM Sizing Requirements
As Enterprise Gateway is backed by Job Queue to serve the user requests, it requires minimal resources (for Redis and Node) to get it running. Apart from this, it depends on how many workers you configure (through config) to run Enterprise Gateway as a cluster to fully utilize your CPU cores, so that parallel actions/tasks can be run by the workers simultaneously. Typically Enterprise Gateway can run smoothly on 1GB VM with a single worker. So, if you want to run four workers of Enterprise Gateway on your machine, you would require minimum 4 GB VM for it.
