North Plugins¶

North plugins are used in North tasks and microservices to extract data buffered in FogLAMP and send it Northbound, i.e. to a server or a service in the Cloud or in an Enterprise data center. We currently have two North plugins, one to send data to an OSIsoft PI Server and one to the OSIsoft Cloud Service.

The OMF Plugin¶

The OMF Plugin is used by a North task to send data to an OSIsoft PI server via a PI Connector Relay or PI Web API, it can also send to Edge Data Store or OSIsoft Cloud Services. All these destinations share a single protocol for communication, OMF. OMF stands for OSIsoft Message Format, it is the JSON format defined by OSIsoft to send IoT data to a PI server via a Connector Relay server.

The plugin is designed to send two streams of data:

The data collected by South microservices and buffered into FogLAMP
The statistics generated by FogLAMP

The streams are managed by two different North tasks using the same plugin, but with a different configuration. The two tasks are registered in the list of scheduled jobs and they can be identified using the schedule API call:

$ curl -sX GET   http://locahost:8081/foglamp/schedule
{
  "schedules": [
    {
      "id": "ef8bd42b-da9f-47c4-ade8-751ce9a504be",
      "name": "OMF to PI north",
      "processName": "north_c",
      "type": "INTERVAL",
      "repeat": 30.0,
      "time": 0,
      "day": null,
      "exclusive": true,
      "enabled": false
    },
    {
      "id": "27501b35-e0cd-4340-afc2-a4465fe877d6",
      "name": "Stats OMF to PI north",
      "processName": "north_c",
      "type": "INTERVAL",
      "repeat": 30.0,
      "time": 0,
      "day": null,
      "exclusive": true,
      "enabled": true
    },
  ...
  ]
}

The output of API call above shows three interesting tasks: the two tasks associated to the OMF plugin, the one to send data (OMF to PI north) and the one to send statistics (Stats OMF to PI north).

The two scheduled tasks are associated to two configuration items that can be retrieved using the category API call. The items are named OMF to PI north and Stats OMF to PI north.

$ curl -sX GET   http://localhost:8081/foglamp/category/OMF%20to%20PI%20north
{
  "enable": {
    "description": "A switch that can be used to enable or disable execution of the sending process.",
    "type": "boolean",
    "readonly": "true",
    "default": "true",
    "value": "true"
  },
  "streamId": {
    "description": "Identifies the specific stream to handle and the related information, among them the ID of the last object streamed.",
    "type": "integer",
    "readonly": "true",
    "default": "0",
    "value": "4",
    "order": "16"
  },
  "plugin": {
    "description": "PI Server North C Plugin",
    "type": "string",
    "default": "OMF",
    "readonly": "true",
    "value": "OMF"
  },
  "source": {
     "description": "Defines the source of the data to be sent on the stream, this may be one of either readings, statistics or audit.",
     "type": "enumeration",
     "options": [
       "readings",
       "statistics"
      ],
    "default": "readings",
    "order": "5",
    "displayName": "Data Source",
    "value": "readings"
  },
...}
$ curl -sX GET   http://localhost:8081/foglamp/category/Stats%20OMF%20to%20PI%20north
{
  "enable": {
    "description": "A switch that can be used to enable or disable execution of the sending process.",
    "type": "boolean",
    "readonly": "true",
    "default": "true",
    "value": "true"
  },
  "streamId": {
    "description": "Identifies the specific stream to handle and the related information, among them the ID of the last object streamed.",
    "type": "integer",
    "readonly": "true",
    "default": "0",
    "value": "5",
    "order": "16"
  },
  "plugin": {
    "description": "PI Server North C Plugin",
    "type": "string",
    "default": "OMF",
    "readonly": "true",
    "value": "OMF"
  },
  "source": {
    "description": "Defines the source of the data to be sent on the stream, this may be one of either readings, statistics or audit.",
    "type": "enumeration",
    "options": [
      "readings",
      "statistics"
     ],
    "default": "readings",
    "order": "5",
    "displayName": "Data Source",
    "value": "statistics"
  },
...}
$

In order to activate the tasks, you must change their status. First you must collect their id (from the GET method of the schedule API call), then you must use the IDs with the PUT method of the same call:

$ curl -sX PUT http://localhost:8081/foglamp/schedule/ef8bd42b-da9f-47c4-ade8-751ce9a504be -d '{ "enabled" : true}'
{
  "schedule": {
    "id": "ef8bd42b-da9f-47c4-ade8-751ce9a504be",
    "name": "OMF to PI north",
    "processName": "north_c",
    "type": "INTERVAL",
    "repeat": 30,
    "time": 0,
    "day": null,
    "exclusive": true,
    "enabled": true
  }
}
$ curl -sX PUT http://localhost:8081/foglamp/schedule/27501b35-e0cd-4340-afc2-a4465fe877d6 -d '{ "enabled" : true}'
{
  "schedule": {
    "id": "27501b35-e0cd-4340-afc2-a4465fe877d6",
    "name": "Stats OMF to PI north",
    "processName": "north_c",
    "type": "INTERVAL",
    "repeat": 30,
    "time": 0,
    "day": null,
    "exclusive": true,
    "enabled": true
  }
}
$

At this point, the configuration has been enriched with default values of the tasks:

$ curl -sX GET   http://localhost:8081/foglamp/category/OMF%20to%20PI%20north
{
  "enable": {
    "description": "A switch that can be used to enable or disable execution of the sending process.",
    "type": "boolean",
    "readonly": "true",
    "default": "true",
    "value": "true"
  },
  "streamId": {
    "description": "Identifies the specific stream to handle and the related information, among them the ID of the last object streamed.",
    "type": "integer",
    "readonly": "true",
    "default": "0",
    "value": "4",
    "order": "16"
  },
  "plugin": {
    "description": "PI Server North C Plugin",
    "type": "string",
    "default": "OMF",
    "readonly": "true",
    "value": "OMF"
  },
  "source": {
     "description": "Defines the source of the data to be sent on the stream, this may be one of either readings, statistics or audit.",
     "type": "enumeration",
     "options": [
       "readings",
       "statistics"
      ],
    "default": "readings",
    "order": "5",
    "displayName": "Data Source",
    "value": "readings"
  },
...}
$ curl -sX GET   http://localhost:8081/foglamp/category/Stats%20OMF%20to%20PI%20north
{
  "enable": {
    "description": "A switch that can be used to enable or disable execution of the sending process.",
    "type": "boolean",
    "readonly": "true",
    "default": "true",
    "value": "true"
  },
  "streamId": {
    "description": "Identifies the specific stream to handle and the related information, among them the ID of the last object streamed.",
    "type": "integer",
    "readonly": "true",
    "default": "0",
    "value": "5",
    "order": "16"
  },
  "plugin": {
    "description": "PI Server North C Plugin",
    "type": "string",
    "default": "OMF",
    "readonly": "true",
    "value": "OMF"
  },
  "source": {
    "description": "Defines the source of the data to be sent on the stream, this may be one of either readings, statistics or audit.",
    "type": "enumeration",
    "options": [
      "readings",
      "statistics"
     ],
    "default": "readings",
    "order": "5",
    "displayName": "Data Source",
    "value": "statistics"
  },
...}
$

OMF Plugin Configuration¶

The following table presents the list of configuration options available for the task that sends data to OMF (category OMF to PI north):

Item	Type	Default	Description
AFMap	JSON	{ }	Defines a set of rules to address where assets should be placed in the AF hierarchy.
compression	boolean	true	Compress readings data before sending to PI server
DefaultAFLocation	integer	/foglamp/data_piwebapi/default	Defines the hierarchies tree in Asset Framework in which the assets will be created, each level is separated by /, PI Web API only.
enable	boolean	True	A switch that can be used to enable or disable execution of the sending process.
formatInteger	string	int64	OMF format property to apply to the type Integer.
formatNumber	string	float64	OMF format property to apply to the type Number
notBlockingErrors	JSON	“{ “errors400” : [ “Redefinition of the type with the same ID is not allowed”, “Invalid value type for the property”, “Property does not exist in the type definition”, “Container is not defined”, “Unable to find the property of the container of type” ] }”	These errors are considered not blocking in the communication with the PI Server, the sending operation will proceed with the next block of data if one of these is encountered.
OCSClientSecret	boolean	ocs_client_secret	Client secret associated to the specific OCS account, it is used to authenticate the source for using the OCS API.
OCSClientId	string	ocs_client_id	Client id associated to the specific OCS account, it is used to authenticate the source for using the OCS API.
OCSTenantId	string	ocs_tenant_id	Tenant id associated to the specific OCS account
OCSNamespace	string	name_space	Specifies the OCS namespace where the information are stored and it is used for the interaction with the OCS API.
OMFHttpTimeout	integer	10	Timeout in seconds for the HTTP operations with the OMF PI Connector Relay
OMFMaxRetry	integer	1	Seconds between each retry for the communication with the OMF PI Connector Relay, NOTE : the time is doubled at each attempt.
PIWebAPIKerberosKeytabFileName	string	piwebapi_kerberos_https.keytab	Keytab file name used for Kerberos authentication in PI Web API.
PIWebAPIAuthenticationMethod	enumeration	anonymous	Defines the authentication method to be used with the PI Web API.
PIWebAPIPassword	password	password	Password of the user of PI Web API to be used with the basic access authentication.
PIWebAPIUserId	string	user_id	User id of PI Web API to be used with the basic access authentication.
PIServerEndpoint	enumeration	Connector Relay	Select the endpoint among PI Web API, connector Relay, OSIsoft Cloud Services or Edge Data Store
plugin	string	OMF	PI Server North C Plugin
producerToken	string	omf_north_0001	The producer token that represents this FogLAMP stream
ServerHostname	string	localhost	Hostname of the server running the endpoint either PI Web API or Connector Relay
ServerPort	integer	0	Port on which the endpoint either PI Web API or Connector Relay or Edge Data Store is listening, 0 will use the default one
source	enumeration	readings	Defines the source of the data to be sent the stream, this may be one of either readings, statistics or audit.
StaticData	JSON	{ “Location” : “Palo Alto”,”Company” : “Dianomic” }	Static data to include in each sensor reading sent to the PI Server.
stream_id	integer	0	Identifies the specific stream to handle and the related information, among them the ID of the last object streamed.

The following table presents the list of configuration options available for the task that sends statistics to OMF (category Stats OMF to PI north):

Item	Type	Default	Description
AFMap	JSON	{ }	Defines a set of rules to address where assets should be placed in the AF hierarchy.
compression	boolean	true	Compress readings data before sending to PI server
DefaultAFLocation	integer	/foglamp/data_piwebapi/default	Defines the hierarchies tree in Asset Framework in which the assets will be created, each level is separated by /, PI Web API only.
enable	boolean	True	A switch that can be used to enable or disable execution of the sending process.
formatInteger	string	int64	OMF format property to apply to the type Integer.
formatNumber	string	float64	OMF format property to apply to the type Number
notBlockingErrors	JSON	“{ “errors400” : [ “Redefinition of the type with the same ID is not allowed”, “Invalid value type for the property”, “Property does not exist in the type definition”, “Container is not defined”, “Unable to find the property of the container of type” ] }”	These errors are considered not blocking in the communication with the PI Server, the sending operation will proceed with the next block of data if one of these is encountered.
OCSClientSecret	boolean	ocs_client_secret	Client secret associated to the specific OCS account, it is used to authenticate the source for using the OCS API.
OCSClientId	string	ocs_client_id	Client id associated to the specific OCS account, it is used to authenticate the source for using the OCS API.
OCSTenantId	string	ocs_tenant_id	Tenant id associated to the specific OCS account
OCSNamespace	string	name_space	Specifies the OCS namespace where the information are stored and it is used for the interaction with the OCS API.
OMFHttpTimeout	integer	10	Timeout in seconds for the HTTP operations with the OMF PI Connector Relay
OMFMaxRetry	integer	1	Seconds between each retry for the communication with the OMF PI Connector Relay, NOTE : the time is doubled at each attempt.
PIWebAPIKerberosKeytabFileName	string	piwebapi_kerberos_https.keytab	Keytab file name used for Kerberos authentication in PI Web API.
PIWebAPIAuthenticationMethod	enumeration	anonymous	Defines the authentication method to be used with the PI Web API.
PIWebAPIPassword	password	password	Password of the user of PI Web API to be used with the basic access authentication.
PIWebAPIUserId	string	user_id	User id of PI Web API to be used with the basic access authentication.
PIServerEndpoint	enumeration	Connector Relay	Select the endpoint among PI Web API, connector Relay, OSIsoft Cloud Services or Edge Data Store
plugin	string	OMF	PI Server North C Plugin
producerToken	string	omf_north_0001	The producer token that represents this FogLAMP stream
ServerHostname	string	localhost	Hostname of the server running the endpoint either PI Web API or Connector Relay
ServerPort	integer	0	Port on which the endpoint either PI Web API or Connector Relay or Edge Data Store is listening, 0 will use the default one
source	enumeration	readings	Defines the source of the data to be sent the stream, this may be one of either readings, statistics or audit.
StaticData	JSON	{ “Location” : “Palo Alto”,”Company” : “Dianomic” }	Static data to include in each sensor reading sent to the PI Server.
stream_id	integer	0	Identifies the specific stream to handle and the related information, among them the ID of the last object streamed.

The last parameter to review is the OMF Type. The call is the GET method foglamp/category/OMF_TYPES, which returns an integer value that identifies the measurement type:

$ curl -sX GET http://localhost:8081/foglamp/category/OMF_TYPES
{
  "type-id": {
    "description": "Identify sensor and measurement types",
    "type": "integer",
    "default": "0001",
    "value": "0001"
  }
}
$

If you change the value, you can easily identify the set of data sent to and then stored into PI.

Changing the OMF Plugin Configuration¶

Before you send data to the PI server, it is likely that you need to apply more changes to the configuration. The most important items to change are:

URL : the URL to the PI Connector Relay OMF. It is usually composed by the name or address of the Windows server where the Connector Relay service is running, the port associated to the service and the ingress/messages API call. The communication is via HTTPS protocol.
producerToken : the token provided by the Data Collection Manager when the PI administrator sets the use of FogLAMP.
type-id : the measurement type for the stream of data.
source : this parameter should be set to readings (default) when the plugin is used to send data collected by South microservices, and to statistics when the plugin is used to send FogLAMP statistics to the PI system.

An example of the changes to apply to the plugins to send data to the PI system is available here here.

Data in the PI System¶

Once the North plugins have been set properly, you should expect to see data automatically sent and stored in the PI Server. More specifically, the process of the plugin is the following:

Assets buffered in FogLAMP are stored as elements in the PI System. - PI Asset Framework is automatically update with the new assets. - JSON objects captured as part of the reading in FogLAMP become attributes in the PI Data Archive
The Producer Token is used to authenticate and create the hierarchy of elements in the PI Asset Framework
The configuration object named as Static Data is added as a set of attributes in the PI Data Archive

System	Object	Value
FogLAMP	Producer Token	readings_001
	OMF Type	001
	Static Data	{ “Company” : “Dianomic”, “Location” : “Palo Alto”}
	Asset	fogbench/accelerometer
	Reading	[{“reading”:{“y”:1,”z”:1,”x”:-1}, “timestamp”:”2018-05-14 19:27:06.788}]
PI	Element Template	[OMF.readings_001 Connector.0001_fogbench/accelerometer_typename_sensor]
	Attribute Template	[OMF.readings_001 Connector.0001_fogbench/accelerometer_typename_sensor]
		Company \| Configuration Item, Excluded, String
		Location \| Configuration Item, Excluded, String
		x \| Excluded, Int64
		y \| Excluded, Int64
		z \| Excluded, Int64
	Element	foglamp > readings_001 > fogbench/accelerometer
	Attributes	Company \| Dianomic \| 1970-01-01 00:00:00
		Location \| Palo Alto \| 1970-01-01 00:00:00
		x \| -1 \| 2018-05-14 19:27:06.788
		y \| -1 \| 2018-05-14 19:27:06.788
		z \| -1 \| 2018-05-14 19:27:06.788