Simple REST with Payload Scripting¶

The foglamp-south-simple-rest plugin uses REST calls to receive API responses from sensors or other sources. The plugin make HTTP or HTTPS GET requests to retrieve API responses, HTTP header fields can also be added via the plugin configuration. It then uses an optional script, written is Python, that converts the message into a JSON document and pushes data to the FogLAMP System. However it also has a set of built in rules for interpreting some common payload formats which enable it to be used without providing a script in a large number of common cases.

When a script is provided it is best practice to do the minimum required to allow the data to be ingested into the FogLAMP data pipeline. Further processing to shape the data to exact requirements can often be done using an existing filter. The advantages of this are twofold; it simplifies the scripts required here and it simplifies maintenance should the data be required in a different format some time later.

Configuration¶

When adding a south service with this plugin the same flow is used as with any other south service. The configuration is divided into a number of tabs, each of which groups related configuration elements.

Default Configuration: Configuration related to asset naming and HTTP connection settings.

Authentication: Optional configuration related to the authentication that will occur with the REST service.

Data Selection: Configuration that defines how the data read from the REST service will be treated.

Script: The optional script that will be run on the result returned from the REST service.

Timestamps: Configuration that deals with the generation of timestamps for the data.

Default Configuration¶

rest_default

Asset Name: The name of the asset the plugin will create for each message, unless the convert function returns an explicit asset name to be used.

URL: The URL of the REST API to be called. This should be a complete URL, including the http or https protocol to use.

Proxy: The address of the HTTP proxy to use when making calls to the REST API. Leave blank if no proxy is to be used.

Headers: An optional set of headers to include in the REST API call. The headers are encoded as a JSON document as a set of name/value pairs within a JSON object.

Collapse: Collapse the returned data to a flat structure, if not enabled a nested reading will be produced.

Asset Field: The name of a field in the response payload that should be treated as the asset name to use for the reading. If this is left empty or the data does not contain a field with this name then the default asset name configured in the Asset Name configuration item will be used.

Authentication¶

The plugin supports optional basic authentication with the REST service. If basic authentication is required then the username and password should be entered in this tab. If no authentication is required then these configuration items should be left empty.

rest_auth

Username: A username to use for basic authentication with the REST endpoint. If the endpoint does not require basic authentication then this should be left blank.

Password: The password to use for basic authentication with the REST endpoint. If the endpoint does not require basic authentication then this should be left blank.

Data Selection¶

The data selection tab provides those configuration items that are used to interact with REST services that offer historical data acquisition. They provides the means to select the appropriate data.

rest_data

Selection Method: The plugin supports a number of methods for selecting the data should be returned. The choices are return all the data, return data based on an ID or return data based on time. See Selection Method for more details.

ID Parameter: An optional URL query parameter to add to each call to the URL. This is expected to be a numeric value that gets passed to the API and is used for implementing ID passing to calls. This is only valid of the selection method ID Based has been chosen.

Initial ID: The initial value to pass for the query parameter. This may be used on the first call only if the ID Based selection method is chosen.

ID Field: This defines a data field that is ingested that will be used for second and subsequent calls to the API as the new value of ID. This is only used with a selection method of ID Based.

Start: The name of the query parameter to add to the URL to indicate the start time if a selection method of Time Based has been chosen.

End: The name of the query parameter to add to the URL to indicate the end time if a selection method of Time Based has been chosen.

Script¶

rest_script

Script: The Python script to execute for message processing. Initially a file must be uploaded, however once uploaded the user may edit the script in the box provided. A script is optional.

Timestamps¶

The plugin has the option of using timestamps returned by the REST service in the payload or using the current time. It is this tab that defines that behavior are how the payload is decoded in order to extract timestamps.

rest_timestamps

Timestamp: The name of the item in the response payload that should be treated as the timestamp for the reading.

Time Format: The format to both pass the timestamps into the query parameters using and also to interpret the timestamps returned in the payload.

Timezone: The timezone to use for the start and end times that are sent in the API request and also when timestamps are read from the API response. Timezone is expressed as an offset in hours and minutes from UTC for the local timezone of the API. E.g. -08:00 for the PST timezone.

Selection Method¶

The plugin supports two methods to select data to be retrieved from the API that is called, these methods are designed for use with an API that is maintaining historic data and provided a mechanism to present the same historic data being read multiple times. If your API does not store historic data then you may select the method None to simply retrieve all the data available via the API.

ID Based¶

The select mechanism ID Based is designed for API that give each value some form of ID that increases over time. When a call is made you pass the value of the ID for the next data item you wish to read. This method is used in conjunction with 3 other parameters. These parameters are used to control the name of the query parameter to add to the URL, ID Parameter. This name will be used to pass in the ID to be read and is added to the URL that is configured. In the first call using the ID Based method the value of Initial ID will be passed as the value. In subsequent calls the maximum value of the data field name as per the ID Field configuration parameter will be used as the value of the ID parameter.

ID Parameter is the name of the parameter that is passed in the requests, it is appended to the configured URL along with the current value for the parameter.

For example if the URL is configured as http://api-server.com/api/v1/data?user=dianomic and the ID Parameter is defined as requestID with the Initial ID of 100, then the full URL that is used in the call will be

http://api-server.com/api/v1/data?user=dianomic&requestID=100

The URL used in the next call will be dependent on the setting of ID Field. If it is left empty then the value last used will be incremented for the next call, provided the previous call was successful. In our above example this would result in the next call using the URL

http://api-server.com/api/v1/data?user=dianomic&requestID=101

If the first call had failed, then the next call would use the same value for our parameter.

A more common case is when the data returned contains ID values for each returned value, in this case the ID Field configuration option is set and the values taken from the response will generate the next ID to use. For example, if the response payload returns sets of readings, each identified by a field called id, then set ID Field to id. A response payload that returned id’s 125, 126 & 127 would then cause the next request to send a value for the parameter of 128.

http://api-server.com/api/v1/data?user=dianomic&requestID=128

Time Based¶

The selection mechanism Time Based is designed for an API that returns values for a time window. It requires two parameters to be passed in the request, Start and End, to specify the time window to the server. The format of the timestamps passed to the server are defined by the Time Format configuration parameter.

%%
The % character.

%a or %A
The name of the day of the week according to the current locale, in abbreviated form or the full name.

%b or %B or %h

The month name according to the current locale, in abbreviated form or the full name.

%c
The date and time representation for the current locale.

%C
The century number (0–99).

%d or %e
The day of month (1–31).

%D
Equivalent to %m/%d/%y. (This is the American style date, very confusing to non- Americans, especially since %d/%m/%y is widely used in Europe. The ISO 8601 standard format is %Y-%m-%d.)

%H
The hour (0–23).

%I
The hour on a 12-hour clock (1–12).

%j
The day number in the year (1–366).

%m
The month number (1–12).

%M
The minute (0–59).

%n
Arbitrary white space.

%p
The locale’s equivalent of AM or PM. (Note: there may be none.)

%r
The 12-hour clock time (using the locale’s AM or PM). In the POSIX locale equivalent to %I:%M:%S %p. If t_fmt_ampm is empty in the LC_TIME part of the current locale, then the behavior is undefined.

%R
Equivalent to %H:%M.

%S
The second (0–60; 60 may occur for leap seconds; earlier also 61 was allowed).

%t
Arbitrary white space.

%T
Equivalent to %H:%M:%S.

%U
The week number with Sunday the first day of the week (0–53). The first Sunday of January is the first day of week 1.

%w
The ordinal number of the day of the week (0–6), with Sunday = 0.

%W
The week number with Monday the first day of the week (0–53). The first Monday of January is the first day of week 1.

%x
The date, using the locale’s date format.

%X
The time, using the locale’s time format.

%y
The year within century (0–99). When a century is not otherwise specified, values in the range 69–99 refer to years in the twentieth century (1969–1999); values in the range 00–68 refer to years in the twenty-first century (2000–2068).

%Y
The year, including century (for example, 1991).

When using the Time Based selection mechanism two parameters may be appended. For example if the URL is configured as http://api-server.com/api/v1/data?user=dianomic, the configuration option Start is defined as startTime and End as endTime, with the Time Format set to be %Y-%m-$dT%H:%M:%s, then the full URL that is used in the call will be

http://api-server.com/api/v1/data?user=dianomic&startTime=2021-07-11T15:12:34&endTime=2021-07-12T12:45:12

If this call succeeds then the next call will use the endTime from this call as the startTime for the next call. The endTime is always the current time.

Request URL Handling¶

The plugin makes HTTP (or HTTPS) GET requests to the configured URL, this may include parameter passing. Parameters used be encoded within the URL of in the plugin configuration, however if a Selection Method other than None is selected extra parameters will be added to the request URL.

Response Payload Handling¶

If the payload of the REST response is a JSON document with simple key/value pairs, e.g.

{ "temperature" : 23.1, "humidity" : 47.2 }

Then no translation script is required, each key/value pair will become a data point within an asset whose name is set in the configuration of the plugin. A working example of this is the /foglamp/ping API call of FogLAMP itself, it produces a response,

{
  "uptime": 27,
  "dataRead": 1063459,
  "dataSent": 617310,
  "dataPurged": 1063024,
  "authenticationOptional": true,
  "serviceName": "FogLAMP",
  "hostName": "foglamp-18",
  "ipAddresses": [
    "192.168.0.173"
  ],
  "health": "green",
  "safeMode": false,
  "version": "1.9.1"
}

This results in an asset which has data points for all the string, numeric and boolean items with the response. In this case the ipAddresses item is ignored as FogLAMP does not currently support string array type data.

If the response payload included nested JSON objects then these will be included also. For example if the response payload was

{
  "motor" : {
      "speed"  : 12345,
      "current" : 1.4
      },
  "gearbox" : {
      "ratio" : 64,
      }
}

The three values would be extracted, speed, current and ratio. How these values are represented will depend on the setting of the Collapse Data configuration option. If this is set to true then a flat reading will be created for each of the three values. If it is set to false then a reading with two objects will be created, one for the motor and one for the gearbox. Within these they will contain the values for the appropriate object. The choice of flattening the data will depend on how the user wishes to use this data upstream within FogLAMP.

If the payload is a JSON document that is an array rather than an object, then it is interpreted as a set of readings. For example a payload that is an array of numbers such as

[
    12.4, 15.8, 18.2
]

Will result in a set of readings, one per value being created. The asset name and data point name will be taken for the Asset configuration option. This same rule applies for arrays of integers, floating point numbers, string or booleans.

The payload may also be an array of objects, in which case each object will be an asset, with the members of the object becoming the data points. These objects may be nested in which case they will follow the same rules as the motor and gearbox example above.

[
     {
       "motor" : {
           "speed"  : 12345,
           "current" : 1.4
           },
       "gearbox" : {
           "ratio" : 64,
           }
     },
     {
       "motor" : {
           "speed"  : 12345,
           "current" : 1.4
           },
       "gearbox" : {
           "ratio" : 64,
           }
     },
 ]

This will create two assets with the name of the asset as the Asset configuration option.

The default asset naming can be overridden by setting a value for the configuration item Asset Field. This can be used to extract a value from the data returned by the API as the name to use for the resultant asset. If the Asset Field configuration item is set to machine and the payload returned by the API calls is as follows.

[
     {
       "machine" : "CNC14698",
       "motor" : {
           "speed"  : 12345,
           "current" : 1.4
           },
       "gearbox" : {
           "ratio" : 64,
           }
     },
     {
       "machine" : "CNC15217",
       "motor" : {
           "speed"  : 12345,
           "current" : 1.4
           },
       "gearbox" : {
           "ratio" : 64,
           }
     },
 ]

The result would be two assets called CNC14698 and CNC15217.

Also if the payload is a simple numeric value the plugin will accept this and create an asset with the data point name matching the topic on which the value was given in the payload.

If the message format is not a JSON document that can be parsed using the built in rules or is in some other format then a Python script should be provided that turns the message into a JSON format.

An example script, assuming the payload in the message is simply a value, might be a follows

def convert(message):
    return {
        'temperature' : float(message)
    }

Note that the message is passed as a string and the data we wish to ingest into FogLAMP in this case is assumed to be a floating point value. The example above of course is unnecessary as the plugin can consume this data without the need of a script.

The script could return either one or two values.

The script should return the JSON document as a Python DICT in the case of a single value.

The script should return a string and a JSON document as a Python DICT in the case of two values, the first of these values is the name of the asset to use and overrides the default asset naming defined in the plugin configuration.

First case sample:

def convert(message):
    return {'temperature_1': 10.2}

Second case sample:

def convert(message):
    return "ExternalTEMP",  {'temperature_3': 11.3}

A single API call be return reading data for multiple assets. In this case the script can return a more complex JSON document that contains both the asset name and the data points for that asset. The return document should return a single JSON objects called readings and within that a set of readings, one per asset, expressed as a number of reading objects. The key of each of these objects becomes the asset name and the value of each is the data points within the asset.

As an example, if a single API call gives us back both data on a motor and on a machine tool, we can process that API response into two distinct assets; motor and tool, each with its own set of data points. In this case the rpm and current of the motor and the temperature and coolant flow rate for the machine tool.

{
    "readings" :
            {
                "motor" : {
                            "rpm"     : 8450,
                            "current" : 1.3
                          },
                "tool" :  {
                            "temperature" : 32.1,
                            "coolant"     : 147
                          }
            }
}

If a script is returning this more complex JSON object it should not return an asset name, if it does return an asset name with this JSON format then the asset name will be ignored.

Timestamp Treatment¶

The default timestamp for a reading collected via this plugin will be the time at which the reading was taken, however it is possible for the API that is being called to include a different timestamp.

Returning a data point called whose name is defined in the Timestamp configuration option will result in the value of that data point being used as the timestamp. This data point will not be added to the reading. The default name of the timestamp is timestamp.

The timestamp data point should be either be a string with the timestamp should be formatted to match the definition given in the Time format configuration parameter, or a numeric value. The format is based on the standard Linux strptime formatting options and is discussed above in the section discussing the Time Based selection method. It should be noted however that this timestamp handling in the payload is independent of the selection method chosen.

If the value of the Timestamp field is numeric then it is assumed to be a Linux epoch value, i.e. the number of seconds since the first of January 1970. A non-integer value will be assumed to contain fractional seconds in the timestamp.

The timezone may be set by using the Timezone configuration parameter to set the offset of the timezone in which the API is running.

The plugin will automatically filter out a second or subsequent readings that have the same timestamp value as previous reading for that same asset. This allows an API which returns the timestamp of the data to be called multiple times and the data will only be ingested once for the given timestamp. The result is the polling rate of the south service can be set independently of the rate the data changes.

Script Error Handling¶

If an error occurs in the plugin or Python script, including script coding errors and Python exception, details will be logged to the error log and data will not flow through the pipeline to the next filter or into the storage service.

Warnings raised will also be logged to the error log but will not cause data to cease flowing through the pipeline.

To view the error log you may examine the file directly on your host machine, for example /var/log/syslog on a Ubuntu host, however it is also possible to view the error logs specific to Fledge from the Fledge user interface. Select the System option under Logs in the left hand menu pane. You may then filter the logs for a specific service to see only those logs that refer to the service which uses the filter you are interested in.

logview_1

Alternatively if you open the dialog for the service in the South or North menu items you will see two icons displayed in the bottom left corner of the dialog that lets you alter the configuration of the service.

logview_2

The left most icon, with the ? in a circle, allows you to view the documentation for the plugin, the right most icon, which looks like a page of text with a corner folded over, will open the log view page filtered to view the service.

Error Messages & Warnings¶

The following are some errors you may see within the log with some description of the cause and remedy for the error.

The supplied Python script does not define a valid “convert” function: The script that has been supplied does not define a Python function called convert. The script must provide a single function called convert that accepts the HTTP response payload and will process that to provide the JSON DICT and an optional asset name to import.
Python error: IndentationError ‘expected an indented block’ in XXXX at line Y of script: The script supplied does not conform to Python requirements for code block indentation. The text XXXX will be replaced with the line of text in error and Y with the line number within the script.
Python error: SyntaxError ‘invalid syntax’ in XXXX at line Y of script: The script supplied does has invalid Python syntax. The text XXXX will be replaced with the line of text in error and Y with the line number within the script.
Python error: ModuleNotFoundError “No module named ‘nosuchpackage’” in supplied script: The script supplied is attempting to import a Python module that is not available.
Python error: TypeError “convert() missing 1 required positional argument: ‘name’” in supplied script: The type of the convert function has been incorrectly defined. The convert function should take a single argument which is the message to process.
Return from Python convert function is of an incorrect type, it should be a Python DICT object or a DICT object and a string: The convert function is returning data of an incorrect type. It may either return a Python DICT, which may be empty, None or a string and a Python DICT.
The plugin is unable to process data without a valid ‘convert’ function in the script.: This warning will periodically be logged following an earlier error that has resulted in an error which prevents the Python convert function from processing the messages. Fix the earlier error to stop this warning being logged.