Bucket Storage Service¶

The FogLAMP Bucket Storage Service is an optional service that can be installed as part of a FogLAMP instance which provides storage services to other components of the FogLAMP instance. These storage services take the form of a single file of contents that are indexed by a set of key/value pair attributes. The contents of the file can be anything required by the client of the bucket storage service and may be of any size.

The motivation behind the bucket storage service is a provide a mechanism for other components of the FogLAMP system to persist arbitrary objects, which may be any size from relatively small items to large binary objects. The overhead of the bucket storage service is such that storing very small objects may be wasteful on resources.

Limitations¶

The storage of the content of the buckets is in files on the file system of the machine on which the bucket storage service is running and is thus dependent upon the amount of storage available on that machine.

There are no limitations as to what the contents of the buckets may be, they can be textual objects, binary data or archives of multiple files.

Indexing¶

When a storage bucket is created a number of key/values pairs are supplied with the content of the bucket, it is these that are used to later locate the bucket for retrieval. These key values pairs must consist of string values and each bucket must have a unique set of attributes. Each bucket must have at least one attribute given when it is created.

The creation of a bucket will return a unique ID for a bucket that can be used to retrieve the contents of the bucket.

Searches of the bucket storage service operate against the attributes of the bucket. A subset of the key/value pairs are given in order to find the set of matching storage buckets. The system then returns the unique identifiers for all those buckets that match this subset of attributes.

API¶

The bucket service server a REST API to all of the other services within FogLAMP, as well as offering a modified version of that API via the public REST API of FogLAMP.

A service within FogLAMP wishing to use the bucket service should first contact the FogLAMP core with which it is registered and request the registration record for the bucket storage. Note that there is nothing stopping one FogLAMP service having multiple bucket storage service, although normally each instance would have a single bucket storage service. Access to the bucket service API would then be via the service port of the bucket service API.

Components outside of the FogLAMP instance may also access the bucket service via the FogLAMP public API, in this case a modified form of the service API would be used with the /foglamp/extension prefix added to the URL path of the REST call.

The examples below assume that the FogLAMP instance is running on a host called foglamp.local, this name should be substituted with the real address or hostname of the FogLAMP machine. The examples also assume all the FogLAMP service are running on the same host, hence the localhost address of 127.0.0.1 is used when accessing the service port. Again this should be substituted with the address obtained from the service registry.

Adding a bucket¶

A bucket is added into the bucket store using the POST method of the API. The URL for adding a bucket is /bucket if using the service API from within FogLAMP or /foglamp/extension/bucket if using the FogLAMP public API. The payload associated with this call should be a multipart message. The data itself is sent in a part named bucket, the content of which is an octet stream. The attributes, used for indexing, should be carried in a part named attributes and are expressed as a single JSON object with the set of key value pairs.

If using the curl command to add a new bucket, the command would be of the form

curl -v -X POST -F "[email protected]" -F '[email protected]' http://foglamp.local:8081/foglamp/extension/bucket

The contains of the attributes.json file would be as follows

{
        "name"   : "Tag Mapping",
        "type"   : "TagSheet",
        "device" : "Acme Press"
}

This will create a bucket whose contents are the file given in the bucket= section with three attributes, name, type and device. These attributes names are arbitrary and have no internal significance within the bucket service, they should however be carefully chosen as they must be unique, as a set, for each bucket and they will be used to match particular buckets.

The return payload, upon successfully adding a bucket is a JSON document that contains the unique ID of the bucket.

{
    "bucket" : 3293
}

The above example shows the URL for adding the bucket via the public REST API of FogLAMP, to add it via the service API of the bucket service, which is the option that would be used only by other services within the same FogLAMP instance, the example would be

curl -v -X POST -F "[email protected]" -F '[email protected]' http://127.0.0.1:<service_port>/bucket

Where <service_port> is the service port of the bucket storage service that has previously be obtain from the service registry of the FogLAMP instance using the management REST API.

Error Responses¶

A bad request response will be generated in the following conditions

The request is missing the attributes set

The request is missing the bucket contents

The attribute JSON document could not be parsed

An empty set of attributes was given

One of the attributes had a value that was not a string

A bucket exists that has the same set of attributes as the request

The response will also contain an explanatory message, within a JSON document that describes the nature of the issue.

{
   "message" : "A bucket must have at least one attribute"
}

Retrieving Bucket Contents¶

The API to retrieve the contents of a bucket is a simple GET HTTP request to either the public REST API of FogLAMP or the Bucket Storage Service’s service API from other FogLAMP services. The call requires that you give the unique identifier of the bucket that was returned as the result of the add request or by using the attribute matching API entry point.

curl http://foglamp.local:8081/foglamp/extension/bucket/3293 -o b3293

The response is an octet stream and may be binary if the bucket contains a binary file. The above example shows using the public REST API, the call from within a FogLAMP instance would use the service port of the Bucket Storage Service and a slightly modified URL.

curl http://127.0.0.1:<service_port>/bucket/3293 -o b3293

Where <service_port> is the service port of the bucket storage service that has previously be obtain from the service registry of the FogLAMP instance using the management REST API.

Error Responses¶

A bad request response will be generated in the following conditions

The identifier given does not related to any bucket stored within the system

The bucket contents could not be read

The response will also contain an explanatory message, within a JSON document that describes the nature of the issue.

{
   "message" : "Invalid bucket id"
}

Matching Bucket Attributes¶

Normally it is expected that clients of the Bucket Storage Server would not know the unique identifiers for the buckets of interest and instead would search for them using the attributes of the buckets. This is done using the match API entry point. The entry point is given a set of attributes to match against and will return all those buckets that have the attributes given in the set with identical values. This set may be a subset of the attributes a bucket actually has associated with it.

Matching is done using a PUT method on the REST API with a payload of those attributes to match. The payload is a JSON document similar to the one given in the API call to create the bucket, however it may be a subset of the attributes that the bucket has. The following payload would return all those buckets that have a type attribute whose value is TagSheet.

{
    "attributes" : {
        "type"   : "TagSheet"
        }
}

The example curl command that would be used to perform the match, via the public FogLAMP REST API would be

curl -X PUT http://foglamp.local:8081/foglamp/extension/bucket/match [email protected]

Assuming the file match.json has the contents should above. In the same way as shown in the other example, other services within the same FogLAMP instance would use the service API of the Bucket Storage Server itself to get the matching set of buckets.

curl -X PUT http://127.0.0.1:<service_port>/bucket/match [email protected]

The return of this call is a JSON document that lists all the matching buckets stored within the Bucket Storage Service.

{
   "matches" : [
      {
         "bucket" : 3293,
         "file"   : "/usr/local/foglamp/data/buckets/3/3293",
         "attributes" : {
             "name"   : "Tag Mapping",
             "type"   : "TagSheet",
             "device" : "Acme Press"
          }
      },
      {
         "bucket" : 2712,
         "file"   : "/usr/local/foglamp/data/buckets/2/2712",
         "attributes" : {
             "name"   : "Drier 002",
             "type"   : "TagSheet",
             "device" : "Drier"
          }
      }
   ]
}

The result of this can then be used to retrieve the particular bucket of interest. Note this return payload includes the internal filename in which the bucket is stored, callers should not normally use this information is the Bucket Storage Service may not be running on the same host and the filename will then not be valid locally. All access should be via the REST API.

Error Responses¶

A bad request response will be generated in the following conditions

The attributes are missing from the payload

One of the attribute values is not a string

The attributes JSON document could not be parsed

The response will also contain an explanatory message, within a JSON document that describes the nature of the issue.

{
   "message" : "Badly formed payload, missing attributes to match"
}

Updating Bucket Attributes¶

It is possible to update bucket attributes by overwriting the values of existing attributes or adding new attributes, it is not possible to remove attributes or to change the content of the bucket itself. The API entry point must be given the unique identifier for the bucket that should be updated and the set of new attributes as a JSON document.

The API entry point for the public FogLAMP API is

curl -X PUT http://foglamp.local:8081/foglamp/extension/bucket/3293 -d '{ "attributes" : { "status" : "current" } }'

Where the 3293 is the unique identifier for the bucket to be updated. The above example will add a new attribute status to the bucket with the value of current. If the bucket already has an attribute names status then the value of that attribute will be updated.

Again a service that is part of a FogLAMP instance should use the service API port of the Bucket Storage Service itself rather than the public REST API.

curl -X PUT http://127.0.0.1:<service_port>/bucket/3293 -d '{ "attributes" : { "status" : "current" } }'

Error Responses¶

A bad request response will be generated in the following conditions

The identifier is not a valid identifier of a bucket stored in the Bucket Storage Server

The attributes are missing form the update

The attributes JSON document could not be parsed

One of the attributes has a value that is not a string

The response will also contain an explanatory message, within a JSON document that describes the nature of the issue.

{
   "message" : "Badly formed payload, missing attributes to update"
}

Deleting a Bucket¶

To delete a bucket the DELETE method should be used, giving the unique identifier of the bucket to be deleted in the URL

curl -X DELETE http://localhost:8081/foglamp/extension/bucket/3292

The above example shows the use of the public FogLAMP API to delete bucket 3292 from the system. As with all entry points internal services would use the Bucket Storage Service API via the service port of the service itself.

curl -X DELETE http://localhost:<service_port>/bucket/3292

Error Responses¶

A bad request response will be generated in the following conditions

The identifier given does not related to any bucket stored within the system

The response will also contain an explanatory message, within a JSON document that describes the nature of the issue.

{
   "message" : "Invalid bucket id"
}