Azure Blob Storage¶

The foglamp-north-azure-blob plugin sends data to Azure Blob Storage, providing flexible organization of data through configurable container structures and file naming patterns.

Configuration Details¶

azure_blob_1

Storage Settings¶

Account Name: Azure Storage account name to which the Containers/Blobs will be published

Account Key: Azure Storage account access key

Data Source: Source of FogLAMP data [readings/statistics]

azure_blob_2

Data Format Settings¶

Image Format: Format for Image data [PNG/JPEG] Used when readings contain image data (bytes, PIL Image, or numpy array).

Data Format: Format for Non-Image data [CSV, JSON, XML, PARQUET] Determines how scalar readings are serialized.

azure_blob_3

Container Organization¶

Container Type: Determines how assets are organized in Azure Storage:

Per Asset - Creates a separate container for each asset, using the asset name as the container name. Example: Asset “temperature” → Container “temperature”

Single Static - Uses one container for all assets, specified by Static Container Name. Example: All assets go to container “my-data”

Single Static with Asset Hierarchy — One container with each asset in its own subfolder. Example: Asset “temperature” → “my-data/temperature/readings.json”

Static Container Name: Name of the container when using Single Static modes. Required if Container Type is “Single Static” or “Single Static with Asset Hierarchy”.

File Naming¶

File Name Pattern: Template for generating blob names using tokens. If not specified, defaults to “{ts}_{asset}” for data and “{ts}_{asset}_{dp_name}” for images.

Available tokens:

Basic tokens:

{ts} - Full timestamp (YYYY_mm_dd_HH_MM_SS_us)

{date} - Date portion (YYYY_mm_dd)

{time} - Time portion (HH_MM_SS_us)

{asset} - Asset name

{datapoint} - Datapoint name (for image data)

{ext} - File extension based on format

Date/Time tokens:

{YYYY} - 4-digit year (e.g., 2025)

{mm} - 2-digit month (e.g., 07)

{dd} - 2-digit day (e.g., 09)

{HH} - Hour in 24-hour format (e.g., 17)

{MM} - Minute (e.g., 51)

{SS} - Second (e.g., 30)

azure_blob_4

Batching Options¶

Batch Mode: Controls how readings are grouped into blobs:

None - Each asset’s readings go to a single blob until plugin restart

Reading Count - Create new blob after specified number of readings

Time - Create new blob after specified time window

Batch Size (rows): Maximum readings per blob when using “Reading Count” mode Example: 1000 readings → create new blob after 1000 readings

Batch Window: Maximum seconds per blob when using “Time” mode Example: 3600 seconds → create new blob every hour

Per-reading Overrides (`blobhint`)¶

For granular control, you can attach a blobhint dictionary to individual readings using azurehint filter. These hints override the plugin’s configuration on a per-reading basis.

Available Hints:¶

container (string) Override the Azure container name. Must follow Azure naming rules (3-63 chars, lowercase letters/numbers/hyphens).

prefix (string) Folder path inside the container (prepended to blob name). Supports all File Name Pattern tokens. Example: “site1/{YYYY}/{mm}/” → “site1/2025/07/data.json”

postfix (string) Text to append before extension. Supports all File Name Pattern tokens. Example: “_{HH}{MM}” → “data_1751.json”

filePattern (string) Custom naming template for this reading. Supports all File Name Pattern tokens. Overrides the global File Name Pattern setting. Example: “{asset}_{YYYY}{mm}{dd}_{HH}{MM}” → “temperature_20250709_1751.json”

metadata (object) Key/value pairs added as Azure blob metadata. Used for searching/filtering in Azure. Example: {“location”: “Building A”, “sensor_type”: “temperature”}

contentType (string) Override the HTTP Content-Type header. Example: “application/json”, “image/png”

Examples¶

Here are examples demonstrating common use cases:

Basic time-based organization:
```
{
  "filePattern": "{YYYY}_{mm}_{dd}_{asset}.{ext}"
}
```
Creates files like: 2025_07_09_temperature.json Good for simple chronological organization.
Hierarchical time-based organization using prefix:
```
{
  "container": "sensor-data",
  "prefix": "{YYYY}/{mm}/{dd}/{HH}",
  "filePattern": "{MM}_{SS}_{asset}"
}
```
Creates paths like: 2025/07/09/17/51_30_temperature.json Ideal for time-based data exploration and retention policies.
Complex pattern with multiple tokens:
```
{
  "filePattern": "{asset}_{YYYY}{mm}{dd}_{HH}{MM}",
  "postfix": "_v1"
}
```
Creates files like: temperature_20250709_1751_v1.json Useful when you need compact filenames with version tracking.
Using with Single Static with Asset Hierarchy:
```
{
  "container": "factory-data",
  "prefix": "{asset}/{YYYY}/{mm}",
  "filePattern": "{dd}_{HH}{MM}_{datapoint}"
}
```
Creates paths like: factory-data/temperature/2025/07/09_1751_sensor1.json Best for organizing multiple assets with time-based subfolders.

Combining with metadata:

{
  "prefix": "site1/{YYYY}/{mm}",
  "filePattern": "{asset}_{HH}{MM}",
  "metadata": {
    "timestamp": "{YYYY}-{mm}-{dd} {HH}:{MM}:{SS}",
    "location": "Building A"
  }
}

Creates files with searchable metadata. Useful for adding context and improving discoverability.

Best Practices¶

Container Organization: - Use “Per Asset” for complete isolation between assets - Use “Single Static with Asset Hierarchy” for unified management with clear organization
File Naming: - Include date/time components for easy chronological browsing - Use consistent patterns across similar assets - Keep names reasonably short while maintaining clarity
Batching: - Use “Time” mode for regular intervals (e.g., hourly files) - Use “Reading Count” for consistent file sizes - Consider storage costs and query patterns when choosing batch sizes
Metadata: - Add relevant search terms as metadata - Include timestamp information for easier querying - Use consistent metadata keys across related assets
Error Handling: - The plugin will sanitize invalid container/blob names - Invalid tokens in patterns fall back to default naming - Failed uploads are retried with exponential backoff