Application Archiving

Application Archiving is a way to save a copy of an Application’s Device data, Events, and/or Data Tables on either Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage. Application archiving configuration can be found under the “Data Archive” link in an application’s subnavigation.

The archive configuration allows you to specify which Devices’, Events’, and Data Tables’ data should be backed up. After your archive configuration has been set, you have the option to backfill your device data archive and fully archive Events and Data Tables.

Note: Application archiving is only available in organization-owned applications.

Configuration

For each storage service provider, you can select a Service Credential to authenticate the request. If selected, you’ll be required to provide a Credential Name from a dropdown list of your Service Credentials.

If a Service Credential is not selected, each storage service provider requires its own connection configuration:

• Amazon S3: Requires “Region”, “Access Key ID”, and “Secret Access Key” for configuration.
• Google Cloud Storage: Requires “Account Key (JSON)” for configuration. You may optionally provide a “Project ID”, which will override the Project ID associated with the service account key.
• Microsoft Azure Blob Storage: Requires “Account Name”, and “Account Key” for configuration.

All three require a “Bucket” (or “Container” for Azure).

All three have the optional field “Directory Inside the Bucket” (or “Container” for Azure), which specifies a directory for archival files to go; if left unset, the files will be appended to the top-level directory.

Templating

All archive configurations are templatable (except for Credential Method and Credential Name) via your application globals.

Note: If a template is provided for a configuration field, it must resolve to a valid value in order to save the archive configuration.

Permissions

Before setting up your configuration to archive, make sure that your 3rd-party user has the correct permissions.

For Amazon S3, set the user’s IAM Policy to the following guidelines, such that bucketname is replaced with the actual bucket name:

{
  "Version": "YYYY-MM-DD",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:getBucketAcl"],
      "Resource": ["arn:aws:s3:::bucketname"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::bucketname/*"]
    }
  ]
}

For Google Cloud Services, create a service account with ObjectAdmin permissions, which are the minimal permissions. More information on permissions within Google Cloud Services can be found in Google Cloud’s Documentation.

For Azure Blob Storage, ensure that the accessing account has at least the role of Storage Blob Data Contributor assigned to it. More information on role assignment in Azure can be found here.

Device Data

If “Enable Device Data Archiving” is selected, Devices matching your Archive Mode will be archived.

Device Data Archive Frequency

When device data becomes older than 30 days, WEGnology creates a directory in the configured storage service that represents the archived date. The directory will contain a CSV for each device.

The reason WEGnology waits for the data to be older than 30 days is because device state data can be overwritten until the data is older than 30 days.

Device Data Archive Modes

When specifying the “Archive Modes,” the following options are available:

• Archive all devices: All Devices within the application will be selected for data to archive.
• Archive all EXCEPT these devices…: A blacklist of Devices and/or Device Tags.
• Archive ONLY these devices…: A whitelist of Devices and/or Device Tags.

There is also an “Archive Backfill” button which will immediately enqueue a job to archive data from 31 days ago.

Device Data Generated CSV

Each generated CSV file is placed within a directory named by the ISO timestamp of the archived date. The files themselves will be named by the applicationId, deviceId, and the start and end time of the data contained in the file. An example directory and file would be:

wegnology-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-568bf74a1ff37b0100f5123e-1512259200000-1512345599999.csv

Each Device Data CSV will have an “ID” column (for the device ID), a “Timestamp” column (where the timestamp will be represented as a Unix timestamp in milliseconds), an ”ISO Date” column (where the time is represented in human-readable form), as well as a column for each attribute on your device. The following is an example of a CSV:

"ID","Timestamp","ISO Date","Current","On","Inuse"
"568bf74a1ff37b0100f5123e",1512259242342,"2017-12-03T00:00:42.342Z",21.666666666666668,1,1
"568bf74a1ff37b0100f5123e",1512259302424,"2017-12-03T00:01:42.424Z",21.733333333333334,1,1
"568bf74a1ff37b0100f5123e",1512259362505,"2017-12-03T00:02:42.505Z",21.8,1,1
"568bf74a1ff37b0100f5123e",1512259422586,"2017-12-03T00:03:42.586Z",21.866666666666667,1,1
"568bf74a1ff37b0100f5123e",1512259482697,"2017-12-03T00:04:42.697Z",21.933333333333334,1,1

Blob Data

If any of your devices have attributes that are of type Blob, the archive will contain the original base64 strings of data within the CSV.

Archives will never contain the Blob Signed URLs because they expire.

Device Metadata Generated JSON

Alongside the CSV files of data for the individual devices, a JSON file will be created containing metadata about any devices archived for the given day. An example directory and file would be:

wegnology-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-device-metadata.json

This file will be an array of device objects, one for each of the devices included in the archive for this day. It will be in the same format as the device metadata export that can be requested through the user interface.

Events

If “Enable Events Archiving” is selected, all Events will be archived.

There is also a “Full Archive” button which will immediately enqueue a job to archive all accessible Events in the current application.

Event Archive Frequency

Events are archived daily. For each day all the Events that have been created or modified the day before, WEGnology will create a directory in the configured storage service that represents the archived date. The directory will contain a CSV containing all Events and updates.

Events Generated CSV

Each generated CSV file is placed within a directory named by the ISO Date of the archived day. The files themselves will be named by the applicationId, the word “events”, and the archived time of the event data contained in the file. An example directory and file would be:

wegnology-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-events-1512345599999.csv

The Events CSV will have the following columns: “ID” (for the event ID),“Creation Date” (ISO Date), “Last Updated” (ISO Date), “Source ID” (where the event originated), “Source Type”, “Level”, “State”, “Subject”, “Message”, “Device ID”, “Data” (in the form of a JSON), and an “Update Number” (0 for original/parent event). The following is an example of a CSV:

"ID","Creation Date","Last Updated","Source ID","Source Type","Level","State","Subject","Message","Device ID","Data","Update Number"
"5d24b2fc84cd730006e40cbe","2019-07-09T15:30:04.040Z","2019-07-09T15:30:04.042Z","5b031db82f4fa100050fdf35","user","critical","new","broken device",,,,0
"5d24b2e62f909b0006e2a145","2019-07-09T15:29:42.612Z","2019-07-09T15:29:42.619Z","5b031db82f4fa100050fdf35","user","critical","new","lost device",,,,0
"5c54cbfd10f81d000bac06aa","2019-02-01T22:45:17.629Z","2019-05-08T16:24:57.204Z","5c54c7ff1ef216000626fc38","flow","info","acknowledged","Checking","this is an event",,,0
"5c54cbfd10f81d000bac06aa","2019-05-08T14:57:24.085Z",,"5b0db05e3a4350000933f993","user",,"acknowledged",,"this is an update",,,1

Data Tables

If “Enable Data Tables Archiving” is selected all Data Tables matching your “Archive Mode” will be archived.

Data Tables Frequency

Data Tables are archived daily. Each day, WEGnology will create a directory in the configured storage service that represents the archived date. The directory will contain a CSV for each Data Table.

Data Tables Archive Modes

When specifying the “Archive Modes,” the following options are available:

• Archive all data tables: All Data Tables within the application will be selected for data to archive.
• Archive all EXCEPT these data tables…: A blacklist of Data Tables.
• Archive ONLY these data tables…: A whitelist of Data Tables.

There is also a “Full Archive” button which will immediately enqueue a job to archive all Data Tables connected to the application.

Data Tables Generated CSV

Each generated CSV file is placed within a directory provided or named by the ISO timestamp of the archived date. The files themselves will be named by the applicationId, the word “data-table-data”, and the Data Table’s ID. An example directory and file would be:

wegnology-bucket/2017-12-03T00:00:00.000Z/568beedeb436ab01007be53d-data-table-data-5d24b2fc84cd730006e40cbt.csv

Or if you provide a directory in the configuration:

wegnology-bucket/2017-12-03T00:00:00.000Z/your-directory/568beedeb436ab01007be53d-data-table-data-5d24b2fc84cd730006e40cbt.csv

Each Data Table CSV will have an “id” column (for the row ID), your data columns, an “updatedAt” column (ISO Date), a “createdAt” column (ISO Date). The following is an example of a CSV:

"id","Hour","PriceScale","updatedAt","createdAt"
"5d2d45183c4fad0006e657b0",0,0.8199146941389697,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b1",1,0.7345838500288444,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b2",2,0.5752394968861998,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"
"5d2d45183c4fad0006e657b3",3,0.44334167226038557,"2019-07-16T03:31:36.593Z","2019-07-16T03:31:36.593Z"