Introduction
Spoony integrate a service able to push datapoint information (ex: Power measurements) to a remote Http server in the form of JSON data.
This service allows the device to send information to the server, without need for complex network configuration, as the device connect itself to the remote server and use w3c standards protocols and data format.
In typical applications, the remote server is available through Internet connection, but using a server on the local network is also possible.
In most case, the server will simply receive data from the device and store it in a database, but other scenarios are possible according to user requirements.
1. Basic requirements
Basic requirements include:
- An Http server infrastructure able to accept POST request containing JSON data
- A network connectivity between the device and the server
2. Mode of operation
Once configured, the device will sample datapoints on a regular basis (generally at 1Hz, ie one sample per second).
Once sampled, the values (called Records) are encapsulated in a JSON frame (the DataChunk) and sent to the remote server at the configured address, using an HTTP POST request.
If connection with the remote server fail, the device is able to store internally up to 14400 DataChunk, representing up to 4 hours of data at a sample rate of 1Hz.
3. Data format
The DataChunks sent by spoony are organized as follow:
|
DataChunk |
||
|
Field name |
Type |
Description |
|
“from” |
DataChunkSource (see below) |
Source of DataChunk. Identify source device and internal unit. |
|
“t” |
Date |
Timestamp of DataChunk. Date at which the DataChunk was sent from the device. ISO 8601 format. |
|
“count” |
Integer |
Number of items in the following array ‘elements’. |
|
“elements” |
DataPointChunk[] (see below) |
Array containing sampled data, regrouped by DataPoint. |
|
DataChunkSource |
||
|
Field name |
Type |
Description |
|
“deviceId” |
String |
Source device identifier. As configured on device. User is responsible for unicity of devices IDs within the target server. |
|
“unit” |
String |
Source of data within the device (Service name). Always ODMDataChunk in current devices revisions. Subject to change without notification. |
|
DataPointChunk |
||
|
Field name |
Type |
Description |
|
“name” |
String |
Name of the datapoint. See below for available datapoints and meanings. |
|
“count” |
Integer |
Number of items in the following array ‘records’. |
|
“records” |
DataPointRecord[] (see below) |
Array of records. Each record is a single datapoint sample. |
|
DataPointRecord |
||
|
Field name |
Type |
Description |
|
“i” |
Integer |
Index of the sample. Index get incremented each time the concerned datapoint is sampled. |
|
“t” |
Date |
Sample Timestamp. ISO 8601 format. |
|
“q” |
String |
Quality of the sample. Always 'good' in current devices revisions. Possible values are: “good”, “bad”, “uncertain”, “unknown” |
|
“v” |
Floating-point |
Value of the sample. All Spoony datapoints are single-precision floating-point numbers. |
4. Transport protocol
JSON DataChunk objects (as described in previous paragraphs), are sent to the remote server using HTTP POST requests.
Spoony always send data using Chunked transfer encoding, as per RFC 2145.
Spoony offer two possibilities for data transfert: raw or compressed JSON data.
4.1 Raw JSON data
Raw JSON data is transported directly over HTTP.
Raw JSON data is sent to the remote server with following header:
Content-Type: application/json
as per RFC 1341 and RFC 4627
4.2 Compressed JSON data
JSON data is compressed before being sent over HTTP.
Spoony use a specific compression algorithm based on LZSS adapted to constrained devices. Embedded (and server-side) compression and decompression library comes from the Open-Source project heatshrink (https://github.com/atomicobject/heatshrink). This compression algorithm use two parameters which must be known for compression and decompression: window and lookahead.
Compressed JSON data is sent to the remote server with following header:
Content-Type: application/octet-stream
As per RFC 1341 and RFC 2046
Body of the http request (binary data) is organized as follow:
|
Bytes |
Field |
Description |
Value |
|
0-5 |
HEADER |
Heatshrink data header |
0x50, 0x41, 0x4E, 0x44, 0x41, 0x5A |
|
6 |
V_MAJOR |
Version Major |
0x01 |
|
7 |
V_MINOR |
Version Minor |
0x00 |
|
8 |
WINDOW |
Heatshrink Window compression parameter |
Variable |
|
9 |
LOOKAHEAD |
Heatshrink Lookahead compression parameter |
Variable |
|
10 |
TL |
MIME-Type length |
Variable |
|
11–(11 + TL) |
MIME_TYPE |
MIME-Type of compressed data |
Variable |
|
(11+TL+1)–End |
Data |
Heatshrink compressed data |
Variable |
MIME_TYPE field value is “application/json” (thus TL = 16) when sending compressed JSON data.
DotVision offers free of charge the Source code used for parsing and decompression of compressed data, included in the sample Open-Source server (see paragraph 5).
Compression allow for network bandwidth reduction and reduced data usage, and is particularly suited for use over 3G/4G networks, or any other kind of pay per use network infrastructure.
While still being compliant with w3c standards, this mode of operation requires a proprietary server-side decompression algorithm.
5. Server implementation
Following are some guidelines regarding the server implementation:
- Server MUST accept HTTP POST requests.
- Server MUST answer 200 OK to POST request if received data is valid. If response is not sent, or if response is different from 200 OK, Spoony will retry to send the same DataChunk until his internal buffer overflow.
- Server MUST answer within 2s.
- Server SHOULD verify that source device is authorized to publish data and SHOULD ignore data coming from unknown or unauthorized devices.
- Server SHOULD verify incoming data and validate it against the provided JSON schema before handling data.
- Server SHOULD accept both compressed and raw JSON data.
DotVision can provide customers with a sample server implementation in the form of an Open-Source Microsoft® IIS Http Handler (IIS 7.0 or newer), storing data into an InfluxDb database.
Sample server include:
- Heatshrink Open-Source c library written by Scott Vokes (https://github.com/atomicobject/heatshrink)
- Heatshrink .NET C# wrapper in form of compression/decompression Stream and utility functions
- .NET C# front-end for receiving both compressed and uncompressed data transparently from devices.
- .NET C# IIS Sample handler for DataChunk Requests.
- .NET C# IIS Sample handler for Ping Request (used for device discovery and diagnostic and described in another document).
Both handlers store received data to a specified InfluxDb Database (v0.13) but can be adapted easily to send received data to any kind of database.
Provided source code can be used free of charge by customers to receive data from DotVision devices.
6. Device Configuration
Device is configured using xml files stored on the embedded MicroSD card.
Following files allow configuration of the DataChunkWebPushService:
|
File: system\services\Device\ODM\ODMDataChunk\serviceSettings.xml |
|
<configuration> <settings> <add key="sourceAutomationService" value="Automation"/> <add key="samplingPeriod" value="1"/> <add key="chunkPoolSize" value="15"/> <add key="datapoint" value="ade7758/FREQ"/> <add key="datapoint" value="ade7758/VRMSA"/> <add key="datapoint" value="ade7758/VRMSB"/> <add key="datapoint" value="ade7758/VRMSC"/> <add key="datapoint" value="ade7758/IRMSA"/> <add key="datapoint" value="ade7758/IRMSB"/> <add key="datapoint" value="ade7758/IRMSC"/> </settings> </configuration> |
This file contains the Sample Period (Parameter “samplingPeriod”, expressed in seconds, value here is 1s), and the list of the datapoints being sampled. Here the following datapoints will be sampled and sent to the remote server:
ade7758/FREQ, ade7758/VRMSA, ade7758/VRMSB, ade7758/VRMSC, ade7758/IRMSA, ade7758/IRMSB and ade7758/IRMSC.
Available datapoints and meaning are explained in paragraph 7.
|
File: system\services\Device\ODM\ODMWebPush\serviceSettings.xml |
|
<WebPushService version="1.0"> <settings> <add key="uri" value="http://www.yourserver.com:port/push"/> <add key="shrink" value="false" /> </settings> <headers size="1024"><!-- size will define the memory footprint to store the headers values --> <add key="User-Agent" value="Mozilla/5.0 (Windows; U; Windows NT 6.1; rv:2.2) Gecko/20110201"/> <!-- key HAS to be a known header --> </headers> <encoder type="DatapointRecordEncoder_Json_t"/> </WebPushService> |
This file contains the address of the target server (Parameter “uri”). Given address can include an optional port number.
The parameter “shrink”, when set to “true”, enable compression of JSON data as described in paragraph 4.2. If parameter is omitted or set to false, Raw (uncompressed) JSON data is sent directly to the server.
|
File: system\services\Device\serviceSettings.xml |
|
<configuration version="1.0">
<!-- Service manager part --> <settings> <add key="rootPath" value="system/services/device"/> <add key="fileName" value="serviceSettings.xml"/> </settings>
<!-- Device declaration --> <device> <serialNumber>123456789</serialNumber> <location>48.8566140,2.3522220,0.0</location><!--PARIS--> <friendlyName>SpoonyDotVision-VirtualDevice</friendlyName> <firmwareVersion>1.5.23</firmwareVersion> <model> <manufacturer>DotVision</manufacturer> <manufacturerUrl>dotvision.com</manufacturerUrl> <name>spoony</name> <number>1.5</number> <url/> <presentationUrl/> </model> </device>
<services> <service> <type>NtpClient_t</type> <name>Ntp</name> <startMode group="2">automatic</startMode> </service>
[ . . . ]
<service> <type>ODMService_t</type> <name>ODM</name> <startMode group="3">automatic</startMode> </service>
<service> <type>CsvLogService_t</type> <name>CsvLog</name> <startMode group="3">manual</startMode> </service> </services> </configuration> |
Service “ODM” must be declared and configured with automatic startMode in order to be started when the device starts.
7. Available datapoints
These datapoints are available in Spoony Automation service and can be sampled.
|
Datapoint (unit/name) |
Description |
Unit |
|
ODMDataChunk/TEMP |
Device temperature |
°C |
|
ODMDataChunk /FREQ |
Line Frequency |
Hz |
|
ODMDataChunk /VRMSA |
VRMS Phase L1 |
V |
|
ODMDataChunk/VRMSB |
VRMS Phase L2 |
V |
|
ODMDataChunk/VRMSC |
VRMS Phase L3 |
V |
|
ODMDataChunk/IRMSA |
IRMS Phase L1 |
A |
|
ODMDataChunk/IRMSB |
IRMS Phase L2 |
A |
|
ODMDataChunk/IRMSC |
IRMS Phase L3 |
A |
|
ODMDataChunk/WATTA |
Active power Phase L1 |
W |
|
ODMDataChunk/WATTB |
Active power Phase L2 |
W |
|
ODMDataChunk/WATTC |
Active power Phase L3 |
W |
|
ODMDataChunk/VAA |
Apparent power Phase L1 |
VA |
|
ODMDataChunk/VAB |
Apparent power Phase L2 |
VA |
|
ODMDataChunk/VAC |
Apparent power Phase L3 |
VA |
|
ODMDataChunk/VARA |
Reactive power Phase L1 |
VAR |
|
ODMDataChunk/VARB |
Reactive power Phase L2 |
VAR |
|
ODMDataChunk/VARC |
Reactive power Phase L3 |
VAR |
|
ODMDataChunk/AWATTHR |
Active Energy Phase L1 |
Wh |
|
ODMDataChunk/BWATTHR |
Active Energy Phase L2 |
Wh |
|
ODMDataChunk/CWATTHR |
Active Energy Phase L3 |
Wh |
|
ODMDataChunk/AVARHR |
Reactive Energy Phase L1 |
VARh |
|
ODMDataChunk/BVARHR |
Reactive Energy Phase L2 |
VARh |
|
ODMDataChunk/CVARHR |
Reactive Energy Phase L3 |
VARh |
|
ODMDataChunk/AVAHR |
Apparent Energy Phase L1 |
VAh |
|
ODMDataChunk/BVAHR |
Apparent Energy Phase L2 |
VAh |
|
ODMDataChunk/CVAHR |
Apparent Energy Phase L3 |
VAh |
|
ODMDataChunk/PFA |
Power Factor Phase L1 |
- |
|
ODMDataChunk/PFB |
Power Factor Phase L2 |
- |
|
ODMDataChunk/PFC |
Power Factor Phase L3 |
- |
Annex 1: JSON Schema of DataChunk JSON objects
|
DataChunk JSON Schema |
|
{ "$schema": "http://json-schema.org/draft-04/schema#", "title": "SpoonyDataChunk", "type": "object", "properties":{ "from": { "description": "Source of Datachunk. Identify source device and internal unit.", "type": "object", "properties":{ "deviceId": { "description": "Source device identifier. As configured on device. User is responsible for unicity of devices IDs whitin the target server.", "type": "string" }, "unit": { "description": "Source of data within the device (Service name). Always ODMDataChunk in current devices revisions.", "type": "string" } } }, "t": { "description": "Timestamp of DataChunk. Date at which the DataChunk was sent from the device. ISO 8601 format.", "type": "string" }, "count": { "description": "Number of items in the following array 'elements'.", "type": "number" }, "elements": { "description": "Array containing sampled data, organized by datapoint.", "type": "array", "items": { "type": "object", "properties": { "name": { "description": "Name of the datapoint.", "type": "string" }, "count": { "description": "Number of items in the following array 'records'.", "type": "number" }, "records": { "description": "Array of records. A records is a single datapoint sample", "type": "array", "items": { "type": "object", "properties": { "i": { "description": "Index of the sample. Index get incremented each time the concerned datapoint is sampled.", "type": "number" }, "t": { "description": "Timestamp of the sample. ISO 8601 format.", "type": "string" }, "q": { "description": "Quality of the sample. Always 'good' in current devices revisions.", "type": "string" }, "v": { "description": "Value of the sample. All Spoony datapoints are single-precision floating-point numbers.", "type": "number" } } } } } } } } }
|
Annex 2: Sample JSON DataChunk Object
|
Sample DataChunk JSON object |
|
{ "from": { "deviceId": "SpoonyDotVisionDev", "unit": "ODMDataChunk" }, "t": "2016-07-05T15:13:54.013Z", "count": 29, "elements": [{ "n": "TEMP", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 25.03737 }] }, { "n": "FREQ", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 50.00000 }] }, { "n": "VRMSA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 220.03842 }] }, { "n": "VRMSB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 219.70237 }] }, { "n": "VRMSC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 220.19691 }] }, { "n": "IRMSA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": -9.85277 }] }, { "n": "IRMSB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 7.34283 }] }, { "n": "IRMSC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 2.99373 }] }, { "n": "WATTHRA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": -1250.21447 }] }, { "n": "WATTHRB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 887.71948 }] }, { "n": "WATTHRC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 382.72506 }] }, { "n": "VARHRA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": -54.76239 }] }, { "n": "VARHRB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 37.80604 }] }, { "n": "VARHRC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 16.43738 }] }, { "n": "VAHRA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": -1251.86120 }] }, { "n": "VAHRB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 888.86926 }] }, { "n": "VAHRC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 383.22750 }] }, { "n": "WATTA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": -2163.50097 }] }, { "n": "WATTB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 1609.89099 }] }, { "n": "WATTC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 658.52893 }] }, { "n": "VARA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": -139.41661 }] }, { "n": "VARB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 103.86761 }] }, { "n": "VARC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 30.00405 }] }, { "n": "VAA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": -2167.98828 }] }, { "n": "VAB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 1613.23815 }] }, { "n": "VAC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 659.21209 }] }, { "n": "PFA", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 0.99793 }] }, { "n": "PFB", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 0.99792 }] }, { "n": "PFC", "count": 1, "records": [{ "i": 2068, "t": "2016-07-05T15:13:53.998Z", "q": "good", "v": 0.99896 }] }] } |
Annex 3: Sample Grafana Dashboards using Spoony data (real-time monitoring and data analysis)
Dashboards rendered in web browser using Grafana
Commentaires
0 commentaire
Vous devez vous connecter pour laisser un commentaire.