TECHZEN Zenoss User Community ARCHIVE  

Processing JSON output from a Data Source

Subject: Processing JSON output from a Data Source
Author: Chris Gregors
Posted: 2017-08-24 17:39

Background:
Zenoss data source is configured with 4 data points
An appropriate graph has been constructed.

Problem
Ain't getting no data satisfaction eh!

Thoughts ?


------------------------------
Chris Gregors
Enbridge Pipelines Inc.
------------------------------


Subject: RE: Processing JSON output from a Data Source
Author: Patrick McMahon
Posted: 2017-08-25 11:01

Could be the format of the JSON your generating? The JSON output you posted seems to not follow the expected structure like nested JSON objects with a "values" key and a key per component

Take a look at http://wiki.zenoss.org/JSON_Parser

------------------------------
Patrick McMahon
Sr. Client Services Engineer
Zenoss
------------------------------


Subject: RE: Processing JSON output from a Data Source
Author: Chris Gregors
Posted: 2017-08-25 11:11

I thought about that but since JSON is parsed in an XML'ish fashion, you can do deep nesting like shown on http://wiki.zenoss.org/JSON_Parser but it's not necessary.

I'm thinking the JSON conforms but is not extensive as the one shown on http://wiki.zenoss.org/JSON_Parser but it's not necessary.

It does conform two levels down like this:

{
        "datapoint1": 123.4,
        "datapoint2": 987.6
      }
which is comparable to what the script is generating

{
 "begin_time": "17/08/15 22:23:35", 
  "begin_time": "17/08/15 22:23:35",
"begin_time_unix": 1502861015,
"end_time": "17/08/24 14:08:25",
"end_time_unix": 1503608905,
"filepath": "f:\\data\\datafile.dt",
"file": "datafile.dt",
"seconds_behind": 6,
"seconds_behind_error": 0,
"time_range": 747890,
"time_range_error": 0 }


------------------------------
Chris Gregors
Enbridge Pipelines Inc.
------------------------------ Zenoss data source is configured with 4 data points
An appropriate graph has been constructed.

Problem
Ain't getting no data satisfaction eh!

Thoughts ?


------------------------------
Chris Gregors
Enbridge Pipelines Inc.
------------------------------


Subject: RE: Processing JSON output from a Data Source
Author: Chris Gregors
Posted: 2017-08-25 11:22

Of course if you have a suggestion to minimal changes to the JSON format that will make it conform with the parser's requirements, I'll give that a shot.

------------------------------
Chris Gregors
Enbridge Pipelines Inc.
------------------------------


Subject: RE: Processing JSON output from a Data Source
Author: Patrick McMahon
Posted: 2017-08-25 17:14

Based on my reading of the latest JSON parser code (https://github.com/zenoss/zenoss-prodbin/blob/8097e1fcdd73da3d0f73c1f2548997f49faf99cd/Products/ZenRRD/parsers/JSON.py) I would suggest you
pre-pend {"values": {"":
and append }}

Like so...

{"values": {"":{
"begin_time": "17/08/15 22:23:35",
  "begin_time": "17/08/15 22:23:35",
"begin_time_unix": 1502861015,
"end_time": "17/08/24 14:08:25",
"end_time_unix": 1503608905,
"filepath": "f:\\data\\datafile.dt",
"file": "datafile.dt",
"seconds_behind": 6,
"seconds_behind_error": 0,
"time_range": 747890,
"time_range_error": 0 }}}

This assumes you're collecting for the device, denoted by the empty string "", if this is for a component add the component id within the double quotes


------------------------------
Patrick McMahon
Sr. Client Services Engineer
Zenoss
------------------------------


Subject: RE: Processing JSON output from a Data Source
Author: Chris Gregors
Posted: 2017-08-25 17:21

I thought it might come to that so I'm one step ahead of you.  Here's the new JSON returned:

{
    "values": {
        "RTUData_Error_Checking": {
            "seconds_behind": 4,
            "seconds_behind_error": 0,
            "time_range": 737880,
            "time_range_error": 0
        }
    }
}

 
I've attached the template definition also.  Still no joy.  No data on the graph


BTW: I tried switching to the AUTO parser and hacked the script to produce the following output.  No luck there either.

OK| seconds_behind=3 seconds_behind_error=0 time_range=738217 time_range_error=0



------------------------------
Chris Gregors
The secrets of the universe are simple. I just can't figure them out!
------------------------------
Based on my reading of the latest JSON parser code (https://github.com/zenoss/zenoss-prodbin/blob/8097e1fcdd73da3d0f73c1f2548997f49faf99cd/Products/ZenRRD/parsers/JSON.py) I would suggest you
pre-pend {"values": {"":
and append }}

Like so...

{"values": {"":{
"begin_time": "17/08/15 22:23:35",
  "begin_time": "17/08/15 22:23:35",
"begin_time_unix": 1502861015,
"end_time": "17/08/24 14:08:25",
"end_time_unix": 1503608905,
"filepath": "f:\\data\\datafile.dt",
"file": "datafile.dt",
"seconds_behind": 6,
"seconds_behind_error": 0,
"time_range": 747890,
"time_range_error": 0 }}}

This assumes you're collecting for the device, denoted by the empty string "", if this is for a component add the component id within the double quotes


------------------------------
Patrick McMahon
Sr. Client Services Engineer
Zenoss
------------------------------
{
        "datapoint1": 123.4,
        "datapoint2": 987.6
      }
which is comparable to what the script is generating

{
 "begin_time": "17/08/15 22:23:35",
  "begin_time": "17/08/15 22:23:35",
"begin_time_unix": 1502861015,
"end_time": "17/08/24 14:08:25",
"end_time_unix": 1503608905,
"filepath": "f:\\data\\datafile.dt",
"file": "datafile.dt",
"seconds_behind": 6,
"seconds_behind_error": 0,
"time_range": 747890,
"time_range_error": 0 }


------------------------------
Chris Gregors
Enbridge Pipelines Inc.
  • Windows powershell invoked batch file that calls a python script that spits out JSON
    • C:\utilities\tail.bat -j f:\data\datafile.dt
  • JSON output
    • {
      "begin_time": "17/08/15 22:23:35",
      "begin_time_unix": 1502861015,
      "end_time": "17/08/24 14:08:25",
      "end_time_unix": 1503608905,
      "filepath": "f:\\data\\datafile.dt",
      "file": "datafile.dt",
      "seconds_behind": 6,
      "seconds_behind_error": 0,
      "time_range": 747890,
      "time_range_error": 0
      }
    • And these are some relevant settings:
Zenoss data source is configured with 4 data points
  • seconds_behind
  • seconds_behind_error
  • time_range
  • time_range_error
An appropriate graph has been constructed.

Problem
Ain't getting no data satisfaction eh!

Thoughts ?


------------------------------
Chris Gregors
Enbridge Pipelines Inc.
------------------------------

Attachments:

ScreenClip.png



Subject: RE: Processing JSON output from a Data Source
Author: Patrick McMahon
Posted: 2017-08-28 08:51

With regards to the JSON, the component id reference "RTUData_Error_Checking"looks like the datasource name? I'd double check you have the right component id (The component id is set by the modeler)... or if this isn't a component and it's a device, use a blank id.

Looking at your test of the generic command parser, that should work... anecdotally (and if you haven't done so already) I'd say tear everything down and start fresh.


------------------------------
Patrick McMahon
Sr. Client Services Engineer
Zenoss
------------------------------


< Previous
Zenoss 5.2.6 services stuck starting in Docker
  Next
RM Graph Rendering Issue
>