Customized Solar Dashboard with Raspberry PI and Thingsboard

Last Updated on 12th May 2024 by peppe8o

In this tutorial, I will share a DIY project to create a fully customizable solar dashboard with a Raspberry PI computer board and ThingsBoard. The project will use Python to collect data from the servers of my inverter producer (Azzurro ZCS) by using their public APIs. I will also give you the procedure to reach the goal so that you can adapt my scripts to your inverter producer and its APIs.

Photovoltaic Dashboard and Raspberry PI Communication

The project aims to get solar data from the Cloud servers of the inverter producer rather than directly from the inverter. The following picture summarizes the communication flows:

This project will show you how to create the scheduled data download and the on-demand dashboard. It doesn’t aim to dig into the communication between the solar inverter and the Cloud servers, as this part is usually already implemented by the inverter producer (that’s how the producer gets data for monitoring in your smartphone app).

The main PRO of this solution is that you don’t have to fight to create complex interfaces between the Raspberry PI and the inverter, usually accessible with serial ports requiring a lot of development time. The main CON of this approach is that you will not have “really real-time” data, as you may get delays of around 10-15 minutes (depending on your inverter producer process).

What We Need

As usual, I suggest adding from now to your favourite e-commerce shopping cart all the needed hardware, so that at the end you will be able to evaluate overall costs and decide if to continue with the project or remove them from the shopping cart. So, hardware will be only:

• Raspberry PI Computer Board (including proper power supply or using a smartphone micro USB charger with at least 3A)
• high speed micro SD card (at least 16 GB, at least class 10)
• a solar system/kit, with the inverter producer capable of giving you access to data APIs

Step-by-Step Procedure

Get the API from your Solar Inverter Producer

This part depends on the producer of your solar kit. Many may give you public documentation about using their API, while others may require you to explicitly ask for it. If there’s no documentation from their web pages, try to reach their customer support to get their availability and their development manuals.

In my case, with the Azzurro ZCS, I simply emailed their customer support and they were available to share their API guides, including the authentication procedure and API usage. I will base this tutorial on this inverter producer so that in this case you can just use my scripts. With other inverters, you may anyway adapt my scripts to your needs.

The Azzurro ZCS API

The API I will use enables you to get data from multiple solar kits. There are 4 different data:

• historicData: to get daily data from history and every device
• realtimeData: to get the latest data from every device
• deviceHistoricAlarm: to get daily alarms from history and every device
• deviceAlarm: to get the latest alarms from every device

Every API request (POST) has a single URL as the destination: https://third.zcsazzurroportal.com:19003/. All the requests must include an authentication header with the following parameters:

• “auth key” and “client“: provided by the Azzurro ZCS support
• “thing key“: the serial number of the inverter (or inverters)

Every request will return a number of measures from your inverter (more precisely, from those stored in the cloud from your inverter). You can filter them with the “requiredValues” key or use a “*” wildcard to get all the parameters. I suggest getting all the datasets so that you can decide from your ThingsBoard which values to show.

You can request data for a single device (inverter) or multiple devices. According to your needs, your “thingKey” variable must be set to your single serial number (like, for example, “ZA1AA222B3C444”) or multiple serial numbers (like, for example, “ZA1AA222B3C444, ZB5AA666B7C888, ZC9AA000B1C222”).

historicData requests must include the following POST JSON payload:

{
  "historicData": {
    "command": "historicData",
    "params": {
      "thingKey": "ZA1AA222B3C444",
      "requiredValues": "*",
      "start": "2021-09-15T00:00:00.000Z",
      "end": "2021-09-15T23:59:59.059Z"
    }
  }
}

Realtime data can be achieved with the following JSON:

{
  "realtimeData": {
    "command": "realtimeData",
    "params": {
      "thingKey": "ZA1AA222B3C444",
      "requiredValues": "*"
    }
  }
}

The following applies to deviceHistoricAlarm parameters requests:

{
  "deviceHistoricAlarm": {
    "command": "deviceHistoricAlarm",
    "params": {
      "thingKey": "ZA1AA222B3C444",
      "requiredValues": "*",
      "start": "2021-09-15T00:00:00.000Z",
      "end": "2021-09-15T23:59:59.059Z"
    }
  }
}

Finally, the following applies to deviceAlarm data:

{
  "deviceAlarm": {
    "command": "deviceAlarm",
    "params": {
      "thingKey": "ZA1AA222B3C444",
      "requiredValues": "*"
    }
  }
}

We’ll see in the following chapters how to deal with these requests.

The Goal of our Data Manipulation

Data collected from the API can’t be sent directly to ThingsBoard as they have different formats. In my case, I get real-time data from Azzurro ZCS in the following format:

{
    "realtimeData": {
        "params": {
            "value": [
                {
                    "ZA1AA222B3C444": {
                        "powerConsuming": 800,
                        "thingFind": "2021-04-29T12:23:20Z",
                        "energyAutoconsuming": 0.57,
                        "energyAutoconsumingTotal": 1942,
                        "batteryCycletime": 161,
                        "energyImportingTotal": 2053,
                        "energyCharging": 0,
                        "energyGeneratingTotal": 3572,
                        "energyExportingTotal": 814,
                        "energyChargingTotal": 816,
                        "powerAutoconsuming": 570,
                        "lastUpdate": "2022-01-10T09:44:11Z",
                        "powerExporting": 0,
                        "powerGenerating": 570,
                        "energyImporting": 4.7,
                        "powerImporting": 230,
                        "energyGenerating": 0.57,
                        "energyDischargingTotal": 769,
                        "energyDischarging": 0.12,
                        "energyConsuming": 5.39,
                        "energyExporting": 0,
                        "energyConsumingTotal": 4764,
                        "batterySoC": 17,
                        "powerCharging": 0,
                        "powerDischarging": 0
                    }
                }
            ]
        },
        "success": true
    }
}

On the other hand, ThingBoard requires data to be transmitted with a “ts” key (which is the TimeStamp) and values into a JSON, with the following format:

{
  "ts": 1641808171000,
  "values": {
    "powerConsuming": 800,
    "thingFind": "2021-04-29T12:23:20Z",
    "energyAutoconsuming": 0.57,
    "energyAutoconsumingTotal": 1942,
    "batteryCycletime": 161,
    "energyImportingTotal": 2053,
    "energyCharging": 0,
    "energyGeneratingTotal": 3572,
    "energyExportingTotal": 814,
    "energyChargingTotal": 816,
    "powerAutoconsuming": 570,
    "powerExporting": 0,
    "powerGenerating": 570,
    "energyImporting": 4.7,
    "powerImporting": 230,
    "energyGenerating": 0.57,
    "energyDischargingTotal": 769,
    "energyDischarging": 0.12,
    "energyConsuming": 5.39,
    "energyExporting": 0,
    "energyConsumingTotal": 4764,
    "batterySoC": 17,
    "powerCharging": 0,
    "powerDischarging": 0
  }
}

As you can see, the measures are stored in the JSON object at the following root path:

[“realtimeData”][“params”][“value”][0][“inveter_id”ZA1AA222B3C444”]

The Python scripts in the following chapters will do this job for Azzurro ZCS systems. If you use a different Solar Inverter with a different API, you’ll need to find your root path and adapt my scripts to your cases, ensuring that the output is similar to the last example. In this case, it will be helpful to run the script into a Python IDE (like Thonny) and test the different paths until you find the correct one for your API.

Prepare the Operating System

Please start installing your favourite Operating System for Raspberry PI (it must be Debian-based, I suggest a 64-bit version). I also recommend using Raspberry PI OS Lite, as it provides a headless, faster environment for low-capabilities Raspberry PI computer boards. But you can also use Raspberry PI OS Desktop, in this case working from its internal terminal. You can check the difference between the two OS from my Raspberry PI OS Lite vs Desktop: comparison between the 2 distributions.

Make your OS up-to-date. From the terminal, please use the following command:

sudo apt update -y && sudo apt upgrade -y

We also need pip3, which we’ll use to install the ThingsBoard MQTT package for Python:

sudo apt install python3-pip -y

Prepare ThingsBoard Device in Raspberry PI

For this task, please start from my tutorials already available at the following links to install Docker and ThingsBoard:

• Setup a Docker environment with Raspberry PI OS Lite
• ThingsBoard and Raspberry PI (part 1): Professional IoT dashboards

At the end of the previous tutorial, you should have your ThingsBoard running and your user logged in with the profile of the System Administrator.

The following steps are creating a ThingsBoard Tenant (with related tenant admin) and a Device.

You can create the Tenant with your preferred name (for example, I will name it “Solar Dashboard”). On the other hand, you can also name the Device as you prefer, but I will name it “Solar Inverter” because it makes more familiar with the object producing data and because you can create different Devices even if their data come from a single board (like our Raspberry PI).

So, please use the ThingsBoard and Raspberry PI (part 2): Add Device telemetry by MTTQ and Python tutorial to create the Tenant, the tenant admin and the device. You can follow these steps till the chapter “Create a new Device”, skipping the “Install and Test Mosquitto” and the following chapters.

At this point, you should have your Device’s token.

Prepare Python on Raspberry PI

To collect data, manage and push them to ThingsBoard, we need to install a few packages.

Go back to the home directory of your user:

cd --

Create a new virtual environment in your Raspberry PI (in my case, I will name this virtual environment “solar”).

python3 -m venv solar --system-site-packages

Activate the virtual environment:

source ./solar/bin/activate

An install Pandas and the ThingsBoard MQTT client:

pip3 install pandas tb-mqtt-client

With Pandas dataframes, it will be easier to manipulate data to match your required output and export them to a JSON compatible with ThingsBoard.

Get Solar Kit Data in your Raspberry PI (Azzurro ZCS real-time)

Now, we can perform our test by getting real-time data from the Azzurro ZCS API.

Let’s create our Python script inside the “solar” folder (please note that I’m running with the virtual environment activated):

(solar) pi@raspberrypi:~ $ cd solar

Get my real-time script in your Raspberry PI storage:

wget https://peppe8o.com/download/python/solar/real-time.py

The following paragraphs will explain my script line by line.

In the beginning we import the required modules:

import requests
import json
import pandas as pd
from tb_device_mqtt import TBDeviceMqttClient, TBPublishInfo
from datetime import datetime, timedelta
from dateutil import tz

As Azzurro ZCS exports the timestamp with the GMT timestamp, I also needed to convert GMT data into my TimeZone. For this reason, the script initializes the UTC and localtz variables which we’ll use in the date_conversion() custom function defined later.

utc = tz.gettz('UTC')
localtz = tz.tzlocal()

Then, we initialize the ThingsBoard variables. Please insert the Device token you got from ThingsBoard when creating your device. If ThingsBoard is installed into the same Raspberry PI computer board, you can use the “0.0.0.0” host (which means localhost). If you use a different computer to host ThingsBoard, you must set the related IP address there:

# Thingsboard variables
token="<<your_Thingsboard_dev_access_token>>"
host="0.0.0.0"
port=1883
topic = "v1/devices/me/telemetry"

The API variables follow. Here I set the required variables for authentication with Azzurro ZCS API. Of course, you should use the values coming from the Azzurro customer service for your (or those coming from your Inverter provider):

inveter_id = "<<your_inverter_serial_number>>"
API_auth = "<<your_API_authentication_token>>"
API_client = "<<your_API_client_token>>"

In the same way, we store the URL where to send our request, authentication head and request payload to variables, so that we can manage in a clear way when sending the API request. Please note that we’ll require all the available values, as we’ll be able to filter them later in our dashboard:

url = "https://third.zcsazzurroportal.com:19003/"
head = {
    "Authorization": API_auth,
    "Content-Type": "application/json",
    "client": API_client,
}
realtime = {
  "realtimeData": {
    "command": "realtimeData",
    "params": {
      "thingKey": inveter_id,
      "requiredValues": "*"
    }
  }
}

As anticipated before, the date_conversion() custom function will keep a given datetime string having the format similar to “2022-01-10T09:44:11Z” into a Python datetime object, finally returning it in your local timezone:

def date_conversion(date_time):
    time_date = date_time
    time_date_object =  datetime.strptime(time_date, '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=utc).astimezone(localtz)
    my_time = int(datetime.timestamp(time_date_object))*1000
    return my_time

Now, we’re ready to issue the Python request to the API server:

response = requests.post(url, json=realtime, headers=head)

We’ll store the response into a Pandas dataframe, by creating it from the proper JSON path:

device = pd.DataFrame(response.json()["realtimeData"]["params"]["value"][0][inveter_id], index=[0])

In my script I will store the values and the TimeStamp into 2 different dataframes.

The dev_values will store all the columns, excluding the “lastUpdate” column. Moreover, I will add a “rt_” prefix in the value keys, so that I will be able to distinguish real-time data from historical data in my ThingsBoard:

dev_values = device.loc[:, device.columns!='lastUpdate']
dev_values = dev_values.add_prefix('rt_')

The dev_ts will store only the timestamps:

dev_ts = device.loc[:,'lastUpdate']

We can now push the values to ThingsBoard via MQTT. First of all, we connect to it:

client = TBDeviceMqttClient(host, username=token)
client.connect()

Then we convert the timestamp to the proper format with the date_conversion function:

my_time = date_conversion(dev_ts.loc[0])

And we can push the telemetry data to ThingBoard. The final row checks that the publish process has been completed successfully:

telemetry = {'ts': my_time, 'values': json.loads(dev_values.loc[0].to_json())}
result = client.send_telemetry(telemetry)
success = result.get() == TBPublishInfo.TB_ERR_SUCCESS

After this step, we can close the ThingsBoard connection:

client.disconnect()

Before going to the dashboard design step, we can check that the script works correctly. We can run it from the Raspberry PI terminal:

(solar) pi@raspberrypi:~/solar $ nano real-time.py

Back to the ThigsBoard page, by selecting the “Latest telemetry” tab from your device, you will be able to see the real-time values pulled:

Get Other Data (Azzurro ZCS Historical)

The same script can also be adapted for other data. For example, you can get Azzurro ZCS historical data with a similar script:

wget https://peppe8o.com/download/python/solar/hist-series.py

This script uses the same approach. of course, it is different because it needs to set the start and end date for the required data. Before running the script, you will need to set the following variables with your Device token and your authentication strings:

token="<<your_Thingsboard_dev_access_token>>"
inveter_id = "<<your_inverter_serial_number>>"
API_auth = "<<your_API_authentication_token>>"
API_client = "<<your_API_client_token>>"

In my script, I require the latest 12 hours, as it allows me to recover data even in those rare moments when the Azzurro API is unavailable:

startTime=datetime.now() - timedelta(hours = 12)
startTime=startTime.strftime('%Y-%m-%dT%H:%M:%SZ')
endTime=datetime.now().strftime('%Y-%m-%dT%H:%M:%SZ')

The JSON request payload will be a bit different compared to the real-time one, according to the AZZURRO ZCS API specifications:

histData = {
  "historicData": {
    "command": "historicData",
    "params": {
      "thingKey": inveter_id,
      "requiredValues": "*",
      "start": startTime,
      "end": endTime
    }
  }
}

For this reason, the Python request uses the histData variable, even if authentication and destination URL remain the same:

response = requests.post(url, json=histData, headers=head)

With the historical data, the Azzurro ZCS API stores the timestamps into the JOSN key named “ts”. Anyway, we’ll use again 2 dataframes to store timestamps separately from the values. Please note that here I don’t add any prefixes to historical data:

device = pd.DataFrame.from_dict(response.json()["historicData"]["params"]["value"][0][inveter_id])
dev_values = device.loc[:, device.columns!='ts']
dev_ts = device.loc[:,'ts']

With historical data, we’ll have multiple timestamps to push to ThingsBoard (we’re collecting the latest 12 hours). For this reason, a “for” loop will scan all the stored data and publish them to ThingsBoard with the related values:

for i in range(0, len(dev_ts)):
    my_time = data_conversion(dev_ts.loc[i])
    telemetry = {'ts': my_time, 'values': json.loads(dev_values.loc[i].to_json())}
    result = client.send_telemetry(telemetry)
    success = result.get() == TBPublishInfo.TB_ERR_SUCCESS

Before testing Again, you can run it from your terminal console with the following command:

(solar) pi@raspberrypi:~/solar $ python3 hist-series.py

Create your Solar Dashboard with ThingsBoard

At this point, you should have your first data loaded into your ThingsBoard instance. You can create a new Dashboard, naming it as you like (I will name it “Solar Dashboard”):

Besides the name, you don’t need specific settings:

From the new dashboard, you can add widgets with the button highlighted in the following picture:

Here, you can add all the widgets you want and configure them as you prefer, the only limit is your imagination! The only things you need to know when adding a Widget are:

• Choose the right device (“Solar Inverter” in my case)
• Choose the right measures, as some may appear similar by name, even if carrying different information

For example, let’s add a Time series chart.

Once you add it, you must select the correct device in the Datasource section:

Also, please remind to set the timewindow from real-time to history and select your desired time interval:

After this, you can select the right series in the following section. With Azzurro ZCS systems, those with a name starting with “power” are the measures of a specific time frame, while others are sums of the day. When you’ll click the key field there, ThingsBoard will propose you the available telemetry measures, so making it simpler to you to choose between the available variables.

I will create a simple chart with the lines of generated, imported and used power by selecting the powerGenerating, powerConsuming, and powerImporting variables. For each variable, you can set the label to show in your chart legend, the line colour, the unit label, and more:

Moreover, by clicking the pencil symbol in the Key field of each line, you can issue more specific settings for each series. For example, you can add a “-” in the “return value;” line, so making it “return –value;” and printing the line in the negative side of the axis:

These settings will give you the following output:

By combining different widgets, you can create complex dashboards with a combination of real-time and historical data.

One possible output is the following:

I don’t have an energy storage to include in this dashboard, but you can add it easily in this dashboard with the multiple widgets available in ThingsBoard for the scope.

You can achieve this dashboard by downloading the following ThingsBoard’s configuration JSON:

solar_dashboard.json

Then you can import it with the “Import dashboard” button as shown in the following picture:

Scheduling the Data Collection

At this point, you need to make the data collection automated instead of manually running the 2 Python scripts just created. For this goal, we can use the cron scheduler available in Raspberry PI OS (as well as in every Linux distribution). From your Raspberry PI terminal console, please your user crontab with the following command (remember to change the “pi” user with your one if different):

crontab -u pi -e

If required, you can choose your favourite text editor, like nano. At the end of the file, please append the following cron jobs:

*/3 * * * * source /home/pi/solar/bin/activate && python3 /home/pi/solar/real-time.py
55 * * * * source /home/pi/solar/bin/activate && python3 /home/pi/solar/hist-series.py

Save and close. These crontab lines will execute the real-time script every 3 minutes and the hist-series script every hour.

What’s Next

If you want to discover many other projects for your Raspberry PI, you can take a look at peppe8o Raspberry PI computer tutorials.

Enjoy!