Jump to content
Search In
  • More options...
Find results that contain...
Find results in...

Append to a json file in python

 Share

I currently have raspberry pi zero 2W running a python scrip I've made which essentially takes measurement of the temperature and humidity every 2 seconds using a DHT20 sensor and stores it alongside the date and time in a json file. I've decided to use json since its convenient and easier to load in another script to plot the data. The way it currently works is by loading in the entire json file, adding the respective readings to it and then essentially overwriting the already existing file. This wasn't an issue at first however over about 2-3 days I've already gathered about 68000 measurements which is about a 3.5mb file and now rather than there being about 2-3 seconds between measurement in the file it's more like 4-5 which isn't a massive issue at the moment however it will be over time.

 

So I was wondering is there any way to append data entries to a json file rather than having to load and rewrite the entire file. The tutorials I can find online that mention appending use the method I previously mentioned. Should I just stop using json and rewrite the code to use a simple text file to which it can append data?

 

This is my code if it's of any use

import time
import board
import adafruit_ahtx0
import json
from datetime import datetime
i2c=board.I2C()
sensor=adafruit_ahtx0.AHTx0(i2c)

def current_date():
    now=datetime.now()
    day=str(now.day)
    month=str(now.month)
    year=str(now.year)
    date=f"{day}/{month}/{year}"
    return date

def current_time():
    now=datetime.now()
    hour=str(now.hour)
    minute=str(now.minute)
    second=str(now.second)
    time=f"{hour}:{minute}:{second}"
    return time
start=time.time()
while True:
    temp=round(sensor.temperature, 2)
    hum=round(sensor.relative_humidity, 2)
    print("Temperature:",temp,"C")
    print("Humidity",hum,"%\n")
    with open('/home/pi/test.json') as f:
        data=json.load(f)
    temps=data["temperatures"]
    hums=data["humidities"]
    dates=data["dates"]
    times=data["times"]
    dates.append(current_date())
    times.append(current_time())
    temps.append(temp)
    hums.append(hum)
    data["temperatures"]=temps
    data["humidities"]=hums
    data["dates"]=dates
    data["times"]=times

    with open('/home/pi/test.json',"w") as f:
        json.dump(data, f, indent=2)    
    final=time.time()-start
    if final > 60:
        start=time.time()
        print("Creating backup")
        with open("/home/pi/backup.json","w") as f:
            json.dump(data, f, indent=2)
        print("Backup created succesfully\n")
    time.sleep(2)

Also I know I could just save to a json every 60 seconds to 2 minutes rather than doing it every time I take a measurement but this doesn't fix the issue it just postpones it.

Link to comment
Share on other sites

Link to post
Share on other sites

21 minutes ago, AndreiArgeanu said:

So I was wondering is there any way to append data entries to a json file rather than having to load and rewrite the entire file.

Yes, but you shouldn't.

 

21 minutes ago, AndreiArgeanu said:

Should I just stop using json and rewrite the code to use a simple text file to which it can append data?

You have several different options:

  • Use a local instance of SQLite, which would require more of a rewrite of your current program, but it's easy to setup and use
  • Use a local instance of MongoDB, which you could just spit JSON into, but if you're unfamiliar with NoSQL than there's a learning curve
  • Use something like Prometheus logging, which is precisely what this is designed for, but requires knowledge of PromQL, but it comes with built in visualization
  • Log to a text file as you suggested, then program an import script. The problem with this solution is that your text file will eventually become unwieldy in size. I recommend against this.

If you wait long enough I can detail out a solution for you.

Link to comment
Share on other sites

Link to post
Share on other sites

3 minutes ago, Sakuriru said:

You have several different options:

  • Use a local instance of SQLite, which would require more of a rewrite of your current program, but it's easy to setup and use
  • Use a local instance of MongoDB, which you could just spit JSON into, but if you're unfamiliar with NoSQL than there's a learning curve
  • Use something like Prometheus logging, which is precisely what this is designed for, but requires knowledge of PromQL, but it comes with built in visualization
  • Log to a text file as you suggested, then program an import script. The problem with this solution is that your text file will eventually become unwieldy in size. I recommend against this.

I would like to use a simple solution but if any of these options provide me with useful skills that I could potentially use in other programs I make in the future I'd be willing to go into them. I don't really need visualization since the pi will spend most of its time away from any display output, I only really use it to log the data and then transfer it to my PC to work on it.

Link to comment
Share on other sites

Link to post
Share on other sites

JSON by design is not meant to be append-able .. but in theory you could, if the structure is easy enough. For example, you could just read the last 1 KB from the json file, find the last } or ]  occurrence (the end of an array/object  holding your records, place your file pointer before the  ]  / }  and add your records and put back the ] or } 

 

The solution is to use a different format, like CSV or TSV (Tab separated values), or some other format where each record has a fixed size or specified size. 

 

TSV would be easiest, as you separate each value with a TAB character (0x09), which is unlikely to appear in regular data, unlike comma .. so you get one record per line,  and each line has  multiple values separated by tab. 

 

Another option would be to serialize each record to json, and store each record as an individual json file... one json / record per line.

 

 

You can have a binary format, where each record is basically  [ 2 bytes : length of record ]  [  record data ]   and in record data you decide how many bytes each value will use ... store a temperature as a 32 bit float value , store a time as an unsigned 32 bit int ( ms from 1970 or whatever)...

 

Link to comment
Share on other sites

Link to post
Share on other sites

I used the Prometheus method to create a solution: https://github.com/sakuriru/pi_sensor_to_prom

 

I chose Prometheus because it requires very little programming on your part. There's no need to write a file parser, no need to move files from your pi, comes with built in visualizations, and can be easily expanded just by editing your py file.

 

First I recommend using docker desktop. I find it easy to work with and you can just let it run in the background anywhere. It's possible to run this anywhere though. It is possible to use Prometheus without docker, but I don't recommend it and docker has a container prebuilt with prometheus, making that setup even easier.

 

image.png.a5e417b9b3bf7eeafc20c8ecc32f447b.png

 

You will need to also install prometheus_client in your python instance on the client where you're running your sensors (on the pi). I use a virtualenv because that's good practice, but on a pi it's probably best to keep its configuration the same with the python install.

image.thumb.png.36b4c3d37a8c10fcb639f4af92e41a67.png

 

sensor.py should be pretty effortless to edit for your uses, I used a stand-in class to generate sensor data for me. This is what I believe it would look like in your use case:

 
import random
import time
from prometheus_client import Gauge, start_http_server
import adafruit_ahtx0
import board

i2c = board.I2C()
sensor = adafruit_ahtx0.AHTx0(i2c)

g_temp = Gauge('temperature', 'Temperature from sensor')
g_humidity = Gauge('humidity', 'Humidity from sensor')

start_http_server(8000)

while True:
    g_temp.set(round(sensor.temperature, 2))        
    g_humidity.set(round(sensor.relative_humidity, 2))

    time.sleep(2)

You'll notice this code is considerably smaller, there's no need for you to set time data yourself or modify JSON. In Prometheus, a gauge is like a gauge in a car that would tell you a single value, like the car's temperature, speed, or RPM. That's what a gauge is in Prometheus.

 

The way Prometheus works is that it starts a HTTP server at the specified port (8000 in this case) and then exposes it on a webpage accessible on your local network. You can view it since it's just readable text at localhost:8000 once you run this script. You can run it as easily as python sensor.py

 

You'll also need to grab your IP address for your pi (local network, so 127.x.x.x) and plug it in to the prometheus.yml file under where it says host.docker.internal. This will allow your Prometheus server to scrape from your pi.

 

On your other machine with docker, you're going to want to run docker build -t prom_server . to build the Prometheus image. This will automatically pull in a Prometheus image and add the needed configurations automatically to it.

image.thumb.png.dd21e3314e87e6ca93b988285b54dad2.png

 

The next steps are all in the GUI. You'll see a docker image for prom_server in your images tab.

 

image.png.796dc07cb28cc8956030800db1403575.png

 

Hit run and fill in the configuration information to allow it to run with the correct port mapped so that you can view the server on your local machine.

image.thumb.png.f7a4bb56bb47ad6d448d73c2d8a8f477.png

 

Finally, hit run and the container will start. You'll be able to open it in the browser with an interface. At this point it should be scraping data so you can view it.

 

image.thumb.png.41152423b1be50a04e33973f9f885bbb.png

 

It uses PromQL, but that shouldn't be too intimidating since just typing in the scalar value "temperature" will allow you to view a graph like you want. Good luck, I hope this helps if you decide to go this route.

 

Link to comment
Share on other sites

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share


×