Append to a json file in python

AndreiArgeanu · May 10, 2022

I currently have raspberry pi zero 2W running a python scrip I've made which essentially takes measurement of the temperature and humidity every 2 seconds using a DHT20 sensor and stores it alongside the date and time in a json file. I've decided to use json since its convenient and easier to load in another script to plot the data. The way it currently works is by loading in the entire json file, adding the respective readings to it and then essentially overwriting the already existing file. This wasn't an issue at first however over about 2-3 days I've already gathered about 68000 measurements which is about a 3.5mb file and now rather than there being about 2-3 seconds between measurement in the file it's more like 4-5 which isn't a massive issue at the moment however it will be over time.

So I was wondering is there any way to append data entries to a json file rather than having to load and rewrite the entire file. The tutorials I can find online that mention appending use the method I previously mentioned. Should I just stop using json and rewrite the code to use a simple text file to which it can append data?

This is my code if it's of any use

import time
import board
import adafruit_ahtx0
import json
from datetime import datetime
i2c=board.I2C()
sensor=adafruit_ahtx0.AHTx0(i2c)

def current_date():
    now=datetime.now()
    day=str(now.day)
    month=str(now.month)
    year=str(now.year)
    date=f"{day}/{month}/{year}"
    return date

def current_time():
    now=datetime.now()
    hour=str(now.hour)
    minute=str(now.minute)
    second=str(now.second)
    time=f"{hour}:{minute}:{second}"
    return time
start=time.time()
while True:
    temp=round(sensor.temperature, 2)
    hum=round(sensor.relative_humidity, 2)
    print("Temperature:",temp,"C")
    print("Humidity",hum,"%\n")
    with open('/home/pi/test.json') as f:
        data=json.load(f)
    temps=data["temperatures"]
    hums=data["humidities"]
    dates=data["dates"]
    times=data["times"]
    dates.append(current_date())
    times.append(current_time())
    temps.append(temp)
    hums.append(hum)
    data["temperatures"]=temps
    data["humidities"]=hums
    data["dates"]=dates
    data["times"]=times

    with open('/home/pi/test.json',"w") as f:
        json.dump(data, f, indent=2)    
    final=time.time()-start
    if final > 60:
        start=time.time()
        print("Creating backup")
        with open("/home/pi/backup.json","w") as f:
            json.dump(data, f, indent=2)
        print("Backup created succesfully\n")
    time.sleep(2)

Also I know I could just save to a json every 60 seconds to 2 minutes rather than doing it every time I take a measurement but this doesn't fix the issue it just postpones it.

AndreiArgeanu · May 10, 2022

3 minutes ago, Sakuriru said:

You have several different options:

Use a local instance of SQLite, which would require more of a rewrite of your current program, but it's easy to setup and use

Use a local instance of MongoDB, which you could just spit JSON into, but if you're unfamiliar with NoSQL than there's a learning curve

Use something like Prometheus logging, which is precisely what this is designed for, but requires knowledge of PromQL, but it comes with built in visualization

Log to a text file as you suggested, then program an import script. The problem with this solution is that your text file will eventually become unwieldy in size. I recommend against this.

I would like to use a simple solution but if any of these options provide me with useful skills that I could potentially use in other programs I make in the future I'd be willing to go into them. I don't really need visualization since the pi will spend most of its time away from any display output, I only really use it to log the data and then transfer it to my PC to work on it.

mariushm · May 10, 2022

JSON by design is not meant to be append-able .. but in theory you could, if the structure is easy enough. For example, you could just read the last 1 KB from the json file, find the last } or ] occurrence (the end of an array/object holding your records, place your file pointer before the ] / } and add your records and put back the ] or }

The solution is to use a different format, like CSV or TSV (Tab separated values), or some other format where each record has a fixed size or specified size.

TSV would be easiest, as you separate each value with a TAB character (0x09), which is unlikely to appear in regular data, unlike comma .. so you get one record per line, and each line has multiple values separated by tab.

Another option would be to serialize each record to json, and store each record as an individual json file... one json / record per line.

You can have a binary format, where each record is basically [ 2 bytes : length of record ] [ record data ] and in record data you decide how many bytes each value will use ... store a temperature as a 32 bit float value , store a time as an unsigned 32 bit int ( ms from 1970 or whatever)...