Plant monitoring system Pi zero - back end advise <python>

[Sharif]

I am making this thread to get some pointers on a personal project of mine, I made a little documentation to make it less intimidating but I still got no clue where to start.

Quote

Devices used:

·         Raspberry pi zero w

·         DHT11 temperature and humidity sensor module

·         YL-69 Soil moisture sensor module

 

Functions

·         Collect soil’s binary moisture value

o   Use it turn on led to remind user to water the plant

o   Use it to send data to my website

·         Collect ambient temperature and humidity

o   Send information along with current time to my website, to be presenting on a graph that can be adjusted to needed time span

Backend:

·         Write code to take the information from the sensors, store them all in a dataset with time stamps (For the graphs on my website) and send their information every 30 minutes.

·         Write code to send current sensor’s, soil moisture status, ambient temperature, and humidity to be sent to the website as an object every minute.

·         Write code to turn LED on when soil is dry.

 

Frontend:

·         Use receiving dataset to implement a graph with adjustable time span to showcase past to current change in temperature, humidity, and soil moisture.

·         A live preview of current stat values on top of page, using information from the received object.

·         A section mentioning last time the plant was watered

 

System enclosure:

Build an enclosure to hold all wiring, raspberry pi zero w, breadboard and needed wiring extensions to reach pot. Solder connections needed and run power to pi.

 

My question, is there a more simpler way to do what I am doing? or how would you do it? 

 

I am trying to get more ideas, I never messed with a micro computer prior to this and this whole thing is supposed to a trip down learning lane

[Zalosath]

39 minutes ago, Sharif said:

I am making this thread to get some pointers on a personal project of mine, I made a little documentation to make it less intimidating but I still got no clue where to start.

 

My question, is there a more simpler way to do what I am doing? or how would you do it? 

 

I am trying to get more ideas, I never messed with a micro computer prior to this and this whole thing is supposed to a trip down learning lane

Cool project idea. Love little IoT devices.

 

I think you've got a pretty good plan there. I would start with getting the moisture sensor working, write some code to parse the data from the sensor and see if you can get some meaningful information out of it. LEDs should be easy from there. I don't know how you plan to run your website but personally I'd use a database of some kind, storing date and moisture content, your website can fetch this data every 30 minutes like you wanted, I would send the data every time you take it, I assume once a minute? 1440 readings per day, and obviously adjust according to traffic. But you can always just store up the data and send it every 30 if that's what you'd prefer. 

 

I'll leave the website design to you, not much of a HTML/CSS/JavaScript person. 

[Sharif]

1 hour ago, Zalosath said:

Cool project idea. Love little IoT devices.

 

I think you've got a pretty good plan there. I would start with getting the moisture sensor working, write some code to parse the data from the sensor and see if you can get some meaningful information out of it. LEDs should be easy from there. I don't know how you plan to run your website but personally I'd use a database of some kind, storing date and moisture content, your website can fetch this data every 30 minutes like you wanted, I would send the data every time you take it, I assume once a minute? 1440 readings per day, and obviously adjust according to traffic. But you can always just store up the data and send it every 30 if that's what you'd prefer. 

 

I'll leave the website design to you, not much of a HTML/CSS/JavaScript person. 

I am reading up about nodejs express, it's a thing I need to use in my group project I am doing for a course, so it's working as double homework lol

 

As for the website it's already up and hosted on netlify, currently figuring out how to actually use express and then figure out how to actually send of data to the website to make use.

 

I already did get the sensor working few days ago (at least the moisture sensor) using a little infinite while loop, it was fun seeing the sensor kick out a little text saying it detected water, it was weirdly way more satisfying than it should be lol

[shadow_ray]

11 hours ago, Sharif said:

As for the website it's already up and hosted on netlify, currently figuring out how to actually use express and then figure out how to actually send of data to the website to make use.

// server endpoint for saving measurement data
app.post('/sample', auth, (req, res, next) => {
	
	const received_data = req.body;
    console.log(received_data); // body can be JSON string XML whatewer u want
    
	// validate the request
  	// make sure it's coming from your device
  	// (it should be an auth middleware)
      
    //save data to DB
  	let db_result = save_to_db(received_data); // use whatewer db u like
  	
  	if(!db_result) res.status(201).send("yeeee, data saved")
  	else res.status(500).send("hellnooo, something went terribyl wrong")
})

Server side would look something like this.  Postman is a great tool for sending requests with dummy data for testing purposes.

[Zalosath]

12 hours ago, Sharif said:

I am reading up about nodejs express, it's a thing I need to use in my group project I am doing for a course, so it's working as double homework lol

 

As for the website it's already up and hosted on netlify, currently figuring out how to actually use express and then figure out how to actually send of data to the website to make use.

 

I already did get the sensor working few days ago (at least the moisture sensor) using a little infinite while loop, it was fun seeing the sensor kick out a little text saying it detected water, it was weirdly way more satisfying than it should be lol

Oh I know the feeling, controlling a motor with a button on a breadboard was one of my first experiments, in no way was it mentally demanding but it was great to play with, even came with some blades that would fly off when the motor got fast enough, good fun.

 

Node.js express is pretty useful, set up a database within it, use a little script on the website to fetch the data and plot it to your hearts content. Your python app can push data to the database whenever you fancy and you can let the website handle the updates. 

[Sharif]

7 hours ago, Zalosath said:

Oh I know the feeling, controlling a motor with a button on a breadboard was one of my first experiments, in no way was it mentally demanding but it was great to play with, even came with some blades that would fly off when the motor got fast enough, good fun.

 

Node.js express is pretty useful, set up a database within it, use a little script on the website to fetch the data and plot it to your hearts content. Your python app can push data to the database whenever you fancy and you can let the website handle the updates. 

Exactly! haha

This is what I am leaning towards, I might try to see if I can use json to send everything over but..... I want to keep 12 months worth of data on a file to create a graph on the website showing changes over the time, that would be one huge json file, wondering to use a csv for that purpose or just use an ever growing json file.

[Sharif]

8 hours ago, shadow_ray said:

// server endpoint for saving measurement data
app.post('/sample', auth, (req, res, next) => {
	
	const received_data = req.body;
    console.log(received_data); // body can be JSON string XML whatewer u want
    
	// validate the request
  	// make sure it's coming from your device
  	// (it should be an auth middleware)
      
    //save data to DB
  	let db_result = save_to_db(received_data); // use whatewer db u like
  	
  	if(!db_result) res.status(201).send("yeeee, data saved")
  	else res.status(500).send("hellnooo, something went terribyl wrong")
})

Server side would look something like this.  Postman is a great tool for sending requests with dummy data for testing purposes.

Thank you! 

I'll get back to this when I reach that point, at the moment I am thinking of just creating a dummy database to play around with and see if everything is doing what it is supposed to, it'll save me a lot of hassle if I can get that postman thing to to pretend to be my pi and try different things and not actually mess with that until the final stage.

[shadow_ray]

9 minutes ago, Sharif said:

it'll save me a lot of hassle if I can get that postman thing to to pretend to be my pi and try different things and not actually mess with that until the final stage.

Exactly

 

12 minutes ago, Sharif said:

I want to keep 12 months worth of data on a file to create a graph on the website showing changes over the time, that would be one huge json file, wondering to use a csv for that purpose or just use an ever growing json file.

Even if you save a measurement in every 15 minutes that's 4 * 24 * 365 = 35 040 data point/sensor, which is not a lot of data. Times series databases are optimized for this kind of workload. Usually u don't need this much resolution for a graph and these databases can calculate averages (and many other things) for you. So when you watch a yearly overview then 365 data points are more than enough. But if you interested to see data from a specific day then you can just query every original data point from that time period. This might be overkill for a small project that uses 1 or 2 sensors, but I just wanted to let you know that these things exist, in case you are interested.

[AndreiArgeanu]

Not relevant to the programming part of the project but it's worth knowing the YL-69 is a resistive soil moisture sensor from what I could find which isn't great especially given that a capacitive one basically costs the same now. First issue is soil moisture detected by the sensor can be affected by fertilizers and the second issues being the most important one is that the resistive soil moisture sensor will corrode over time due to the way it works and as a result the data you collect will become erroneous over time to the point where you won't even be able to take a reading any more. If you already bought/have this sensor then it's fine, it's just worth considering switching it for a capacitive one if you want to use the device long term.

If you want to know more, this video goes in good depth about this.

 

The DHT11 is a decent sensor and it's suitable for this project, but if you haven't bought it already, I would suggest going for an I2C temperature sensor.

https://learn.adafruit.com/modern-replacements-for-dht11-dht22-sensors

There's this article from adafruit which explains the problems with the DHT11 and DHT22 sensors and shows you alternatives like the AHTXX sensors (AHT10, AHT20, AHT21). Adafruit sells the AHT20 in a PCB package, capsule package like the DHT11 and DHT22 and wired capsule like the AM2302. And you can find them on aliexpress as well but the PCB model requires soldering. If you want the capsule package similar to DHT11 that doesn't require soldering you'll have to search DHT20 on aliexpress instead of AHT20.

 

The AHTXX chips beside their I2C interface which makes them much more easier to read than the DHT11/22 also have a number of improvements such as lower power consumption, higher temperature range, higher temperature and humidity resolution, higher temperature and humidity repeatability, higher temperature and humidity accuracy, lower prolonged drift and a wider range of operating voltages.

[Sharif]

On 3/6/2022 at 1:17 PM, AndreiArgeanu said:

Not relevant to the programming part of the project but it's worth knowing the YL-69 is a resistive soil moisture sensor from what I could find which isn't great especially given that a capacitive one basically costs the same now. First issue is soil moisture detected by the sensor can be affected by fertilizers and the second issues being the most important one is that the resistive soil moisture sensor will corrode over time due to the way it works and as a result the data you collect will become erroneous over time to the point where you won't even be able to take a reading any more. If you already bought/have this sensor then it's fine, it's just worth considering switching it for a capacitive one if you want to use the device long term.

If you want to know more, this video goes in good depth about this.

 

The DHT11 is a decent sensor and it's suitable for this project, but if you haven't bought it already, I would suggest going for an I2C temperature sensor.

https://learn.adafruit.com/modern-replacements-for-dht11-dht22-sensors

There's this article from adafruit which explains the problems with the DHT11 and DHT22 sensors and shows you alternatives like the AHTXX sensors (AHT10, AHT20, AHT21). Adafruit sells the AHT20 in a PCB package, capsule package like the DHT11 and DHT22 and wired capsule like the AM2302. And you can find them on aliexpress as well but the PCB model requires soldering. If you want the capsule package similar to DHT11 that doesn't require soldering you'll have to search DHT20 on aliexpress instead of AHT20.

 

The AHTXX chips beside their I2C interface which makes them much more easier to read than the DHT11/22 also have a number of improvements such as lower power consumption, higher temperature range, higher temperature and humidity resolution, higher temperature and humidity repeatability, higher temperature and humidity accuracy, lower prolonged drift and a wider range of operating voltages.

Yep aware, right now it's mainly a pretty thing for my portfolio first. In the summer once it did its job, get me into my internship, I'll turn this into the actual thing I wanted to do, more an automated thing for my plants, my overkill project justification being for my tiny indoor plant collection, that doesn't really need a lot of tlc is.... because I can kind of reason hahaha, I always wanted to do something like this as a kid. 

 

I was aware of their drawback, didn't know a lot of info about them existed, I'll go through that rabbit hole when the time comes, thanks a lot for the info! 

[Sharif]

On 3/2/2022 at 5:42 PM, shadow_ray said:

Exactly

 

Even if you save a measurement in every 15 minutes that's 4 * 24 * 365 = 35 040 data point/sensor, which is not a lot of data. Times series databases are optimized for this kind of workload. Usually u don't need this much resolution for a graph and these databases can calculate averages (and many other things) for you. So when you watch a yearly overview then 365 data points are more than enough. But if you interested to see data from a specific day then you can just query every original data point from that time period. This might be overkill for a small project that uses 1 or 2 sensors, but I just wanted to let you know that these things exist, in case you are interested.

Thank you!

 

I don't really have anything relatively for scale right now, everything being so new to me. Would you say it be extreme overkill to use mongodb to store the data? 

[shadow_ray]

50 minutes ago, Sharif said:

Would you say it be extreme overkill to use mongodb to store the data? 

No, mongo is fine

[jackieAZ]

I know you said you don't care that much about the using the best possible sensor... but if you wanted to the Veggetronix VH400 is a pretty decent capacitive soil moisture sensor. It's an analog read, like those resistive ones everyone uses. It's 40 bucks which is a bit pricier than a lot of resistive sensors but it will be a lot more accurate for longer.

[Sharif]

4 hours ago, jackieAZ said:

I know you said you don't care that much about the using the best possible sensor... but if you wanted to the Veggetronix VH400 is a pretty decent capacitive soil moisture sensor. It's an analog read, like those resistive ones everyone uses. It's 40 bucks which is a bit pricier than a lot of resistive sensors but it will be a lot more accurate for longer.

Thanks for the recommendation. Once I get to that bit, for my use case because I normally water my plant once it's dry to at least inch deep (Plant in a pot, indoor) it doesn't have to be all that accurate, since the difference will be big enough. I'll probably find out about drawback as I work on it in the summer, right now a binary value will do the trick and I'll likely take it off before that current sensor does anything to my soil on my actual plants. 

[Sharif]

So I ended up doing it this way or stopped till something kinda put a nail in my plans, in a bad way. 

 

Took the sensor data from the sensors and directly pushed them to Mongodb from the pi, the problem arise where the pi zero doesn't support mongo on it's CPU's architecture, since I did most of the code and pseudo code already written up, thinking of just buying a raspberry pi 3 off the marketplace when I see one. Since I am broke right now, I decided to put the project to a pause but I was so close to calling it done-ish lol 

[shadow_ray]

I don't get it. Do you want to run MongoDB on the pi zero? I thought you already set up a database for your website.

You can get a M0 shared MongoDB Atlas instance for free: https://www.mongodb.com/pricing Data storage is limited to 512 megs but that's more than enough for a small personal project.