data logger

[sgschwend]

I did not see a relay power spec. It is nice if they can handle 8-16amps, otherwise you might need another relay to switch the higher currents.

At first I didn't think it had a data log file, but it does. It outputs the data every 24hrs, I guess if your monitor is not there the log data is overwritten. It does mention "your software" does that mean low level machine control or object orientated code? Looks like a good place to start (are you OK with the system requirements? They look a little thin).

I think you can control pretty well with two channels of temperature that would leave you two other for storage data. I did find that the Control by Web data logger only provides 8 channels when in logging mode, if it is placed in control mode it only has two temperature channels as control channels.

 

[Hunderliggur]

If they just made those nifty kits with a Web and Ethernet interface I would be all set. In the meantime, I'll keep saving pennies for the Controlbyweb W-300. I need at least one 8 channel in the house (tank top/middle/bottom x 2; heat exchange in/out = 8), one 4 -channel fo the boiler (boiler temp, flue temp, in temp, out temp = 4), and maybe another 8 with relays to monitor the house (3 zones, outside air temp, freezers - why not).

 

[jebatty]

I"ll add more later as to success/issues/failure, but I decided to buy a ready-made 4-channel data logger using the Dallas one-wire sensors for $30 (two sensors included). It's on order and will arrive in a few days. For monitoring purposes, I also bought ready-made digital thermometer panel meters which also use the Dallas one-wire. I bought 4 of these, each with a two sensor input (monitor 8 sensors), and switchable to show either channel or automatically alternate back and over between channels. Each meter is $18. I got the DOW version (DOW sensors included) thinking that in the future I also could use these with a to be determined computer system, including hopefully the data logger. Serial display chips alone run about $15, so this looked to be a good deal, if the thermometers perform well. Lastly, for $20 I bought a ready-made digital thermometer panel meter which uses a K-type thermocouple to monitor stack temp. All the thermometers run on 12vdc. The data logger draws power from the serial port. I intend to use a USB to RS232 adaptor, if it works, and then have the possibility of running more than one data logger via the USB port on my computer, if that works. It might be pie in the sky, hopefully not, but for $125 will have a 4-channel data logger, equivalent of 9 digital panel meters, and a USB/serial converter. Maybe I'll end up with light weight boat anchors.

 

[EricV]

Where did you find these

ready-made 4-channel data logger using the Dallas one-wire sensors

Thanks
Eric

 

[jebatty]

To find the ready-made data logger, look a few posts above or here. Good luck.

 

[jebatty]

To find the ready-made data logger, look a my post a couple of posts above.

 

[mwk1000]

I took some time this week to go from the logged data to some charts on a home made web page. I must say I am very impressed with how far the open source space has come in presenting the graphs!. This was about 2 4 hour sessions to work out the kinks and go from data in a database to the pictures.

It's all queried live from a mysql db using the OpenFlashCharts modules I mentioned before. The navigation and charts are quite good looking and have live tool tips with the sample details. It makes for a nice looking page, each sensor line "expands" to show a chart. They are resizable and you are supposed to be able to right mouse and save as an image. ( Haven't worked that out yet.

I wish I had I site that was easy to hit but my home is connected via a cellular. I will wrap it all up together and repost it on the wiki so a complete "kit" is out there from controller -> logger -> database -> web Pages.

Cheers

 

[mwk1000]

I took some time this week to go from the logged data to some charts on a home made web page. I must say I am very impressed with how far the open source space has come in presenting the graphs!. This was about (2) 3 hour sessions to work out the kinks and go from data in a database to the pictures.

It's all queried live from a mysql db using the OpenFlashCharts modules I mentioned before. The navigation and charts are quite good looking and have live tool tips with the sample details. It makes for a nice looking page, each sensor line "expands" to show a chart. They are resizable and you are supposed to be able to right mouse and save as an image. ( Haven't worked that out yet.)

I wish I had I site that was easy to hit but my home is connected via a cellular. I will wrap it all up together and repost it on the wiki so a complete "kit" is out there from controller -> logger -> database -> web Pages.

Cheers

Updated : https://www.hearth.com/econtent/index.php/wiki/TankControl_Information/

It's all out there to download if you like.

 

[sgschwend]

Wow, it's Christmas all over again.

Looks like I could use that module too, must take a closer look at it. Isn't that the way it is, so many cool things if you can only find them.

 

[deerhntr]

I took some time this week to go from the logged data to some charts on a home made web page. I must say I am very impressed with how far the open source space has come in presenting the graphs!. This was about (2) 3 hour sessions to work out the kinks and go from data in a database to the pictures.

It's all queried live from a mysql db using the OpenFlashCharts modules I mentioned before. The navigation and charts are quite good looking and have live tool tips with the sample details. It makes for a nice looking page, each sensor line "expands" to show a chart. They are resizable and you are supposed to be able to right mouse and save as an image. ( Haven't worked that out yet.)

I wish I had I site that was easy to hit but my home is connected via a cellular. I will wrap it all up together and repost it on the wiki so a complete "kit" is out there from controller -> logger -> database -> web Pages.

Cheers

Updated : https://www.hearth.com/econtent/index.php/wiki/TankControl_Information/

It's all out there to download if you like.

Impressive!

I have been lurking on the thread for a bit, but you guys have got me going. I tried to take advantage of Sparkfun's free day last week to get an arduino board for free, but to no avail. So I just ordered the board, and some dallas one-wire temp sensors after the "Free Day" was over. Now you've got me resurrecting an old 266Mhz PII PC to run linux for the data logger project. I found the DSL(damn small linux) distro, and that old PC booted right up on it. This is great stuff when it is cold out, and there is nothing better to do than fiddle with projects, and stoke the boiler. Oh yea, I should be cutting some firewood for next year. In do time!

Great Thread, and thanks for the wiki stuff. Hopefully I will be able to add a little when I'm up and running.

 

[mwk1000]

Have at it. I have not been this interested in the Charting for a while. It's fun to play again. There are a bunch of data files in that download that you can use to load up the database with some data and have a try at setting up the web while you are working on the controller. I had to wait to get the data , it's a pain to try and design with an empty data or even made up data.

See the readme in the tar file for a quick start. You can load up all of my data as a sample. like this:

cd /<whereever you are starting>/ctl/ttymon/log
for FILE in $(ls *.sql)
do
mysql -uoccam -p1qaz2wsx temperature< $FILE
done

That will dump them all in the database and you can play and change quicker.

Just run the command that creates the database again when you are ready for the real stuff. It clears all the tables and starts again.

Mark

I am off on the hunt for how to hook up my Taylor 9940 digital sensors to get the tank top and bottom readings into the mix. There was supposed to be a way to read them but I lost the doc. I was wondering if they use the same sensors but I'm not quite ready to clip the line and find out.

Meanwhile I'll spend some time and try and make the pretty graphs contain some more meaningful data!

 

[jebatty]

I decided to buy a ready-made 4-channel data logger using the Dallas one-wire sensors for $30 (two sensors included).

Logger arrived and works perfectly. This is ultra-simple: plug into the Com port, use HyperTerminal to set comm parameters, starts log on call, ends on disconnect, capture text to file. Logs all four sensors about every second. At least gets me started to move in a more sophisticated direction.

Here's a sample of the output with two sensors attached. I put one sensor in a glass of cold water.

1 70.5 F
2 --
3 72.1 F
4 --
1 68.1 F
2 --
3 72.1 F
4 --
1 65.8 F
2 --
3 72.1 F
4 --
1 64.0 F
2 --
3 72.1 F
4 --
1 62.4 F
2 --
3 72.1 F
4 --
1 60.9 F
2 --
3 72.1 F
4 --
1 59.5 F
2 --
3 72.1 F
4 --
1 58.4 F
2 --
3 72.1 F
4 --
1 57.3 F
2 --
3 72.1 F
4 --

 

[mwk1000]

That works! And the price is right. You can use the ttymon script I posted as an example of how to :

Automatically open the com port and collect the data to a file name that is date and time stamped.
Date and time stamp the measurements
Roll the data every X min/hours/days into a new file
Create database insert statements from the raw text.
Run the database inserts on every log file turnover.

.....

You will have to address the differences from windows to unix but it is usable on either platform. COM1: in Windows is /dev/ttyABC in unix. Or just use it to get an Idea of what needs to happen and roll your own. I use a crossover product called CYGWIN on windows to create a unix like environment and run unix oriented apps on a windows system. More open source stuff but a nice way to have both environments available at the same time. I'm not sure how it presents serial ports, never had to do it.

 

[jebatty]

Some, maybe most, of you are way ahead of me on this, but fwiw, here is what I did today. I located an inexpensive graphing program (shareware about $14.00) that will graph multiple temperature inputs. I held my breath, as it is from a Russian programmer (shouldn't have a bias, but too much press about computer hackers in Russia, probably no more though than are in the US). It installed and runs without issue. It downloaded from Digital River, which itself is reputable. Check (broken link removed to http://www.rominten.com/VIGraph/) out, if you wish. It really is pretty slick and usable for graphing many types of data.

I should learn an updated programming language; instead I went back to DBIII (DOS) and did a quick program to read in the data file like that shown a couple of posts above. The data logger produced 11,322 lines of data for the four sensors in one hour, or 2,825 reading for each of the four sensors,or about 47 readings for each sensor per minute. The DBIII program I did reduces that to one reading for each sensor for each 5 minutes, and then outputs a csv file which the graphing program reads to produce a graph of all four sensor readings.

I don't think the very basic 4 channel data logger is the best way to go, but for me it was the way to start to get my head around some of this stuff again. Can't believe that my last programming was done in the early 1980's. Showing my age. Real-time, ethernet, web-based, with time stamping, ability to set time between data log reads, all would be much better. But I guess what should one expect fo $30 bucks?

 

[Gooserider]

There are a lot of good hackers in Russia, they are doing some really high quality coding there, nothing wrong with the stuff they produce... There are also a lot of CRACKERS living in Russia, and unfortunately the Russian legal system does not make it easy to prosecute them - of course if it wasn't for the poor quality products of a certain US software company, the crackers and script kiddies would have a lot harder time of it...

That said, $14.00 for a graphing program seems exorbitant... In the Linux world, the toughest problem is often deciding WHICH free / open source language / database / office suite / program one wants to use... I just looked at the Gentoo Packages site, and they had 97 packages listed under /dev/lang (the computer language category - from dev-lang/bashforth to dev-lang/yasm - and that was just for X86 boxes, and doesn't include programs with their own scripting languages, or stuff like the ability to do shell scripts and so forth... I even found

dev-db/xbase
Latest version available: 2.0.0
Latest version installed: [ Not Installed ]
Size of files: 481 kB
Homepage: http://www.rekallrevealed.org/
Description: XBase is an xbase (i.e. dBase, FoxPro, etc.) compatible C++ class library
License: GPL-2

and

dev-db/xbsql
Latest version available: 0.11
Latest version installed: [ Not Installed ]
Size of files: 300 kB
Homepage: http://www.rekallrevealed.org/
Description: XBSQL: An SQL Wrapper for the XBase library
License: GPL-2

Though I don't have them installed on my machine (no need for it)

(And if you are worried about crackers from whatever country, the source code for all of this is part of the deal...)

This is one of the reasons why I like Linux so much - there isn't much in the way of stuff that isn't available on it, and it is all just a download away...

Gooserider

 

[jebatty]

Alive and still learning. My limited knowledge also limits my ability to understand exactly what some programs/languages etc do and how to use/learn them. A Google on programs to graph csv and other delimited file types was not productive in locating graphing programs with results I could understand. After spending more than an hour on Google, when I found something for $14, it was a bargain. So, have you seen something like this lately: 256K bytes minimum of memory to run a database program, one billion records per file (table), 10 files open simultaneously, lightning fast sorts, no crashes, hangs or mysterious hourglass symbols while you wait ....... and wait .... etc. ? I know none of that is true with Windows. Linux may be free of all of that also, and yet for many things DOS is so elegant.

EDIT: forgot, reference is to dBASE III. I also have Clipper, which accelerates DBIII by a couple of orders of magnitude.

 

[Gooserider]

Alive and still learning. My limited knowledge also limits my ability to understand exactly what some programs/languages etc do and how to use/learn them. A Google on programs to graph csv and other delimited file types was not productive in locating graphing programs with results I could understand. After spending more than an hour on Google, when I found something for $14, it was a bargain. So, have you seen something like this lately: 256K bytes minimum of memory to run a database program, one billion records per file (table), 10 files open simultaneously, lightning fast sorts, no crashes, hangs or mysterious hourglass symbols while you wait ....... and wait .... etc. ? I know none of that is true with Windows. Linux may be free of all of that also, and yet for many things DOS is so elegant.

EDIT: forgot, reference is to dBASE III. I also have Clipper, which accelerates DBIII by a couple of orders of magnitude.

Don't have anything to match those specs exactly, but considering that Linux will run on everything from an IBM experimental wristwatch to most of the TOPS 500 supercomputers, I'd be surprised if there wasn't something in there somewheres that did what you wanted... There are getting to be a great many very tiny dedicated Linux setups for very small systems - including a lot of the new "smartphones" of which the 'droid' is just the latest... It wouldn't surprise me at all if that $30 data logger wasn't already running Linux under the hood, a great many of your current consumer electronics devices are. (It is a significant issue according to the GPL enforcement folks, as many of the companies involved are NOT properly complying with the software license, and giving back their source code)

Gooserider

 

[deerhntr]

Alive and still learning. My limited knowledge also limits my ability to understand exactly what some programs/languages etc do and how to use/learn them. A Google on programs to graph csv and other delimited file types was not productive in locating graphing programs with results I could understand. After spending more than an hour on Google, when I found something for $14, it was a bargain. So, have you seen something like this lately: 256K bytes minimum of memory to run a database program, one billion records per file (table), 10 files open simultaneously, lightning fast sorts, no crashes, hangs or mysterious hourglass symbols while you wait ....... and wait .... etc. ? I know none of that is true with Windows. Linux may be free of all of that also, and yet for many things DOS is so elegant.

EDIT: forgot, reference is to dBASE III. I also have Clipper, which accelerates DBIII by a couple of orders of magnitude.

Don't have anything to match those specs exactly, but considering that Linux will run on everything from an IBM experimental wristwatch to most of the TOPS 500 supercomputers, I'd be surprised if there wasn't something in there somewheres that did what you wanted... There are getting to be a great many very tiny dedicated Linux setups for very small systems - including a lot of the new "smartphones" of which the 'droid' is just the latest... It wouldn't surprise me at all if that $30 data logger wasn't already running Linux under the hood, a great many of your current consumer electronics devices are. (It is a significant issue according to the GPL enforcement folks, as many of the companies involved are NOT properly complying with the software license, and giving back their source code)

Gooserider

Actually, there is a small free version called DSL(Damn Small Linux) that has a bunch of packages, but is intended for small systems, and will boot from a CD,Thumb drive,pen drive whatever. Very small, and can be found at Dan Small Linux. Pretty neat, and the price is right!

 

[Gooserider]

Yes, that is one of many choices for small distros - there are MANY more... If you want lists of distributions, many of which can have some interesting focuses, you can try either Linux Weekly News or Distrowatch Weekly Might be worth looking for one of the science oriented distros to see if it has what you want...

Incidentally on the database side, most stuff these days tends to be using some flavor of SQL, - again there are a bunch of competing Linux programs that do SQL type databases, which pretty much don't have limits on the number of records - we are talking "Enterprise Class" software here...

Gooserider

 

[jebatty]

The auduino-type route in a Linux environment using MySQL would be a very good route to go. Since I have a need to be resourceful, my very low bucks approach may be accomplishing just about the same thing for datalogging (control can come next). A few posts above I mentioned the $30 4-channel datalogger which I got. This plugs into the serial port, which my desktop had but my laptop does not. For $1.00 I got a Ser/USB converter, and now the datalogger is fully USB compatible. The datalogger produces an output text stream through the serial port. Hyperterminal Capture grabs this and produces a *.txt file of the data. The datalogger reads all 4 channels 47 times/minute, each channel read is a line of data (the temp reading), so it is producing lots of lines of data. This simple datalogger itself is not programmable.

Since I had a database program, dBase III, I structured a very simple database and 3 short programs, run successively, which 1) read the *.txt file created by the datalogger, 2) process the data to produce a temperature reading at a user-selected interval (I chose every 5 minutes), and 3) output a *.txt file which can be read by the $14 graphing program. The graphing program produces a separately colored graph line for each of the 4 channels. The y-axis is the temperature scale and the x-axis, based on my data, was a 5 minute scale (5, 10, 15, etc.), but it just as easily could be a time-stamped scale by adding a couple of lines of code.

Example: I ran the datalogger for 6 hr - 14 minutes. It produced a *.txt file with 70,492 lines of data, each line being a temperature reading for each channel. Total time was 4 seconds to give the command to load dBase, run the dBase programs to process the 70,492 lines of data, and output a *.txt file with a temp reading every 5 minutes for the graphing program to read.

dBase III produces *.dbf data files. The programming language for dBase III is quite similar to SQL. For an excellent discussion of dBase III and its relationship to SQL see dBase III on Wikipedia.

As someone probably said before me, you do what you can with what you got, I think for my purposes I'm close to logging and graphing all the data I want. And because the datalogger now uses the USB port, I could probably add as many temperature channels as I wanted at $30 per 4 channels. To add control, I see a kit is available with 4 channel temperature/relay control for $80.

In the next day or two I'll do a temp logging on my boiler system and provide the results.

 

[Nofossil]

Thanks a bunch, guys! I didn't have enough to do already, without the distraction of OpenFlashCharts. I chose jpgraph a few years back to do my charting. It works pretty well, but it depends on some pretty heavy-duty PHP on the server to create the actual graph image. Not a problem on my server, but no way it will run on my controller.

Creating the graph with ShockWave puts the load on the user's computer (with the web browser) so it could be a nice solution. It looks really nice, but unfortunately it appears to be bug infested at the present level of development. I ended up spending a whole day that I don't have. The killer problem is that I graph discrete signals (on/off) as a horizontal line that's present if the load or signal is 'on' and not present if it's 'off'. To display a line with missing sections, you use null values anywhere that you don't want the line. OpenFlashChart (or the shockwave plugin) will crash if more than a certain number of nuill values are received. Works great for analog plots, but I can't reliably show more than a single discrete plot. Here are jpgraph and OpenFlashChart samples.

Mwk1000 (and any others) - have you run into this problem, and do you have any suggestions?

 

[deerhntr]

The auduino-type route in a Linux environment using MySQL would be a very good route to go. Since I have a need to be resourceful, my very low bucks approach may be accomplishing just about the same thing for datalogging (control can come next). A few posts above I mentioned the $30 4-channel datalogger which I got.

If conserving cash is your primary goal, then the aurdino route is the way to go. Your data logger is 4 channels for $30 or $7.50/channel. An arduino is $30, for as many one-wire devices you can put on your network until you run out of the 32K of programming memory. I currently have 5 on a 150' network and plan to add more once I order some more sensors. Currently, that's $6/channel. Let's say I go to 10 sensors, that is $3/channel. Your data logger is $30 with no sensors, and so is the arduino. If you want to expand beyond 4 channels, you have to shell out another $30 for another data logger, not sure how much for control. Not sure if you can daisy chain or you will need another serial port. The nice thing about the arduino is you get 14 digital pins, and 6 analog pins to configure how ever you chose. You will easily become resource limited in your current setup.

Now for the stuff you will really like. All the software is FREE. The development environment for the arduino if FREE. If you chosoe to use linux it is FREE. if your choose to use MSQL it is free. Also, there is a huge user community out there in the "Web World" that is doing a lot of the programming for the arduino already. All you have to do is tap into that resource. And did I mention, it is all FREE. If you want to stick to the DbaseIII route, great you can simply write your arduino data to a txt file and import it to dbase like you are currently doing.

All kidding aside, If someone has a little programming background, or not, the arduino is the most cost effective route to implement a data logging/simple all-in-one control system. It is flexible, programmable, and fairly inexpensive when compared to other similar solutions. Cost per channel is significantly less once you cross the 4 channel need, and it is easy to expand.

My 2cents.

 

[WoodNotOil]

I haven't reviewed this whole thread (as it is lengthy) and I am not sure I have a ton to contribute... That said, I have been working a lot with mysql and did a great deal of experimenting with the free web space offered by batcave web hosting. It comes with a free database and php, mysql, etc all installed already. It also has phpmyadmin for you to use to setup and manage tables, fields, and data initially. It saved me the time and effort of converting an old PC to a linux server and trying to get mysql running on it and it is way more accessible to the outside world. I was able to start right in on writing the php instead of wasting time configuring hardware. Not sure how easy it would be to get your txt or other files to post into the data set from your boards though.

I run kubuntu linux exclusively at home and I am a big supporter of the open source movement. I love to hear about others making good use of these collective resources. I am looking forward to seeing what you all come up with and possibly doing some data logging from my arduino eventually. Thanks jebatty for suggesting a way to do that.

 

[Nofossil]

For sure a full-blown PC is an easy way to get going, and with Linux / MySQL it's all free. My issue is that PCs suck a bunch of power. If the PC would be running anyway it doesn't matter, but I hate to add a power load like that if I don't have to.

For things that have to run 24/7, I like little embedded systems like the Arduino.

My personal experience with Linux based systems from full-blown servers to embedded systems is that it's really reliable but sometimes a challenge to figure out. Lots of people willing to help, and Google is a great resource as well. Good luck.

 

[jebatty]

Lots of things to think about and different ways to go. Using my laptop rather than desktop was for the primary purpose of saving power, although I would guess it still takes a lot more power than an embedded arduino/Linux/MySQL etc. I haven't seen DS18B20 sensors for less than about $4-5 each + s/h. The 4-ch board I mentioned comes with two of these, and two left to user-provide. So up-front cost is still a little more, perhaps. Ease of use could not be beat - no new operating system, no new programming, to get the data stream. I already have my database program; approximate equivalent of MySQL, plus I believe *.dbf files can be fully queried through MySQL. I'm not sure that I really want to log 24/7; more likely 1) to set up the boiler system to peak and tweak performance; 2) then to periodically monitor performance to verify operational efficiency; and 3) to play around and experiment just for the fun of it -- something I keep coming back to about the time I think I'm "done" and ready to rock on the rocker, although the last rocking I did was at a Metallica concert with my wife last summer -- I think we were just about the oldest rocking head bangers there.

At this point my control system is very adequate and dependable, being a combination of aquastat's, a digital differential controller, and a 2-stage digital controller. Yet, adding my own computerized control system would really be fun. Also, my shop does not have a telephone/DSL line, so a full hosted Web based system is not practical, although it could run locally. My new shop might be close enough to run Wifi, but that will have to wait until the new shop is put in operation this summer.

Yet, I have so much fun with computers, programming, experimenting, etc. that it is very likely the time will come, probably sooner rather than later, that I will take the jump into Linux. Maybe first install on my PC to get the hang of it, then move to another embedded platform. All this makes my future brighter, more challenging, and the prescription to keep my brain from stratifying with a dementia (maybe already demented) my wife says I don't already have.

.... ready to head over and fire up the Tarm and monitor storage loading. I let the tank run down a fair amount and getting time/data from top, middle and bottom for a full charging would be quite interesting. We're headed into some really cold weather, about -20F at night, so this would be a good time anyway to load up storage to 190F.

I also just finished installing 5 digital temperature readouts on a panel: boiler supply, storage middle, storage bottom, boiler return, and flue. I will post later a pic of the readout panel + graphs from the new datalogging.