Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: 1 2 [3] 4 5

Author Topic: RaspberryPi - gui-less stats logger  (Read 20028 times)

G.DMT

  • Member
  • **
  • Posts: 74
Re: RaspberryPi - gui-less stats logger
« Reply #30 on: October 30, 2014, 01:01:38 PM »

Thanks for the 7800N log :)
You are welcome! I hope it helps you.  :)

Quote
That is a good point about the SD card write cycles and mysqld.
To be honest, whether it uses a different embedded db, or even wrote to a flat text file, whilst the scale of the write cycles might differ, the underlying potential problem will remain  - the only way to be sure of the longevity of the card in this situation is to leave it running for many months and see if it kills the card.

The last thing I would like to be known as is a "Pi Card killer" - it's probably prudent that I pull this project from public access and let it run here for many months and see what happens, so that is what I will do.
Eeek! I didn't expect to cause a flap! :-[
I was only suggesting that if users were advised of the possiblities, then they could make an informed choice.

There exist multiple mitigation options for the SD card write issue.
Easiest for a non technical user is to save to a separate device, as I indicated above.
That device _could_ be fuse-ssh mounted over the network as I described in a previous post.
A local mount can be to a memory backed file system e.g. tmps or loop.

A traditional disk-based database engine makes many,many distributed updates (writes) due to its hash-table or b+ tree based file content indexing.

If the goal is to minimise the number of writes-to-flash then appending a single data block to a data store is obviously far less write intensive.

e.g. Creating any intermediary files on a RAM backed filesystem and then appending only the data of interest to your data store sounds like it is getting close to a 'good enough' solution.
 
A default Pi install takes some account of this already
i.e.

root@raspberrypi:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
rootfs          7.4G  2.0G  5.1G  28% /
/dev/root       7.4G  2.0G  5.1G  28% /
devtmpfs        215M     0  215M   0% /dev
tmpfs            44M  268K   44M   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            88M     0   88M   0% /run/shm
/dev/mmcblk0p1   56M  9.6M   47M  18% /boot

Maybe it would be enough to just allow the user to specify where they want their temporary working directory to be (in /run/shm, /tmp, -o loop, whatever)...
 and you would be half way there already.  ;D
Logged

roswellgrey

  • Member
  • **
  • Posts: 73
Re: RaspberryPi - gui-less stats logger
« Reply #31 on: October 30, 2014, 02:06:41 PM »

Eeek! I didn't expect to cause a flap! :-[

I am glad you brought it up  :)

There are options to alleviate this potential problem, but it's getting complicated for the potential end-user :-[
What I was trying to achieve was a very easy install of a self-contained, gui-less logger with a very low power footprint.
Now, it looks like some form of (a) more write tolerant external storage or (b) a potentially sacrificial external storage device (such as a usb stick) is required to counter the fears. And unfortunately this is going against what I was trying to achieve (i.e. the easy, self-contained bit - lol)

I know people can make up their own minds about this issue, but I don't want anyone pointing fingers at me saying
"I cooked their SD card" ("cooked their Pi" is too corny ...) .... life is too short  >:D

In the meanwhile, I will continue to abuse my card - it has 9K entries in the db at the moment - will see how many it can get to before it fails :)
« Last Edit: October 30, 2014, 02:19:16 PM by roswellgrey »
Logged
Billion 8800NL

roswellgrey

  • Member
  • **
  • Posts: 73
Re: RaspberryPi - gui-less stats logger
« Reply #32 on: November 02, 2014, 10:04:34 AM »

Being curious about the actual number of writes that mysqld causes, I conducted a little experiment (results attached)

The conclusion is that my daemon install on a Pi causes approx. 3x the number of writes/second (0. 19 w/s) that a vanilla Rasbian Pi install does (0.06 w/s) over a 10 minute test period . 

My daemon is polling the modem every 1 minute, hence 10 stats db inserts are created during the test.

My test was slightly biased against my dameon install as I am running vnstatd on that install and that process's writes are included in the daemon install write stats. However, it does give a good general idea.

Due to the real-world uncertainties of wear-leveling (as a singular example, I not sure if my card actually has it) and as I cannot determine how many writes to the same physical locations on the card a new row insert into the database actually causes, I am unable to draw any firm conclusions as to how much my daemon will decrease the life of the SD card.

Long term soak test seems the only realistic way forward to see if a very early card failure happens :)
Logged
Billion 8800NL

G.DMT

  • Member
  • **
  • Posts: 74
Re: RaspberryPi - gui-less stats logger
« Reply #33 on: November 06, 2014, 04:03:56 PM »

@roswellgrey Thank you for contributing some concrete information to the discussion.  ;D

Your test file has prompted me to revisit this issue and I spent a few hours reading up on the current situation and checking it out on a long running Pi here.
I now have two large text files filled with commands and data, but I suspect this would be far too technical to post here.

So ... for those of you who are not Linux experts, I will try to formulate a short version.

 On a RasPi the Operating System and all the user files and log files are stored on a small, cheap SD Flash memory card plugged into the mmc controller.
 
Flash memory is written in blocks of 512 bytes

SSD Flash drives are fast, expensive and use 'dynamic wear levelling' to spread the writes out evenly and maximise the useful life of the flash cells.

SD cards are cheap, slow and some have no wear Levelling at all, most implement 'static wear levelling' via proprietary algorithms embedded in the controller on the card.
This Flash Translation Layer is opaque and cannot be directly observed or interacted with.

They have a 'write pool' of writable blocks and _if_ the OS can tell the controller that a block is no longer needed, it can be added back to the pool of available blocks, and thus be recirculated to help share out writes among flash blocks.

The Linux mmc driver can issue a TRIM command which tells the flash controller to erase a block, and thus add it back to the free pool.
 Unfortunately this functionality of the mmc controller on the RasPi has been broken for years.  :(
see: 'FStrim does not work on RPI'
http://www.raspberrypi.org/forums/viewtopic.php?f=71&t=47484

root@raspberrypi:~# sudo fstrim -v /
fstrim: /: FITRIM ioctl failed: Operation not supported
 
TL;DR
1) Buy a (genuine)  SanDisk Ultra SDHC card much bigger than you need.
2) DO NOT write a huge image file that fills up the whole partition, because every block that is written to is removed from the wear levelling pool.
 INSTEAD copy the minimal  OS image onto it, THEN expand the filesystem.
e.g. DD the <4 Gig image onto the card, then expand it to 8Gig.
If you bought a 16 Gig card that 'unused' space is not 'wasted' as it should be contributing to the wear levelling pool.

3) update to the latest kernel  ('sudo aptitude update kernel')
root@raspberrypi:~# uname -a
Linux raspberrypi 3.12.28+ #709 PREEMPT Mon Sep 8 15:28:00 BST 2014 armv6l GNU/Linux

And Broadcom binary blob. ('sudo aptitude update firmware-brcm80211')

root@raspberrypi:~# /opt/vc/bin/vcgencmd version
Sep  8 2014 19:02:48
Copyright (c) 2012 Broadcom
version 3f2f2607186be72e4945cfa8edc77872dfc73195 (clean) (release)

Enable the Newly Fixed TRIM  ;D

'The latest rpi-update kernel now includes a new MMC driver. This driver is completely rewritten'
http://www.raspberrypi.org/forums/viewtopic.php?f=63&t=85061#

root@raspberrypi:~# sudo fstrim -v /
/: 4681015296 bytes were trimmed

Automate the running of trim occasionally.
e.g.
Add this to your /etc/crontab
00 00   * * *   root    ionice -c 3 fstrim -v / | logger -t fstrim

For bonus points you could also reduce the wear from services which continually append to their log files -such as web server by e.g. installing ramlog:
http://www.tremende.com/ramlog/

I also came across this handy service called backup2mail where you can have your Pi email a backup of your database:
 http://www.backup2mail.com/

So due to the magic of the Open Source development model, everything now should 'just work' as it is supposed to, and you didn't have to pay for the upgrade.  ;)

« Last Edit: November 06, 2014, 05:07:31 PM by G.DMT »
Logged

NewtronStar

  • Kitizen
  • ****
  • Posts: 4839
Re: RaspberryPi - gui-less stats logger
« Reply #34 on: November 06, 2014, 05:44:17 PM »

This so helpfull G.DMT  :) had moved from 8GB MSD card with Noobs to 32GB kingston MSD card with just the Raspbian OS installed and extended the files system.
 
Logged

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 39528
  • Penguins CAN fly
    • DSLstats
Re: RaspberryPi - gui-less stats logger
« Reply #35 on: November 06, 2014, 06:45:41 PM »

That's very useful information, G.DMT, thank you.
Logged
  Eric

NewtronStar

  • Kitizen
  • ****
  • Posts: 4839
Re: RaspberryPi - gui-less stats logger
« Reply #36 on: November 08, 2014, 11:24:49 PM »

Just did the Kernal update tonight on the RPi using sudo rpi-update a newer version was downloaded.

pi@raspberrypi ~ $ sudo uname -a
Linux raspberrypi 3.12.32+ #721 PREEMPT Fri Nov 7 16:50:31 GMT 2014 armv6l GNU/Linux
and sudo fstrim -v / trimmed 5000000000 bytes

Cheers
Logged

roswellgrey

  • Member
  • **
  • Posts: 73
Re: RaspberryPi - gui-less stats logger
« Reply #37 on: November 12, 2014, 07:23:35 PM »

Just curious as to what router stats sample/poll rate people typically set up dslstats (or similar) with ?

1 min, 10 mins, 1 hour ?

Was just trying to get a guesstimate as to how many samples / month  is a typical figure ....
Logged
Billion 8800NL

NewtronStar

  • Kitizen
  • ****
  • Posts: 4839
Re: RaspberryPi - gui-less stats logger
« Reply #38 on: November 12, 2014, 07:53:40 PM »

I have Dslstats set to sample each minute
Logged

kitz

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 32223
  • Trinity: Most guys do.
    • http://www.kitz.co.uk
Re: RaspberryPi - gui-less stats logger
« Reply #39 on: November 12, 2014, 07:59:32 PM »

Every minute here
Logged
Please do not PM me with queries for broadband help as I may not be able to respond.
-----
How to get your router line stats :: ADSL Exchange Checker

MartinGoose

  • Member
  • **
  • Posts: 92
Re: RaspberryPi - gui-less stats logger
« Reply #40 on: November 12, 2014, 08:17:58 PM »

Every 5 minutes here.
Logged

burakkucat

  • Global Moderator
  • Senior Kitizen
  • *
  • Posts: 27839
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: RaspberryPi - gui-less stats logger
« Reply #41 on: November 12, 2014, 08:59:06 PM »

Every minute.
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

roswellgrey

  • Member
  • **
  • Posts: 73
Re: RaspberryPi - gui-less stats logger
« Reply #42 on: November 13, 2014, 06:00:58 PM »

Thank you :)
I thought you would be in the ballpark of 1 min samples, and it seem that most of you are (I am as well) :)
This makes for 525K rows in the database per annum. It will be interesting to see how the Pi behaves as it's database grows (just from a performance point of view), assuming the SD card doesn't nuke itself  :lol:

I do have an (almost) "no writes to the SD card option" available if anyone wants to have a play .... ("almost" as it only adds to an error log if something unforeseen happens). Shout if interested, and I will make it publicly available.
Logged
Billion 8800NL

Axel Foley

  • Member
  • **
  • Posts: 49
  • Memento audere semper
Re: RaspberryPi - gui-less stats logger
« Reply #43 on: December 23, 2014, 12:39:48 PM »

I do have an (almost) "no writes to the SD card option" available if anyone wants to have a play .... ("almost" as it only adds to an error log if something unforeseen happens). Shout if interested, and I will make it publicly available.

This is a great idea, I hope you will release it publicly. :)

I have no problem with sdcard, since I have a NAS, with a central mysql tha hosts the DBs of all the software I use. I was also thinking if the software could actually be ported so I could run it directly on the NAS, which is the only computer always turned on in my house. If it's not possible I'll turn on the raspberry. :)

I use a Netgear DGN2200 router (as a pure ADSL modem/bridge), please find attached the log as requested, with telnet authenticaion and all the adsl commands required. Please take note the in the prompt string you will find the hostname, it's not the classic, plain-style prompt.

Thanks for your work.



« Last Edit: December 23, 2014, 01:09:52 PM by Axel Foley »
Logged

Axel Foley

  • Member
  • **
  • Posts: 49
  • Memento audere semper
Re: RaspberryPi - gui-less stats logger
« Reply #44 on: January 16, 2015, 10:42:01 AM »

Sounds interesting to me too.  :)

I have a Pi connected to a Billion 7800 and a Netgear DGND3700
with scripts collecting the modem data from them.

Its all modular so you would be welcome to pick up any pieces that might be of use.

At The moment:

cron calls a script which interrogates the modem via telnet > modem_out_file
A script parses modem_out_file and to extract the data of interest.

optionally script functions can split out into individual files and create graphs
optionally script functions can archive selected data files
optionally script functions can archive the collected data
optionally script functions can extract data from the archives

PHP classes written & running under Eclipse remote debugger / nginx / php-fpm to access the data.
;-)

G.DMT, since I noticed this GUI-less project is not being brought ahead, actually I found your post very interesting.

What I hoped existed was a php/web/python solution, that queried the modem/router, gathered the output, parsed it and then send data to MySQL and to MDWS (it would be great if it also showed a GUI with graphs like MDWS does, in case a user does not want to send data externally).

PHP/Python solution could be easily run on a home nas, or raspberry, or another device that is always turned on and it would be accessible via any browser to analyze data, also remotely.

Reading your post, I thought what you developed could be a good start. :)



Logged
Pages: 1 2 [3] 4 5