Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2 3 ... 5

Author Topic: RaspberryPi - gui-less stats logger  (Read 32134 times)

roswellgrey

  • Member
  • **
  • Posts: 73
RaspberryPi - gui-less stats logger
« on: October 25, 2014, 08:25:16 AM »

Just a poke if to see if anyone would be interested in this ....

I run my Pi with no graphical environment i.e. as a gui-less server.

Now, DSLStats is a great bit of software (major kudos to the developer), and it does run really well on the Pi, but I am having to run up a vnc session just for this purpose, which for me personally is not ideal ( have been using ssh for far too long to change my ways - lol). Hence, I decided to develop a gui-less stats gathering daemon. It stores all data captured in a mysql database (obviously hosted on the pi itself), and the stats view is html (nginx server) .

The performance hit on the Pi running this is minimal - mysql runs like a dream, and offloading the graph rendering to the client by using Google Charts means even when browsing the data the additional load on the Pi is, on average,  very low.

Now, if anyone else likes to run their Pi without a graphical environment, and hence would be interested in this, post here and please let me know what router you are using. I might need you to give me a telnet dump of a command/responses if I cannot find that info by other means. At the moment, it only knows about a Billion8800NL. It's probably a week off being fit for release of basic level functionality (needs a few more days soak test)

I should point out that it will never be as comprehensive as DSLStats ; it's solely a solution to fit a particular niche need of mine :) and I was just wondering if anyone else would find it useful....
Logged
Billion 8800NL

MartinGoose

  • Member
  • **
  • Posts: 93
Re: RaspberryPi - gui-less stats logger
« Reply #1 on: October 25, 2014, 09:18:15 AM »

I should point out that it will never be as comprehensive as DSLStats ; it's solely a solution to fit a particular niche need of mine :) and I was just wondering if anyone else would find it useful....

I am interested and am already running a script to access the stats on my Billion 8800. My log format is as follows:

Code: [Select]
Time Status Sync kbps----- SNR (dB) ----- Atten (dB) --- Power (dBm)--- FEC (15 min)-- CRC (15 min)-- ES (15 min)--
Down Up Down Up Down Up Down Up Down Up Down Up Down Up
00:00:01 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 3 6 0 4 0
00:05:01 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 6 3 1 2 1
00:10:01 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 6 3 1 2 1
00:15:01 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 6 3 1 2 1
00:20:02 Showtime 56614 14361 6.1 5.6 19.1 0.0 13.1 7.4 0 18 1 4 1 3
00:25:02 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 18 1 4 1 3
00:30:02 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 18 1 4 1 3
00:35:01 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 0 5 0 4 0
00:40:01 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 0 5 0 4 0
00:45:01 Showtime 56614 14361 6.1 5.7 19.1 0.0 13.1 7.4 0 0 5 0 4 0

Happy to share my script for constructive criticism ;D
Logged

loonylion

  • Reg Member
  • ***
  • Posts: 723
Re: RaspberryPi - gui-less stats logger
« Reply #2 on: October 25, 2014, 12:19:25 PM »

I'd be interested in something like this for an x86 processor. I run a headless firewall on a low power atom device and it has no gui (except a web based one) If there was a version of dslstats that could be configured with a conf file and ran without a gui it would be perfect.
Logged

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43467
  • Penguins CAN fly
    • DSLstats
Re: RaspberryPi - gui-less stats logger
« Reply #3 on: October 25, 2014, 01:23:53 PM »

I'm afraid that a GUI-free version of DSLstats would need a substantial rewrite, and there just aren't enough hours in the day...
Logged
  Eric

boost

  • Reg Member
  • ***
  • Posts: 768
Re: RaspberryPi - gui-less stats logger
« Reply #4 on: October 25, 2014, 02:05:09 PM »

HG612 + Pi here, what you need? :)
Logged

NewtronStar

  • Kitizen
  • ****
  • Posts: 4898
Re: RaspberryPi - gui-less stats logger
« Reply #5 on: October 25, 2014, 07:08:26 PM »

( have been using ssh for far too long to change my ways - lol).

Thats something i'll need to understand as got myself into a predicament with the HDMI to VGA converter on the PI selected hdmi_mode=4 720p  60 Hz and got a blank screen on VGA monitor and aslo with HDMI TV so I was blind could not even use the composite video, if I new how to use ssh could have got myself out of this instead of going down the re-install os root with Noobs  :-[
Logged

roswellgrey

  • Member
  • **
  • Posts: 73
Re: RaspberryPi - gui-less stats logger
« Reply #6 on: October 25, 2014, 10:21:38 PM »

Martin:  - thanks for the interest :)
In a few days I will post some install instructions and a download link for you to give it a try

Loony : what are you running exactly ?  and more importantly is there a freebee version available that I could play with ? Would be interested to explore what options are available, if any, for what you are running .....

boost : think there is enough info here and elsewhere for the HG612 for me to make a stab at it.  Will let you know when you can play guinea pig  ;D

newtron : post away with any ssh questions - will certainly help if I can ....
Logged
Billion 8800NL

loonylion

  • Reg Member
  • ***
  • Posts: 723
Re: RaspberryPi - gui-less stats logger
« Reply #7 on: October 25, 2014, 10:28:33 PM »

I'm running smoothwall express 3.1 on a fit-pc 2i. It's an opensource firewall distro.
Logged

roswellgrey

  • Member
  • **
  • Posts: 73
Re: RaspberryPi - gui-less stats logger
« Reply #8 on: October 25, 2014, 10:33:26 PM »

thanks - will do a little research on what might be feasible ...
Logged
Billion 8800NL

NewtronStar

  • Kitizen
  • ****
  • Posts: 4898
Re: RaspberryPi - gui-less stats logger
« Reply #9 on: October 25, 2014, 11:01:41 PM »

newtron : post away with any ssh questions - will certainly help if I can ....

Cheers roswellgrey at the moment ssh is disabled in the config will do my best to research into the ssh before taking the plunge into uncertain waters, this Raspbian OS has a simulatory to the Amiga boot configs which had no Bios either the PI has defo rekindled my enthusiasm for computing  :)
« Last Edit: October 25, 2014, 11:16:00 PM by NewtronStar »
Logged

G.DMT

  • Member
  • **
  • Posts: 76
Re: RaspberryPi - gui-less stats logger
« Reply #10 on: October 26, 2014, 08:31:45 PM »

Sounds interesting to me too.  :)

I have a Pi connected to a Billion 7800 and a Netgear DGND3700
with scripts collecting the modem data from them.

Its all modular so you would be welcome to pick up any pieces that might be of use.

At The moment:

cron calls a script which interrogates the modem via telnet > modem_out_file
A script parses modem_out_file and to extract the data of interest.

optionally script functions can split out into individual files and create graphs
optionally script functions can archive selected data files
optionally script functions can archive the collected data
optionally script functions can extract data from the archives

PHP classes written & running under Eclipse remote debugger / nginx / php-fpm to access the data.
;-)
Logged

G.DMT

  • Member
  • **
  • Posts: 76
Re: RaspberryPi - gui-less stats logger
« Reply #11 on: October 26, 2014, 09:16:27 PM »

It occurred to me that some of you that are using RasPis might be interested in a simple technique that I used to sidestep a range of issues related to connectivity and file storage on a Pi, which makes the whole process significantly simpler by using two brilliant features of SSH.  ;)

1) by adding the following line to ~/.ssh/config on your PC
LocalForward 127.0.0.1:10123 192.168.3.1:23

you can telnet directly to the modem locally from your PC by using the sshd on the Pi as a proxy
In this example the modem at 192.168.3.1 is firewalled on a separate subnet from the LAN and can not be directly be contacted from the LAN.
{The same can be accomplished on a M$ winblows machine using putty.}

2) On the linux client ( Pi or a Linux PC) use SSH_File_System to locally mount storage that is physically located elsewhere on the netwiork.

http://fuse.sourceforge.net/sshfs.html

On Raspbian 'aptitude show sshfs'
On Pidora 'yum info fuse-sshfs'

Thus I can either
a) Access data files stored on a Pi as if it were local to my PC.
or
b) Not have to save data onto memory card/usb stick on the Pi, because the locally visible file system can actually be fuse-mount to a directory on my PC.

;D
 
Logged

NewtronStar

  • Kitizen
  • ****
  • Posts: 4898
Re: RaspberryPi - gui-less stats logger
« Reply #12 on: October 26, 2014, 10:08:41 PM »

Yes got the PC to ssh commands to RPi using two small programs this is great to get me out of impossible situation on the RPi as you can see and interact with RPi's desktop GUI from the PC or get down an dirty with commands in the terminal  ;D 
Logged

roswellgrey

  • Member
  • **
  • Posts: 73
Re: RaspberryPi - gui-less stats logger
« Reply #13 on: October 26, 2014, 10:24:52 PM »

Thanks for the responses,and the offers for the work you have done ....

Now, the architecture of this is as follows:

A stats gathering daemon, which is a linux console app written in C, telnets to the router, requests the a/vdsl stats and parses the response. It then populates a mysql database (schema/table definition will be available to you), sleeps and repeats at a configurable rate.

With the exception of Loony's particular requirement (more on that in a bit), the mysql database and nginx server are hosted on your Pi on your local Lan. (although, with proper security measures taken, you could obviously open up the Pi to the wider internet world should you so wish, and hence access it from anywhere)

Now, using the mysql databse, an nginx server (although it could be apache/lighttpd etc) processes php scripts and uses Google Charts javascript to pass the data to the webclient to produce charts and the like.

Now, as of now , and for the first beta release, the charts are pretty crude/basic.
 
What would be great however is if those with interest in php/google charts could develop charts of their own and feed them back so that all could inherit their efforts. (you obviouly get the current php code with the install for use as a basis/template)

Now, doing this might mean that the daemon would need to capture more/different data, and obviously I would incorporate any such mods into the stats daemon.

Now, loony :)

I have had a play with smoothwall in a virtual machine, and there is a way forward, but it is probably more complex that you would like ideally, for the following reasons. I recompiled the daemon for what I believe is your host system, and it runs just fine on a virtual machine running smoothwall (i386). It's a special build as I needed to statically link in the mysql libraries as, unlike a standard distro, one cannot just add required libraries to smoothwall (understandably - it's a security device after all)

The special case is the fact we need a mysql/webserver somewhere (as it's normally on the Pi) , and running a mysql server on smoothwall would appear to be a no-no (ignoring the technical aspects, it's probably a bad idea to even try)

So, you need a mysql server to talk to.

Now, the stats daemon can talk to any mysql server that it can connect to, whether local or on the web

3 options spring to mind to overcome this problem.

1. I could, with the devs approval and help, make the stats daemon update the MyDslWebStats solution, removing the requirement for you to supply a mysql server/webserver (any of the MyDslWebStats team care to chip in with a comment about this in principle ?)
2. You could, for approx $15 US (approx £8 per annum) rent a virtual private server (VPS) with a good enough spec to host the mysql server/webserver yourself. (sry if this is teaching you to suck eggs, but you would need to look after it though - keep it patched and keep it secure - you cannot ignore an internet-facing device/server)
3. You could host a mysql server/nginx server on your lan on a 24/7 machine (buy a Pi ?  >:D), but this is probably a non-realistic option as if you had a linux machine on 24/7, you wouldn't have asked about the smoothwall option - lol

However, being technically curious, I would like to prove it would actually work in the real world on a proper smoothwall device, rather than just in my virtualization.
If you wanted to test this out, I would be more than happy to temporarily host a database/site for your data just to see if it really works as anticipated.
I don't ever want to get into hosting data storage for this solution - the whole point was to utilise the users Pi as it fits the requirements so well, but I would be intrigued to see if actually works in your case, hence the offer.

Let me know if you want to take this forward by any of the above means, and if so, we will see what we can work out...


« Last Edit: October 27, 2014, 08:05:05 AM by roswellgrey »
Logged
Billion 8800NL

sheddyian

  • Kitizen
  • ****
  • Posts: 1159
    • My Shed Blog
Re: RaspberryPi - gui-less stats logger
« Reply #14 on: October 26, 2014, 11:53:58 PM »

( have been using ssh for far too long to change my ways - lol).

Thats something i'll need to understand as got myself into a predicament with the HDMI to VGA converter on the PI selected hdmi_mode=4 720p  60 Hz and got a blank screen on VGA monitor and aslo with HDMI TV so I was blind could not even use the composite video, if I new how to use ssh could have got myself out of this instead of going down the re-install os root with Noobs  :-[

If you're having problems with the display on a Pi, there are some things you can change in the config.txt file that can help.

If you're already remotely logged into the pi, you can find the file at /boot/config.txt

OR

unplug the SD card from the PI, pop it into a PC, and you will see a small selection of files.  config.txt is one of them, so you can edit on a PC, then plug it back into the pi.

The lines of interest in config.txt are :

Code: [Select]
# uncomment if hdmi display is not detected and composite is being output
#hdmi_force_hotplug=1

# uncomment to force a specific HDMI mode (this will force VGA)
#hdmi_group=1
#hdmi_mode=1

# uncomment to force a HDMI mode rather than DVI. This can make audio work in
# DMT (computer monitor) modes
#hdmi_drive=2

# uncomment to increase signal to HDMI, if you have interference, blanking, or
# no display
#config_hdmi_boost=4

# uncomment for composite PAL
#sdtv_mode=2

I had to remove the # from config_hdmi_boost=4 to make a pi work with a particular DVI monitor using an HDMI to DVI cable, for example.

Ian
Logged
Pages: [1] 2 3 ... 5