Kitz Forum

Broadband Related => Router Monitoring Software => Topic started by: roseway on August 09, 2012, 04:50:32 PM

Title: rs-ux and rs-w v0.4 released
Post by: roseway on August 09, 2012, 04:50:32 PM
(I've moved this subject to the Software section as more appropriate)

rs-ux and rs-w are programs which monitor and graph the performance of DSL routers, inspired by Routerstats. rs-ux comes in 32-bit and 64-bit versions and runs on Linux systems. At risk of offending my penguin-hugging associates, rs-w is a Windows port of rs-ux, and at present is only available in a 32-bit version.

*** These programs only work with Broadcom-based DSL routers which have a fully functioning telnet interface.

Compared to version 0.3, changes in this version are:

- corrected "List index exceeds bounds" error when wrong stats command prefix is chosen
- corrected display error on bitloading page when resizing
- added feature to take manual and automatic snapshots of the graphs
- added indicator of time when recording was started
- minor cosmetic changes

All versions can be downloaded from https://docs.google.com/folder/d/0BxbUtOYVZ_SCZ1BnY3RKYnN2b1U/edit
Title: Re: rs-ux and rs-w v0.4 released
Post by: renluop on August 09, 2012, 08:07:00 PM
Quote
*** These programs only work with Broadcom-based DSL routers which have a fully functioning telnet interface.
So not a Technicolorful friend then! ;) But which in particular Broadcomrouters will the rs-w be compatible with?
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 09, 2012, 08:11:28 PM
There's a partial list here: http://wiki.kitz.co.uk/index.php?title=Broadcom_CLI

but in the case of Thomson/Speedtouch I believe that some of the more recent firmware versions may have disabled the telnet capability.
Title: Re: rs-ux and rs-w v0.4 released
Post by: burakkucat on August 09, 2012, 09:21:01 PM
Thank you for this latest update, Eric. It has already been downloaded to The Cattery.  :)
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 09, 2012, 10:41:50 PM
By the way, I should have mentioned that I've partially implemented your request for intermediate marks on the graph axes, but only at the moment on the bitloading graph. The others will follow when I've worked out how to do it (the other two graphs use an off-the-shelf graphing component, and I have to delve into the innards to do this).
Title: Re: rs-ux and rs-w v0.4 released
Post by: burakkucat on August 09, 2012, 11:40:21 PM
As I have already noticed. Thank you for all of your hard work.  :)

One minor issue is that the Take Snapshot button reads as lake snapsnot. Please see the example image, below.

I guess a small tweak will soon sort that out.
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 07:27:06 AM
Quote
lake snapsnot
    ;D

Fonts!  rs-ux uses the system's default fonts, and it looks as though yours don't fit in the available space. Yes, I can tweak it. :)


Title: Re: rs-ux and rs-w v0.4 released
Post by: Bald_Eagle1 on August 10, 2012, 08:46:57 AM

One minor issue is that the Take Snapshot button reads as lake snapsnot. Please see the example image, below.



That doesn't seem to be an issue on my superior Windows system  :lol:
Title: Re: rs-ux and rs-w v0.4 released
Post by: Bald_Eagle1 on August 10, 2012, 09:04:30 AM
Hi Eric,


At risk of offending my penguin-hugging associates, rs-w is a Windows port of rs-ux, and at present is only available in a 32-bit version.



This program is looking pretty polished indeed - much better looking & possibly can be further developed to be much more efficient/easier to set up & use than a few "rough & ready" Windows batch file scripts.

As mentioned elsewhere, here is a little feedback from my Windows 7 (64Bit) system using a HG612 on a VDSL2 connection:-


It would be nice to be able to see an overall view of Bit-loading across all 4096 tones without needing to scroll back & forth.
The current "zoomed in" view is however useful for seeing bit-loading details for individual tones, so maybe a facility to see a default "overview" with an option to zoom in & scroll around?

I don't seem to be able to use the snapshot facility for any of the graphs. Error message screenshot attached.

How much data can be stored if running the program 24/7?
i.e. does it accumulate in a log file somewhere, or is it overwritten as a maximum sized data "window" in memory?

Do you intend to use "xdslcmd info --stats" data to calculate & graph changes in stats over each sampling period e.g. each 30 seconds, each minute etc?

SNRM & SYNC graphs & config screenshots attached for reference.


Cheers,


Paul.
Title: Re: rs-ux and rs-w v0.4 released
Post by: broadstairs on August 10, 2012, 10:09:57 AM
Just installed the new 32 bit Linux version again on Fedora 17 and the snapshot does not create any files on my system, I presume I should get some .png files or some such in the snapshot folder but nothing appears.

Stuart
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 10:25:16 AM
Just installed the new 32 bit Linux version again on Fedora 17 and the snapshot does not create any files on my system, I presume I should get some .png files or some such in the snapshot folder but nothing appears.

Stuart

On Linux, by default the snapshots are stored in ~/.rsux (a hidden directory). The filenames will be something like /home/eric/.rsux/2012-08-10-10:17:32-SNRMargin.png which is one I took a few minutes ago. Could you try changing the snapshot directory to something else, e.g. ~/Documents to see if that works?
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 10:53:04 AM

It would be nice to be able to see an overall view of Bit-loading across all 4096 tones without needing to scroll back & forth.
The current "zoomed in" view is however useful for seeing bit-loading details for individual tones, so maybe a facility to see a default "overview" with an option to zoom in & scroll around?

That would certainly be possible. I would have to throw away quite a lot of the data of course, because I'm sure you haven't got a > 4096 pixels wide screen. :)

Quote
I don't seem to be able to use the snapshot facility for any of the graphs. Error message screenshot attached.

I'll have to look more at this. It worked on the XP machine that I built the Windows version on, which is the only Windows machine available to me. I'm grasping at straws here, but could you try changing the snapshot directory to one of the standard user directories, such as your Documents directory?

Quote
How much data can be stored if running the program 24/7?
i.e. does it accumulate in a log file somewhere, or is it overwritten as a maximum sized data "window" in memory?

In the case of the SNR margin and Connection speed graphs, the historical data is stored internally by the off-the-shelf graphing component which I'm using. I'm going to check with the component author to try to get a definitive answer to this question.

In the case of the bitloading graph, no historical data is stored, so there isn't an issue. But I will be adding an option to store historical data as text files if required.

Quote
Do you intend to use "xdslcmd info --stats" data to calculate & graph changes in stats over each sampling period e.g. each 30 seconds, each minute etc?

That's what I do use to get the data. I use xdslcmd info --Bits to get the bitloading data.

Quote
SNRM & SYNC graphs & config screenshots attached for reference.

Thanks for those. Two other things I notice:

- the Attenuation figures in the status bar are shown as zero. If you could copy here the first 20 lines or so of the output of xdslcmd info --stats that will help me to see where the error is.

- I see in the bitloading chart which you showed above, that there are no blue coloured sections. I would have expected the tones from 32 to 95 to be blue because they could be either upstream or downstream, but this is an area I have to give more attention to.
Title: Re: rs-ux and rs-w v0.4 released
Post by: Bald_Eagle1 on August 10, 2012, 01:54:53 PM

It would be nice to be able to see an overall view of Bit-loading across all 4096 tones without needing to scroll back & forth.
The current "zoomed in" view is however useful for seeing bit-loading details for individual tones, so maybe a facility to see a default "overview" with an option to zoom in & scroll around?

That would certainly be possible. I would have to throw away quite a lot of the data of course, because I'm sure you haven't got a > 4096 pixels wide screen. :)



I have attached a high res (4096 pixels for the plot area) & a low res version of the same bit-loading data as generated via my scripts.

A lot of individual tone detail is definitely lost when plotting at a size that doesn't need to be scrolled, but it does give a reasonable overview that MAY dictate a need to "zoom in" & scroll around.


Quote
Quote
I don't seem to be able to use the snapshot facility for any of the graphs. Error message screenshot attached.

I'll have to look more at this. It worked on the XP machine that I built the Windows version on, which is the only Windows machine available to me. I'm grasping at straws here, but could you try changing the snapshot directory to one of the standard user directories, such as your Documents directory?


Nope, that didn't fix it.
I'll try this on an old 32 bit XP machine. It may just be a Windows 64 bit issue?


Quote

Quote
How much data can be stored if running the program 24/7?
i.e. does it accumulate in a log file somewhere, or is it overwritten as a maximum sized data "window" in memory?

In the case of the SNR margin and Connection speed graphs, the historical data is stored internally by the off-the-shelf graphing component which I'm using. I'm going to check with the component author to try to get a definitive answer to this question.

In the case of the bitloading graph, no historical data is stored, so there isn't an issue. But I will be adding an option to store historical data as text files if required.


The ability to store/graph infinite historical data would be most useful (particularly SNRM, Sync, Attenuation, Connection duration beyween resyncs, error counts etc.).
It certainly assisted me when "proving" ongoing faults that were not seen by my ISP & BT.

Quote

Quote
Do you intend to use "xdslcmd info --stats" data to calculate & graph changes in stats over each sampling period e.g. each 30 seconds, each minute etc?

That's what I do use to get the data. I use xdslcmd info --Bits to get the bitloading data.


Useful additions to bit-loading for static snapshots would be SNR (not SNRM), QLN & Hlog.
Data for these is obtained via "xdslcmd info --linediag".

Of course, the more data being collected, the longer the data collection period needs to be, via my scripts requiring a pause between commands to ensure data isn't overwritten.

Quote
SNRM & SYNC graphs & config screenshots attached for reference.



Quote

Thanks for those. Two other things I notice:

- the Attenuation figures in the status bar are shown as zero. If you could copy here the first 20 lines or so of the output of xdslcmd info --stats that will help me to see where the error is.


xdslcmd info --stats
xdslcmd: ADSL driver and PHY status
Status: Showtime
Retrain Reason:   0
Max:   Upstream rate = 4492 Kbps, Downstream rate = 31988 Kbps
Path:   0, Upstream rate = 4637 Kbps, Downstream rate = 28533 Kbps

Link Power State:   L0
Mode:         VDSL2 Annex B
VDSL2 Profile:      Profile 17a
TPS-TC:         PTM Mode
Trellis:      U:ON /D:ON
Line Status:      No Defect
Training Status:   Showtime
      Down      Up
SNR (dB):    5.3       5.5
Attn(dB):    0.0       0.0
Pwr(dBm):    12.4       6.3


This is a feature/bug of the Broadcom modems on VDSL2 connections.
Nowhere in the stats or GUI can a single value be seen for attenuation.
Line & Signal Attenuation is reported separately for each tone band (via xdslcmd info pbParams).
This data may be obtainable by querying the modem's "hidden" logs, but I don't know how to do that.

Quote
- I see in the bitloading chart which you showed above, that there are no blue coloured sections. I would have expected the tones from 32 to 95 to be blue because they could be either upstream or downstream, but this is an area I have to give more attention to.

Yep, I noticed that too.
They are indeed blue higher up the tone bands.
While you are sussing out that aspect, be mindful that Huawei & ECI DSLAMS use slightly different tone bands.

Title: Re: rs-ux and rs-w v0.4 released
Post by: Bald_Eagle1 on August 10, 2012, 04:36:57 PM

Quote
I don't seem to be able to use the snapshot facility for any of the graphs. Error message screenshot attached.

I'll have to look more at this. It worked on the XP machine that I built the Windows version on, which is the only Windows machine available to me. I'm grasping at straws here, but could you try changing the snapshot directory to one of the standard user directories, such as your Documents directory?


Nope, that didn't fix it.
I'll try this on an old 32 bit XP machine. It may just be a Windows 64 bit issue?



Unfortunately it still doesn't work on a XP Pro machine.

Title: Re: rs-ux and rs-w v0.4 released
Post by: burakkucat on August 10, 2012, 06:26:03 PM
My latest report on rs-ux V0.4.

I have created a directory structure ~/tmp/Eric_rsux/ into which I placed the downloaded tarball. Extraction of the contents gave me a directory ~/tmp/Eric_rsux/rs-ux64-04/ which contained the binary file, rsux, along with HISTORY and README files.

The utility was entered by double left-clicking upon its icon. Taking the Configuration tab, I defined the Snapshot directory to be /home/bcat/tmp/Eric_rsux/rs-ux64-04/. On clicking any one of the three Take snapshot buttons, the respective snapshot graph is saved in the designated directory. If the Snapshot directory is not defined with a suffix "/", then the snapshot graphs are stored in the directory one level above, i.e. in the /home/bcat/tmp/Eric_rsux/ directory.

I have now returned to the Configuration tab and have selected the three boxes to automagically take a snapshot of each page, once completed. This aspect is currently undergoing a test. (As I am sampling at a rate of once per minute and am displaying one hour's worth per screen, I should have a result in less than an hour from now.)
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 06:33:50 PM
Unfortunately it still doesn't work on a XP Pro machine.

I'll have to revisit the relevant section of code then. I'll let you know what progress I make.
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 06:38:51 PM
If the Snapshot directory is not defined with a suffix "/", then the snapshot graphs are stored in the directory one level above, i.e. in the /home/bcat/tmp/Eric_rsux/ directory.

A small coding error on my part. I'll make sure that the trailing directory delimiter is included.

Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 07:13:33 PM
Unfortunately it still doesn't work on a XP Pro machine.

I'll have to revisit the relevant section of code then. I'll let you know what progress I make.


I've found the problem. The snapshot filenames included the time in hh:mm:ss format. The colon is acceptable in filenames on Linux systems, but not on Windows systems. I had corrected this in the Windows version, but unfortunately uploaded the earlier executable. :-[
I've now uploaded the corrected Windows version and changed the archive filename to rs-w32-0.4a.zip.

https://docs.google.com/folder/d/0BxbUtOYVZ_SCZ1BnY3RKYnN2b1U/edit

(In the next release I'll change the Linux filenames in the same way to avoid confusion.)
Title: Re: rs-ux and rs-w v0.4 released
Post by: burakkucat on August 10, 2012, 07:23:50 PM
On the subject of date / time in file names, I have explained to Baldy_Bird how using the [CC]YYMMDDhhmm[ss] format is the only true sensible format . . .  ::)

I regret that I now have to report a failure. Although the snapshot graphs are saved correctly when initiated via "The Button", the automagical save feature only saves black rectangles.  ???

Please see the examples, attached below.
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 07:47:49 PM
Thanks for the report. They're very nice black rectangles, but I guess they're not what's wanted. :)
Title: Re: rs-ux and rs-w v0.4 released
Post by: burakkucat on August 10, 2012, 09:49:09 PM
They're very nice black rectangles, but I guess they're not what's wanted. :)

  :no:
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 10:05:14 PM
It seems that the snapshots only produce results when the graph being snapped is visible. This why the buttons work but the automated snapshots only work in specific circumstances. I'm testing a solution now.
Title: Re: rs-ux and rs-w v0.4 released
Post by: kitz on August 10, 2012, 10:36:48 PM
Download and had a look at it today, just to have a look at the interface - its looking very smart. ;D

Unfortunately it doesnt work with my router (TG582n) which doesnt use any of the 3 supplied cmd options in the dropdown.
Ive been having a poke around the CLI to see what commands are available and got the following info if its any use to you

Code: [Select]
xdsl (200:20008)
 - info (200:20008)     , [expand 200:20008], [g.997.1 200:20008], [counter_peri
od_filter 200:20008], [counters_reset 200:20008]
 - config (200:20008)   , [adslmultimode 0], [detect-lop 200:20008], [syslog 200
:20008], [status 200:20008]
 - maxspeed (200:20008) , [tx 200:20008], [rx 200:20008]
 - version (200:20008)
xdsl debug (204:20008)
 - bitloadinginfo (204:20008)
 - deltconfig (204:20008)       , [state 204:20008]
 - deltinfo (204:20008)
 - modemoptioninfo (204:20008)
 - multimode (204:20008)        , [config 204:20008]
 - traceconfig (204:20008)
xdsl qual (200:20008)
 - alb (200:20008)
 - bitloadinginfo (200:20008)
 - lov (200:20008)      , [signal_type 200:20008]
 - lcl (200:20008)
 - mib (200:20008)
 - qln (200:20008)
 - aisf (200:20008)
 - freeze-showtime (200:20008)  , [state 200:20008]

xdsl info is very basic but the following command gives full line stats inc errors
{Administrator}=>xdsl info expand=enabled

The following command gives bit loading
{Administrator}[xdsl debug]=>bitloadinginfo

If you want to take it any further I can give you some sample outputs.


Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 10, 2012, 11:11:09 PM
Quote
If you want to take it any further I can give you some sample outputs.

Thank you, I'd like to see them to find out if this type of router can be incorporated into the program. I seem to recall that Thomson/Speedtouch use their own custom CLI and don't expose the regular Broadcom CLI. But I wonder if there's a command which will take you to the BusyBox shell - it might be 'sh' or 'shell'?
Title: Re: rs-ux and rs-w v0.4 released
Post by: Bald_Eagle1 on August 11, 2012, 08:18:26 AM
Unfortunately it still doesn't work on a XP Pro machine.

I'll have to revisit the relevant section of code then. I'll let you know what progress I make.


I've found the problem. The snapshot filenames included the time in hh:mm:ss format. The colon is acceptable in filenames on Linux systems, but not on Windows systems. I had corrected this in the Windows version, but unfortunately uploaded the earlier executable. :-[
I've now uploaded the corrected Windows version and changed the archive filename to rs-w32-0.4a.zip.

https://docs.google.com/folder/d/0BxbUtOYVZ_SCZ1BnY3RKYnN2b1U/edit

(In the next release I'll change the Linux filenames in the same way to avoid confusion.)



Manual snapshots are now working, being saved in the same folder as rsw.exe

I have only just started using v 0.4a so am awaiting the automated snapshots.
I have ticked all 3.
How often should these be generated?


I notice that rsux.ini is only actually updated on quitting rsw.exe (Triviality, but should it be named rsw.ini for Windows systems?)

Also, I notice that although ticked in the GUI (screenshot attached), the bit-loading snapshot isn't included in rsux.ini


[login]
IPaddress=192.168.1.1
Telnetname=admin
Telnetpassword=admin
Usesecondstage=1
Shelltext=sh

[stats]
Commandindex=2

[graphs]
Samplerate=30000
Hoursperpage=2
Showbitloading=1

[colours]
Graphcolour=14221311
Upcolour=32768
Downcolour=255
Othercolour=16711680

[misc]
DSLType=3

[snapshots]
Snapshotdir=C:\HG612_Modem_Stats\Scripts\rs-w32-0.4\
SNRsnapshot=1
Speedsnapshot=1



Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 11, 2012, 09:00:53 AM
Quote
Manual snapshots are now working, being saved in the same folder as rsw.exe

Good, thank you.

Quote
I have only just started using v 0.4a so am awaiting the automated snapshots.
I have ticked all 3.
How often should these be generated?

Automated snapshots don't work correctly in this version, so you might as well untick those options for the time being. Automated snapshots are generated with a frequency defined by the 'Time per page' setting, with a small overlap to ensure that nothing is missed.

Quote
I notice that rsux.ini is only actually updated on quitting rsw.exe (Triviality, but should it be named rsw.ini for Windows systems?)

Fair point. I'll change this in the next version.

Quote
Also, I notice that although ticked in the GUI (screenshot attached), the bit-loading snapshot isn't included in rsux.ini

Noted. Will be corrected. I'll also have the .ini file saved when the 'Save' button is pressed.
Title: Re: rs-ux and rs-w v0.4 released
Post by: kitz on August 11, 2012, 12:10:10 PM

Thank you, I'd like to see them to find out if this type of router can be incorporated into the program. I seem to recall that Thomson/Speedtouch use their own custom CLI and don't expose the regular Broadcom CLI. But I wonder if there's a command which will take you to the BusyBox shell - it might be 'sh' or 'shell'?

Unfortunately I think they do and afaik you cant get direct access to the shell.  This I believe is why when they released one of the versions of the ST585 (6.2 iirc ?) that you couldnt get access to certain information in DMTtool, nor the CLI despite many attempts to try and hack that particular version. :/

Theres no hurry on this eric, as I know that you are probably still tweaking things on the 'main' part of the prog.   I only mentioned it because a few of the ISPs do use the TG582n, which is why I thought I'd bring it up as a possible for the future.. Ive only just put the TG582n back on which is why I thought Id go have a look see what info/commands were available in their CLI.  At first glance it didnt look like full stats and bitloading were available, but using mlp interaction I was able to get a list of the above 'hidden' commands if you do want to make use of them at some time.
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 11, 2012, 12:51:57 PM
Quote
Theres no hurry on this eric, as I know that you are probably still tweaking things on the 'main' part of the prog.   I only mentioned it because a few of the ISPs do use the TG582n, which is why I thought I'd bring it up as a possible for the future.

Thanks for that. It does seem a good idea to try to incorporate the TG582n, but as you say, it will have to wait a bit.
Title: Re: rs-ux and rs-w v0.4 released
Post by: tonyappuk on August 11, 2012, 01:31:53 PM
Just to add my three penn'orth for what it's worth, it doesn't want to log in on my old Speedtouch 546, no matter what i've tried. Routerstats Lite and DMT both log in OK.
Tony
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 11, 2012, 03:25:34 PM
Thanks Tony. I'll put it on the list for investigation, but I'm afraid it's becoming a long list. :)
Title: Re: rs-ux and rs-w v0.4 released
Post by: tonyappuk on August 11, 2012, 06:28:20 PM
Quite understand and it was only for your information, really.
Tony
Title: Re: rs-ux and rs-w v0.4 released
Post by: roseway on August 12, 2012, 01:38:47 PM
To return to a question asked earlier:

How much data can be stored if running the program 24/7?
i.e. does it accumulate in a log file somewhere, or is it overwritten as a maximum sized data "window" in memory?

The only limit is available memory. By my calculations, if you sample every 30 seconds you'll be holding about 1.3 MB of data after a week, so there isn't an issue there at all. It's possible that there could be some speed issues with scrolling when a very large number of points have been plotted, but that shouldn't happen within any realistic time frame.