Kitz ADSL Broadband Information
adsl spacer  
Support this site
Home Broadband ISPs Tech Routers Wiki Forum
 
     
   Compare ISP   Rate your ISP
   Glossary   Glossary
 
Please login or register.

Login with username, password and session length
Advanced search  

News:

Pages: [1] 2

Author Topic: The OR Provided VDSL Modem  (Read 21520 times)

burakkucat

  • Respected
  • Senior Kitizen
  • *
  • Posts: 38300
  • Over the Rainbow Bridge
    • The ELRepo Project
The OR Provided VDSL Modem
« on: June 29, 2011, 07:18:29 PM »

[TD -- If I've posted this in the wrong location, sorry. Please re-home, as appropriate. :-[ ]

Those persons who currently have a FTTC broadband connection will have been provided with a router (with a WLAN input) by their ISP and a VDSL modem provided by OR at service activation.

There have been grumbles regarding the VDSL modem:

1) It is impossible to obtain statistics from it
2) It is unreliable and tends to overheat unless mounted vertically
3) The EU is supposed to only use it and no other
4) It is an unknown device

In the thinkbroadband fora there are two interesting threads. The first is The BT FTTC line stats thread, especially the fork of it, created by asbokid on Wed 22-Jun-11 @ 23:34:23 by changing the subject headed to "re-enabling web interface on HG612". The second is 8200M suitable for UK's FTTC?, espcially asbokid's post of Tue 28-Jun-11 @ 23:50:44

From analysis of both of those threads it can be determined that the OR supplied VDSL modem is a hobbled Huawei Echolife Home Gateway 612 (to be referred, hereafter, as a HG612) and it seems it is possible to modify it back to some form of "normality".

[The other point (but OT for this thread) is that a Billion 8200N, with both POTS VDSL and Ethernet WAN inputs may prove to be a useful replacement of both the OR provided modem and the ISP provided router.]

« Last Edit: July 18, 2011, 03:15:44 AM by burakkucat »
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

Azzaka

  • Reg Member
  • ***
  • Posts: 572
  • SysAdmin
    • A Designers Work in Progress
Re: The OR Provided VDSL Modem
« Reply #1 on: July 11, 2011, 04:35:00 PM »

As far as I have heard, there are rumours the Old OR will be replaced by a new and improved. It does have its issues and the reason you cannot see the line stats is due to OR locking them down as BTW does not own the equipment.

As soon as I know more I will of course post here, however if anyone hears more in the meantime please dont wait for me.
Logged
I Sync', I Auth', therefore I am.
Online

burakkucat

  • Respected
  • Senior Kitizen
  • *
  • Posts: 38300
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: The OR Provided VDSL Modem
« Reply #2 on: July 11, 2011, 05:02:36 PM »

Quote
As soon as I know more I will of course post here <snip>

Thank you. :) I'm hoping that asbokid will provide some more information on the OR hobbled HG612 in due course, when the moment is ripe, etc.
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

asbokid

  • Kitizen
  • ****
  • Posts: 1286
    • Hacking the 2Wire
Re: The OR Provided VDSL Modem
« Reply #3 on: July 17, 2011, 01:16:28 AM »

I'm hoping that asbokid will provide some more information on the OR hobbled HG612 in due course, when the moment is ripe, etc.

Hi..

I've found a way to unlock the HG612 without the need to solder anything.  I will test a bit more, just to check it's not going to brick the device.

If anyone wants to hack along with me, then there is a tarball of Linux tools that I created to dismantle and modify the firmware. The tools would probably build in Microsoft Windows, maybe in the Cygwin environment.

Once you've teased open the firmware for the HG612, it's pretty clear what needs modifying to re-enable the web interface.

With the web interface reinstated, you can get xDSL line stats from the HG612 and you can configure various bits and bobs, including the firewall.

How it works:

The modem runs a configuration management server (CMS). That server reads a configuration file. Amongst other things, that configuration file contains the firewall ruleset to use.  The current configuration file is stored in a 64KB block of the Huawei's flash memory that is reserved for NVRAM purposes.

When you press the reset button on the front of the modem, that region of NVRAM is cleared and the modem reboots.  The default configuration file is then read from the (read only) squash root file system of the router.  By default, that configuration file disables LAN-side web access, secure shell access, etcetera.

While the consumer is denied access to line stats from the HG612,  Openreach and Huawei have left themselves access to the device via TR-069 (cwmp).

On the WAN side, there is a VLAN channel (301) which provides access to the httpd, as well as to dhcpd (for obtaining an IP address), sshd (for shell access with a default Huawei username:password) and tftpd (for uploading new files to the modem).

To open up web access from the LAN-side of the HG612 involves modifying the default configuration file in the root filesystem of the modem (/etc/defaultcmg.xml).

When the device is reset using the button on the front panel, the default configuration is read from that file.  The original CMS config file looks like this.

That's about it really!

The toolkit to make the modification (at your own risk) is on a dedicated blog for hacking the hg612...

http://huaweihg612hacking.wordpress.com/about/

cheers,
asbokid


P.S. It would be really useful to know whether Huawei/Openreach has released any other firmware versions for the HG612 since 22 February 2010.

All the HG612s that I've encountered have the firmware version "HG612V100R001C01B028SP06".  I'm surprised that the firmware hasn't been updated since then because that version is not very polished (it's full of bugs).

« Last Edit: November 27, 2011, 01:48:26 AM by asbokid »
Logged

roseway

  • Administrator
  • Senior Kitizen
  • *
  • Posts: 43598
  • Penguins CAN fly
    • DSLstats
Re: The OR Provided VDSL Modem
« Reply #4 on: July 17, 2011, 07:25:32 AM »

Thanks for all that information asbokid.
Logged
  Eric

razpag

  • Reg Member
  • ***
  • Posts: 622
Re: The OR Provided VDSL Modem
« Reply #5 on: July 17, 2011, 12:18:34 PM »

Wow asbokid !!! That is impressive reading, right there.

I haven't got a clue what most of it meant but am envious of your talents. ;D
Logged

burakkucat

  • Respected
  • Senior Kitizen
  • *
  • Posts: 38300
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: The OR Provided VDSL Modem
« Reply #6 on: July 17, 2011, 05:04:56 PM »

Extremely useful and interesting information. I do declare that asbokid is a top-class software hacker (using the word in its correct sense) and, possibly, is a fellow "penguin hugger". ;)

It was only yesterday that I called in on my neighbour (FTTC enabled) to ask if I could note down all the details from his OR provided modem. For those who have never seen one in real life, or the box in which it is delivered, here follows a transcript from my note pad --

On the Box

Huawei Home Gateway, EchoLife HG612
Support VDSL2
2 Ethernet Ports
Advanced Management via HTTP, TR-069
Support Router Mode and Built-In Firewall

On a Sticky Label Affixed to the Right Hand End of the Box

ITEM: 53030428
MODEL: EchoLife HG612
NAME: FTTC VDSL NTE
DATE: 2011-01-24
NOTES:

On a Stickly Label Affixed to the Left Hand End of the Box

UOI (PRIMARY PACK)
BT Item Code 049586
Description FTTC HG612
Quantity 1
Unit of Issue PCS
Contract No
Date of Man/Del
Use by Date
Calibration Date
Huawei BOM code 53030428

On the Underside of the Device

Serial Number: 21530304288K11031127 Y2 HG612
Firmware Version: V100R001C01B028SP10
MAC: <elided by b*cat>

Two questions:

(1) Has anyone obtained access to a copy of the original Huawei documentation for this device? :help:
(2) Is there anyone willing to send an OR HG612 to b*cat @ BSE? ;)
« Last Edit: July 18, 2011, 03:17:40 AM by burakkucat »
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

asbokid

  • Kitizen
  • ****
  • Posts: 1286
    • Hacking the 2Wire
Re: The OR Provided VDSL Modem
« Reply #7 on: July 17, 2011, 09:53:59 PM »

Quote from: burakkucat
For those who have never seen one in real life, or the box in which it is delivered, here follows a transcript from my note pad --
....

On the Underside of the Device

Serial Number: 21530304288K11031127 Y2 HG612
Firmware Version: V100R001C01B028SP10
MAC: <elided by b*cat>

Two questions:

(1) Has anyone obtained access to a copy of the original Huawei documentation for this device? :help:
(2) Is there anyone willing to send a OR HG612 to b*cat @ BSE? ;)

Wow! Thank you for going to the trouble of getting all that information!

I'm a bit worried now though.

Both of the HG612s that I've been allowed to hack have firmware version V100R001C01B028SP06, whereas your friend's has firmware version....SP10..   Which is presumably four revisions later.

That said, maybe the revision numbers don't mean much.

The Broadcom CFE bootloader in the modem supports double firmware images in the flash. Two different system images are stored in the flash memory - the MAIN image and the SLAVE image.

The bootloader selects which of those two images to load. Here we see it booting the MAIN (latest) image..

Code: [Select]
CFE version 1.0.37-102.6 for BCM96368 (32bit,SP,BE)
Build Date: Mon Mar  2 15:45:35 CST 2009 (root@localhost.localdomain)
Copyright (C) 2000-2008 Broadcom Corporation.

Parallel flash device: name MX29LV640BT, id 0x22c9, size 8192KB
CPU type 0x2A031: 400MHz, Bus: 160MHz, Ref: 64MHz
CPU running TP0
Total memory: 33554432 bytes (32MB)
Boot Address 0xb8000000

Board IP address                  : 192.168.1.1:ffffff00  
Host IP address                   : 192.168.1.100  
Gateway IP address                :  
Run from flash/host (f/h)         : f  
Default host run file name        : vmlinux  
Default host flash file name      : bcm963xx_fs_kernel  
Boot delay (0-9 seconds)          : 3  
Boot image (0=latest, 1=previous) : 0
Board Id (0-4)                    : 96368MVWG  
Number of MAC Addresses (1-32)    : 11  
Base MAC Address                  : 00:e0:fc:09:09:09  
PSI Size (1-64) KBytes            : 64  
Main Thread Number [0|1]          : 0  

*** Press any key to stop auto run (3 seconds) ***
Auto run second count down: 0

Support Double system.

Flash boot Flag: MAINSS
Boot from main system!
Decompression OK!
Entry at 0x8024b000
Closing network.
Disabling Switch ports.
Flushing Receive Buffers...
0 buffers found.
Closing DMA Channels.
Starting program at 0x8024b000
Linux version 2.6.21.5 (root@g40420m) (gcc version 4.2.3) #43 Thu Jan 7 10:23:47 CST 2010
......

The weird thing is that I extracted both firmware images (MAIN and SLAVE) from the flash memory and found that they are identified as versions ...SP06 and ...SP05, and yet they are byte identical.

Perhaps Huawei periodically increments the firmware version number, even when it has made no changes?

However, it leaves me a bit wary of updating the configuration of the firmware with an older version.

Ideally, we would flash an unlocked firmware into the MAIN image slot and use it to dump an image of the locked firmware from the SLAVE image slot. That locked firmware version could then be archived for examination. If necessary, the locked version could always be flashed back.

It's easy enough to dump the flash memory from the bootloader prompt, it's just a slow task. Here is a flash dump of the 'tag' header from an HG612 firmware image:

Code: [Select]
CFE> help
Available commands:

sm                  Set memory or registers.
dm                  Dump memory or registers.
w                   Write the whole image start from beginning of the flash
e                   Erase [n]vram or [a]ll flash except bootrom
r                   Run program from flash image or from host depend on [f/h] flag
p                   Print boot line and board parameter info
c                   Change booline parameters
f                   Write image to the flash
i                   Erase persistent storage data
b                   Change board parameters
reset               Reset the board
flashimage          Flashes a compressed image after the bootloader.
help                Obtain help for CFE commands

For more information about a command, enter 'help command-name'
*** command status = 0
CFE> dm b8010000 256
b8010000: 37 00 00 00 42 72 6f 61 64 63 6f 6d 20 43 6f 72    7...Broadcom Cor
b8010010: 70 6f 72 61 74 69 6f 00 76 65 72 2e 20 32 2e 30    poratio.ver. 2.0
b8010020: 00 00 00 00 00 00 36 33 36 38 00 00 39 36 33 36    ......6368..9636
b8010030: 38 4d 56 57 47 00 00 00 00 00 00 00 31 00 33 34    8MVWG.......1.34
b8010040: 37 33 31 34 35 00 00 00 30 00 00 00 00 00 00 00    73145...0.......
b8010050: 00 00 00 00 30 00 00 00 00 00 00 00 00 00 33 32    ....0.........32
b8010060: 31 37 30 39 36 39 36 30 00 00 32 36 37 38 37 38    17096960..267878
b8010070: 34 00 00 00 33 32 31 39 37 37 35 37 34 34 00 00    4...3219775744..
b8010080: 37 39 34 33 36 31 00 00 00 00 00 00 00 00 45 63    794361........Ec
b8010090: 68 6f 4c 69 66 65 5f 00 00 00 00 00 00 00 00 00    hoLife_.........
b80100a0: 00 00 56 31 30 30 52 30 30 31 43 30 31 42 30 32    ..V100R001C01B02
b80100b0: 37 53 50 30 35 00 00 00 00 00 00 00 00 00 00 00    7SP05...........
b80100c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................
b80100d0: 00 00 00 00 00 00 00 00 d1 df b4 69 d6 34 da d6    ...........i.4..
b80100e0: bf 02 72 a9 00 00 00 00 00 00 00 00 1e 1a 8c 65    ..r............e
b80100f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00    ................

*** command status = 0
CFE>

However, getting access to the CFE> bootloader prompt involves soldering the UART/JTAG pins onto the modem PCB, which most people aren't prepared to do.

As yet, a way to access the flash memory from userspace hasn't been discovered. It's obviously possible and there are library functions that the config tools must use to update the NVRAM region of the flash, but nothing is documented.

If the ltrace tool could be built for MIPS Linux, then it could be used to monitor all the function calls as they are made to discover the exact flash reading/writing mechanisms.  That could be useful for hacking other modems and routers that have a Broadcom 63xx processor (e.g. the BT Infinity HomeHub 2.0b).

There are lots of things to hack on these devices but Huawei is not a Linux-friendly company. Ironically, all the Huawei modems and routers run Linux, and almost all of the code in the HG612 is GNU GPL.  Many people have asked Huawei for documentation, source code and build configurations (which the company is obliged under the GPL to provide) but noone has ever got a response.

The only documentation I have seen for the HG612 is a CE electrical compliance certificate on Huawei's website, and a template PhD thesis (in Chinese) from the Harbin Institute of Technology that makes a brief reference to the design of the device.

Any way, thanks once again for going to the trouble of getting the firmware details.  Can you borrow the device briefly, and fit the UART/JTAG pins without anyone noticing?  Not wishing to put anyone off from doing that, but there is also small amount of PCB drilling to do as well  ;)

Cheers,
asbokid
« Last Edit: July 17, 2011, 10:50:24 PM by asbokid »
Logged

burakkucat

  • Respected
  • Senior Kitizen
  • *
  • Posts: 38300
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: The OR Provided VDSL Modem
« Reply #8 on: July 18, 2011, 03:13:39 AM »

Quote
Wow! Thank you for going to the trouble of getting all that information!

Pure co-incidence and, to be honest, a fluke. Someone over at the ThinkBroadband fora is compiling details to see if there is any logic behind those devices that overheat and those that do not. Hence my collection of that information was so that I could post it in the relevant thread over there . . . :)

(Incidently, my neighbour's HG612 is not mounted vertically and only runs warm. He has had no problems with it, so far.)

Quote
The weird thing is that I extracted both firmware images (MAIN and SLAVE) from the flash memory and found that they are identified as versions ...SP06 and ...SP05, and yet they are byte identical.

So if they both compare byte-identical, then their version identifiers (SP05, SP06) are not stored within the images. I wonder if that discrepancy in names is just a Huawei phoo-bah.

I confess that although I understand exactly what you have done / are doing at the "bigger picture" level, when it comes down to the finer details I am a little bit lost. ???

My "butterfly" mind has been speculating if it might be possible to set up a "spoof" TR-069 system and so access the device that way. ;) (Though that process really originated from when I was contemplating the 2Wire 2700HGV -- the BT Business Hub.)

So now there are two similar ideas floating around in my mind -- unlocked, generic firmware for the 2700HGV and the HG612. ;D
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

razpag

  • Reg Member
  • ***
  • Posts: 622
Re: The OR Provided VDSL Modem
« Reply #9 on: July 20, 2011, 09:14:03 PM »

Just a heads-up lads. Huawei have upgraded their Fibre modem and the new 'Echolife Modem B' has been launched and is in our stores hubs now.
I can't add anything else to the debate I'm afraid due to business protocols.  :)
Logged

burakkucat

  • Respected
  • Senior Kitizen
  • *
  • Posts: 38300
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: The OR Provided VDSL Modem
« Reply #10 on: July 20, 2011, 10:04:42 PM »

I am not sure if I am understanding you correctly, RP, but perhaps you could either roll your eyes or shake your head to the following:

Huawei have made a different "EchoLife" modem available to OR (i.e. not a HG612)?
This "EchoLife Modem B" is for EUs who are receiving a FTTC service?
And it is not for EUs receiving FTTP?

Quote
::) == yes.
:no: == no.

Of course, if the Huawei HG612 is now considered to be "old equipment", I don't suppose there would be any possibility of one being sent to "b*cat, At the end of pair nnn, DP1032, EABSE", so that I might have a new "play-thing" . . .  :hmm:
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

razpag

  • Reg Member
  • ***
  • Posts: 622
Re: The OR Provided VDSL Modem
« Reply #11 on: July 21, 2011, 06:24:48 PM »

Nice try B*Cat, but i'm pretty sure BT Security wouldn't need the Enigma to 'crack' your code pal.  ;D

What I'm trying to say is, it's 'business as usual' in all respects, other than there is a newer version of the VDSL Hub, which they are referring to as VDSL Hub B.

Is that ok for you B*Cat ?? ......  :no:=No and  ;D=Yes.
Logged

burakkucat

  • Respected
  • Senior Kitizen
  • *
  • Posts: 38300
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: The OR Provided VDSL Modem
« Reply #12 on: July 22, 2011, 12:36:40 AM »

 ;D Excellent! Thank you for taking the time to ensure that there is a happy cat at the end of this line pair. :dance:
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.

asbokid

  • Kitizen
  • ****
  • Posts: 1286
    • Hacking the 2Wire
Re: The OR Provided VDSL Modem
« Reply #13 on: August 03, 2011, 12:13:19 AM »

Someone over at the ThinkBroadband fora is compiling details to see if there is any logic behind those devices that overheat and those that do not. Hence my collection of that information was so that I could post it in the relevant thread over there . . . :)
(Incidently, my neighbour's HG612 is not mounted vertically and only runs warm. He has had no problems with it, so far.)

I think there must be something wrong with the power supply circuitry on the earlier model of the HG612 because the processor has a substantial heatsink stuck on it, and still it overheats. Yet the same processor, the MIPS-based Broadcom 6368, is used in a number of other modems and routers, without a heatsink. And the processor is a highly integrated System on a Chip (SOC), so the board variations are quite small.  Even the ethernet switch is integrated on the same silicon, as is the Analog Front End (AFE) that handles the DSP processing of the xDSL signal. So Huawei only had to bolt on some flash memory, and a DRAM, and still they got it wrong.

The ChinaTelecom VDSL2 equivalent to the Huawei HG612, is called the DareTech DB120-B2 [1] It's actually quite a bit better than the HG612 with 4 ethernet ports, wlan, USB, twice the RAM and twice the flash memory. But it's most interesting because it also uses the Broadcom 6368, and it does so without a heatsink.

The retail price of the DareTech is just 258 yuan (£25). So the crippled HG612 must have cost BT Openreach considerably less, perhaps just half that price. There is also a high-flying £200 Zyxel which uses the 6368, and Telus and Bell Fiber in Canada both supply CPE modems based on the 6368. Respectively, these are the Actiontec V1000H and the Lucent CellPipe 7130.
 
The fact that Huawei has had to slap on a processor heatsink, whereas all of those other manufacturers run the 6368 "bare", suggests that Huawei has done something very wrong with the board design.

Quote
I confess that although I understand exactly what you have done / are doing at the "bigger picture" level, when it comes down to the finer details I am a little bit lost. ???

I'm totally lost! It's just a bit of fun really, to see what's under the bonnet, and to find out what happens if you poke your finger in the places that say "keep out!"

Quote
My "butterfly" mind has been speculating if it might be possible to set up a "spoof" TR-069 system and so access the device that way. ;)

Netstat and iptables reveal that there is a hole in the HG612 firewall on the LAN-side. Whether or not that is intentional,  it allows LAN-side access to the TR-069 software, the chief component of which is called BTAgent.[2]   The BTAgent listens on UDP port 161 (SNMP).   I haven't really looked at it yet. An inquisitive fellow discovered that an snmp server is listening. That server is BTAgent, but he didn't manage to evoke a response from it[3] The strace tool is now built for the MIPS architecture[4] so it can be used to study the system calls that are being invoked by BTAgent and other components. That might shed a little light.

It's quite challenging to hack this device. A lot of the code for the executables in the firmware has been moved into shared library files which makes it hard to see what it's doing.  Ideally I would get ltrace running on the platform, so it could report all the library calls.  I finally built ltrace as a static executable (nightmare.. since it requires lib elf, lib iberty (part of binutils), and lib supc++, which bloats it to 500KB)  After compiling ltrace as a static binary from that heap of libraries, I ran it and it immediately segfaulted!  It turns out that ltrace will never work with the libraries in the HG612 firmware since Huawei has "stripped" them of all non-essential symbols to save space on the flash chip! For the same reason, the GNU debugger, gdb and the target server gdbserver, won't work properly either.

Anyway, talking of minutiae, most of the non-proprietary software libraries for the Huawei (uclibc, libm, libcrypto, etc..) have been rebuilt, as has the dynamic library loader (ld-uclibc), and I made sure not to strip any of them.  So ltrace should theoretically work. Either those libraries can be shoehorned into the flash memory of the Huawei, or hopefully, the root file system could be moved over from flash to a network file system on a PC.

A few more of the HG612s are knocking about on ebay at the moment, if you fancy a kickabout! They are the SP10s firmwares as well..  Go on, you know you want to!

[1] http://www.openwrt.org.cn/bbs/archiver/?tid-1802.html  (in English)
[2] http://huaweihg612hacking.wordpress.com/2011/08/01/what-does-the-btagent-do/
[3] http://www.kerneldump.org.uk/
[4] https://docs.google.com/leaf?id=0B6wW18mYskvBNmE2YTE4OTMtZTNmMC00NjgwLWI3ODMtMDE3ODk5NWI1MzQ1&hl=en_US
« Last Edit: August 03, 2011, 01:24:30 AM by asbokid »
Logged

burakkucat

  • Respected
  • Senior Kitizen
  • *
  • Posts: 38300
  • Over the Rainbow Bridge
    • The ELRepo Project
Re: The OR Provided VDSL Modem
« Reply #14 on: August 03, 2011, 02:06:33 AM »

Thank you for yet another very informative update. :thumbs:

There are many things that I would like to do, the problem is finding sufficient free time! However, I have downloaded a copy of your PDF format file ("Unlocking the BT Openreach HG612 VDSL2 modem router") and devoured its contents with relish. :yum: I am just a little puzzled as to what you mean at the second bullet point of the notes on page six -- "The dropbear sshd is configured incorrectly . . .". "Dropbear"? :hmm:

Quote
A few more of the HG612s are knocking about on ebay at the moment, if you fancy a kickabout! They are the SP10s firmwares as well..  Go on, you know you want to!

I'm tempted. Quite a bit . . . :dance: One for me, one for Eric, one for Walter . . . :silly:

(b*cat now departs to check eBay.)
Logged
:cat:  100% Linux and, previously, Unix. Co-founder of the ELRepo Project.

Please consider making a donation to support the running of this site.
Pages: [1] 2
 

anything