Last updated: Thu Aug 6 15:36:14 CDT 1998

Abstract

We constructed a console server for the following reasons:

  1. we can reboot servers from home by dialing in (even from prom mode).
  2. Get rid of machine room clutter (monitors and keyboards everywhere). Now we have one monitor and one keyboard.
  3. Now we can grab the console of any machine remotely. Even if the machine is network unreachable.
  4. We can now have all the servers (Sparc 10's and 20's, etc) rack mounted and get rid of cables everywhere.
  5. We wanted something that was ASCII based so that we could login from home.
  6. We wanted security, so that not just any user could grab our consoles.
  7. We wanted scrollback. On a traditional server/console (servers scattered around on tables all with monitors) if your machine panics while nobody is in the room, you are very likely to lose all the panic information off the top of the monitor. This can also be a problem with some of the hardware based console servers that you may see advertised in the backs of magazines. With the console server and screen, this won't happen and you can set scrollback as large as you want.

Parts

  1. 1 Serial port expander (we've had excellent luck with Aurora)
  2. a copy of screen (we have v3.5.1) click here to download.
  3. miscellaneous scripts.
  4. a modem or two. _Image_
  5. a restricted access machine.

What to avoid

In my estimation, typical TCP/IP based terminal servers from vendors (e.g. Xyplex, Cisco, Livingston) should be avoided for this type of installation. They are excellent for dial-in setups where you want multi-protocol capability (SL/IP, PPP, Xremote, ARA).
here's why:

  1. They can be non-transparent (8 bit clean)
  2. Even for dial-in, I prefer the serial port expander for easier authentication and custom login programming
  3. Ease of customization.. Try writing a special script or C program which triggers an event on something happening over a TCP/IP connection to a terminal server. It's doable, but messy.
  4. We want to be able to reach the server even when it is network unreachable. If the hub should happen to fail, how do you get to your console if it's on a TCP/IP terminal server?
  5. Limited securing capabilities. These terminal servers can be authenticated with IP addresses and tacacs and things like that, but once you get beyond these built-in capabilities, there's nothing else. We prefer to use things like stel or ssh to access the console server in a more secure fashion.
  6. The break signal. Most terminal servers will take you back to some kind of prompt on receipt of break signal. Then, you have to set a special software switch on the server to tell it to send the break through remotely. It's often difficult or impossible to do both at the same time.

Our setup

We have two Aurora 16 port serial expansion SBUS cards which give us access to all of our servers and give speeds capable up to 124KB/s. We have one modem on one of these ports which has dial-back built into hardware. It is a telebit trailblazer. Each server has an entry in the /etc/remote file by server name. (thus you can tip server.) This comes in very handy when it comes time for script writing.

The machine

When picking a machine, we chose 4.1.3_U1 as the Operating system because of its relative advantages over Solaris 2.3 in interactive mode. This has changed with 2.5, but the machine has worked so flawlessly for the last 2 years that there has been no reason to change. It is also much easier to setup a modem on a 4.1.3 machine. It is important that you have a machine with enough memory. We have a sparc 2 with 48 MB of memory and it handles things quite adequately. It used to be on a Sparc1, but once we got about 16 consoles (16 windows), it became noticeably sluggish when switching windows. If we were running X11R5 instead of OpenWindows3.0 (setup installed pre-R6) it would have probably fixed that problem. Our machine is its own NIS master so it does not depend on any other machines for NIS, and we do not automount mail. So, if the mailserver fails (of which the console is on one of the local ports) we don't run into a catch-22 situation.

We also have a special account setup on the console server. Consoles may only be grabbed by using this account. This was setup because all of our home directories are automounted from somewhere else. This would have introduced dangerous dependency circles (console server dependent on home directory server dependent on console server...)

Ideally, the machine would have local everything, no NIS, and be on a separate UPS from the rest of the servers. In practice it makes things easier for us if we can rlogin to this server from time to time to do updates. So, a compromise was reached, and it has proved very effective. In the event of a UPS failure, everything goes down anyway, so the additional expensve of a separate UPS (however small) seemed overkill.

We have been very satisfied with the Aurora portmaster series. It is easily installed with a loadable kernel module, and offloads the CPU processing burden from our host machine to the SBUS card. It also uses standard TTY's for interfaces so you can think of it just as another serial port. The management of it is quite easy to. They have included a special program which will change the maximum baud rate to any arbitrary number. Since 38400 is the maximum kernel baud rate on SunOS machines, (built into the hardware divisor chip - you can replace one entry with 76800 - click here for details), you set your getty up to 38400, and use the routine "mset" to change this value in the loadable kernel module to the actual baudrate you want to use. So, now your getty is running at this higher baudrate (e.g. 76800, 115200) even though it looks like it's running at 38400 in ttytab. I say it's possible, but I don't recommend it. Keep your consoles at 9600 and things will work best. We briefly tried to do 19200 for a while, and it worked fine, but you end up having to switch back when you are in PROM mode anyway, so it wasn't worth it. There may be a correct combination of PROM and serial port settings that would allow one to increase the baud rate, but I have yet to find it, and the relative gains seem to be small.

cabling

All cables from the serial ports of the Aurora to a sun ttya port must be null-modem cables. Really, you only need 3 pins (2,3,7) for _picture of DB25-RJ45 converter_ your cables (add pin 1 if you want chassis ground). We opted to make our own very cheap cables by using a spool of satin flat cable and some RJ-45 crimping tools with male and female connectors. This makes it very easy to run the cables since these flat cables are very compact. We bought a bag each of the male and female (the serial expander requires female connectors, while the Sun's require male.) Each blank connector has a DB25 connector on one side, and an RJ-45 on the other. It comes with 8 pins, that you can wire into the RJ-45 however you want. A picture is available of it to the right.
The pins come out of the plug into the empty DB25 connector on the right which snaps into place.

The last major piece you'll want it something to manage your sessions so that you can connect to them remotely and see what the console looks like. We refer to this as grabbing the console. The easiest way I found to do this was with the screen software. Click here for a short overview of screen's capabilities. Here's the man page.

Grabbing the console is accomplished via a small script that provides locking so that only one person can have grabbed a console at a time. Normally, all the consoles are iconized in xterm windows on the console master. When a person grabs a console, this window on the console server goes blank with a message saying "screen grabbed away". and then the user gets the screen. Advantages of using screen:

  1. well done and well documented software
  2. Terminal independent viewing. Uses built in unix terminal emulation facilities
  3. Stable software that is well supported
  4. Lots of features for logging, hardcopy, and having multiple sessions at once. We use 500 lines of scrollback

The disadvantage (for some) is that it's not GUI based. Some people like GUI's, but I feel an xterm window is adequate. I'm sure a quick Tcl/Tk application could be whipped up for the GUI devotee.

Other things that are present in the OSU/Purdue console server that are not present in this implementation are time stamps on console messages (we use syslog mark instead), and that multiple people can not conveniently grab a console simultaneously (one person RW, and the rest RO).

Personally Contributed items

Here's the grab script. When a person is done with the console, they just use the screen release code (Ctrl-a d) and the screen reappears on the console master a maximum of 5 seconds later.

Here's the X startup script I use to bring up all the Xterminals and start things off when Xwindows is started via a line in my .xinitrc file.

This is a miscellaneous script for putting something meaningful in the title bar that matches the name of the console of the machine being managed. You don't need it here, but it's useful in general for xterm windows.

The following script is called from the X startup script mentioned above. It's name is check. it's purpose is to manage the console's on the console server by:

  1. Starting a new screen and tip session to the console when none is present.
  2. Checking periodically if a screen is available for recapture to the console window.
  3. Displaying messages when a screen has successfully been grabbed away.

This should be all that is needed to setup a console server for < $1000 (US). (the price of the Aurora serial port expander). Though you may want to invest in more memory for the console management machine, or a nice large 19" screen.

Tricky stuff

We use tip since it is included with the OS. You can use any communications program you want. Tip uses the ~# escape sequence to send a break to the console. Be careful when using the ~ escapes. They can be very powerful, but they can also cause you problems if you are not careful.

If you must power down your console server, most serial port expanders will send the equivalent of a break signal down each line to the machine hooked up there. This will halt all your machines on each port. To avoid this there are several techniques. 1) unplug all your consoles (either at the console server or at the machine) before rebooting the console server! (This is the beauty of RJ45 jacks! They make it very easy!) Plug them back in when the server comes up. If you have old machines, like Sun4/280's or less than a sparc 10, you will probably have to type 'go' or 'c' or equivalent to take your Sun out of prom mode when you plug the console in. We have two machines that require this, they are 4/280's.

Aurora has a console driver for Solaris and a few other operating systems that effectively handle this break signal and allow you to not worry about your console server being powered down for whatever reason. To date, I haven't tried it, but I've been meaning to. It hasn't been an issue since our console server is on UPS and never goes down.

2) Another alternative is available by purchasing a serial port expander like the one available from GNP. The speed of these things (230KBps) will be over kill, and they are more expensive but they are also supposed to prevent that break signal from travelling down the line and taking your server to PROM mode. They have streams drivers for Solaris and for SunOS.

Other Features

Mike Russell modified things slightly to make grab work with a terminal server instead of directly connect SBUS or SCSI serial devices. Here's what you do:

  1. Put each hostname/port mapping in /etc/services. Then you can simply put telnet termserver $1 in place of tip $1 in the check scripts.
  2. Since the screen method of sending a break CTRL-A B doesn't work for telnet connections, you must use the terminal server equivalent instead. For example: CTRL-]
  3. The following changes to the grab script will show who has the port locked:
    12,15c12,1312,15c12,13
    	<       
    	<	who=`ls -l /tmp/lock.$1 | awk '{print $3}'`
    	<	  host=`cat /tmp/lock.$1`
    	<	  echo "Console already in use by user $who at $host"
    	---
    	>	  who=`cat /tmp/lock.$1`
    	>	  echo "Console already in use by user at $who"
    
    	<       
    	<	who=`ls -l /tmp/lock.$1 | awk '{print $3}'`
    	<	  host=`cat /tmp/lock.$1`
    	<	  echo "Console already in use by user $who at $host"
    	---
    	>	who=`cat /tmp/lock.$1`
    	>	echo "Console already in use by user at $who"
    

It's also possible to have screens within screens. You could have multiple consoles and use the capabilities of screen from home to switch quickly back and forth between them. Screen also allows you to asynchronously send commands to a screen session. In this way you could do a certain thing to all the consoles at once. Care should be taken when nesting screen inside itself. This introduces another level of indirection. So, to detach a console, which is in a window of your screen session (the console itself is a screen session), you would need to do 'Ctrl-A a d'. You have to tell screen that you want to send a literal control-a through to the terminal (in this case the second screen) and not to interpret it as the local escape character. It should be noted that you can set the screen escape character to anything you want, but that 'Ctrl-A' is the default. If you have a screen within a screen within a screen, you would need to do 'Ctrl-A a a d' to detach the bottom-most screen.

Yes, but can you turn off the power remotely?

I've also developed a method whereby you can use X10 modules to remotely turn your computer equipment on and off. I have a writeup of my solution and the reasons we chose not to go with some of the more popular commercial solutions (APC MasterSwitch, WTI devices, etc) here.

If you have any questions, feel free to send me email or comments.

Other Console Servers (this information may be out of date. I keep it as current as I can given feedback from others)

For comments from a person who implemented this with slight variations on Linux, click here

An old version of the Purdue console server conserver (recent) is at this UUnet config-mgmt archive. In production for at least 2 years.

Ohio State also has the original Unix Console Server program available here. Tom Fine is the author of the OSU console server upon which the revised purdue console server is based. The history of the modern Unix console server is quite interesting.

The Conserver is supposed to be quite similar to Aurora's commercial offering. Also see the Greater Scroll of Console Knowledge web pages.

There's also one available at iastate.edu that is based on the Ohio state version. This particular link appears to be for the client source only, though.

For an excellent discussion of 100baseT and vcons, a console server implementation at Elementel Telecommunications Systems Laboratories, Sweden, see this paper by Henrik Martin. Or send him email at henrik.martin@eua.ericsson.se

Sun has done away with their ConsoleServer product and integrated this service into Solstice Suite. Go to Sun's WAIS search page and search for 'Console Server' for more information on it.

Greg Baker has a console server that looks sort of like mine (also uses screen) but also adds ttywatcher to give you the ability to trade read/write access with other individuals.

Aurora has a commercial console server which is apparently loosely based on what I've written here.

Console Works at www.tditx.com is a commercial application to manage consoles and other serial and non serial devices via a web interface

Table showing 9 pin to 25 pin serial RS232 wiring