Considerations for running a headless FreeBSD server

[assumes competentcy in kernel config, general freebsd use,
networking(if applicable) etc] this document does not explain how
to run explicit services on a freebsd headless server, but the
mechanism as to how one can get a headless machine running as a
ground for providing such services.

What is a headless server?

 A headless server is a computer, or other automated device that
 runs without any means of external user input/output devices, such
 as mice, keyboards, monitors and the like. Headless servers usually
 complete tasks that require no local user at the console, yet this
 does not exempt them from a user interacting with the machine in
 another form. Interaction can be in the form of, but not limited
 to network connections, such as telnet and ssh, or a serial terminal
 to provide interaction with the machine. Headless servers might
 not even need interaction in the form of a remote user, but at the
 very least, this should be accounted for as a high level form of
 debugging any problems that may arise with the machine. Machines
 used for routing, file serving, X servers, and many other tasks
 can be run in a headless manner.

Why run a headless server?

 Headless servers have many advantages, but as with any solution,
 they also have their disadvantages. We will discuss these particular
 merits in this section.

 By using a headless server, the actual footprint and/or space that
 the machine occupies can be significantly lower, making them ideal
 for cabinets, or small spaces that one might not originally intend
 to be used for a computer. These factors are particularly useful
 where space is of a premium.

 Energy consumption can also be reduced compared to that of a normal
 computer station, as monitors and other external power consuming
 devices can be removed altogether. Of course, energy consumption
 merits can vary between setups, depending on the power configuration
 of the machines such as power saving methods. In any case consumption
 should be lower than a typical computer station setup, even if the
 difference is not entirely noticeable.

 Reduced costs not only because of energy savings, but also because
 of lack of components required to run a headless machine. Outside
 of the costs of the actual machine itself, very few components
 should be needed. Necessary items of noteworthiness would be a
 rack or cabinet, as well as a power back up unit, if it is felt
 that the operating system or the hardware is not tolerant of power
 cuts in which it can recover upon power restoration. A power back
 up unit should also be considered even if the system is decided
 to be tolerant of power loss, but still consumes time to bring the
 system back up after power restoration, for instance, checking
 file systems for integrity.

 It could be argued that extra security could be gained from a
 headless machine, as a cracker would be unable to sit down at the
 console, and attempt to break into the system. This of course would
 have no bearing if connections for a keyboard and monitor were
 available on the headless machine. With the lack of a console, and
 if the system is on an internal LAN, it would typically be behind
 a firewall, giving a little extra security. Of course, these
 security benefits should only ever be coupled with other security
 measurements provided by the operating system and/or LAN environment.

 A particular disadvantage of a headless machine would be when it
 crashes. For what ever reason the crash, it could be harder to
 trace and debug the problem, as when an operating system crashes,
 it only displays information relating to the crash on the local
 console. To external or remote users, it would seem to appear the
 system had hung, thus needing a manual reset. Logging should be
 fine for 3rd party and high level applications, (these applications
 should also display any of their crash messages to the remote user
 anyway), but problems arise when dealing with kernel panics, as
 these work in the nature discussed earlier.

 A work around to this problem is to run the system with means of
 input and output locally, through the means of a monitor and
 keyboard, for a test period, still using the system for the purpose
 it was designed for, with the hardware that is expected to be
 running in the system. This will then ensure the hardware can
 co-exist with the operating system in its role of normal use. Of
 course unexpected crashes will continue to be a problem, but in
 the course of modern computing this is always the case.

 Reliability studies of individual hardware components can help
 give an overall summary of a computer expectancy in hardware terms.
 This then leaves operating system crashes. If the machine passed
 the testing phase, discussed earlier, successfully, then the only
 other time an operating system can be introduced to a bug (in
 theory, anyway), is through source code changes. This can then
 hopefully give insight into a possible timing to when a crash
 (however rare they are) may happen.

 Sometimes it may not be apparent when a system has crashed. For
 instance, a crash at night may not be noticed, or even logged for
 reasons above, if the user only uses the machine between normal
 office hours. It is good practice to regularly check the uptime
 of the system, with output from such commands as w(1) or uptime(1)
 on freebsd systems. Other methods of checking system uptime and
 outages is discussed in CHAPTER X of 'Unix Hints and Hacks', by
 AUTHOR.

How to run a headless freebsd server

 Making a machine headless

  In order to make the machine run in a headless configuration, a
  few items need to be checked. Configure the machine with the
  monitor and keyboard plugged in, as in normal operation. After
  completing the following instructions, these inputs should be
  able to be safely removed, allowing the machine to operate headless.

The actual specification of the machine should be determined by the
role of the machine, and not the requirements of the operating
system, if these are greater. FreeBSD will run on a 386, with 8mb
of RAM, but the role of the server may demand a lot more physical
resources. It is the user's responsiblity to ensure the machine
specification meets demand of the designated role.

Install FreeBSD via any method suitable. Installation for a normal
running version of freebsd should be possible inside a 400mb HD,
including kernel source code and swap. Of course, this excludes 3rd
party software the user may want to run, such as the X Window System.

Settings in the BIOS need to be altered in order to let the machine
boot without a video card or keyboard. Ideally, the BIOS should be
told to continue booting with any error found at boot time. This
setting can be found in Award BIOSes in the same screen selection
where options, such as time can be set, under the heading `Halt
On`. This needs to be toggled to `No Errors`. Other BIOSes should
allow booting with errors, check the system manual.

Other BIOS settings can be 'tweaked', for instance, disabling serial
or parallel port, secondary IDE channels if they are not in use,
and the like. Of course, the user must be entirely sure they wont
be needing to call on these disabled resources later on in the life
of the machine without knowing the implications (eg. rebooting,
fitting a monitor and keyboard to gain access to the BIOS in order
to enable ports).

Also check to see if the system speaker works, as this can be a
good indicator of any problems at boot time. Award BIOSes have only
two types of POST (power on self test) beep code. One informs the
user there is no video card (3 long beeps, and one short), and any
other tones indicate RAM problems, which alert the user of any
problems with the system prematurely.

This next paragraph is optional. It may be beneficial to customise
the freebsd kernel, so it will not probe for devices, or for instance
remove support for devices that are not present in the machine. For
instance, there may be no floppy drive, CDROM, etc., present, so
removing these lines from the kernel should make the kernel slightly
smaller, but also not waste time probing for devices that are not
present in the system. Removing things like apm (Advanced Power
Management) would also be a good idea, so the operating system
doesnt try to send a sleep signal to a non-existant piece of hardware,
or cause a BIOS call that confuses the system. If the user decides
to customise the kernel, it is a good idea to have two usable kernels
in the system. One may be the new customised kernel, created by the
user, for running in a headless configuration, and the other, a
kernel with support for extra devices, like CDROMs. This can be
acheived by creating the custom kernel, and having it installed as
/kernel, and then using /kernel.GENERIC when the machine needs to
fall back to a normal monitor/keyboard/other device configuration.
The GENERIC kernel can be booted by pressing another key when the
freebsd bootloader appears. This drops the user to a basic prompt.
All that needs to be typed here is `unload`, followed by `load
/kernel.GENERIC`.

After these steps are complete, the machine needs to be configured
for the purpose it will actually serve. The user must also check
to see if the software designed for the role will run correctly
headless. This should be proven in the final test stage. It is a
good idea that the program does not log solely to the console, as
this will of course be absent on a headless machine.

This leads to the topic of syslogd. Syslogd logs output from programs
on the system, and writes them to the particular file descriptor
specified in the syslog.conf file. This file should altered to allow
logging of all messages to a file called `all.log`. Simply uncomment
the line:

	*.*                                             /var/log/all.log.

save changes to the file, and restart syslogd with `killall -1
syslogd`. The line that defines output to /dev/console can be left
in. Take care when running the machine that /var does not get full
with the all logging option enabled. The user may want to alter
/etc/newsyslog.conf to alter pruning sizes of all.log.

Disabling daemons (or not enabling them at all) such as moused(8),
would also be wise, as these daemons will be listening for a device
that should never needed to be connected to a machine in a headless
configuration. Make sure the user can log into the system in the
way they expect (such as ssh'ing over a network). Enabling daemons
such as sshd should be done now. Check security features such as
ipfw so that it is not denying all packets, if the user intends to
log in remotely.

Other small touches can be made to rc.conf, such as the
`fsck_y_enable="YES"` directive, if running FreeBSD 4.3-RELEASE or
greater. This allows fsck to complete upon booting from a system
crash without interaction (if needed) from the user to complete the
checks.

Once the user is satisfied the machine can run headless, the machine
should be run with just a monitor, so output can be veiwed by the
user. The machine should then be put into service on a trial period,
while the user watches for output from the monitor for information
on kernel panics, verbose logging, etc.

 Logging in remotely

  Administration will probably be elected to be done remotely, and
  therefore, a remote user must be allowed to somehow use root.
  Either make sure there is a `normal` user, accessed by the
  adminstrator who can then log in, su to root, by adding the user
  account to the wheel group in /etc/group (known as inviting the
  user to another group, by adduser(1)), or allow direct root logins.
  The latter is a potential security threat, but some times can not
  be avoided. sshd(8) allows remote root login. For details read
  the sshd man page, and alter the /etc/ssh/sshd_config file to
  suit.

 Final steps

 Once the system has proved it can run `semi-headless` with a
 monitor, the system should be powered off, and the video card
 removed. The machine should now be totally independent of local
 input, such as keyboards, monitors, video cards and the like.
 Sometimes, it may be beneficial to leave the video card in place,
 so a monitor can be plugged in without needing to reboot. However,
 in some machines, slots may be of a premium, and therefore dictating
 that the video card be removed.

 Upon re-powering the system, the BIOS may produce a series of beeps
 telling the user there is no video card present. Check the manual
 against these POST beep codes. Despite the missing components, the
 BIOS should carry on booting the system, if the BIOS was configured
 to do so earlier. Once the FreeBSD bootloader is called, there is
 a pause of 10 seconds, before continuing boot up. (This 10 second
 delay can be removed, or shortened when configuring a build of the
 operating system world.)

 During boot up, the network should come up (if applicable), after
 which point the machine should be reachable. However, be aware it
 may not be accessible in the form of web serving, ssh, or whatever
 other network based tasks the machine has been given to do, until
 the binary for each task has been called, which, depending on
 configuration, could be noticibly later than when the system is
 initally reachable from the network.

 Powering down the system

  When powering off, the `shutdown -h time` command should be issued
  as root. After this command is issued, some form of beeps from
  the pc speaker should indicate that the system is shutting down.
  To be safe, do not remove power for a further 10 minutes, to make
  sure all binaries have been stopped, and file systems sync(2)'d
  and unmounted.


 Tips

 Monitoring a headless system can be useful, as hard disk activity
 or other signs of load may not be apparent to the user when logged
 in remotely. I find these (somehwta FeeBSD exclusive) commands
 useful especially on headless machines.

`systat -vmstat`        monitors a lot of system activity, interupt
actiivty, and the like. man systat(8) `uptime`                        brief
overview of system load trends over the past 1-15 minutes, and users
logged in `top`                           interactive process list

Reading dmesg, and also checking /var/log/messages cannot be stressed
enough, as this often alerts the user to any problems that might
be occuring within the system.