Does this site look plain?

This site uses advanced css techniques

NOTE: This procedure has not been updated in some time and is mostly out of date; consult the iSystems Evo Technical Guide for official word if you care to do this yourself.

Evolution Logo

The Evolution Payroll service-bureau system from iSystems, LLC employs a three-tiered architecture, with the backend being one or more Linux servers running the Firebird SQL database. Evolution is able to split the load across multiple DB servers on a per-client basis.

Table of Contents

As Service Bureaus grow, it's common to add new DB servers to the system, and this Evo Tip covers the steps required to take a fresh Fedora Core system and prepare it for Evolution.

This Evo Tip is similar to (and loosely based on) the instructions provided by iSystems for their Franklin database upgrade, but we focus on building a machine from scratch rather than upgrading from one database version to another.

We use these very notes to configure our own machines, and we incorporate best practices from iSystems and from the rest of the Evolution technical user community.

Important Disclaimers

Prepare the machine

On a naked machine with no OS, one must often do a bit of configuration before the operating system can be loaded. This includes RAID array setup and BIOS configuration.

First, avoid RAID 5: this is the most space efficient (only one drive is "wasted" for redundancy), but our experience has shown that it has the worst write performance. This is particularly the case where the RAID controller has no battery backup, which usually disables the write cache. This means atrocious disk performance.

We normally use RAID 10, which is mirrored, striped drives: this requires essentially doubling the disk space, but the performance improvements are enough that we simply won't consider RAID 5 again.

Second, many machines offer Intel processors sporting HyperThreading™, and we recommend enabling it for DB servers (though iSystems recommends disabling it for Windows machines). Our testing indicates that it provides a modest improvement in real performance, and we've not found any reason not to do this.

Systems with dual-core Xeon processors should always enable the dual core - these are real live extra processors and make a tremendous difference in processing speed.

Install Linux

We typically install Linux from CDs made from downloadable ISOs from the Fedora site and boot them on the system being built. Though it's possible with modern server equipment to use network-based booting and system building (Kickstart/PXE), it's unlikely that most service bureaus will need to build so many machines that it's worth the trouble to set this all up.

Our preference is for heavily stripped down machines — just the base operating system plus the compilers for building tools — and we normally install with in linux text mode. The graphical installer may be used instead by those who prefer it.

Recommended filesystem layout is discussed in the next section.

We typically de-select almost every package during installation, preferring to manually install the few we actually need. We do

We normally disable the firewall and the SELinux facilities as well. SELinux is a Security-Enhanced Linux, and it was developed in the main by the National Security Agency.

Though a powerful augmentation to system security, one should always look at security in terms of the threats it's going to face. SELinux mainly protects against hostile local users, something we don't expect much of in a payroll service bureau.

Likewise with the firewall: we normally expect that the enterprise has firewall protecton at the border, not requiring it inside.

Prepare the Filesystem

Evolution typically keeps its data files in the /db/evolution directory, and this location is configured into the Registration Daemon on the Windows server. Though this location is not strictly required, technical support is probably going to be a lot easier if your installation relies on things being in the usual places (we've never seen a service bureau not use these locations). The rest of this Evo Tip assumes the default location.

It's a good practice to make /db/ a separate filesystem on its own partition, isolating it from the rest of the sytem. Should the root partition run out of space for whatever reason, it won't impact the client database file storage.

The size of the /db/ partition is normally defined during operating installation time, and plenty of space should be allocated for these very large files. Furthermore, choose a stable filesystem type (ext2 preferred over ext3). We have no experience with ReiserFS in a production environment and cannot yet recommend it for Evolution.

A representative new system with around 130 gigabytes of storage looks like this:

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda5              4032092    647864   3179404  17% /
/dev/sda2               101105     11067     84817  12% /boot
/dev/sda6            132628932     94272 125797404   1% /db
none                   1036320         0   1036320   0% /dev/shm

We note that iSystems recommends putting /opt on its own partition so that a Firebird log filling up won't trash the whole machine, and this doesn't seem like a bad idea. It's not really necessary, though, if the Firebird configuration has been set properly.

Be sure to create the /db/evolution directory and make sure it's owned by Firebird: this allowed Evo to create new client databases.

# mkdir -p /db/evolution
# chown firebird:firebird /db/evolution

Install some tools

We publish quite a few tools on our website, and it's not a bad idea to install them in each Linux DB server. We'll be using some of them during installation.

We suggest:

Install Firebird

FirebirdSQL logo

The Firebird - formerly Interbase - database is used by Evolution, and for now the older "Classic Server" is required rather than the newer "Super Server" version. The latter has a much better architecture, but iSystems reports serious data corruption issues with it and is not supported. DO NOT attempt to use the Super Server.

For Jamaica and prior releases, FirebirdCS-1.5.3 was used, but Killington and beyond uses version FirebirdCS-2.0.1; these can all be downloaded from the iSystems support site:

Though it can be downloaded via the GUI (KDE/Gnome) on the Linux machine, or via a standard Windows browser and then copied to the db server, we prefer to fetch the file directly from the command line.

Using a regular Windows browser, visit the iSystems support site and right-click on the link name.

Then, using a terminal emulation program (Secure Shell, preferably) login as root to the database server and use the evo-wget program to fetch the RPM file directly to the server; you should have set this up in a previous step to include your FTP/sbupdate credentials.

# cd /tmp

# evo-wget

Alternately, simply save the file to the local Windows machine and get it to the Linux database box somehow.

However: Some builds of FireBird 1.5 were constructed with an older version of the compiler, and requires an older standard C++ library that is not normally included with Fedora releases. The failure to find this library manifests itself with an error of this nature:

/opt/firebird/bin/gsec: error while loading shared libraries: cannot open shared object file:
    No such file or directory

The library must be installed, and we do so using the yum installer tool to fetch a compatibility package. The version required depends on the Fedora release:

# yum install compat-libstdc++-8    # Fedora Core 3


# yum install compat-libstdc++-33   # Fedora Core 5, RHEL5.1, CentOS 5

Red Hat Enterprise Linux 4 also requires the compat-libstdc++-33 package, but the Yum installer is not typically found: it's likely that the up2date system will fetch this package.

We've not seen this problem at all with Firebird 2.0 installations.

With the proper compatibility library in place, we can install Firebird:

# rpm -i FirebirdCS-

This installs Firebird in the /opt/firebird/ directory and makes some adjustments to the operating system (we'll be making more of them later as well).

Setting up the Firebird database users

Firebird supports its own notion of users independent of those maintained by the operating system or Evolution itself, and Evo's Franklin release (2004) introduced the EUSER id to augment the system-provided SYSDBA.

Both are used by Evolution, and they must be configured using the gsec tool:

# /opt/firebird/bin/gsec

GSEC> modify SYSDBA -pw pps97

GSEC> add EUSER -pw pps97 -fname Evo -lname User

GSEC> display
  user name        uid   gid     full name
SYSDBA             0     0
EUSER              0     0      Evo User

GSEC> control-D

Note that it's possible to use some other password for these accounts as long as Evolution is configured in the EEC to use them.

We don't customarily change the default password.

Configure Firebird

Firebird's turning is mainly driven by the file /opt/firebird/firebird.conf, and there are three settings that should be tuned. This is a straight ASCII text file that can be easily edited with vi, and the file saved.

These parameters are documented in the file in comments, so it should be a simple matter to find them and update the parameter.

TcpNoNagle = 0
Setting this option to zero insures that Firebird doesn't attempt to disable the use of the Nagle algorithm for the TCP/IP connection. Since Firebird runs out of xinetd, it's not able to change this option anyway, but some installers default to doing so.
Failure to disable this option leads to the Firebird.log file filling up with these messages:
inet server err:  setting NODELAY socket option
The errors themselves are benign, but it's annoying and fills up the log; this should always be disabled.
LockHashSlots = 2039
This increases the number of lock-manager hash slots to the largest prime less than 2048, and is much larger than the default of 101. This setting somewhat improves performance under heavy load, especially on multiprocessor machines.
LockSemCount = 128
This increases the number of semaphores allocated to Firebird, up from the default value of 32. In rare cases, presumably when Evolution is very busy, Firebird will run out of semaphores and display this message in the logfile:
Fatal lock manager error: semaphores are exhausted
When this happens, Evolution stops dead in its tracks, requiring a full restart of everything. It's hoped that upping this value to 128 will avert future occurrences of this calamity, though iSystems and I are researching the recommended value (which may increase in the future).

Set up the UDFs

Evolution extends the power of Firebird with functions defined by iSystems: These UDFs (User Defined Functions) provide features not supported by the stock database, such as the ability to remotely query all the client database filenames, as well as supporting Evolution's time-domaining system on all DB records.

The User-Defined Functions are provided in a single file, EvolUDFs and must be placed in the /opt/firebird/UDF directory; Firebird will use these automatically once installed.

This file is found on the iSystems support website, and as of the Irasburg release, it was in the Irasburg_Upgrade_Files/EvolUDFs.tar.bz2 This should be downloaded to the Linux machine, unpacked and untar'd, and moved to the /opt/firebird/UDF/ location.

Firebird daemons

Once Firebird is installed, it must be configured to listen for network connections from the middle tier servers. This is done via the xinetd service, which is not always installed with Fedora by default.

First: insure that the xinetd service is installed properly:

# yum install xinetd

If it's not installed, agree to do so.

Next, the default firebird configuration must be updated, as it's not really suitable for the manner in which Evolution accesses it. This is found in the file /etc/xinetd.d/firebird file, and it can be edited with vi:

service gds_db
        flags           = KEEPALIVE NOLIBWRAP
        socket_type     = stream
        wait            = no
        user            = firebird
        instances       = UNLIMITED
        per_source      = UNLIMITED
        cps             = 400 5
        server          = /opt/firebird/bin/fb_inet_server
        log_on_success  =
        disable         = no

We'll touch on what these options mean.

These set parameters that govern the network connection. KEEPALIVE asks that Linux periodically make sure the network connection to the Evo middle tier server is still there, and it allows for detection of crashed package servers.
NOLIBWRAP disables TCP Wrapper support, a security mechanism that provides no benefit in an evolution environment, and avoids a small amount of overhead.
Some configs include NODELAY, which disables TCP's Nagle algorithm, and for most Evolution installations this has no effect one way or the other.
This says that this service is a TCP service (as opposed to UDP).
This instructs xinetd to run the spawned process as the firebird Linux user, which is also the owner of the files in /db/evolution.
Xinetd knows how many xinetd processes are running at any given time, and it can enforce a limit. We don't want one, and UNLIMITED specifies this.
This is like the instances keyword, but specifies a per connected IP address limit. This should also be set to UNLIMITED lest a busy system provoke failures like this in the /var/log/messages logfile:
Jan 13 21:01:02 evolinux xinetd[1683]: FAIL: gds_db per_source_limit from=
This sets yet a third limitation on the connection rate: if xinetd receives too many connections from one source within a one-second period, it considers something wrong (such as an attack) and will disable the service for a short period with a message in the logfile:
Jan 9 14:20:15 evolinux xinetd[2622]: Deactivating service gds_db due to excessive incoming connections. Restarting in 30 seconds.
We set the value 400 5, which disables the service for 5 seconds if 400 connections arrive per second. In the past this was set to 200, but Plymouth's Evo DB Maintenance tool would sometimes exceed this during check-and-patch runs.
This provides the full pathname to the program that serves connection requests: it's the Firebird DB server process.
This entry, with nothing after the equal sign, asks xinetd not to make any log entries for successful firebird service connections. A busy Evo installation will get thousands of these per day, and they serve no useful purpose to log them.
Setting this to no enables the service.

Once done, start/restart the service and insure that it launches when the system boots:

# service xinetd restart
# chkconfig --level 2345 xinetd on

Operating system tuning

Now that Firebird is configured, we turn to some base Linux configuration issues: none of these is strictly necessary, but we have found them to be helpful in the past.

Shorten the TCP keepalive timers
The use of TCP keepalives (as configured in xinetd) can help keep Evolution from getting terribly out of sync if a package server reboots unexpectedly by insuring that DB connections associated with the previous crashed sessions are not retained.
By lowering the keepalive timer to less than the normal middle-tier reboot time, we can be sure that locks held by the previous run are never held over. The keepalive traffic is essentialy zero, so it presents no performance hit to the network.
The file /etc/sysctl.conf contains a whole list of tuning parameters, and we're just going to add one to the end:

net.ipv4.tcp_keepalive_time = 300
This setting takes effect upon the next reboot.
Disable IPv6 support
IPv6 is the next-generation of IP addressing, and though it's certainly going to be useful someday, it has positively zero use to an Evolution service bureau. Only IPv4 - "regular" internet addresses - are supported by Evo and by the internet at large, so we can reduce the complexity of the IP stack by disabling this advanced but unneeded feature. We do so by updating two files:
alias net-pf-10 off
alias ipv6 off     — Fedora 8
/etc/modprobe.d/blacklist — Fedora Core 7 & 8
blacklist net-pf-10
NOZEROCONF=yesRHEL4 or FC5/6, at least
The NOZEROCONF=yes token disables support for the 169.254.X.X network, which is not needed for Evolution installations. It won't hurt to add this for any Linux installation.
These also take effect on the next reboot.
Disable internationalization support
A few tools provided by the system (and here at use the perl language, but modern versions of perl get terribly slow when the full internationalized character set support is enabled. We typically disable UTF-8, making things run quite a bit faster.
# LANG="en_US.UTF-8" -- this part disabled
This has no effect on Evolution or Firebird itself and can be skipped if desired (we never do: we use a lot of perl tools).
Added Firebird to the system PATH
The Firebird tools are all found in /opt/firebird/bin/, but this is not normally part of the command-search path: this means that manually running commands such as isql or gbak will fail to locate them.
We typically add this

# Path manipulation
if [ `id -u` = 0 ]; then
        pathmunge /sbin
        pathmunge /usr/sbin
        pathmunge /usr/local/sbin

pathmunge /usr/X11R6/bin after

pathmunge /opt/firebird/bin     # add this

This takes effect on the next login.
Uninstall unnecessary packages
Though it's possible to do a minimal OS install, it's still common to have many more packages than could ever be used on an Evolution database server. Our feeling is to keep the machine as stripped-down as possible to avoid running useless processes or creating additional complexity, but the process of whittling down a machine takes time and experience to know what can go and what must stay.
However, one package has proven to be troublesome: ODBCunix. This provides database connectivity unrelated to Evolution's use, and it includes an isql command unrelated to Firebird - this is simply asking for trouble, so we always remove it.
# rpm -e ODBCunixFedora Core

# rpm -e unixODBC unixODBC_kdeRed Hat Enterprise

# rpm -q -a | grep -i odbclook for any ODBC pkgs
Some versions of Red Hat Enterprise Linux preinstalled on Dell systems have packages installed more than once, and it requires the use of the --allmatches parameter to actually remove them (we don't know why):

# rpm -q -a | grep -i odbc

# rpm -e unixODBC mysql-connector-odbc
error: "unixODBC" specifies multiple packages

# rpm -e --allmatches unixODBC mysql-connector-odbc
We also normally remove the krb5-workstation package: this contains Kerberized versions of the FTP client, something that provides no value to the SB but gives more warning messages during use. After removing this package, the "regular" FTP client will continue to be used.
Remove machine name from localhost
As configured by the installer, the machine's hostname is added to the /etc/hosts file with an IP address of this is localhost, and is designed to make connect-to-self connections faster by bypassing the lower part of the IP stack.
This, however, has badly broken Evolution in the past due to a curious interaction with the middle-tier machine. We observed that Firebird would provide the requesting middle-tier machine with its own IP address in localhost format, expecting the Windows machine to call on that IP, but Windows would just call itself.
/etc/hosts - the old way evodb localhost localhost.localdomain
/etc/hosts - the correct way     localhost evodb
This was all tracked down several Evolution releases ago, and we don't know whether it's still required or not, but it was such an awful mess to debug that we don't take any chances.
Disable SELinux
The SELinux facility ought to be disabled during the initial installation process, or else (at best) the logs will get filled with annoying messages, and it may well interfere with Firebird operation.
Disabling SELinux is done by editing the GRUB boot-time file by adding selinux=0 to the kernel line:
title Fedora Core (2.6.15-1.2054_FC5smp)
    root (hd0,0)
    kernel /vmlinuz-2.6.15-1.2054_FC5smp ro root=/dev/VolGroup00/LogVol00 selinux=0
    initrd /initrd-2.6.15-1.2054_FC5smp.img
This takes effect upon the next reboot.
Disable IPTables
IPTables is the built-in Linux firewall, and though it's much easier to configure than SELinux (and not unreasonably difficult to set up with Evolution), but this is also more trouble than it's worth in most cases. As with the previous section, the security of the system depends on external factors such as the enterprise firewall.
If IPTables is installed, it should be stopped and marked to not restart at boot time.
# service iptables stop
# chkconfig --del iptables
Those who are comfortable with iptables and wish to use them need to add a rule allowing inbound 3050/tcp from the package servers and the machine running ADR; this is the Firebird client/server protocol.
Verify/Update Timezone Rules
In Spring 2007, the new rules for Daylight Savings Time took effect, and this required teaching computer systems all over the planet about those rules. Fedora Core 6 already ships with the updated rules, but it's wise to check to be sure. By dumping the existing timezone rules, one can look for the time changes happening on March 11 and November 4th:
# zdump -v /etc/localtime | grep 2007
/etc/localtime  Sun Mar 11 09:59:59 2007 UTC = Sun Mar 11 01:59:59 2007 PST isdst=0 gmtoff=-28800
/etc/localtime  Sun Mar 11 10:00:00 2007 UTC = Sun Mar 11 03:00:00 2007 PDT isdst=1 gmtoff=-25200
/etc/localtime  Sun Nov  4 08:59:59 2007 UTC = Sun Nov  4 01:59:59 2007 PDT isdst=1 gmtoff=-25200
/etc/localtime  Sun Nov  4 09:00:00 2007 UTC = Sun Nov  4 01:00:00 2007 PST isdst=0 gmtoff=-28800
If the first two lines reflect April 1 — the old rules — then it will be necessary to update your system. We've found these instructions to be pretty helpful.

This Evo Tip is not endorsed, approved, or supported by iSystems, LLC.

First published: 2006/09/23, but updated frequently