Does this site look plain?

This site uses advanced css techniques

Stopping and starting of Evolution services is a routine part of running an Evolution instance, and this Evo Tip means to spell out some best practices for managing them.

Table of Contents

These practices are driven by a desire to avoid a poor experience for the Evolution users who this gear is here to support, all based on experience in multiple environments.

Note: This Tip is intended only for IT staff who actively administer an Evolution environment: if you are a service bureau hosted on any SaaS environment, Asure staff handles this for you.


Evo Management Console

Services are typically managed by the Evolution Management Console, a service that itself is managed with a web browser. Many configuration and monitoring operations are available, but we're looking at only a few of them. This is not a comprehensive guide to the Evo Management Console.

We mainly focus on the Service Control page, available from the Maintenance tab on the left:

Evo MC services page; 6 services shown

All services are listed, including their running status, and with obvious start and stop options.

However: Never use the [Restart] option, ever, for any reason.

Though conceptually a restart should be the same as a Stop followed by a Start, the Windows services API provides an explicit "Restart" interface that Evolution appears not to implement correctly. Sometimes a restart will work, other times it won't, and failure is not immediately apparent to the administrator (the service will run, but Evolution won't work correctly).

Services control page, but restart marked with danger

Many, many years of experience have made it clear: never, ever use [Restart]. We have requested that this option be removed from the Evo Mgmt Console interface to avoid confusion.

Stopping Evo Services

Stopping services is necessary before a routine update as well as routine Windows maintenance, and this is straightforward by just clicking the [Stop] on All Services in the upper left of the right pane:

Click [Stop] in All Services

When clicking [Stop], the Evo Management Console sends a request to the Deployment Manager service on every machine asking it to stop all the Evo services, and it waits for all to stop before returning.

This process can take some time, and occassionally a service will get stuck while stopping, needing a bit of help. This is particularly the case with Remote Relay, which often takes quite a long time due to what I believe is a bug.

In this case, you can manually kill Remote Relay via the Windows Task Manager; this must be done from the machine hosting the service itself (and some Evo instances have more than one Remote Relay, each installed on a different middle tier Windows machine):

Manually kill Remote Relay in task manager

This is common enough that when I'm stopping all services while doing customer maintenance, I immediately go to the task manager and kill RR, not waiting to see if it will stop on its own. On the classic SaaS platform, the orchestration for pod management kills Remote Relay as part of the process as well.

We don't know why Remote Relay hangs in spite of a fair effort of research.

Also note that the Evo MC occasionally misrepresents service status: even after all services have genuinely stopped, it may still show them "running" with the green check. Just re-click the "Service Control" tab on the left to refresh the status.

Starting Evo Services

Unlike stopping, which you can do in one shot, starting is a bit more complicated due to the requirement that services all be started in a particular order.

Important: the order of starting Evolution services must be:

  1. [Start] Request Broker
  2. [Start] All Request Processors
  3. [Start] All other services
Slow RB Startup

When the Request Broker starts, it scans the entirety of the Task Queue directory (often named C:\EvoQueue\, and for bureaus with many current and recent tasks, this can take a long time (I've seen > 5 minutes before!).

The broker is not ready until it's fully scanned this directory, and there's nothing you can really do to hurry it along while at the Evo Mgmt Console screen.

The most common cause of very long start times has been a large number of scheduled tasks for Webapps Middleware that phones home employee census data to the Kronos timeclock mothership. When scheduled to run for many customers at high repeat intervals, this can create an enormous task queue directory taking many minutes to scan.

In this case, I have some tooling that can help this issue by purging obsolete Webapps tasks from the queue at regular intervals: it has made an enormous difference in startup time for a number of my customers.

Contact me to see if this might be for you.

It appears that the Request Broker simply needs to get fully started and settled into place before the other services start and phone home. If this start order is not respected, I believe that the Broker and Request Processors get out of sync with each other and fail to communicate properly.

When this out-of-sync thing happens, Evolution appears to be running properly from the view of the Evo Mgmt Console and the system administrator, but Evolution users see a locked up environment.

This is most painful when the admin performs maintenance in the off hours, but Evo users don't trip across it until business hours the following morning. A full and proper (and interruptive) restart is required to fix this.

We have no real sense for what's the root cause of this issue, but it happens often enough (let's say 25% of the time), that as a matter of practice I always start in the proper order. Classic SaaS pod orchestration enforces this start order also.

A note about the third step: starting all the other services—Remote Relay, API Adapter, and the ADR Service— you can use the [Start] in "All Services". This button is smart enough to not touch services that are already running

Start order shown on EMC screen

Some customers, when doing their own maintenance, use the individual [Start] buttons on each service and start each one in order, from top to bottom. This works reliably also.


Restarting Evo Services

"Restart" versus "Reboot"

Terminology matters, especially when asking your IT staff to kick the servers:

Restarting services is done in the Evolution Management Console and does not change the state of Windows. This is usually only a brief interruption.

Rebooting Windows restarts the entire operating system and is much more invasive and takes longer. Sometimes this is actually required, but usualy is not.

Do not use these terms interchangeably.

Sometimes to clear up an Evolution stuck condition, you decide (or are told by Support) to restart Evolution services. As noted above, never ever click the [Restart] button in the Evo Mgmt Console; it's simply not reliable and should never be used.

Therefore, when one needs to restart all Evo services, you perform a proper stop followed by a proper start as noted in the previous sections, but the steps can be summarized as:


Rebooting Middle Tier Servers

The above scenarios—starting, stopping, and restarting—all presume that the underlying Windows machines stay running the whole time, but all systems undergo maintenance such as applying Windows Updates. This involves rebooting the systems in question, and special procedures must be employed to make sure Evo starts properly.

Even if Evolution is fully stopped properly prior to rebooting, the services all come up in their own random order: this is essentially a [Start] All Services, and it often leads to Evo hangups requiring a proper restart.

There are two approaches to handle this:

Easy but unreliable
In the easy method, you stop Evolution in the usual way, then reboot the various middle tier machines as needed, allowing Evolution to start in its own random order.
Once all machines are back up, you'll stop all services, then start them in the proper manner.
The downside of this is that once Evo is running after reboot, it may well start scheduled tasks (including catching up on tasks missed during the maintenance). Doing a proper restart interrupts all of these tasks.
Evolution is robust enough that killing these tasks won't generally cause data loss, but the tasks that didn't run might be important, and payroll staff will have to re-run them in the morning; sometimes these tasks (such as the "List of Liabilities for Period" report, or pretty much any Audit Report) take long enough that it's interruptive to payroll operations to run them during the day. We want to avoid that.
More work but more reliable
In order to prevent these potential scheduled tasks—which you can't predict by looking at the Evo Mgmt Console—from starting after a reboot, the safe approach is to use the Windows control panel "Services" applet to mark the "Evolution Request Broker" service for manual start.
Caution for Updates

If activating a new Evo version in the Evo Mgmt Console, always activate the version before setting the RB startup mode to Manual.

The very act of activating a new version in the Evo MC sets the RB Startup to Auto, so setting Startup=Manual has to be done after to avoid having all of Evolution start after the next reboot.

You'll typically do this after a proper stop of Evolution tasks just before rebooting the machine hosting the RB. This done, you can reboot this or any Evolution middle tier servers and handle all the maintenance required (including multiple server reboots).
Though the other Evo services such as the Request Processors, Remote Relay, and the rest will all autostart, the fact that the Request Broker (the "traffic cop") is down means that Evo is down, and none of the tasks run.
Once all of the Windows machines are back running, we get Evolution started in this order:
  • Log into the Evo Mgmt Console as usual
  • [Stop] All Services in the usual way (though the RB will be down already)
  • Start services in the usual order (RB; RPs, the rest)
  • Mark the "Evolution Request Broker" service as Automatic Start
This mechanism insures that Evolution does not start in the possibly-random state and fire off tasks that will get killed.
Your Evolution users will thank you for this.

For those of a scripting bent, it's easy to automate some of this with PowerShell: I've been using a Set-EvoRBSTartup.ps1 script to manage the startup state when performing maintenance at my customers.

First published: 2018/10/05