The Core Technologies Blog
Our Software // Windows Services // 24×7 Operation
Tod Daniels of d’innovative reported a strange problem. His JScript file (run with Microsoft’s CSCRIPT.EXE) started fine from his desktop but failed when run as a windows service with AlwaysUp.
By troubleshooting the situation from the command line, Tod was able to narrow the problem down to a problem when reading from the registry. Specifically:
regedit /e F:\test1.reg “HKEY_LOCAL_MACHINE\Software\Broadcom\BACS” run outside AlwaysUp produces a file containing the registry export.
regedit /e F:\test2.reg “HKEY_LOCAL_MACHINE\Software\Broadcom\BACS” run inside an AlwaysUp CMD session does not produce a file.
Both commands were run in the same user account, so why the different results?
The discrepancy is due to the different views of the registry presented to 32-bit and 64-bit applications.
Most modern versions of Windows are 64-bit. All the major applications and supporting DLLs distributed with the OS are 64-bit. To ensure that older 32-bit applications continue to run fine on these new operating systems, Microsoft engages in some “creative trickery”:
32-bit applications see a 32-bit version of the System32 Folder
Even though the Windows System32 folder (typically, C:\Windows\System32) is stocked with 64-bit applications, a 32-bit application has that folder “mapped” to a counterpart (C:\Windows\System32\Wow64) filled with 32-bit versions instead. So when a 32-bit application runs the “DIR” or “REGEDIT” commands, it is actually invoking the 32-bit version in the Wow64 folder. This silent mapping ensures compatibility when a 32-bit application invokes one of those standard Windows utilities.
AlwaysUp is a 32-bit application and is constrained by this behavior. When we’re troubleshooting, the command prompt is launching the 32-bit version of regedit.
64-bit applications work with a (slightly) different view of the Registry
In the 64-bit operating systems, some registry keys actually have a 32-bit version and a 64-bit version! One such key is HKLM\Software. 32-bit applications can write to this key normally, however the changes show up under HKLM\Software\Wow64 instead. A 64-bit application can see both versions of the keys and can choose which version to access.
Now Tod is using Windows Server 2012 R2 which is 64-bit. Our “a-ha” moment came when we noticed that this key exists:
but the corresponding 32-bit key does not:
This discrepancy means that 64-bit applications can access HKEY_LOCAL_MACHINE\Software\Broadcom\BACS while 32-bit applications cannot see a registry entry with that same name.
Tod was able to start the 64-bit version of regedit from AlwaysUp by exploiting another bit of Microsoft trickery — the Sysnative folder. This is the full path that enabled regedit to find the Broadcom key:
C:\Windows\Sysnative\regedit /e F:\test1.reg “HKEY_LOCAL_MACHINE\Software\Broadcom\BACS”
Ultimately he was able to launch the 64-bit version of CSCRIPT from the same path and his application is now functioning as expected as a Windows Service!
Windows Services promise 24/7 operation and it can be very frustrating when they don’t start as expected! Here are the top five reasons why a service may fail to launch when your server boots:
Your Windows Service isn’t set to Start Automatically
A window service can be configured NOT to start when your computer reboots. Indeed, a service can be set to startup only on demand, or entirely disabled so it cannot run at all.
To check that your service is properly configured:
Start the Services Control Panel application.
Find your service in the list and double-click it to show its properties.
Ensure that the Startup type field is set to Automatic.
Note that Automatic (Delayed Start), where your service starts 1-2 minutes after all Automatic services have been launched, may also be acceptable.
A Dependent Service Failed to Start
Some windows services depend on other services to support their work. Windows enforces these dependencies when booting. For example, suppose there are two services, A and B, both set to start automatically at boot. If A depends on B, then Windows will launch service B before starting service A. If service B fails to start, Windows will not start service A.
Open your service in services.msc and switch to the Dependencies tab to see if your service relies on others. If so, the problem may be with one of those dependent services.
The Service Account’s Password has Changed
If your service runs in a specific user account, was the password for that account changed recently? If so, you should edit your service and enter the new password (on the Log On tab):
There is a Problem with the Domain Account’s Group Policy
If your service is running in a domain account, ensure that the domain account’s group policy has the “Log on as a service” right. As described in this article, the service may run fine at first but suddenly stop working when the local policy (which has the right) is overwritten by the global policy (which does NOT have the right).
Look for the telltale “The service did not start due to a login failure” message in the Event Logs to identify this situation.
There is another problem, reported in the Event Logs
Your service may be shutting down because it has encountered a fatal error. The devil will be in the details so be sure to scour the Event Logs for any helpful messages from your service.
Your service may also maintain its own log files, separate from what is available in the Event Viewer. Please be sure to consult those as well!
Why Windows Services Should Avoid Showing Popups/Dialog Boxes
Unlike a regular application (like Microsoft Word) that you launch from a desktop icon and actively work with to accomplish a specific task, a Windows Service is designed to start when your computer boots and run entirely in the background — even if you never log on.
Yet despite this design philosophy, there is no restriction that actually prevents a Windows Service from displaying windows and trying to interact with users logged on to the PC. Indeed, such “Interactive Services” were encouraged in Windows XP and Windows Server 2003, but Microsoft changed their tune in Windows Vista. In our opinion, a couple of glaring problems caused the folks in Redmond to reverse course and actively discourage Windows Services from trying to get the desktop user’s attention:
What happens when no one is logged on?
It is unclear what should happen when a Windows Service shows a prompt and no user is there to answer it. Holding up the action while waiting for someone to log on and click a button can be problematic for tasks designed to perform important tasks, 24/7.
What happens when several users are logged on?
When a service throws up a prompt, which of the multiple users logged on should be allowed to respond to that window? One? All?
These thorny issues should discourage anyone from writing an interactive service today!
But what is you have inherited a legacy service implemented under the more permissive Windows XP or Server 2003 rules? What if commercial realities force you to run an interactive application as a windows service with a service wrapper like AlwaysUp? How do you keep those services running without interruption, getting around annoying prompts that would otherwise stall an important process?
Three Ways to Automatically Dismiss Dialog Boxes and Popups
Write a Custom AutoIt Script
AutoIt is a free scripting language designed for automating the Windows GUI. With this powerful tool, you can easily write scripts to activate windows, click buttons, check boxes, enter text and much more.
For example, this AutoIt script clicks the “OK” button on a window named “OCR Error”:
; Try to bring the window to the top, to prepare it for input.
; Click the OK button in the window.
ControlClick("OCR Error", "OK", "");
Note that an AutoIt script can be compiled into an executable to run on any machine — even those without AutoIt installed.
Buy Buzof, the “Annoying Windows Eliminator”
Buzof is a commercial utility that makes it easy for non-programmers to automatically answer Windows prompts. Its point and click interface enables you to “train” the software to take the appropriate action on those annoying popups. Here is what Buzof looks like when configured to watch for four popups, clicking the “Yes” or “No” buttons:
Note that Buzof is really meant for clicking buttons and other simple actions. More complex situations, such as checking a box and subsequently clicking a button or typing in a user name, are beyond its abilities.
Develop your own Windows GUI Utility
If you are an experienced Windows programmer, then it won’t be too difficult to create your own solution. This may be a good investment of your time if using AutoIt or purchasing Buzof are not viable options.
How to Run your Automated Solution on the Isolated Session 0 Desktop
Once you have settled on a method to automatically dispatch the dialog boxes and popups, you must now set it up to run on the desktop where Windows Services run — in Session 0. There are a couple of ways to achieve this:
Run your Utility as a Windows Service
We recommend using our AlwaysUp application to run your chosen utility as a Windows Service. If you are using an AutoIt script, set it up to run periodically, perhaps once every minute. For Buzof, which should be started once and kept running continuously, any service wrapper (such as Microsoft’s Srvany) will perform the basic job.
Run your Utility with the Task Scheduler
Setup a basic scheduled task to start your solution. It will run entirely in Session 0.
Note that you can also launch your utility manually in Session 0 using Microsoft’s PsExec command line tool.
Best of luck keeping your Windows Services operating 24×7, without annoying interruptions!
Backups are an essential part of any professional system. Schedule them regularly or risk losing valuable data (and precious time) when a component fails.
But backing up a live system can often be tricky. The backup software must be “lock” each file to capture a consistent snapshot, and that exclusive access, though temporary, can cause another application to panic and throw up its hands in failure. Or the reverse can happen — an application can lock a file and prevent the backup software from capturing it. And what good is a backup if it doesn’t capture all your important data?
So it’s always best to have a “quiet” system when performing a backup. Close all non-essential applications in advance. But what about Windows Services? Unlike regular desktop programs, those shouldn’t be stopped indefinitely.
This is the dilemma faced by one of our customers. Backups run every weekend constantly fail because his windows service prevents the backup from accessing key files. He tried stopping the service before he left work on Friday evening and restarting it on Monday morning, but users complained loudly about the extended downtime.
The obvious solution is to have a much smaller downtime window for the windows service — only for the duration of the backup. Here is how to do so in an automated fashion, with a couple of well-timed scheduled tasks that leverage the useful NET command.
Part 1: Create a Scheduled Task to Stop your Windows Service before the Backup Starts
Start the Windows Task Scheduler by running taskschd.msc from a command prompt. (You can also find it in Control Panel by searching for “Schedule tasks”).
In the window that comes up, click Create Basic Task… on the right to launch the Create Basic Task Wizard:
Enter a name for your new task. Something like Stop service before backup starts should work nicely. Click Next > when you are done.
In the next couple of steps, we’ll specify when to stop the windows service. Since our backup occurs every Saturday at 10 PM, we’ll setup the task to shut down the service a minute before, at 9:59 PM:
Click Next > to proceed.
On the subsequent screen, select Start a program and click Next >:
Here is where we must specify the program to shut down the service. We’ll use the NET STOP command.
- Enter NET.EXE in the Program/Script line.
- And in the arguments field, enter
where <Service-Name> is the name of the service. We have used “Spooler” in this tutorial but you will obviously use your own service instead. Be sure to enclose that name in quotes if it contains a space!
Click Next > to continue.
We’re almost done! You should see a screen confirming your settings. Click Finish to record this new scheduled task.
Part 2: Create a Scheduled Task to Restart your Windows Service after the Backup Completes
Basically, repeat Part 1 but with the following changes:
When naming the task in Step 3, call it something like Restart service after the backup is done instead.
In Step 4, set the time so that the task will run AFTER the backup completes.
In Step 6, run the NET START command instead. That is, set the arguments field to
Here is what the ending summary screen should look like once you are done:
And that’s it. Since implementing this tactic a few weeks ago, our customer’s backups have been problem-free. We’re confident that the same solution will work for you!
We’re big fans of Oracle’s VirtualBox software. It’s powerful and free, and it enables our small company to efficiently test and validate our software on various versions of Windows without having to invest in costly, dedicated hardware.
However, we noticed a strange problem with VirtualBox last week. After following our tutorial to setup a VirtualBox Virtual Machine (VM) as a Windows Service with AlwaysUp, we noticed that even though the machine was working normally, the VirtualBox Manager application incorrectly listed the VM as “Powered Off”. But it had been on for more than 10 days!
And it wasn’t just the VirtualBox Manager. Even the powerful VBoxManage command line utility thought that our VM wasn’t running:
However, when we ran a command prompt in Session 0, VBoxManage saw the VM there:
So it seems that the VirtualBox tools can only see virtual machines running in the same session that they are in.
Therefore, we simply can’t rely on the state shown in the VirtualBox Manager application when running our VM in the background (in Session 0) with AlwaysUp.
Customer Theodore Bogucki recently reported strange behavior when running Dropbox as a windows service with AlwaysUp:
If I run Dropbox as a normal application, the CPU will run at about 10%. Following the directions to set up Dropbox in AlwaysUp as a service, I will get Dropbox running at 100% CPU and the file upload is extremely slow.
I am using the latest version of Dropbox and the latest version of AlwaysUp on a Windows 2012 R2 server.
What can be causing Dropbox to run amok like this when running as a windows service?
After some back and forth to investigate, Theodore determined that his antivirus program (Symantec Endpoint Protection) was conflicting with Dropbox — but only when running as a service! He said:
The Dropbox folders currently stores SQL backup files from my SQL server. I was able to fix the issue by adding an exclusion in SEP for the file extension used for the SQL backup files. I just as easily could have specified the folder to exclude.
Once this was done I stopped Dropbox on the desktop and started it using AlwaysUp and the CPU only spiked for minute or so while Dropbox started up and then settled down to its normal level. I also observed that network activity was the same as when Dropbox was running on the desktop.
If you are experiencing a similar problem, here are the instructions for excluding files and folders from scans in Symantec Endpoint Protection. A simple Google search should turn up the same information for other popular antivirus packages.
Finally, thanks to Theodore for bringing this problem (and the solution) to our attention!
The mystery: “The service did not start due to a login failure”
One of our customers reported a very strange problem last week. After about a day of running flawlessly, their windows service would suddenly fail to start after a reboot. The error reported by the Event Viewer hinted at a problem with the service user’s account:
Trying to start the service directly from the Windows Services Control Panel application produced the same unsatisfying result:
The service account’s password had not changed, and the user had no problem logging into the server interactively. Why was the windows service failing to login?
Luckily we were able to get the service going again by re-entering the user’s password:
When doing so, we noticed that the “Log on as a service” right had to be granted again. Very suspicious…
But a mere 24 hours later, the problem resurfaced! Once again, the service failed to start after a reboot.
The problem: Group Policy overwriting Local Policy
The message about the “Log on as a service” right lead us to the root of the problem.
Entering the password in services.msc updated the user’s rights in the machine’s Local Group Policy — a collection of settings that define how the system will behave for the PC’s users. However, since the user and server were part of a domain, those local settings were periodically overwritten by the domain’s group policy, which had not been updated with the new permission. And because the necessary permission “disappeared” on the machine, the service failed the next time it tried to start.
The solution: Modify the Domain Group Policy
To fix the problem, we must update the domain group policy and explicitly give the service user the “Log on as a service” right. To do so:
Start the Group Policy Management application.
Open Control Panel, navigate to System and Security > Administrative Tools, and double-click Group Policy Management on the left.
(Note: Don’t search for “group” in Control Panel. That will lead you to the “Edit group policy” link, which opens the local group policy!)
Find your default domain policy on the left. Right-click it and select Edit to bring up the Group Policy Management Editor window.
On the left, navigate to Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > User Rights Assignment and select
the Log on as a service entry on the right.
Double-click Log on as a service to bring up its Properties window.
Add the user running your windows service to the list and click OK to record the change.
Next time your domain policy is copied to your server, it will bring along the Log on as a service right for the user. You shouldn’t encounter the “logon failure” error again!
A closing note for the folks at Microsoft: A better error message please!
Instead of reporting the generic “logon failure”, why not be more precise and say something like “The user doesn’t have the necessary rights to start the service”? You could provide even more guidance by listing the missing rights.
The bottom line is a that a helpful error message highlighting the true problem would have led us straight to the solution and avoided a few anxious days for us and our client.
Start Software is an award-winning software development and consulting business with offices in the UK and Australia. They create easy-to-use software systems that their clients enjoy using and which deliver real benefits. The Start Software team are specialists in Alpha Anywhere, Alpha Five and Mobile Data Studio developments. Alpha Tracker – the latest addition to the Tracker family – is a web browser-based version of Tracker for asbestos surveyors, asbestos consultants and asbestos consultancy businesses of any size.
Start Software has been using AlwaysUp regularly since 2007.
Why does Start Software use AlwaysUp?
We have created a variety of software systems using Alpha Five and Alpha Anywhere from Alpha Software and Visual Basic from Microsoft. Many of these systems need to be “always up” – operating like 24x7x365 windows services. The AlwaysUp product enables us to deploy these applications with confidence that the services they provide will always be available to our customers.
How are you using AlwaysUp today?
We install AlwaysUp along with our software as a matter of course. We have more than 50 clients using AlwaysUp on Windows Server 2008 and Windows Server 2012 on a daily basis all around the world.
What are your favorite features of the software?
AlwaysUp is easy to install and configure, and has flexible options for the automatic restarting of applications (services) under a variety of conditions. Technical support provided by Core Technologies is also excellent and they are a joy to deal with.
Any other comments?
Our Tracker and Alpha Tracker systems have been used by asbestos consultancies to capture data about more than 25 million samples in buildings worldwide and so, indirectly, AlwaysUp is helping to manage this deadly substance across the globe.
Do you want to run a script or application every minute? And have you decided that creating an AlwaysUp Windows Service will be better than using the built-in Windows Task Scheduler? If yes, here are the key settings to specify with AlwaysUp.
Wait One Minute Before Restarting
By default, AlwaysUp will immediately restart your application whenever it exits. To have AlwaysUp “pause” for a minute before restarting, make the following changes on the Restart tab:
Avoid Verbose Event Logging on each Run/Restart
AlwaysUp is designed to record detailed information whenever your application exits and has to be restarted. This is often useful information for applications that should run continuously but the logging can become overwhelming when your application is stopping and restarting every minute!
Check the Minimize event logging as the application stops & starts on the Restart tab to avoid extra chatter as your application/script cycles each minute.
If you are interested in email notifications, be sure NOT to check the Whenever the application stops and has to be restarted box on the Email tab! If you do, expect to be inundated with emails as your application goes up and down every minute.
Recently, a few of our customers running Dropbox as a windows service with AlwaysUp have reported failures after manually updating to the latest version of Dropbox. These are happening because Dropbox changes the location of its executable file during the update. Let’s dig into the details and see how to fix the problem.
The Old Dropbox Location
Prior to version 3.5.58, released in April 2015, Dropbox stored its application executable file (Dropbox.exe) in a user-specific folder. If you installed Dropbox in the Administrator account, you could find the file in C:\Users\Administrator\AppData\Roaming\Dropbox\bin:
The New Dropbox Location
If you install a fresh copy of version 3.5.58 or later, you will notice that Dropbox.exe now lives underneath the Program Files folder, in C:\Program Files (x86)\Dropbox\Client (or C:\Program Files\Dropbox\Client on 32-bit versions of Windows):
We have already noted the new location in our step-by-step tutorial to guide future customers, but the story is more complicated for existing installations…
Upgrading Dropbox Changes its Location (and Breaks your AlwaysUp Configuration)
Unfortunately the impact of this change to the file locations isn’t limited to new installations. Manually updating an existing Dropbox installation will cause the files to be moved to the new folder. This isn’t a problem for Dropbox itself, but since AlwaysUp is blissfully unaware of the change, it will continue to run Dropbox.exe from its old location (as you originally specified in the Application field on the General tab):
Because the executable is no longer present there, AlwaysUp will fail to start Dropbox and report errors like this:
The Fix: Set the New Dropbox Location in AlwaysUp
Fortunately normal operation can be restored quite easily. Simply edit Dropbox in AlwaysUp and update the Application field on the General tab to specify the new path to the Dropbox executable (C:\Program Files (x86)\Dropbox\Client\Dropbox.exe):
Save your settings and AlwaysUp should be able to start Dropbox normally again!