The Core Technologies Blog
Our Software // Windows Services // 24×7 Operation
The next version of Windows, dubbed Windows 10 is expected to be released in the second half of 2015. On October 1, Microsoft provided software makers with an early look at the new OS, so that we can test and certify our applications in advance of the launch. And like a kid in a candy store, we downloaded and eagerly installed the Windows 10 Technical Preview on a VirtualBox VM. The most noticeable change? The return of the Start menu! Finally, Microsoft has come to its senses.
Next, it was on to testing our software. Service Protector, which monitors and automatically restarts failing windows services, passed with flying colors. There with no issues whatsoever. Here it is happily monitoring the Print Spooler service:
However, things did not go as smoothly for AlwaysUp, our popular utility that runs any application as a windows service. We set up Notepad.exe as a service but quickly ran into trouble starting it:
After a couple of hours of debugging the code, it turns out that the problem is related to to the little-used user reboot notification feature recently introduced in Windows 8 & Server 2012. To avoid getting too technical, let’s just say that the new capability is simply not accepted by Windows 10. Using it causes AlwaysUp to fail to transition the service into the running state and remain in the starting state. And since no controls are accepted in the starting state, all the action buttons are grayed out and the service will remain stuck starting forever! The only way to stop the service is to disable all recovery actions for the service and forcibly terminate AlwaysUpService.exe from the Task Manager. A real mess!
We have reported the problem to Microsoft and hopefully they will resolve it before the official release of Windows 10 next year. If not, a change to our code to eliminate the user reboot notification feature will also fix the problem, so not to worry. We’ll stay on top of it
A windows service, designed to run “headless” and unattended in the background, cannot easily employ conventional popup windows to report its activities as a user may not even be logged on. Instead, a service is encouraged to send important communication to the Windows Event Log – an administrative utility that collects and stores messages and events. Once recorded, these messages can be very helpful in troubleshooting problems, for example when a service stops unexpectedly or when it fails to start at all.
Viewing Events from Windows Services
Use Microsoft’s Event Viewer to see messages written to the Event Log. Start the application by clicking on the Start button and typing in Event Viewer, or from the Control Panel (search for it by name). The somewhat cluttered window should come up after a few seconds:
The left hand side shows a tree grouping the various logs captured on your machine. The events from Windows Services (and other applications running on your PC) are filed under Windows Logs > Application. Navigate to that section to load the events in the center of the window, with the entire list in the top and details of the highlighted event underneath:
Messages from your windows service will have the display name of the service in the Source column.
Important Components of an Event
The Event Viewer shows over 10 pieces of information associated with each event, including:
- Level – How important is this event?
Each event is classified into one of three categories:
Information: An informative yet unimportant event. You will probably see a lot of these, and they can be safely ignored unless you are digging into a specific issue from an application or service.
Warning: A moderately important event. These don’t necessarily signify a failure, and your software will probably limp along, but they should be reviewed regularly to see if anything mentioned can be resolved.
Error: Indicates a critical problem or failure that may deserve your immediate attention!
- Date and Time – When did this event occur?
- Source – Which application reported this event?
As mentioned before, an event written by a Windows Service will contain the service’s display name as the Source.
- Description – Which happened?
The full description shown prominently in the lower pane will (hopefully) provide the relevant details of the event.
For example, this information event is from the Interactive Services detection service (“UI0Detect”) reporting that Notepad is showing itself in Session 0:
Viewing Events about Windows Services
While the Application log keeps track of events from a running service, the Windows Logs > System area records when services are started, stopped, crash or fail to start. Look for events with the Source set to Service Control Manager (SCM). For example, here is the SCM telling us that the Windows Print Spooler service has crashed:
Viewing Events from AlwaysUp and Service Protector
Both AlwaysUp and Service Protector write messages to the Application section of the event logs (Windows Logs > Application).
For AlwaysUp, events from your application named “My Application” will be logged with Source set to My Application (managed by AlwaysUpService). The Event Log Messages Page lists and explains the events reported.
For Service Protector, events related to your service named “MyService” will have a Source of ServiceProtector: MyService.
And for both applications, events related to the starting and stopping of the underlying services themselves appear in the Windows Logs > System section. Look there if you have a problem with AlwaysUp itself failing to start at boot.
You need to start your application at boot, even if no one logs on
A regular application will only run after a user has logged in and started it. Not so for Windows Services, which are automatically started by the operating system whenever the computer reboots and run happily without anyone having to do anything. Services are the true set-it-and-forget-it approach.
Your application must be available 24/7
The automatic, extensible failure protection system built into AlwaysUp will help to minimize the downtime for your legacy application – even if it is buggy, leaks memory or stops working for no good reason!
You don’t have the time, resources or access to the source code to re-write the application as a Windows Service
The boss needs your software to be a windows service to win the next contract, but your development team is swamped and no one wants to mess around with the code anyway. Use AlwaysUp to “wrap” your application to operate as a windows service so you can satisfy the client and keep your boss happy, all in a day’s work.
You are managing an unreliable application that keeps crashing
Support calls for buggy applications have ruined too many evenings and nights! AlwaysUp automatically restarts unstable applications whenever they stop for any reason, so enjoy an uninterrupted dinner with your family for a change!
You prefer to run your program in the background, away from inexperienced users
Mission-critical applications visible on the desktop run the risk of being accidentally shutdown by a user or closed when he logs off. However, a windows service runs on a “hidden” desktop where it is well insulated from those using the computer.
We are happy to announce the availability of the AlwaysUp Troubleshooter – our online, step-by-step guide to help you identify and resolve the most common problems encountered when configuring your application as a windows service.
The troubleshooter is very easy to use and is packed with illustrative screenshots and helpful tips to quickly guide you to a successful outcome. However, if your issue is complex and demands more than can be achieved in that format, the interview ends with a summary of what has been learned and a contact form that makes it easy to engage our support team for additional help.
So if you are having difficulty setting up your application, script or batch file as a windows service with AlwaysUp, click here to start troubleshooting now!.
Starting with Windows Vista in 2007 and continuing with every subsequent release, Microsoft has banned users from logging in to the first session created as Windows boots. This “Session 0 Isolation” has been accompanied by a steady erosion of the interactive capabilities of Session 0, rendering the Session 0 desktop virtually unrecognizable in the post-XP world. The prevailing wisdom in Redmond: Why support a proper working desktop in Session 0 when no one in their right mind is expected to use it?
Unfortunately those of us working with interactive applications and Windows Services in Session 0 don’t have the luxury of ignoring GUI-related issues. And users of the excellent AutoIt automation toolkit have to make special accommodations when creating scripts to click buttons and fill in forms in Session 0. Here is what we have learned while troubleshooting AutoIt with our customers:
WinActivate, WinWaitActive and WinActive Often Fail
Essential AutoIt functions like WinActivate, WinWaitActive and WinActive rely on the operating system to track the active window. Unfortunately it seems that the active window is not tracked in Session 0 unless a user is actively viewing the desktop! Therefore a script that takes some action only if a window has been activated may never do its work. In this sample code, WinActivate will always return 0 when run in an unattended Session 0 and the button will never be clicked:
; Activate the window and click the button.
if (WinActivate($windowName)) Then
; Click the button on the window.
ControlClick($windowName, $buttonName, "")
Our advice: Don’t depend on the Win* functions to find and activate a window. Simply operate on your window regardless of its active state. For example, restructure the code above to look like this:
; Try to activate the window if possible. May fail in Session 0.
; Click the button on the window.
ControlClick($windowName, $buttonName, "")
The Mouse Functions (MouseMove, MouseClick, etc.) Don’t Work
As with window activation, it seems that the mouse is not tracked when session 0 is unattended. Functions that move and manipulate the mouse can fail unexpectedly.
Our advice: Don’t use the mouse functions in your scripts.
AutoIt Scripts created with Au3Record Are Unreliable
The Au3Record utility bundled with AutoIt will easily record a set of actions for later replay. It captures mouse movements, mouse clicks, key presses and more. However the script created makes heavy use of the problematic activation and mouse related functions cited above, thus rendering Au3Record scripts virtually useless in Session 0.
Our advice: Don’t use scripts created by Au3Record in Session 0.
You can Rely on ControlSend and ControlClick in Session 0
Fortunately functions like ControlSend and ControlClick continue to work as expected in Session 0. Neither of them relies on the target window being active nor the mouse being in a specific position, so they sidestep trouble related to those areas on the isolated desktop.
Are you using AutoIt in the isolated Session 0? Have you run into other functionality that does not work as expected? Please let us know what you have found! Your feedback will be much appreciated.
If you are comfortable working with Windows Services from the command line, take a look at Microsoft’s TASKLIST utility. This often overlooked tool — available on all versions of Windows — can help you answer the following questions as you troubleshoot your Windows Services:
What is the PID Associated with a Running Service?
You can use the name of the service to look up the identifier of the process associated with it. For example, to find the PID of the Print Spooler service (named “Spooler”), use:
TASKLIST /SVC /FI "SERVICES eq Spooler"
Here is the result you can expect, with the PID listed in the central column:
What Windows Services are Being Hosted by a Specific Process?
If you are considering forcibly terminating a process, it may be wise to confirm that doing so will not unexpectedly kill related services! TASKLIST.EXE comes to the rescue by easily showing what services are running in a given process. For example, to see what services are behind the process with PID 1234, type:
TASKLIST /SVC /FI "PID eq 1234"
Here we can see that the svchost process with PID 1284 is managing five (!) core services:
Note that TASKLIST can interrogate remote machines as well. Simply add the /S /U and /P flags to the command line to specify the system, user name and password respectively.
The Problem: Dropbox Automatic Updates
Recently, a few customers reported “issues” with Dropbox automatically updating itself and causing havoc when running under AlwaysUp in their 24×7 environments. Apparently the auto-upgrade would restart Dropbox outside of AlwaysUp and that unmanaged instance would prevent AlwaysUp from starting a new copy under its control. The result was hundreds of Explorer windows open on the screen — a great experience!
Our first instinct was to turn off automatic updates, which are usually a bad idea in a controlled, 24×7 environment anyway. Unfortunately Dropbox does not allow that. Auto-updates are totally managed by Dropbox and there are no exposed options for adjusting the behavior.
The lack of controls thwarted our second brilliant idea as well — scheduling auto-upgrades for a specific time each day/week/month. Apparently changes can take place at any time, at the program’s discretion. Again, not suitable in a managed environment.
A polite email to the Dropbox support folks explaining the situation and requesting help & advice yielded a most unsatisfying generic response:
Thanks for writing in. While we'd love to answer every question we get,
we unfortunately can't respond to your inquiry due to a large volume of
support requests. Here are some resources for resolving the most common
Restore files or folders - www.dropbox.com/help/969
Reset your Dropbox password - www.dropbox.com/forgot
Reset/Disable two-step verification - www.dropbox.com/help/364
Learn about sharing files or folders - www.dropbox.com/help/category/Sharing
Learn about Dropbox's desktop app - www.dropbox.com/help/category/Desktop
Learn about Dropbox's mobile apps - www.dropbox.com/help/category/Mobile
For all other issues, please check out our Help Center - www.dropbox.com/help
We're sorry for the inconvenience,
The Dropbox Team
So time to do some reverse engineering!
The Investigation: How Dropbox Auto-Update Works
We installed an outdated version of Dropbox (2.47, thanks OldApps.com!) on our Windows Server 2012 machine, started it under AlwaysUp and patiently waited with Microsoft’s excellent Process Explorer open. It took about 8 minutes for the action to start…
First, Dropbox.exe launched dropbox-upgrade-2.8.2.exe, which it must have silently downloaded into the private cache folder (C:\Users\<UserName>\AppData\Roaming\Dropbox\.dropbox-cache).
- A few seconds later, the update program spawned a second copy of Dropbox.exe, passing the ominous /killeveryone flag.
Next, all instances of Dropbox subsequently shut down (including the one managed by AlwaysUp), and the updater launched a new copy of Dropbox before it too exited.
Notice the peculiar command line arguments.
This is all fine when Dropbox is running normally on your desktop, but it is a problem when Dropbox is being managed by AlwaysUp. Indeed, once AlwaysUp noticed that the Dropbox process it was managing was closed, it started up a second copy:
Now Dropbox abhors multiple instances, so after a few seconds the AlwaysUp-controlled instance exits — but not before throwing up an Explorer window browsing the Dropbox folder. AlwaysUp, persistent fellow that it is, detects that Dropbox has once again exited, starts it up again and the whole process repeats. It is easy to see that if left unchecked, we may end up with hundreds of redundant Explorer windows!
The Solution: Configuring AlwaysUp to Tolerate Automatic Dropbox Updates
To prevent the auto-update from triggering ten billion Explorer windows, we must have AlwaysUp close the instance of Dropbox spawned during the upgrade prior to starting its own copy. To do so:
Download close-application.bat, a DOS batch file that will close a given application. Save it to the AlwaysUp folder (C:\Program Files\AlwaysUp, or C:\Program Files (x86)\AlwaysUp)
Edit Dropbox in AlwaysUp
Click over to the Startup tab. Check the Run the following program/batch file box and enter the following command line:
“C:\Program Files\AlwaysUp\close-application.bat” Dropbox.exe
Check the Also run it whenever the application is restarted box as well.
This will cause AlwaysUp to close all running instances of Dropbox before starting the one that it will monitor.
And finally, to give Dropbox some time to complete its auto-update before AlwaysUp tries to fire it up again, click over to the Restart tab and:
- Check the Whenever the application stops, restart it box
- Check the Not immediately, but box
- Select the After option and enter 5 minutes in the adjacent field.
Click the Save >> button to record your settings.
That’s it! The next time Dropbox decides to automatically update itself, AlwaysUp will quickly bring it back under control — for continued 24×7 operation.
We are happy to report that AlwaysUp now integrates with the Windows power management system, to provide the ability to restart your applications when the host computer resumes from a low-power state (sleep or hibernation). This option is useful if your application being run as a Windows Service fails to properly handle the “time lapse” that occurs when a PC is put to sleep and later revived.
This new, easy to use functionality is available on the Monitor tab. Simply check the box highlighted below and you will be ready to go:
Windows Services are designed to start up at boot and run 24×7, but that framework is overkill for server software that must occasionally come alive to process specific events and go back to a waiting state. Service Triggers were introduced in Windows 7 to remedy the situation but at this point, only a handful of triggering events are supported. We look forward to more capabilities (and better documentation!) of this useful feature in future versions of Windows.
If you have a Windows Service that works with USB flash drives, you can use our free Service Trigger Editor utility to configure your service to start when a flash drive is inserted. To do so:
Download Service Trigger Editor from our web site. Note that it is a standalone executable, ServiceTriggerEditor.exe, so just save it in a new folder on your hard drive.
Run ServiceTriggerEditor.exe. You should see the standard security prompt if User Account Control (UAC) is enabled on your PC:
Service Trigger Editor is digitally signed by our company for your safety & security so please click Yes to proceed.
Once the utility’s main window comes up, listing all services installed on your PC, find and highlight the service you wish to modify. We have selected the SyncToy service (created by our AlwaysUp product) for this tutorial.
Select Trigger > Add… to summon the Add Trigger window. Adjust the settings to start the service when A specific device arrives (or is present at startup).
Next, click the Add… button on the right, enter USBSTOR\GenDisk in the window that comes up, and click OK to record that value.
We’re now done configuring this new trigger so click the Save >> button.
That’s it. Next time you insert a USB flash drive, your service will start. Enjoy!
While the useful NET.EXE utility is great for starting an stopping windows services, it cannot do much beyond that. Enter Microsoft’s SC.EXE – a versatile command-line utility built into Windows that can help you start, stop, restart or configure any Windows Service.
Type SC at a command prompt to see the extensive set of options available:
SC is a command line program used for communicating with the
Service Control Manager and services.
sc <server> [command] [service name] <option1> <option2>...
The option has the form "\\ServerName"
Further help on commands can be obtained by typing: "sc [command]"
query-----------Queries the status for a service, or
enumerates the status for types of services.
queryex---------Queries the extended status for a service, or
enumerates the status for types of services.
start-----------Starts a service.
pause-----------Sends a PAUSE control request to a service.
interrogate-----Sends an INTERROGATE control request to a service.
continue--------Sends a CONTINUE control request to a service.
stop------------Sends a STOP request to a service.
config----------Changes the configuration of a service (persistent).
description-----Changes the description of a service.
failure---------Changes the actions taken by a service upon failure.
failureflag-----Changes the failure actions flag of a service.
sidtype---------Changes the service SID type of a service.
privs-----------Changes the required privileges of a service.
qc--------------Queries the configuration information for a service.
qdescription----Queries the description for a service.
qfailure--------Queries the actions taken by a service upon failure.
qfailureflag----Queries the failure actions flag of a service.
qsidtype--------Queries the service SID type of a service.
qprivs----------Queries the required privileges of a service.
qtriggerinfo----Queries the trigger parameters of a service.
qpreferrednode--Queries the preferred NUMA node of a service.
delete----------Deletes a service (from the registry).
create----------Creates a service. (adds it to the registry).
control---------Sends a control to a service.
sdshow----------Displays a service's security descriptor.
sdset-----------Sets a service's security descriptor.
showsid---------Displays the service SID string corresponding to
an arbitrary name.
triggerinfo-----Configures the trigger parameters of a service.
preferrednode---Sets the preferred NUMA node of a service.
GetDisplayName--Gets the DisplayName for a service.
GetKeyName------Gets the ServiceKeyName for a service.
EnumDepend------Enumerates Service Dependencies.
The following commands don't require a service name:
sc <server> <command> <option>
boot------------(ok | bad) Indicates whether the last boot should
be saved as the last-known-good boot configuration
Lock------------Locks the Service Database
QueryLock-------Queries the LockStatus for the SCManager Database
Stopping/Starting a Service with SC
To stop a windows service from an elevated DOS prompt, run:
SC STOP <Service-Name>
where <Service-Name> is the name of the service. Be sure to enclose the name in quotes if it contains a space!
For example, to stop the Print Spooler service (named “Spooler”), run:
SC STOP Spooler
Notice that the SC command will simply make a request for the service to stop and return immediately, before the service has actually stopped. This is evidenced by the STOP_PENDING state (which means that the service is in the process of winding down) returned in the screenshot above. If you plan to use this command in a batch file, you may need to add a sleep/pause after calling SC to give the service some time to respond. (The NET.EXE command, which will wait/block until the service has completely stopped, may be a better choice in this respect.)
Similarly, to start a windows service, use:
SC START <Service-Name>
Again, the request will be made but SC will not wait for the service to complete its startup before returning.
Using SC to Check the Status of a Service
To discover the state of your service, run SC with the QUERYEX option:
SC QUERYEX <Service-Name>
Check on the Spooler service like this:
SC QUERYEX Spooler
If the service is running, SC will return the underlying process identifier (“PID”) which can be used to manipulate the service’s process. This is very handy information when a service is stuck or unresponsive and must be forcibly terminated!
Disabling a Service
The CONFIG option enables you to modify a service’s settings. If you wish to disable a naughty service, preventing anyone from starting it, type:
SC CONFIG <Service-Name> start= disabled
For example, this command disables the infamous Interactive Services Detection Service (named “UI0Detect”):
SC CONFIG UI0Detect start= disabled
Note that the space in between “start=” and “disabled” is required!
How to Create a New Service with SC
SC can be used to create a new service as well. Type “SC CREATE” to see the many settings that can be applied but at a minimum you must specify:
- the name of the service,
- the display name of the service (a more descriptive moniker),
- the full path to the executable hosting the service
For example, the following command creates a service called “MyService” with an executable located in “C:\MyService\MyService.exe”:
SC CREATE MyService binPath= “C:\MyService\MyService.exe” DisplayName= “My very cool service”
Once installed, you can work with the new service as normal in the Services application:
Note that only executables explicitly written to interface with the Windows Service Control Manager should be installed this way. While SC will happily accept a regular, non-service binary, you will receive the fatal Error 1053 when you attempt to start the service. Employ a service wrapper like AlwaysUp in that situation.
Using SC to Delete a Service
The command to remove a service with SC is straightforward:
SC DELETE <Service-Name>
To discard the service named “MyService” that we installed above, use:
SC DELETE MyService
Needless to say, please use this command with caution!. Once a service is deleted it cannot be easily re-instated. Removing the wrong service can render your computer unusable!
SC is the Complete Command Line Utility for Windows Services
So whenever you need to work with a service via a batch file or from a DOS command prompt, look to SC for support. This versatile, essential tool has earned its reputation as the “Swiss Army Knife” for Windows Services!