The Core Technologies Blog
Our Software // Windows Services // 24×7 Operation
Crashes and memory leaks are particularly offensive when the occur in windows services – applications designed to run unattended, 24/7 in the background to perform important tasks. Our AlwaysUp and Service Protector utilities go a long way to diminishing the effect of these insidious failures, but using them is no substitute for finding and fixing a problem at its root.
Developers looking to diagnose failures in their windows services should consider using Microsoft’s Debug Diagnostic Tool. This free desktop application, nicknamed DebugDiag, will monitor your windows service process and create a “dump” describing the state of the application when it crashed (or started using too much memory). You can use this information to narrow down the problem in your own code, or pass it on to the folks who developed the failing service for their expert analysis.
Using the Debug Diagnostic Tool on your Windows Service
Download the Debug Diagnostic Tool from microsoft.com.
Install it. Setup should be quite straightforward, with the usual prompts for a folder, etc.
From Windows Explorer, navigate to the folder where DebugDiag was installed (C:\Program Files\DebugDiag by default). Start DebugDiag.Collection.exe – the application that monitors and collects information from a running process or service.
The Debug Diagnostic Tool window should come up and you will be launched immediately into setting up a new “rule” describing what to watch for. From here, you can specify to monitor for crashes, leaks, and even performance-related issues. We’ll select Crash for this tutorial and click Next to move on.
Next, select the A Specific NT service option:
Select the service you wish to debug from the list. We chose “AA MyService”, a windows service we developed to test our Service Protector application. Click Next.
Almost there! Leave the Advanced Configuration settings at their default values and click Next to continue:
You don’t have to make any changes on the Select Dump Location screen either – just make a note of where the dump file will be saved.
And finally, let’s activate the rule now and click Finish:
After a few seconds, the tool will be monitoring your windows service for crashes:
Analyzing a Windows Service Crash
When the service crashes, DebugDiag will record a dump file and increment the Userdump count column. Here we can see that 1 crash has occurred and been captured (with the full path to the dump file noted on the right):
To dig into the dump file:
Start DebugDiag.Analysis.exe from Windows Explorer:
The DebugDiag Analysis window will open momentarily:
To configure analysis:
- Check the CrashHangAnalysis box near to the top.
- Click the Add Data Files button, navigate to the location of the dump file (it is mentioned in the Diagnostic tool we set up previously) and select it. A new entry should show up in the lower pane.
And click the Start Analysis button to get going! After a period of “thinking”, the application will open its report in your browser:
Scan the report for any smoking guns. Thread call stacks may be particularly useful. For example, this stack trace tells us that our service crashed while handling a button press. (This is not surprising though – that is what we did to force the crash in our testing!)
Hopefully you will find a telltale sign to help you identify and fix your problem.
Also Find Memory Leaks in your Windows Service
If your windows service is consuming too much memory instead of crashing, configure the collector to watch for memory leaks:
The final report will detail all outstanding memory allocations.
We are pleased to announce the availability of AlwaysUp version 9.0, our market leading run-anything-as-windows-service solution used by thousands worldwide!
New features include:
Better support for Dropbox, Google Drive and other Applications
Programs that allow you to run only a single copy at a time would occasionally prevent AlwaysUp from starting its own copy in the background. For example, if someone started a copy of Dropbox on their desktop, trying to launch a second copy under AlwaysUp would lead to Dropbox exiting quickly and the service failing to start.
Checking the new Stop all copies of the application running on this computer setting on the Startup will resolve the issue. Instead of simply trying to launch a second copy of your program, AlwaysUp will first stop any existing instances, ensuring that its own copy will start smoothly.
We recommend using this setting with the following applications:
… and any others that should only run a single instance.
Additional Power Management Options
To prevent your computer from automatically going to sleep while your important program is being run by AlwaysUp, check the corresponding box on the Extras tab:
Reduced Event Logging
As it runs your application as a Windows Service, AlwaysUp writes information, warnings and errors to the Windows Event Log. Usually this is a good thing, keeping you up-to-date on your application’s comings and goings, but it becomes rather “noisy” for applications designed to start and stop frequently. For example, if you have a batch file configured to run every ten minutes, more than 20 events will be generated per hour!
Check the new Minimize event logging as the application stops & restarts box on the Restart tab to improve the situation. Afterwards, you should only see a message when the application starts or terminates unexpectedly. Normal/expected stops and restarts will not be recorded.
Be sure to check out the AlwaysUp Version History for the full list of features, fixes and improvements included in this release.
If you find that your application is starting and then quickly stopping when you run it with AlwaysUp, your application may be trying to report an error message that you simply cannot see. In this situation, we recommend starting your application from a command prompt running as a service, to find out if something useful is being written to the console.
How to start CMD.EXE as a service on your desktop
Highlight your troublesome application in AlwaysUp and select Add Copy… from the Application menu.
(In this guide, our “StartServer” batch file is failing for some unknown reason so we’ll work with that entry…)
In the Add Application window that comes up:
In the Application field, enter the full path to the windows command prompt. This is:
If there is anything in the Arguments field brought over from the application you copied, please remove it.
Change the Name field to something meaningful. We have entered “CMD Testing” in this guide.
Set the Start the application field to Manually, from AlwaysUp. We don’t want this command prompt service to start automatically at boot.
If you have specified a user on the Logon tab, switch over there and re-enter the password (a security measure).
Those are all the changes we need to make so click the Save >> button to record your new service. It will show up on the AlwaysUp console soon.
Next, start the new CMD service on your desktop by selecting Application > Start “CMD Testing” in this session:
In a few seconds, the familiar black box will pop up on your desktop:
And finally, in the command box, type in the full command line you gave to AlwaysUp – application & arguments. Be sure to specify the exact values that you had supplied to your AlwaysUp service! Please cut & paste to ensure accuracy.
Hit Enter to run your application and pay special attention to what is written to the command prompt.
Hopefully at this point your application will tell us what is wrong! In our case (pictured below), the batch file can’t find the JAR file it is expecting…
Resolving the problem
Common errors include:
- Running the wrong executable (doh!)
- Not properly quoting command-line parameters (smacks forehead)
- Not running from the correct directory (which you can fix by adding the folder on the General tab)
- Insufficient permissions to access a required resource (are you using the right user on the Logon tab?)
The AlwaysUp Frequently Asked Questions (FAQ) will help you resolve these and other common issues. And if you’re still stuck, please feel free to contact us for fresh ideas and expert advice
And don’t forget to cleanup when you’re done…
Once you’ve fixed the problem and verified that your application can start normally with AlwaysUp, please feel free to remove the CMD service by highlighting the entry in AlwaysUp and selecting Application > Remove.
The next version of Windows, dubbed Windows 10 is expected to be released in the second half of 2015. On October 1, Microsoft provided software makers with an early look at the new OS, so that we can test and certify our applications in advance of the launch. And like a kid in a candy store, we downloaded and eagerly installed the Windows 10 Technical Preview on a VirtualBox VM. The most noticeable change? The return of the Start menu! Finally, Microsoft has come to its senses.
Next, it was on to testing our software. Service Protector, which monitors and automatically restarts failing windows services, passed with flying colors. There with no issues whatsoever. Here it is happily monitoring the Print Spooler service:
However, things did not go as smoothly for AlwaysUp, our popular utility that runs any application as a windows service. We set up Notepad.exe as a service but quickly ran into trouble starting it:
After a couple of hours of debugging the code, it turns out that the problem is related to to the little-used user reboot notification feature recently introduced in Windows 8 & Server 2012. To avoid getting too technical, let’s just say that the new capability is simply not accepted by Windows 10. Using it causes AlwaysUp to fail to transition the service into the running state and remain in the starting state. And since no controls are accepted in the starting state, all the action buttons are grayed out and the service will remain stuck starting forever! The only way to stop the service is to disable all recovery actions for the service and forcibly terminate AlwaysUpService.exe from the Task Manager. A real mess!
We have reported the problem to Microsoft and hopefully they will resolve it before the official release of Windows 10 next year. If not, a change to our code to eliminate the user reboot notification feature will also fix the problem, so not to worry. We’ll stay on top of it
A windows service, designed to run “headless” and unattended in the background, cannot easily employ conventional popup windows to report its activities as a user may not even be logged on. Instead, a service is encouraged to send important communication to the Windows Event Log – an administrative utility that collects and stores messages and events. Once recorded, these messages can be very helpful in troubleshooting problems, for example when a service stops unexpectedly or when it fails to start at all.
Viewing Events from Windows Services
Use Microsoft’s Event Viewer to see messages written to the Event Log. Start the application by clicking on the Start button and typing in Event Viewer, or from the Control Panel (search for it by name). The somewhat cluttered window should come up after a few seconds:
The left hand side shows a tree grouping the various logs captured on your machine. The events from Windows Services (and other applications running on your PC) are filed under Windows Logs > Application. Navigate to that section to load the events in the center of the window, with the entire list in the top and details of the highlighted event underneath:
Messages from your windows service will have the display name of the service in the Source column.
Important Components of an Event
The Event Viewer shows over 10 pieces of information associated with each event, including:
- Level – How important is this event?
Each event is classified into one of three categories:
Information: An informative yet unimportant event. You will probably see a lot of these, and they can be safely ignored unless you are digging into a specific issue from an application or service.
Warning: A moderately important event. These don’t necessarily signify a failure, and your software will probably limp along, but they should be reviewed regularly to see if anything mentioned can be resolved.
Error: Indicates a critical problem or failure that may deserve your immediate attention!
- Date and Time – When did this event occur?
- Source – Which application reported this event?
As mentioned before, an event written by a Windows Service will contain the service’s display name as the Source.
- Description – Which happened?
The full description shown prominently in the lower pane will (hopefully) provide the relevant details of the event.
For example, this information event is from the Interactive Services detection service (“UI0Detect”) reporting that Notepad is showing itself in Session 0:
Viewing Events about Windows Services
While the Application log keeps track of events from a running service, the Windows Logs > System area records when services are started, stopped, crash or fail to start. Look for events with the Source set to Service Control Manager (SCM). For example, here is the SCM telling us that the Windows Print Spooler service has crashed:
Viewing Events from AlwaysUp and Service Protector
Both AlwaysUp and Service Protector write messages to the Application section of the event logs (Windows Logs > Application).
For AlwaysUp, events from your application named “My Application” will be logged with Source set to My Application (managed by AlwaysUpService). The Event Log Messages Page lists and explains the events reported.
For Service Protector, events related to your service named “MyService” will have a Source of ServiceProtector: MyService.
And for both applications, events related to the starting and stopping of the underlying services themselves appear in the Windows Logs > System section. Look there if you have a problem with AlwaysUp itself failing to start at boot.
You need to start your application at boot, even if no one logs on
A regular application will only run after a user has logged in and started it. Not so for Windows Services, which are automatically started by the operating system whenever the computer reboots and run happily without anyone having to do anything. Services are the true set-it-and-forget-it approach.
Your application must be available 24/7
The automatic, extensible failure protection system built into AlwaysUp will help to minimize the downtime for your legacy application – even if it is buggy, leaks memory or stops working for no good reason!
You don’t have the time, resources or access to the source code to re-write the application as a Windows Service
The boss needs your software to be a windows service to win the next contract, but your development team is swamped and no one wants to mess around with the code anyway. Use AlwaysUp to “wrap” your application to operate as a windows service so you can satisfy the client and keep your boss happy, all in a day’s work.
You are managing an unreliable application that keeps crashing
Support calls for buggy applications have ruined too many evenings and nights! AlwaysUp automatically restarts unstable applications whenever they stop for any reason, so enjoy an uninterrupted dinner with your family for a change!
You prefer to run your program in the background, away from inexperienced users
Mission-critical applications visible on the desktop run the risk of being accidentally shutdown by a user or closed when he logs off. However, a windows service runs on a “hidden” desktop where it is well insulated from those using the computer.
We are happy to announce the availability of the AlwaysUp Troubleshooter – our online, step-by-step guide to help you identify and resolve the most common problems encountered when configuring your application as a windows service.
The troubleshooter is very easy to use and is packed with illustrative screenshots and helpful tips to quickly guide you to a successful outcome. However, if your issue is complex and demands more than can be achieved in that format, the interview ends with a summary of what has been learned and a contact form that makes it easy to engage our support team for additional help.
So if you are having difficulty setting up your application, script or batch file as a windows service with AlwaysUp, click here to start troubleshooting now!.
Starting with Windows Vista in 2007 and continuing with every subsequent release, Microsoft has banned users from logging in to the first session created as Windows boots. This “Session 0 Isolation” has been accompanied by a steady erosion of the interactive capabilities of Session 0, rendering the Session 0 desktop virtually unrecognizable in the post-XP world. The prevailing wisdom in Redmond: Why support a proper working desktop in Session 0 when no one in their right mind is expected to use it?
Unfortunately those of us working with interactive applications and Windows Services in Session 0 don’t have the luxury of ignoring GUI-related issues. And users of the excellent AutoIt automation toolkit have to make special accommodations when creating scripts to click buttons and fill in forms in Session 0. Here is what we have learned while troubleshooting AutoIt with our customers:
WinActivate, WinWaitActive and WinActive Often Fail
Essential AutoIt functions like WinActivate, WinWaitActive and WinActive rely on the operating system to track the active window. Unfortunately it seems that the active window is not tracked in Session 0 unless a user is actively viewing the desktop! Therefore a script that takes some action only if a window has been activated may never do its work. In this sample code, WinActivate will always return 0 when run in an unattended Session 0 and the button will never be clicked:
; Activate the window and click the button.
if (WinActivate($windowName)) Then
; Click the button on the window.
ControlClick($windowName, $buttonName, "")
Our advice: Don’t depend on the Win* functions to find and activate a window. Simply operate on your window regardless of its active state. For example, restructure the code above to look like this:
; Try to activate the window if possible. May fail in Session 0.
; Click the button on the window.
ControlClick($windowName, $buttonName, "")
The Mouse Functions (MouseMove, MouseClick, etc.) Don’t Work
As with window activation, it seems that the mouse is not tracked when session 0 is unattended. Functions that move and manipulate the mouse can fail unexpectedly.
Our advice: Don’t use the mouse functions in your scripts.
AutoIt Scripts created with Au3Record Are Unreliable
The Au3Record utility bundled with AutoIt will easily record a set of actions for later replay. It captures mouse movements, mouse clicks, key presses and more. However the script created makes heavy use of the problematic activation and mouse related functions cited above, thus rendering Au3Record scripts virtually useless in Session 0.
Our advice: Don’t use scripts created by Au3Record in Session 0.
You can Rely on ControlSend and ControlClick in Session 0
Fortunately functions like ControlSend and ControlClick continue to work as expected in Session 0. Neither of them relies on the target window being active nor the mouse being in a specific position, so they sidestep trouble related to those areas on the isolated desktop.
Are you using AutoIt in the isolated Session 0? Have you run into other functionality that does not work as expected? Please let us know what you have found! Your feedback will be much appreciated.
If you are comfortable working with Windows Services from the command line, take a look at Microsoft’s TASKLIST utility. This often overlooked tool — available on all versions of Windows — can help you answer the following questions as you troubleshoot your Windows Services:
What is the PID Associated with a Running Service?
You can use the name of the service to look up the identifier of the process associated with it. For example, to find the PID of the Print Spooler service (named “Spooler”), use:
TASKLIST /SVC /FI "SERVICES eq Spooler"
Here is the result you can expect, with the PID listed in the central column:
What Windows Services are Being Hosted by a Specific Process?
If you are considering forcibly terminating a process, it may be wise to confirm that doing so will not unexpectedly kill related services! TASKLIST.EXE comes to the rescue by easily showing what services are running in a given process. For example, to see what services are behind the process with PID 1234, type:
TASKLIST /SVC /FI "PID eq 1234"
Here we can see that the svchost process with PID 1284 is managing five (!) core services:
Note that TASKLIST can interrogate remote machines as well. Simply add the /S /U and /P flags to the command line to specify the system, user name and password respectively.
The Problem: Dropbox Automatic Updates
Recently, a few customers reported “issues” with Dropbox automatically updating itself and causing havoc when running under AlwaysUp in their 24×7 environments. Apparently the auto-upgrade would restart Dropbox outside of AlwaysUp and that unmanaged instance would prevent AlwaysUp from starting a new copy under its control. The result was hundreds of Explorer windows open on the screen — a great experience!
Our first instinct was to turn off automatic updates, which are usually a bad idea in a controlled, 24×7 environment anyway. Unfortunately Dropbox does not allow that. Auto-updates are totally managed by Dropbox and there are no exposed options for adjusting the behavior.
The lack of controls thwarted our second brilliant idea as well — scheduling auto-upgrades for a specific time each day/week/month. Apparently changes can take place at any time, at the program’s discretion. Again, not suitable in a managed environment.
A polite email to the Dropbox support folks explaining the situation and requesting help & advice yielded a most unsatisfying generic response:
Thanks for writing in. While we'd love to answer every question we get,
we unfortunately can't respond to your inquiry due to a large volume of
support requests. Here are some resources for resolving the most common
Restore files or folders - www.dropbox.com/help/969
Reset your Dropbox password - www.dropbox.com/forgot
Reset/Disable two-step verification - www.dropbox.com/help/364
Learn about sharing files or folders - www.dropbox.com/help/category/Sharing
Learn about Dropbox's desktop app - www.dropbox.com/help/category/Desktop
Learn about Dropbox's mobile apps - www.dropbox.com/help/category/Mobile
For all other issues, please check out our Help Center - www.dropbox.com/help
We're sorry for the inconvenience,
The Dropbox Team
So time to do some reverse engineering!
The Investigation: How Dropbox Auto-Update Works
We installed an outdated version of Dropbox (2.47, thanks OldApps.com!) on our Windows Server 2012 machine, started it under AlwaysUp and patiently waited with Microsoft’s excellent Process Explorer open. It took about 8 minutes for the action to start…
First, Dropbox.exe launched dropbox-upgrade-2.8.2.exe, which it must have silently downloaded into the private cache folder (C:\Users\<UserName>\AppData\Roaming\Dropbox\.dropbox-cache).
- A few seconds later, the update program spawned a second copy of Dropbox.exe, passing the ominous /killeveryone flag.
Next, all instances of Dropbox subsequently shut down (including the one managed by AlwaysUp), and the updater launched a new copy of Dropbox before it too exited.
Notice the peculiar command line arguments.
This is all fine when Dropbox is running normally on your desktop, but it is a problem when Dropbox is being managed by AlwaysUp. Indeed, once AlwaysUp noticed that the Dropbox process it was managing was closed, it started up a second copy:
Now Dropbox abhors multiple instances, so after a few seconds the AlwaysUp-controlled instance exits — but not before throwing up an Explorer window browsing the Dropbox folder. AlwaysUp, persistent fellow that it is, detects that Dropbox has once again exited, starts it up again and the whole process repeats. It is easy to see that if left unchecked, we may end up with hundreds of redundant Explorer windows!
The Solution: Configuring AlwaysUp to Tolerate Automatic Dropbox Updates
To prevent the auto-update from triggering ten billion Explorer windows, we must have AlwaysUp close the instance of Dropbox spawned during the upgrade prior to starting its own copy. To do so:
Edit Dropbox in AlwaysUp
Click over to the Startup tab.
If you have AlwaysUp version 9.0 (released in December 2014) or later, check the Stop all copies of the application running on this computer and the Also whenever the application is restarted boxes as pictured here:
Otherwise, if you have an earlier version of AlwaysUp without the “single instance” feature:
Download close-application.bat, a DOS batch file that will close a given application. Save it to the AlwaysUp folder (C:\Program Files\AlwaysUp, or C:\Program Files (x86)\AlwaysUp)
Check the Run the following program/batch file box and enter the following command line:
“C:\Program Files\AlwaysUp\close-application.bat” Dropbox.exe
Check the Also run it whenever the application is restarted box as well.
These changes will cause AlwaysUp to close all running instances of Dropbox before starting the one that it will monitor.
And finally, to give Dropbox some time to complete its auto-update before AlwaysUp tries to fire it up again, click over to the Restart tab and:
- Check the Whenever the application stops, restart it box
- Check the Not immediately, but box
- Select the After option and enter 5 minutes in the adjacent field.
Click the Save >> button to record your settings.
That’s it! The next time Dropbox decides to automatically update itself, AlwaysUp will quickly bring it back under control — for continued 24×7 operation.