The Core Technologies Blog
Our Software // Windows Services // 24×7 Operation
Trinium Technologies provides business software for two primary verticals: intermodal trucking companies and fuel marketers. Its applications enable companies to reduce cost, grow revenue, and improve customer service. Customers range in size from small businesses to large enterprises with both cloud services and on-premises deployment options available.
Barry Assadi, the Chief Technical Officer at Trinium, has been using AlwaysUp since 2006. He knows AlwaysUp very well and was interviewed for this article.
Why does Trinium use AlwaysUp?
Trinium uses AlwaysUp to handle processes for file transfer (FTP), file import & export interfaces, and report generation. We used to deploy automated background processes with Windows Task Scheduler, but there were constant stability issues along with a lack of visibility of the status of these processes. AlwaysUp runs our executables without fail, and its detailed reporting tell us how our applications are doing.
Customer service is very important to us, so we have a dedicated AlwaysUp process per customer that is used to keep a heartbeat on all of the services we’ve created. This way we can see if any processes are hung or stopped to avoid any downtime for our customers.
How is AlwaysUp deployed in your business?
Each non-hosted customer will get an installation of AlwaysUp to run our software as a service in their environment. We also run AlwaysUp in Trinium’s hosted private cloud, which has a few dedicated servers reserved for processing using AlwaysUp.
What are your favorite features of AlwaysUp?
Over the years, we’ve come to appreciate the depth of settings to manage the processes we model as a service. There seems to be a solution for everything we’ve thrown at AlwaysUp!
Our more technical customers are happy that they can now stop and restart our applications from their batch files (using the NET command).
And we especially like the ability to multi-select to start/stop/restart processes. It is very convenient as our techs install and work with multiple services per server.
Incorporated in 2007 and driven by new opportunities in the “Internet-of-Things” (IoT) market, CartaSense offers end-to-end monitoring and alerting capabilities for cold-chain, agricultural and asset security applications. CartaSense provides complete field proven systems using wireless sensors, gateways and cloud based computing that deliver real-time information to support real-time decision making for increased efficiency and profits.
Chaim Belfer, the Vice President of R&D at CartaSense, has been using AlwaysUp with his Java programs since 2010. He currently manages more than 20 installations and was interviewed for this article.
How does CartaSense use AlwaysUp?
CartaSense deploys several Java applications that manage a world wide network of sensors. For security and management purposes, we need to run our windows installations as windows services. Looking for a solution we found AlwaysUp which solves the problem — plus helps with recovery and logging.
For the past few years, AlwaysUp has been included whenever we install our software on a Windows Server (2008 or 2012). We also recommend AlwaysUp to our own customers.
Why did you choose AlwaysUp for your Java applications?
We found your company through a Google search a few years ago. I tested several open source solutions but none of them really did the job, or at least needed a lot of development to make it work reliably. AlwaysUp worked well from the beginning (the 30-day trial) so we started out with a single license and bought more over time, as needed.
What are your favorite features of AlwaysUp?
Beyond solving the big problem of running our Java applications as windows services, AlwaysUp’s GUI is very nice and self explanatory. The documentation is also good, and the software is easy to use. Prompt service and quick response to our support emails is a big plus too!
First, let’s start with some definitions…
What is a regular application?
A regular application is one that you start and run on your Windows desktop. You launch it from a desktop icon, by double-clicking an associated document, or by typing its name at a command prompt. Examples include Microsoft Excel, Adobe Reader, Google Chrome and virtually any other program you consciously interact with on your PC. These are the programs that you know, love and use every day — the ones that make your computer a remarkably powerful tool.
What is a windows service?
A Windows Service is a special program designed to “plug into” Windows. It typically chugs along doing its work in the background, never revealing itself, with little or no attention from anyone using the PC. It turns out that a whole host of these invisible workers are necessary to make your PC work smoothly! For example, the Print Spooler windows service ensures that your documents arrive at the printer, while the Workstation service makes the Internet available for all applications. Important tasks indeed!
What’s the difference?
So with those basics in place, here are the 10 most important ways that a windows service differs from a regular application:
A windows service can start automatically at boot, without anyone having to log on. A regular application only runs when a user is logged in. This key distinction makes the windows service framework the better choice for software that must run 24×7, surviving the inevitable reboot along the way.
A windows service won’t exit when you or anyone else logs off the PC. A regular application will stop when you log out.
Windows services don’t run visibly on your desktop (not since Vista). They are confined to the isolated Session 0 and can not (usually) show their windows or tray icons to anyone logging into the PC. This restriction makes it very difficult for a windows service to interact with someone using the computer and consequently Microsoft strongly discourages implementing GUI services.
A windows service can be easily started or stopped from the command line using the NET or SC commands. This capability is a boon to sysadmins and technical folks, who can conveniently manage a service’s lifetime through unattended, administrative scripts.
Only a single instance of a windows service can be started at any time. That single copy must serve all users on the PC. This is very different than a regular application, which typically permits multiple copies, especially when several people are logged in.
Services typically run in the Local System account, which has administrative powers over many aspects of the PC. Thus your average service will have more rights and control over your computer than your typical application running in the context of a regular user’s account.
When run as LocalSystem, a service has access to UNC paths (for example, “\\server\data”) but cannot access mapped drive letters. You should run the service in a regular user account to access network drives, and even then the service may have to map drives explicitly (for example, with the NET USE command) to gain access.
All windows services support sophisticated recovery options for when they stop unexpectedly. You can restart the service, run a command line or even reboot the computer.
While each regular application is backed by a single, unique process, it is possible for many services to share a single running process. This situation makes it possible for some services to start rapidly and make efficient use of resources — important characteristics for critical components that must always be available.
A windows service must contain special code to respond to instructions from the Windows Service Control Manager (SCM). For the technical folks, this means implementing the ServiceMain (or equivalent) entry point in your application.
AlwaysUp is designed to run your application as a Windows Service and automatically restart it when it crashes or stops for any reason. Even though this frees you from having to take any action on a failure (eliminating those unwelcome “server-not-working” calls at 3 AM), you may still want to:
- Find out how often your application is failing
- Know the time when your program stops working (for diagnostic purposes)
- See how your application is using CPU and memory over the course of the day
- Find out whenever your server has been rebooted
Automatic email from AlwaysUp will tell you all that, and more!
Specifying Who to Email & When to Send a Message
To setup email in AlwaysUp, edit your application and switch to the Email tab:
Enter the following:
- The email address to notify. To send to multiple addresses, just separate each one with a comma (,).
- When AlwaysUp should send you an email — either in response to an important event, or at a regular time. The daily and weekly summary emails are a good way to monitor your application’s activities without having to receive multiple emails throughout the day.
- A line of text to be included in the email. This value is optional, but you can use it to explicitly label a server & application when you have many installed with the same name (for example, when you are running the same application on many different workstations).
AlwaysUp doesn’t require that you have Outlook or any other email software installed. Instead, it deals directly with your SMTP server – the machine actually responsible for routing your email over the Internet and delivering it to your recipients. Every major provider (such as Gmail, Yahoo and Office 365) supports SMTP mail and it is simply a matter of providing those settings to AlwaysUp so that it can send messages on your behalf.
Click the Configure… button to reveal the details:
The Configure Email Settings window that comes up requests quite a bit of information, but don’t be scared! Your systems administrator will be able to fill this in while balancing a bowl of water on his head and dancing the Macarena, but in case you are entering these values yourself here is what each field means:
- Address: The email address that will be displayed in the “From” field of the email messages sent by AlwaysUp.
- Name: The name that will be displayed in the “From” field of the email messages sent. This value is optional.
- Name/IP: The name (or IP address) of your mail server.
- Port: The numeric port on which the mail server is available. Note that the default is 25 for regular SMTP servers and 465 for servers using SSL – please specify one if these if in doubt.
- This server requires encryption: Check and make the appropriate selection if the mail server uses TLS, SSL or STARTTLS.
- This server requires authentication: Check if the server requires authentication, and select the authentication method. The choices are CRAM-MD5, AUTH-LOGIN, AUTH-PLAIN, and AUTH-NTLM. A login and a password will be required for all but AUTH-NTLM.
- Login: The user name/login for the mail server. Usually your full email address.
- Password: The password for your email account.
How to use your Gmail Account
To send email with your Gmail account, please specify the following settings:
||Your full gmail address (email@example.com)
|SMTP Server Name/IP:
||Your full gmail address (firstname.lastname@example.org)
||Your gmail password
So for someone named Mike Jones, whose Gmail address is email@example.com, the Configure Email window should look something like this:
How to use your Microsoft Office 365 Account
If you have an Office 365 mail account, please specify the following settings:
||Your full Office 365 email address
|SMTP Server Name/IP:
||Your full Office 365 email address
||Your Office 365 password
So for Mike Jones, whose Office 365 email address is firstname.lastname@example.org, the Configure Email window should look something like this:
Using Another Email Service
This list documents SMTP settings for many other email providers.
So that’s it! Please be sure to get in touch if you have any trouble setting up automatic email from AlwaysUp.
Crashes and memory leaks are particularly offensive when the occur in windows services – applications designed to run unattended, 24/7 in the background to perform important tasks. Our AlwaysUp and Service Protector utilities go a long way to diminishing the effect of these insidious failures, but using them is no substitute for finding and fixing a problem at its root.
Developers looking to diagnose failures in their windows services should consider using Microsoft’s Debug Diagnostic Tool. This free desktop application, nicknamed DebugDiag, will monitor your windows service process and create a “dump” describing the state of the application when it crashed (or started using too much memory). You can use this information to narrow down the problem in your own code, or pass it on to the folks who developed the failing service for their expert analysis.
Using the Debug Diagnostic Tool on your Windows Service
Download the Debug Diagnostic Tool from microsoft.com.
Install it. Setup should be quite straightforward, with the usual prompts for a folder, etc.
From Windows Explorer, navigate to the folder where DebugDiag was installed (C:\Program Files\DebugDiag by default). Start DebugDiag.Collection.exe – the application that monitors and collects information from a running process or service.
The Debug Diagnostic Tool window should come up and you will be launched immediately into setting up a new “rule” describing what to watch for. From here, you can specify to monitor for crashes, leaks, and even performance-related issues. We’ll select Crash for this tutorial and click Next to move on.
Next, select the A Specific NT service option:
Select the service you wish to debug from the list. We chose “AA MyService”, a windows service we developed to test our Service Protector application. Click Next.
Almost there! Leave the Advanced Configuration settings at their default values and click Next to continue:
You don’t have to make any changes on the Select Dump Location screen either – just make a note of where the dump file will be saved.
And finally, let’s activate the rule now and click Finish:
After a few seconds, the tool will be monitoring your windows service for crashes:
Analyzing a Windows Service Crash
When the service crashes, DebugDiag will record a dump file and increment the Userdump count column. Here we can see that 1 crash has occurred and been captured (with the full path to the dump file noted on the right):
To dig into the dump file:
Start DebugDiag.Analysis.exe from Windows Explorer:
The DebugDiag Analysis window will open momentarily:
To configure analysis:
- Check the CrashHangAnalysis box near to the top.
- Click the Add Data Files button, navigate to the location of the dump file (it is mentioned in the Diagnostic tool we set up previously) and select it. A new entry should show up in the lower pane.
And click the Start Analysis button to get going! After a period of “thinking”, the application will open its report in your browser:
Scan the report for any smoking guns. Thread call stacks may be particularly useful. For example, this stack trace tells us that our service crashed while handling a button press. (This is not surprising though – that is what we did to force the crash in our testing!)
Hopefully you will find a telltale sign to help you identify and fix your problem.
Also Find Memory Leaks in your Windows Service
If your windows service is consuming too much memory instead of crashing, configure the collector to watch for memory leaks:
The final report will detail all outstanding memory allocations.
We are pleased to announce the availability of AlwaysUp version 9.0, our market leading run-anything-as-windows-service solution used by thousands worldwide!
New features include:
Better support for Dropbox, Google Drive and other Applications
Programs that allow you to run only a single copy at a time would occasionally prevent AlwaysUp from starting its own copy in the background. For example, if someone started a copy of Dropbox on their desktop, trying to launch a second copy under AlwaysUp would lead to Dropbox exiting quickly and the service failing to start.
Checking the new Stop all copies of the application running on this computer setting on the Startup will resolve the issue. Instead of simply trying to launch a second copy of your program, AlwaysUp will first stop any existing instances, ensuring that its own copy will start smoothly.
We recommend using this setting with the following applications:
… and any others that should only run a single instance.
Additional Power Management Options
To prevent your computer from automatically going to sleep while your important program is being run by AlwaysUp, check the corresponding box on the Extras tab:
Reduced Event Logging
As it runs your application as a Windows Service, AlwaysUp writes information, warnings and errors to the Windows Event Log. Usually this is a good thing, keeping you up-to-date on your application’s comings and goings, but it becomes rather “noisy” for applications designed to start and stop frequently. For example, if you have a batch file configured to run every ten minutes, more than 20 events will be generated per hour!
Check the new Minimize event logging as the application stops & restarts box on the Restart tab to improve the situation. Afterwards, you should only see a message when the application starts or terminates unexpectedly. Normal/expected stops and restarts will not be recorded.
Be sure to check out the AlwaysUp Version History for the full list of features, fixes and improvements included in this release.
If you find that your application is starting and then quickly stopping when you run it with AlwaysUp, your application may be trying to report an error message that you simply cannot see. In this situation, we recommend starting your application from a command prompt running as a service, to find out if something useful is being written to the console.
How to start CMD.EXE as a service on your desktop
Highlight your troublesome application in AlwaysUp and select Add Copy… from the Application menu.
(In this guide, our “StartServer” batch file is failing for some unknown reason so we’ll work with that entry…)
In the Add Application window that comes up:
In the Application field, enter the full path to the windows command prompt. This is:
If there is anything in the Arguments field brought over from the application you copied, please remove it.
Change the Name field to something meaningful. We have entered “CMD Testing” in this guide.
Set the Start the application field to Manually, from AlwaysUp. We don’t want this command prompt service to start automatically at boot.
If you have specified a user on the Logon tab, switch over there and re-enter the password (a security measure).
Those are all the changes we need to make so click the Save >> button to record your new service. It will show up on the AlwaysUp console soon.
Next, start the new CMD service on your desktop by selecting Application > Start “CMD Testing” in this session:
In a few seconds, the familiar black box will pop up on your desktop:
And finally, in the command box, type in the full command line you gave to AlwaysUp – application & arguments. Be sure to specify the exact values that you had supplied to your AlwaysUp service! Please cut & paste to ensure accuracy.
Hit Enter to run your application and pay special attention to what is written to the command prompt.
Hopefully at this point your application will tell us what is wrong! In our case (pictured below), the batch file can’t find the JAR file it is expecting…
Resolving the problem
Common errors include:
- Running the wrong executable (doh!)
- Not properly quoting command-line parameters (smacks forehead)
- Not running from the correct directory (which you can fix by adding the folder on the General tab)
- Insufficient permissions to access a required resource (are you using the right user on the Logon tab?)
The AlwaysUp Frequently Asked Questions (FAQ) will help you resolve these and other common issues. And if you’re still stuck, please feel free to contact us for fresh ideas and expert advice
And don’t forget to cleanup when you’re done…
Once you’ve fixed the problem and verified that your application can start normally with AlwaysUp, please feel free to remove the CMD service by highlighting the entry in AlwaysUp and selecting Application > Remove.
The next version of Windows, dubbed Windows 10 is expected to be released in the second half of 2015. On October 1, Microsoft provided software makers with an early look at the new OS, so that we can test and certify our applications in advance of the launch. And like a kid in a candy store, we downloaded and eagerly installed the Windows 10 Technical Preview on a VirtualBox VM. The most noticeable change? The return of the Start menu! Finally, Microsoft has come to its senses.
Next, it was on to testing our software. Service Protector, which monitors and automatically restarts failing windows services, passed with flying colors. There with no issues whatsoever. Here it is happily monitoring the Print Spooler service:
However, things did not go as smoothly for AlwaysUp, our popular utility that runs any application as a windows service. We set up Notepad.exe as a service but quickly ran into trouble starting it:
After a couple of hours of debugging the code, it turns out that the problem is related to to the little-used user reboot notification feature recently introduced in Windows 8 & Server 2012. To avoid getting too technical, let’s just say that the new capability is simply not accepted by Windows 10. Using it causes AlwaysUp to fail to transition the service into the running state and remain in the starting state. And since no controls are accepted in the starting state, all the action buttons are grayed out and the service will remain stuck starting forever! The only way to stop the service is to disable all recovery actions for the service and forcibly terminate AlwaysUpService.exe from the Task Manager. A real mess!
We have reported the problem to Microsoft and hopefully they will resolve it before the official release of Windows 10 next year. If not, a change to our code to eliminate the user reboot notification feature will also fix the problem, so not to worry. We’ll stay on top of it
A windows service, designed to run “headless” and unattended in the background, cannot easily employ conventional popup windows to report its activities as a user may not even be logged on. Instead, a service is encouraged to send important communication to the Windows Event Log – an administrative utility that collects and stores messages and events. Once recorded, these messages can be very helpful in troubleshooting problems, for example when a service stops unexpectedly or when it fails to start at all.
Viewing Events from Windows Services
Use Microsoft’s Event Viewer to see messages written to the Event Log. Start the application by clicking on the Start button and typing in Event Viewer, or from the Control Panel (search for it by name). The somewhat cluttered window should come up after a few seconds:
The left hand side shows a tree grouping the various logs captured on your machine. The events from Windows Services (and other applications running on your PC) are filed under Windows Logs > Application. Navigate to that section to load the events in the center of the window, with the entire list in the top and details of the highlighted event underneath:
Messages from your windows service will have the display name of the service in the Source column.
Important Components of an Event
The Event Viewer shows over 10 pieces of information associated with each event, including:
- Level – How important is this event?
Each event is classified into one of three categories:
Information: An informative yet unimportant event. You will probably see a lot of these, and they can be safely ignored unless you are digging into a specific issue from an application or service.
Warning: A moderately important event. These don’t necessarily signify a failure, and your software will probably limp along, but they should be reviewed regularly to see if anything mentioned can be resolved.
Error: Indicates a critical problem or failure that may deserve your immediate attention!
- Date and Time – When did this event occur?
- Source – Which application reported this event?
As mentioned before, an event written by a Windows Service will contain the service’s display name as the Source.
- Description – Which happened?
The full description shown prominently in the lower pane will (hopefully) provide the relevant details of the event.
For example, this information event is from the Interactive Services detection service (“UI0Detect”) reporting that Notepad is showing itself in Session 0:
Viewing Events about Windows Services
While the Application log keeps track of events from a running service, the Windows Logs > System area records when services are started, stopped, crash or fail to start. Look for events with the Source set to Service Control Manager (SCM). For example, here is the SCM telling us that the Windows Print Spooler service has crashed:
Viewing Events from AlwaysUp and Service Protector
Both AlwaysUp and Service Protector write messages to the Application section of the event logs (Windows Logs > Application).
For AlwaysUp, events from your application named “My Application” will be logged with Source set to My Application (managed by AlwaysUpService). The Event Log Messages Page lists and explains the events reported.
For Service Protector, events related to your service named “MyService” will have a Source of ServiceProtector: MyService.
And for both applications, events related to the starting and stopping of the underlying services themselves appear in the Windows Logs > System section. Look there if you have a problem with AlwaysUp itself failing to start at boot.
You need to start your application at boot, even if no one logs on
A regular application will only run after a user has logged in and started it. Not so for Windows Services, which are automatically started by the operating system whenever the computer reboots and run happily without anyone having to do anything. Services are the true set-it-and-forget-it approach.
Your application must be available 24/7
The automatic, extensible failure protection system built into AlwaysUp will help to minimize the downtime for your legacy application – even if it is buggy, leaks memory or stops working for no good reason!
You don’t have the time, resources or access to the source code to re-write the application as a Windows Service
The boss needs your software to be a windows service to win the next contract, but your development team is swamped and no one wants to mess around with the code anyway. Use AlwaysUp to “wrap” your application to operate as a windows service so you can satisfy the client and keep your boss happy, all in a day’s work.
You are managing an unreliable application that keeps crashing
Support calls for buggy applications have ruined too many evenings and nights! AlwaysUp automatically restarts unstable applications whenever they stop for any reason, so enjoy an uninterrupted dinner with your family for a change!
You prefer to run your program in the background, away from inexperienced users
Mission-critical applications visible on the desktop run the risk of being accidentally shutdown by a user or closed when he logs off. However, a windows service runs on a “hidden” desktop where it is well insulated from those using the computer.