The Core Technologies Blog
Our Software // Windows Services // 24×7 Operation
The mystery: “The service did not start due to a login failure”
One of our customers reported a very strange problem last week. After about a day of running flawlessly, their windows service would suddenly fail to start after a reboot. The error reported by the Event Viewer hinted at a problem with the service user’s account:
Trying to start the service directly from the Windows Services Control Panel application produced the same unsatisfying result:
The service account’s password had not changed, and the user had no problem logging into the server interactively. Why was the windows service failing to login?
Luckily we were able to get the service going again by re-entering the user’s password:
When doing so, we noticed that the “Log on as a service” right had to be granted again. Very suspicious…
But a mere 24 hours later, the problem resurfaced! Once again, the service failed to start after a reboot.
The problem: Group Policy overwriting Local Policy
The message about the “Log on as a service” right lead us to the root of the problem.
Entering the password in services.msc updated the user’s rights in the machine’s Local Group Policy — a collection of settings that define how the system will behave for the PC’s users. However, since the user and server were part of a domain, those local settings were periodically overwritten by the domain’s group policy, which had not been updated with the new permission. And because the necessary permission “disappeared” on the machine, the service failed the next time it tried to start.
The solution: Modify the Domain Group Policy
To fix the problem, we must update the domain group policy and explicitly give the service user the “Log on as a service” right. To do so:
Start the Group Policy Management application.
Open Control Panel, navigate to System and Security > Administrative Tools, and double-click Group Policy Management on the left.
(Note: Don’t search for “group” in Control Panel. That will lead you to the “Edit group policy” link, which opens the local group policy!)
Find your default domain policy on the left. Right-click it and select Edit to bring up the Group Policy Management Editor window.
On the left, navigate to Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > User Rights Assignment and select
the Log on as a service entry on the right.
Double-click Log on as a service to bring up its Properties window.
Add the user running your windows service to the list and click OK to record the change.
Next time your domain policy is copied to your server, it will bring along the Log on as a service right for the user. You shouldn’t encounter the “logon failure” error again!
A closing note for the folks at Microsoft: A better error message please!
Instead of reporting the generic “logon failure”, why not be more precise and say something like “The user doesn’t have the necessary rights to start the service”? You could provide even more guidance by listing the missing rights.
The bottom line is a that a helpful error message highlighting the true problem would have led us straight to the solution and avoided a few anxious days for us and our client.
Start Software is an award-winning software development and consulting business with offices in the UK and Australia. They create easy-to-use software systems that their clients enjoy using and which deliver real benefits. The Start Software team are specialists in Alpha Anywhere, Alpha Five and Mobile Data Studio developments. Alpha Tracker – the latest addition to the Tracker family – is a web browser-based version of Tracker for asbestos surveyors, asbestos consultants and asbestos consultancy businesses of any size.
Start Software has been using AlwaysUp regularly since 2007.
Why does Start Software use AlwaysUp?
We have created a variety of software systems using Alpha Five and Alpha Anywhere from Alpha Software and Visual Basic from Microsoft. Many of these systems need to be “always up” – operating like 24x7x365 windows services. The AlwaysUp product enables us to deploy these applications with confidence that the services they provide will always be available to our customers.
How are you using AlwaysUp today?
We install AlwaysUp along with our software as a matter of course. We have more than 50 clients using AlwaysUp on Windows Server 2008 and Windows Server 2012 on a daily basis all around the world.
What are your favorite features of the software?
AlwaysUp is easy to install and configure, and has flexible options for the automatic restarting of applications (services) under a variety of conditions. Technical support provided by Core Technologies is also excellent and they are a joy to deal with.
Any other comments?
Our Tracker and Alpha Tracker systems have been used by asbestos consultancies to capture data about more than 25 million samples in buildings worldwide and so, indirectly, AlwaysUp is helping to manage this deadly substance across the globe.
Do you want to run a script or application every minute? And have you decided that creating an AlwaysUp Windows Service will be better than using the built-in Windows Task Scheduler? If yes, here are the key settings to specify with AlwaysUp.
Wait One Minute Before Restarting
By default, AlwaysUp will immediately restart your application whenever it exits. To have AlwaysUp “pause” for a minute before restarting, make the following changes on the Restart tab:
Avoid Verbose Event Logging on each Run/Restart
AlwaysUp is designed to record detailed information whenever your application exits and has to be restarted. This is often useful information for applications that should run continuously but the logging can become overwhelming when your application is stopping and restarting every minute!
Check the Minimize event logging as the application stops & starts on the Restart tab to avoid extra chatter as your application/script cycles each minute.
If you are interested in email notifications, be sure NOT to check the Whenever the application stops and has to be restarted box on the Email tab! If you do, expect to be inundated with emails as your application goes up and down every minute.
Recently, a few of our customers running Dropbox as a windows service with AlwaysUp have reported failures after manually updating to the latest version of Dropbox. These are happening because Dropbox changes the location of its executable file during the update. Let’s dig into the details and see how to fix the problem.
The Old Dropbox Location
Prior to version 3.5.58, released in April 2015, Dropbox stored its application executable file (Dropbox.exe) in a user-specific folder. If you installed Dropbox in the Administrator account, you could find the file in C:\Users\Administrator\AppData\Roaming\Dropbox\bin:
The New Dropbox Location
If you install a fresh copy of version 3.5.58 or later, you will notice that Dropbox.exe now lives underneath the Program Files folder, in C:\Program Files (x86)\Dropbox\Client (or C:\Program Files\Dropbox\Client on 32-bit versions of Windows):
We have already noted the new location in our step-by-step tutorial to guide future customers, but the story is more complicated for existing installations…
Upgrading Dropbox Changes its Location (and Breaks your AlwaysUp Configuration)
Unfortunately the impact of this change to the file locations isn’t limited to new installations. Manually updating an existing Dropbox installation will cause the files to be moved to the new folder. This isn’t a problem for Dropbox itself, but since AlwaysUp is blissfully unaware of the change, it will continue to run Dropbox.exe from its old location (as you originally specified in the Application field on the General tab):
Because the executable is no longer present there, AlwaysUp will fail to start Dropbox and report errors like this:
The Fix: Set the New Dropbox Location in AlwaysUp
Fortunately normal operation can be restored quite easily. Simply edit Dropbox in AlwaysUp and update the Application field on the General tab to specify the new path to the Dropbox executable (C:\Program Files (x86)\Dropbox\Client\Dropbox.exe):
Save your settings and AlwaysUp should be able to start Dropbox normally again!
PC Commander 4.0, our free tool that helps you manage your remote computers, is now available for download. This convenient software allows you to control your PC entirely from your web browser. Google Chrome, Mozilla Firefox, Internet Explorer and even Microsoft’s new Edge browser are all supported.
Here are a few things you can do with PC Commander:
Securely access your PC
Type in the URL and specify your software-configured password to login to your computer. Note that you can setup PC Commander to use HTTPS to ensure that all communications are encrypted.
Understand what is running, restart your PC and more
After you login, the System page will present a general overview of what is running on (and how many user are logged on to) your computer. From there you can even restart your server — a very convenient feature if the PC is hung or cannot respond to an interactive login!
Manage all processes
The Processes page lists everything running on your computer. Easily see who started each process and how much CPU and memory is being consumed by each one. Click on a column’s title to sort the table by that column — very useful when looking for memory or CPU “hogs”!
Stop or modify any process
From the table of processes, click a name to “drill down” into a single process. From there you can stop the process or change its priority to give it more or less resources on the computer. You can also quickly find some information about the process at ProcessLibrary.com if you don’t recognize it…
Manage all Windows Services
Go to the Services page to see all windows services installed on your PC. The table shows which services are running (via the green circle on the left), how services are started and the user account that runs each service. Columns are sortable for your convenience and you can easily start, stop or restart any service directly from the table.
Start, stop or restart any Windows Service
Click into a specific service to see additional details about it. The full command line used to launch the service is shown, and all dependent services are identified. You can easily start, stop or restart your service from this page.
View all users logged in
The Users page lists all users currently logged on to the computer. With a couple of clicks, you can quickly disconnect an account to allow another user to log on.
Log off, disconnect or send a message to a user
Drill down into a specific user entry to see additional options. Send a message directly to a user’s screen to notify him that you are about to log him off!
Browse your hard drive and view/download any file
Open the Drives page to list all storage devices attached to your computer. Navigate to any folder and view or download any file you like! And you can see detailed information on a file by clicking on its name.
View the Windows Event Logs
The Event logs page comes in handy when you want to investigate system errors or see what your windows services are up to. Click on a log to review its detailed records in a friendly tabular format.
And that’s it! We hope that you will find this free software useful when managing your computers. Please be sure to get in touch if you have ideas on how we can make PC Commander even better for you.
We all know to fire up the Windows Task Manager when we want to terminate a process, see how much memory is being used or find that rogue program hogging the CPU, but do you know that Task Manager also shows important details of Windows Services? For some situations, the Task Manager, with its focus on what is actually running on the PC, is even more useful than the standard Services Control Panel!
Information about your PC’s services can be found in the Task Manager’s Services tab. If you don’t see any tabs across the top of the Task Manager window, click the More details link at the bottom to reveal them:
The services table shows all the windows services installed on your PC. Helpful columns show the description, status and group of each service. And, perhaps most importantly, services that are running will show the numeric process identifier (PID) of the process hosting the service. That PID is useful when you want to terminate an unresponsive service, boost its priority or perform other low-level operations.
Right-click an entry to summon the context menu offering several operations you can perform on the service:
Start, Stop or Restart your Windows Service
Take action on your service right here! No need to open services.msc, or run the NET START or NET STOP commands from a prompt.
Selecting this entry will open the Windows Services Control Panel. Note that you can also open Services by clicking on the Open Services link at the bottom of the window.
Choosing this entry opens your browser and searches for the service’s name and display name. For example, if you have selected the service named Spooler with display name Print Spooler, selecting “Search online” will open your browser and launch a convenient Bing search (or Google, if that is your default search engine) for Spooler Print Spooler:
Go to details
Selecting this entry, which is only available when your service is running, switches you to the Task Manager’s Details tab with the service’s underlying process highlighted.
For example, here is the VirtualBox service (VBoxService) running with PID 624 as we select “Go to details”:
We are immediately transported to the Details tab to provide us with more information on the process with PID 624 (named VBoxService.exe):
As you can see, Task Manager provides some decent tools for controlling your windows services. Be sure to fire it up next time you have to work with a service!
Wellsoft Corporation, developer and provider of the industry-leading Emergency Department Information System (EDIS) since 1990, is consistently ranked #1 in KLAS user surveys of EDIS time and again. Recognized for providing the most complete functionality and most configurable product on the market, Wellsoft is the specialist in Emergency Department Information Systems.
Wellsoft has been including AlwaysUp for its clients since 2007.
Why does Wellsoft use AlwaysUp for all implementations?
Wellsoft’s Emergency Department Information System (EDIS) is a mission critical application for hospital emergency departments. The software must operate 24 hours a day, with unparalleled availability and reliability. AlwaysUp helps us maintain superior uptimes for clients by:
- Automatically starting the Wellsoft application if the server is rebooted
- Automatically restarting the application if the server hangs or uses too much memory
- Restarting instances of Wellsoft during routine Microsoft memory or CPU spikes that close a running copy
- Sending email alerts to report potential problems and support our IT staff
We selected AlwaysUp based on its feature set, simplicity and cost — it was the best solution for our clients.
How are you using AlwaysUp today?
AlwaysUp is a required part of every implementation of Wellsoft EDIS. To date, we manage over 70 licenses of AlwaysUp in various versions.
What are your favorite features of AlwaysUp?
For Wellsoft: The ease of use, immediate notifications and customer service are right at the top of the list. When we do require assistance, the level of support from Core Technologies is unrivaled.
For our clients: Ease of use; we have found that even our least “techy” clients can use the information provided by AlwaysUp to determine what they can handle and when they need to involve Wellsoft.
Any other comments on working with our company, support, etc?
Yes! We recently used the Activity Report feature to help us locate a memory issue that was causing several of our clients to experience program disruptions. We were able to narrow our focus thanks to AlwaysUp’s weekly statistics, saving us incalculable time.
Our month-long evaluation of AlwaysUp 9.1 running windows services on Windows 10 started soon after the new operating system was released on July 29th. Testing was comprehensive, targeting all major features including:
- Installation & setup
- Sending emails in response to key events
- Running programs in the background in Session 0
- Starting applications interactively in the current session
- Restarting an application if it uses too much memory
- Detecting CPU spikes
- Responding to customized failure detection scripts (“sanity checks”)
- Automation in the current session with AutoIt
- Interaction with other windows services installed on the PC
… and much more.
We also focused on running popular applications that have already earned compatibility with Windows 10.
Dropbox continued to synchronize files as normal when run as a windows service:
And Google Drive operated flawlessly as well:
The end result is that AlwaysUp 9.1 is fully compatible with Windows 10!
The only trouble we noticed was when interacting with applications running in Session 0…
The Keyboard and Mouse didn’t work in Session 0
We were able to switch to the isolated Session 0 but couldn’t interact with the application running there. The mouse cursor vanished and the keyboard was unresponsive. We couldn’t even click the “Return now” button to go back to the normal desktop!
Fortunately this bug has already been reported to Microsoft and a resolution may be on the way.
We were unable to see Session 0 when logged in through RDP
The second oddity we encountered was when logged in to our server via RDP. Switching to Session 0, either from the Interactive Services Detection Dialog or from AlwaysUp, resulted in an unhelpful “black screen”. The black screen remained for one minute, until the Session 0 timeout elapsed and we were unceremoniously disconnected from the server. We experienced this with RDP versions 6.2.9200 and 6.3.9600 and our recommended driver updates didn’t resolve the problem. Hopefully Microsoft will fix this one quickly too.
September 19 2015: Switching to Session 0 over RDP is now working on our Windows 10 machines! We think that the cumulative update for Windows 10 issued on September 8 (KB3081455) may have done the trick so please apply that update if you are still having trouble.
Trinium Technologies provides business software for two primary verticals: intermodal trucking companies and fuel marketers. Its applications enable companies to reduce cost, grow revenue, and improve customer service. Customers range in size from small businesses to large enterprises with both cloud services and on-premises deployment options available.
Barry Assadi, the Chief Technical Officer at Trinium, has been using AlwaysUp since 2006. He knows AlwaysUp very well and was interviewed for this article.
Why does Trinium use AlwaysUp?
Trinium uses AlwaysUp to handle processes for file transfer (FTP), file import & export interfaces, and report generation. We used to deploy automated background processes with Windows Task Scheduler, but there were constant stability issues along with a lack of visibility of the status of these processes. AlwaysUp runs our executables without fail, and its detailed reporting tell us how our applications are doing.
Customer service is very important to us, so we have a dedicated AlwaysUp process per customer that is used to keep a heartbeat on all of the services we’ve created. This way we can see if any processes are hung or stopped to avoid any downtime for our customers.
How is AlwaysUp deployed in your business?
Each non-hosted customer will get an installation of AlwaysUp to run our software as a service in their environment. We also run AlwaysUp in Trinium’s hosted private cloud, which has a few dedicated servers reserved for processing using AlwaysUp.
What are your favorite features of AlwaysUp?
Over the years, we’ve come to appreciate the depth of settings to manage the processes we model as a service. There seems to be a solution for everything we’ve thrown at AlwaysUp!
Our more technical customers are happy that they can now stop and restart our applications from their batch files (using the NET command).
And we especially like the ability to multi-select to start/stop/restart processes. It is very convenient as our techs install and work with multiple services per server.
Incorporated in 2007 and driven by new opportunities in the “Internet-of-Things” (IoT) market, CartaSense offers end-to-end monitoring and alerting capabilities for cold-chain, agricultural and asset security applications. CartaSense provides complete field proven systems using wireless sensors, gateways and cloud based computing that deliver real-time information to support real-time decision making for increased efficiency and profits.
Chaim Belfer, the Vice President of R&D at CartaSense, has been using AlwaysUp with his Java programs since 2010. He currently manages more than 20 installations and was interviewed for this article.
How does CartaSense use AlwaysUp?
CartaSense deploys several Java applications that manage a world wide network of sensors. For security and management purposes, we need to run our windows installations as windows services. Looking for a solution we found AlwaysUp which solves the problem — plus helps with recovery and logging.
For the past few years, AlwaysUp has been included whenever we install our software on a Windows Server (2008 or 2012). We also recommend AlwaysUp to our own customers.
Why did you choose AlwaysUp for your Java applications?
We found your company through a Google search a few years ago. I tested several open source solutions but none of them really did the job, or at least needed a lot of development to make it work reliably. AlwaysUp worked well from the beginning (the 30-day trial) so we started out with a single license and bought more over time, as needed.
What are your favorite features of AlwaysUp?
Beyond solving the big problem of running our Java applications as windows services, AlwaysUp’s GUI is very nice and self explanatory. The documentation is also good, and the software is easy to use. Prompt service and quick response to our support emails is a big plus too!