Working in the field of application virtualization as a Citrix consultant, problems with IMA sometimes are the bulk of my support calls. Citrix Xenapp is completely dependent on the IMA service. When IMA fails to start on a server, it can really put a damper on your day. Often we find ourselves asking why Citrix doesn’t just tell you what it means when there is a failure, because Citrix error messages and codes can border on the ridiculously generic. You may encounter a variety of error messages. These errors could range from IMA related errors in the event viewer on the server, to cryptic number code errors when trying to start the IMA service via the services console. There are many reasons that IMA may crash, or choose not to start on a server. For the purposes of troubleshooting Citrix Xenapp, we’ll cover a few of these basic reasons in today’s blog.
Reasons IMA may not start:
- Heavy Load: Sometimes if there is heavy load on a database, it can take longer than the 6 minutes IMA is allotted to start. In these rare cases, IMA may eventually start anyway, and therefore the failed error messages can be disregarded as benign.
- Corrupt Local Host Cache: As most Citrix administrators know, the local host cache on a Citrix server can sometimes get out of line with what is in the central data store. This seems to occur randomly most often in multi-server environments, and sometimes when servers have not been rebooted in a long time. When IMA won’t start, often the first step a savvy Citrix administrator will take is to recreate the local host cache. It’s considered fairly safe and commonplace to recrate the local host cache from the command prompt by issuing the “DSMAINT RecreateLHC” command.
- Corrupt Rade Components: Whether your environment does streaming or not, you might be surprised to know that IMA won’t start if the Rade components are corrupt. If you suspect that this is a possibility, it’s a fairly safe procedure to recreate the Rade components from a command prompt by issuing the “DSMAINT RecreateRade” command. I’ve personally seen this command resolve issues even in environments that weren’t using any of the streaming capabilities included in recent versions of Citrix.
- Corrupt Service Account: Citrix IMA uses the network service account, which sometimes goes corrupt. To verify if this account is working properly, try going into the service console and changing the account that IMA uses from the NSA to the local account. If this resolves the issue, you’ll know you need to rebuild your account.
- Corrupt Datastore: Citrix, after all, is just a big database – and database corruption is a fact of life for those of us in IT. If you’re experiencing IMA problems across multiple servers, you may be experiencing a corrupt data store. There are many options to attempt to repair a corrupt database (I recommend running “DSCHECK /FULL ?” to see the full syntax of commands available), and these commands should be exhausted completely before proceeding further. If you are in a small farm environment with an access database, you can easily recover a backup of the database by issuing the “DSMAINT recover” command from the CLI. Sadly, most of us in the real world are using SQL and recovery options are not so easy. Many of you rely on help from a DBA team, and therefore recovery will involve putting a ticket in to the appropriate group to restore your datastore from a backup. This should be a last resort, as restoring a SQL database from backup can bring it’s own set of headaches.
- Rebuild Link to Data Store: For unknown reasons, sometimes the link between a Xenapp server and it’s SQL datastore can become broken or corrupted. In order to re-establish this link and rebuild ODBC connections, it is sometimes recommended for you to reinitiate connectivity from the command line. You can accomplish this by running the “DSMAINT CONFIG /user:username /pwd:password /dsn:DSNFilename” command from the CLI.
- XML Service Corruption: The degree to which XML and IMA rely on each other is debatable, but what is not up for debate is the experience I have had successfully re-registering the XML service in attempts to solve IMA issues. Sometimes you may notice that the XML service is not started on a trouble server, or sometimes it will show that it’s started in the service console – but if you do a “Netstat –a” command from the CLI, you may see that it’s not listening on the XML port. If you have exhausted other options (Even if you don’t explicitly see XML errors in the event viewer), re-registering the XML service may be worth a shot. Overall, it’s a pretty low risk procedure that can be accomplished by running the, “CTXXMLSS.EXE /u” first to unregister the service, and then the “CTXXMLSS.EXE /r80” command to re-register the service on port 80, or any port of your choice.
- IP/Netbios conflict: A rare cause of IMA issues occurs when there is another server on the network with the same Netbios name or IP address. This can disrupt communication over IMA, and can cause the service to fail to start up.
- Print Spooler: What does IMA have to do with printing, you may ask? Well, if the print spooler is hung – Citrix Admins have experienced problems with IMA operation. You may fix this by restarting the print spooler, or in some cases by locating print drivers on the terminal server that are in a stuck state and deleting them.
- Service Account Permissions: You may think that there is no way permissions could have changed for the service account being used for IMA, but in the field we often find that to be incorrect. You’ll want to verify that the account you have set up for IMA has the “Log on as a service” permission on the local machine, or you may see specific error codes like “2147483647”. Also open Regedit, and give full control rights to the local administrators group on the following key: “HKEY_Local_Machine\Software\Citrix\IMA\Data”.
- Missing Temp Directory: Believe it or not, Citrix IMA can fail to start if the C:\Winnt\Temp folder is missing. If this directory doesn’t exist, try creating it and verifying that the TMP and TEMP system environment variables are pointed to it.
- Was your Xenapp downgraded from Enterprise?: If so, there is a script from Citrix available to correct the resulting issues that may occur: http://support.citrix.com/article/CTX113836
- Event Viewer: Sometimes the Windows event viewer can provide you with some valuable information on why IMA failed to start. More often than not it’s only going to tell you that you’re in trouble (Which, if you are looking in the EV – chances are, you already know).
- Check the registry: Citrix developers did Citrix Admins a favor by including a special registry key on every Citrix server that can help you determine the reason why IMA may fail. In order to use this registry key, simply launch regedit and browse to the “HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\IMA\Runtime\CurrentlyLoadingPlugin” and pay attention to the value in that field. If the value mentions a particular DLL file, it may give you a clue on something that has gone corrupt or is no longer working in the environment. According to Citrix, it should be OK to copy working DLL’s from other servers in the environment to a trouble server for troubleshooting purposes.
- Network Ports: If you suspect that a port that IMA relies on (Such as 2512) may be blocked in your environment, try telnet’ing from one Xenapp server to another over that port. You can use the command, “telnet <server name> 2512” to test connectivity. If the result is a blank black window with a flashing cursor, then your traffic is flowing. If the CLI hangs after typing in the command, chances are there may be some network hardware or a software firewall in the way.
My Server Was Running FINE! What Happened?
Many support calls surrounding Citrix IMA are relayed from Citrix Admins who immediately state that their server has been running fine for years and nothing has changed. While this may be true in cases where the problem was caused by random corruption, this may not always be the case. For this reason, it’s very important to tightly control what Microsoft patches and updates go onto a Citrix server. I’ve seen issues in the field caused by the order of patch install, or by allowing patches after certain Citrix hotfixes have been applied, etc. There does not always seem to be logic to the cause of these types of support calls, so sometimes they’ve got to be troubleshot from the ground up like any other.
The Sledgehammer Solution:
The reason why we spend so much time troubleshooting is to try to avoid having to run a dreaded Citrix repair. When you have exhausted 100% of the other options, and you’ve limited your IMA problems to a single server in the environment – it may be time to push the red button and begin this emergency process. By inserting the Citrix media into the CD-Rom of the server, you can complete the repair wizard in the hopes that it resolves your issue. If you are dealing with operating system corruption or database problems, this process probably won’t help.
Best Practice Tip:
In the future, to avoid having to resort to emergency measures, be sure to keep recent virtual snapshots of all of your Citrix Xenapp servers. This will enable you to roll back to a working snapshot at any given time in a matter of minutes rather than hours of troubleshooting.
CTX101917 – Error: Error: Windows could not start the Independent Management Architecture on Local Computer .. and refer to service-specific error code �2147483647.
CTX105166 – IMA Service Hangs In a Starting State
CTX103015 – IMA Service Fails on MFPrintss.dll
CTX113836 – Downgraded from Enterprise
CTX104200 – Could not start IMA Service in CTX_MF_IMA_StartIMAService
CTX103253 – Error: IMA service failed to start with error 2147483649 and failed to load plug-ins
CTX032712 – Error: IMA Service Error Message -2147483647
CTX101667 – IMA service failed on startup. Service specific error 2147483649
CTX103048 – IMA Service fails to start when a domain or local user is configured for logon
CTX101877 – Error: An error occurred while attempting to start the IMA Service.
CTX735338 – IMA failed to start with error code 2147483649