Over the course of our time in the field spent working on Citrix Xenserver issues, we have encountered networking problems within Xenserver a few times. These networking problems included symptoms like packet loss in VM to VM communications across a Xenserver host. Some of the Xenserver networking problems included packet loss or latency from a Xenserver host’s physical interface out to the WAN. In order to make troubleshooting a little easier for us next time, and to share with the community – We’ve compiled this little Xenserver networking troubleshooting guide:
Step 1: Collect a server status report from the Xenserver host, and upload it to the Citrix automated tool for analysis online at http://taas.citrix.com — This is a new feature that Citrix has added to their website, which will analyze Citrix Xenserver status reports and provide quick feedback. Some of the feedback you might get would be to update your BIOS on the xenserver host, or maybe change a network interface driver out, etc. We’ve seen recommendations to change BIOS settings also, as well as recommendations to engage the physical hardware manufacturer for support. Overall this has been a very useful tool that Citrix has made available via the web!
Step 2: If the TAAS site comes back with recommendations for you, that is the easy part. You can follow those recommendations and re-test to see if you have resolved the issue. If no major recommendations are made, which seems to be the case at least half the time — You’ll need to continue troubleshooting the old fashioned way, by manually guessing and checking. Some things that are worth a try anyway would be to verify that you have the latest BIOS version on your server, and see if there are any alternative network adapter drivers available. While you might think that contacting the host manufacturer would be the best path here, we’ve actually found that Xenserver NIC drivers need to be custom packaged by Citrix most of the time, so you might as well start there. You can either search http://support.citrix.com for a new driver, or you may need to call into support in order to request that an unreleased driver be given to you, or a new driver be packaged for your specific issue.
Step 3: It’s always a good idea to check for available hotfixes from Citrix to see if something has been released for the particular Xenserver networking problem you have encountered. While the service packs provided for Xenserver usually fix all of the major issues, there are post service pack hotfixes that come out between service packs which may need your consideration for resolving issues in the meantime.
Step 4: If software, drivers and BIOS issues have been ruled out completely, you should start double and triple checking physical hardware for issues. This might include requesting a field engineer from your hardware supplier to come out and test your Xenserver hosts for problems. More than once we’ve found faulty NIC cards to be the cause of a Xenserver networking problem. Beyond that, check cabling, switches, and anything plugged into your Xenserver host for issues. By using guess and check, you should be able to quickly rule out networking as the cause of your issue.
If you end up talking to Citrix support on a case involving Xenserver network problems, it will be a good idea to provide proof of the packet loss in the form of a Wireshark or other type of network trace. They may need to collect a copy of your Vdisk(s) in order to test the problem you are having internally, and help rule hardware in or out. In the meantime, always make sure that the hardware you are using is on the hardware compatibility list for Xenserver, and that you have support available from your hardware manufacturer. Many cases that come in regarding a Xenserver networking problem don’t turn out to be Citrix related at all, so it’s good to keep an open mind and troubleshoot methodically.