ArenaNet:Developer updates/20070701

This is an archive of Developer Updates, originally posted through the official website. It includes semi-official announcements, clarifications, and news from the developers.

Latency (Lag) Issues -- 01 July 2007; 1:00 PM Pacific Time (-7 GMT)

We wanted to let players know that we are aware that some of them have experienced latency (lag) over the last few days when playing Guild Wars. These issues appear to be nearly exclusively related to playing on or connecting through the European servers for Guild Wars. What we have learned is that there is an issue affecting many services that route through the Internet hub in Frankfurt, Germany. Some programs are not affected, but unfortunately, for some players, latency has become a factor within Guild Wars.

Our network teams report that resolving the issue is, with regrets, outside our direct control. However, the team members are tracking the matter and doing what they can to assure that it is prioritized and resolved. The network teams suggest that running a network diagnostic and submitting it with a Support ticket would help very much, as it allows them to learn and relay precisely where the bottlenecks are. You can learn about obtaining a DxDiag here. Simply attach that file to a Support ticket--please be sure to include your physical location and the server on which you are playing--and you will have helped greatly in working through this matter.

We apologize for this inconvenience and will do all that we can to assure that this problem will be resolved as quickly as possible.

Addendum 1 -- 1 July 9:30 PM Pacific (-7 GMT): The Network Team has just sent word that the issue is broader than once believed, and they're focusing on a previously-scheduled maintenance in the system that manages the hand-off between the core Internet switch and ArenaNet's infrastructure that connects Guild Wars hardware. This maintenance had already been scheduled for tomorrow night, but it's now hoped that beyond the original objectives of the maintenance, it will also assist in clearing up the current latency problem. I will post as/when we have more information.

Addendum 2 -- 2 July 10:30 PM Pacific (-7 GMT): Our network team reports that today's scheduled maintenance went as planned and so far, all looks well. There are some configuration and compatibility matters that are now full aligned, and the stability for about the last hour has been solid. The logs they were getting from earlier errors are not reoccurring, which is a positive sign that the maintenance fixed the issues.

If any European player has comments or experience reports since the maintenance was complete, I'd be delighted to pass them along to the network team. Thank you again for your patience while we worked to resolve this issue.

Addendum 3 -- 7 July 4:00 PM Pacific (-7 GMT): The network teams are continuing to work on the latency/lag problem. I'm seeing emails pass by every few hours, and there are at least 50 people receiving these mails, many of whom are working directly or indirectly on trying to find a fix. Support has confirmed that sending the DxDiag (mentioned in the initial post) via a support ticket would be extremely helpful. Including a tracert is good, too, for as they say, "The more information we have, the better." The reports Support previously received are assisting in a process of elimination about the causes of this issue. I'm not at all sure if this is internal, external, or a combination of both. We're aware that the same person can play in the same house, same ISP, but a different computers and experience two differing levels of latency.

The team is very sorry that this problem has lasted so long, and you can be assured that the longevity of this issue is not related to a lack of concern about it, but is directly connected to the complexity of finding out what the issue is.

Addendum 4 -- 10 July 6:00 PM Pacific (-7 GMT): The latest from the network teams is that they have implemented a work-around with some of the components involved with our external hosts. This is because the hosts state that it will be a week before they can work on the problem. Our network teams believe that if the work-around their members devised proves to be effective, then they can use the information to expedite vendor response time. So far, network feels that the work-around has been helpful for about 50% of players, but they're tracking the incoming support tickets and looking for more -diag files and tracerts to establish why the other 50% continue to have problems.

It is unclear if lag reports from a few Americans are connected to this problem. That seems unlikely since the problem is sourced in Germany, but we will look into those matters, as well, particularly if we have the needed tracerts or -diag to help us better pin down the details. Thank you for your continued patience.

Addendum 5 -- 14 July 10:00 AM Pacific (17:00 GMT): The lag issue was resolved yesterday, as reflected in the Network News page update of Friday, July 13. However, by early morning today, we started getting reports of lag incidents, and lag continues to be an issue amongst European players. We have alerted network teams on both side of the Atlantic and are working with our vendor to resolve this issue. They would be very grateful for the receipt of any tracerts or -diag files via the Support system, as the more information they have, the more clearly they can identify and resolve this problem. We are deeply sorry for the continuation of the issue and the network teams will work on this around the clock until it is resolved.

Addendum 6 -- 16 July 12:30 AM Pacific (7:30 GMT): The Network team undertook emergency maintenance measures at approximately Midnight PDT July 15 (7:00 GMT) as part of the effort to correct the lag issue. This maintenance will have disconnected each player, but he or she should have been able to reconnect within two or three minutes. We will update with more information later. Players with continuing issues are encouraged to submit or update a Support Ticket with as many details as possible. Thank you.

Addendum 7 -- 17 July 2:50 PM Pacific (21:50 GMT): Here is some very good news, which we are posting on the Network News Page of the official website: Over the course of the last week or so, an issue with a core router in Frankfurt, Germany, has resulted in latency for a significant number of Guild Wars players as well as those playing other NCsoft games. We have worked with the router vendor and they have made changes to the router that has corrected the issue. We are very grateful to the players who submitted reports and network diagnoses files to our support team, for those reports were useful in determining the cause of the problem and in helping us initiate the necessary repairs. Game service is now back to normal, and we thank all players for their patience as we resolved this issue.