Connectivity issues across our platform

September 3, 2024 at 5:20 PM UTC

Game Shield Remote Shield API Panel

---

At 17:19 UTC monitoring alerted us of several connection issues mainly concerning EU and US East.

At 17:24 UTC our head staff was aware of the issues and started investigating

At 17:35 UTC the main issue had been identified by our staff

At 17:38 UTC we started rolling out updates globally

At 17:59 UTC the rollout was finished

At 18:05 UTC the issue refaced but only with limited impact to Frankfurt

As of ~18:10 UTC other issues started to take place (API unavailability etc.)

We addressed all of those till ~20:00 UTC

Next day (the 4th) -

At 9:46 UTC issues returned to appear, as it was the day before, we only saw issues isolated to Frankfurt and partially Paris

At 9:48 UTC our staff was aware of the incident happening again and was taking action to prevent further impact as best as possible

At 12:10 UTC no issues were appearing anymore

At midnight UTC we then put Amsterdam into production to help reduce the load towards Frankfurt

Post-mortem:

The incidents and issues were caused by a major attack that was complex in both the vector and distribution of the attack.

It consisted of a total of ~1 Billion packets per Second peak, all TCP, more than 700 thousand source IPs participating (we cannot tell you the exact number), opening legitimate TCP connections, then sending a lot of packets within that connection. The main issue was the sheer size/balancing of the attack. More than 65% came from Russia, thus landed in Frankfurt and caused issues solely there as we just did not have the performance needed to defend ~600 Mpps in that PoP alone. We are looking forward to major expansions in Frankfurt soon, those will also be expedited due to this incident.

The attack vector itself was already known, detected and blocked.

Urgent efforts and steps to help eradicated the attack and balance the sheer load of the attack were done since the first few minutes of the attack taking place. We were able to find and implement a more performant algorithm, dropping the attack faster. We also tuned our affected prefixe’s community profiles better to ensure better anycast distribution. Along with that, we mobilized all staff members available to make sure Amsterdam was being put in production asap.

Attack details and other details were reported to the German State Office of Criminal Investigation (LKA) and we cooperated thoroughly with them to ensure they are aware of all available information.

We forwarded the most effective algorithm to eradicate this attack to our Upstream who will implement it for all our announced prefixes in a timely manner which will help to reduce the load on our hardware by a lot so the impact is zero. Please know that we are continuously working on protection enhancements, performance tests and are debugging thoroughly with clients on any issues arising.

Asia was not at all affected by this, part of the US was affected for ~10 seconds. Bucharest or Warsaw also were not affected, “only” Frankfurt was affected and Paris partially as well.