A series of unfortunate configuration mistakes?
Last Thursday, June 6, 2019 at 12:00 a.m. Dutch time, a two-hour outage began at various customers. Both business customers for fiber optic internet as well as private internet connections. At first, the extent of the disruption was not yet completely clear.
After several reports from customers I noticed that also on Dutch websites (such as allestoringen.nl) the complaints came in from people with connection problems. It soon turned out to be a national outage that primarily affected KPN (and indirectly many other parties including national payment systems).
A first traceroute showed that the traffic between Haarlem and Amsterdam went through different (completely illogical) paths. This had all the symptoms of a BGP hijack, The Dutch website Tweakers.net also mentioned this and expressed this suspicion.
Doing a bit more digging on the route my traffic was taking, i saw that my VPN connecting between Haarlem and Amsterdam was suddenly taking a de-tour through AS4134, known as the Chinanet Backbone Network.
For two hours, much of Europe’s Internet traffic passed through Chinese networks
This incident occurred after a Swiss hosting provider (Safe Host SA) started leaking more then 70.000 faulty routes to the Chinese backbone network. This Chinese network, in turn, forwarded these IP address announcements as valid to various large Tier-1 internet providers causing a huge traffic shift toward the Chinese Backbone network. The incident caused huge impact on the networks of Swisscom (AS3303), KPN (AS1130), Bouygues Telecom (AS5410) and SFR (AS21502).
Error or intention?
The official statement is; a configuration error by the Swiss hoster SafeHost. The remarkable thing is that the incorrectly advertised ranges were smaller and more specific then the ones advertised legitimatly. Some websites mention this could indicate the use of route optimizers but Safe Host SA confirms on twitter that they do not use bgp optimalisation software.
Now that Safe Host itself indicates that it is not the use of route optimizers that caused this disruption, the question arises how it is possible that such specific routes have been (wrongly) advertised. And why only to Chinanet Backbone? Safe Host SA is still investigating and not responding to questions on Twitter.
I am very curious about the explanation of this incident, if it ever comes.
Worldwide concern
Chris C. Demchak and Yuval Shavit released a publication in 2018 entitled “The Hidden Story of China Telecom’s BGP Hijacking”. In this publication they write how China has been able to divert specific traffic through Chinese POPs several times via BGP hijacks and the possible implications this has.
They also describe their concerns about the large (infrastructural) network presence of the Chinese Backbone Network in America (which is no different in the EU) while no other international network has similar presence in China. This large presence is one of the things that makes it easy for the Chinese to conduct a BGP Hijack on large scale and at the same time protecting their own infrastructure.
Using these numerous PoPs, CT (China Telecom) has already relatively seamlessly hijacked domestic US and cross US traffic and redirected it to China over days, weeks, and months as demonstrated in the examples below. The patterns of traffic revealed in traceroute research suggest repetitive IP hijack attacks committed by China Telecom.
The report argues that the Chinese government is using local ISPs for intelligence gathering by systematically hijacking BGP routes to reroute western traffic through its country, where it can log it for later analysis and provides the following examples;
- Starting from February 2016 and for about 6 months, routes from Canada to Korean government sites were hijacked by China Telecom and routed through China
- On October 2016, traffic from several locations in the USA to a large Anglo-American bank headquarters in Milan, Italy was hijacked by China Telecom to China
- Traffic from Sweden and Norway to the Japanese network of a large American news organization was hijacked to China for about 6 weeks in April/May 2017.
- Traffic to the mail server (and other IP addresses) of a large financial company in Thailand was hijacked several times during April, May, and July 2017.
Doug Madory from Oracle confirmed that AS4134 was redirecting traffic in a blogpost posted on 5th of november 2018.
In this blog post, I don’t intend to address the paper’s claims around the motivations of these actions. However, there is truth to the assertion that China Telecom (whether intentionally or not) has misdirected internet traffic (including out of the United States) in recent years. I know because I expended a great deal of effort to stop it in 2017.
How to prevent/secure this?
The problem with incidents like this is that the internet is running on the BGP protocol. BGP is a global protocol running between organizations and country’s crossing international borders. There is no single centralized authority, just internet providers that collaborate based on trust. The internet is intended to be open, transparant and so all ISP’s are trusted to play nice. Furthermore, there are initiatives that can improve the security of BGP in general, but these must be introduced on a large scale.
On the 12th of June, the Dutch National Cyber Security Center issued a report in which it once again shows how great the (digital) risks are for the Dutch society. In this report they also specifically mention the influence of China (among others).
“Countries such as China, Iran and Russia have offensive cyber programs against the Netherlands. This means that these countries are digital use resources to achieve both geopolitical and economic objectives to be achieved at the expense of Dutch interests “
It is time to immediately acknowledge and address these risks.
