SecurityBrief Asia logo
Asia's leading source of cybersecurity and cyber-attack news
Story image

Top internet outages of 2019 - ThousandEyes

By Newsdesk
Wed 15 Jan 2020
FYI, this story is more than a year old

ThousandEyes has announced last year’s most disruptive outages and what the industry can learn from them.

When Internet outages happen, they can be extremely disruptive to a business.

By preventing users from reaching applications and services, outages can cause major revenue and reputation damage.

While application delivery is dependent on many Internet Service Providers (ISPs), it also increasingly relies on a large and complex ecosystem of Internet-facing services — such CDN, DNS, DDoS mitigation, and public cloud.

These services work together to provide exceptional digital experiences to users and even brief disruptions can have a significant impact.

“At the same time, enterprises are increasingly relying on Internet transport to connect their sites and reach business critical applications and services. Gone are the days in which applications are solely hosted in private data centers and office locations are connected primarily by MPLS circuits,” says ThousandEyes product marketing director Angelique Medina.

“The Internet is replacing or supplementing services like MPLS as enterprises embrace SD-WAN technologies. As a result, the Internet is now effectively the enterprise backbone, which as a “best effort” transport can have significant yet unforeseen consequences for businesses.”

Over the last year, ThousandEyes has reported on a number of large-scale outages that had ripple effects across the global Internet, impacting enterprises and consumers alike.

The most significant of these outages took place over the northern hemisphere summer and disrupted nearly every top tech company in some fashion.

May 13, 2019 — China telecom outage reveals its global reach

While not the most disruptive outage of 2019, a global and fairly long lasting outage in China Telecom’s network proved to be a harbinger of incidents to come, while also revealing a lesson in how China Telecom’s reach extends far beyond mainland China.

For nearly five hours on May 13, 2019, China Telecom experience substantial packet loss across its backbone, primarily impacting network infrastructure in mainland China, but also affecting China Telecom’s network in Singapore and multiple points in the US, including Los Angeles. Over one hundred services were disrupted. Though not exclusively impacting western sites and services, many users of major US brands such as Apple, Amazon, Microsoft, Slack, Workday, SAP, would have experienced disruptions over the course of the outage window.

This incident illustrated some important realities about China and its impact on the global Internet that many folks aren’t aware of. Specifically, it highlighted that many of the censorship policies that apply to Chinese Internet users may actually be implemented far beyond China’s borders and in countries that have very different attitudes and policies related to Internet use.

June 2, 2019 — Summer of outages begins with Google Cloud

On June 2, 2019, Google Cloud Platform experienced a significant network outage that impacted services hosted in parts of us-west, us-east and us-central regions. This outage impacted Google’s own applications, including GSuite and YouTube. The outage lasted more than four hours, which becomes notable given the criticality of its service to business customers.

Google issued an official report of the incident several days later. ThousandEyes vantage points were able to see the outage as it unfolded in real time, revealing its characteristics and scale ahead of more detailed information becoming publicly available.

Beginning at approximately 9am ET in the US, ThousandEyes observed 100% packet loss from global monitors attempting to connect to a service hosted in GCP us-west2-a. Similar losses were seen for sites hosted in several portions of GCP US East, including us-east4-c.

The complete unavailability of parts of Google’s network, as seen by ThousandEyes, turned out to be due to Google’s network control plane inadvertently getting taken offline. Google later revealed that during the outage period, a set of automated policies determined which services were or were not reachable through the unaffected parts of its network.

One of the most important takeaways from cloud outages is that it’s vitally important to ensure your cloud architecture has sufficient resiliency measures, whether on a multi-region basis or even multi-cloud basis, to protect from future recurrence of outages. It’s reasonable to expect that IT infrastructure and services will sometimes have outages, even in the cloud.

June 6, 2019 — An unfortunate series of events takes down WhatsApp for many users

On June 6, 2019, a large number of users around the globe attempting to access the WhatsApp service experienced connectivity issues. ThousandEyes was able to immediately see that 100% packet loss was preventing the service’s reachability.

Upon further analysis, ThousandEyes determined the root cause of this packet loss was a massive route leak that steered traffic to China Telecom — a service provider that does not forward any Facebook-related traffic.

The incident was triggered when a Swiss colocation company called Safe Host announced to the Internet that the best way to reach WhatsApp and thousands of IP prefixes was through its network, AS 21217. When Safe Host advertised these routes, they were accepted by China Telecom and further propagated through other ISPs such as Cogent.

Users whose traffic was routed to Cogent and ultimately handed off to China Telecom would have been completely unable to reach the service.

It’s unclear why China Telecom would accept routes to a service that it censors, but what is clear is the lesson of this outage. BGP route leaks are not uncommon on the Internet. When you rely on the Internet, an ecosystem that is deeply interconnected and vulnerable, you must understand how it works and expect that a glitch in one service provider can have cascading effects on another.

The unfortunate reality is that business risks associated with BGP route leaks and other Internet flaws are greater given the modern enterprise and service delivery landscape.

June 24, 2019 — Cloudflare users fall victim to routing mishap

Just a couple of weeks after the massive route leak that impacted WhatsApp users, the Internet experienced yet another route-related incident - this one far more damaging.

On June 24, 2019, for nearly two hours, a significant BGP routing error impacted users trying to access services fronted by CDN provider Cloudflare, including gaming platforms Discord and Nintendo Life.

ThousandEyes analysis found that a significant BGP route leak affected a variety of prefixes from multiple providers. DQE, a transit provider, was the original source of the route leak, which was propagated through Allegheny Technologies, a customer of both DQE and Verizon. Unfortunately, Verizon further propagated the route leak, magnifying the impact.

Sites served through the CloudFlare CDN were impacted for nearly two hours.

This major Internet disruption affected about 15% of Cloudflare’s global traffic and impacted services like Discord, Facebook and Reddit. The route leak also affected access to some AWS services.

The root cause of the incident was eventually traced to DQE’s use of a BGP optimiser software that created routes to Cloudflare services that were only meant to be used within DQE’s internal network. When these routes were accidentally leaked to one of its customers, mayhem ensued.

This incident was yet another reminder of how incredibly easy it is to dramatically alter the Internet service delivery landscape. In a cloud-centric world, enterprises must have visibility into the Internet if they’re going to be successful in delivering services to their users.

July 4, 2019 — Apple Services impacted on fourth of July

On July 4, 2019, just before 9am PT, users connecting to Apple’s website and some of its services, such as Apple Pay, began experiencing significant packet loss for a period of over 90 minutes. This issue prevented many users from successfully connecting to Apple. ThousandEyes route visibility demonstrated that the packet loss was caused by a BGP route flap. A BGP route flap is caused when a routing announcement is made and withdrawn in quick succession, often repeatedly.

While Apple services are certainly important for many Internet users, the fact that the incident occurred early on a holiday seems to have prevented the incident from sparking more than a few user complaints. The lesson from this incident is that outages don’t happen in a vacuum. Sometimes even significant outages may go unnoticed (or conversely create significant uproar) simply based on their timing and context.

September 6, 2019 — DDoS attackers target the Internet’s knowledge base

On September 6, 2019, access to Wikipedia sites from around the world was disrupted for close to nine hours, the result of a massive and sustained Distributed Denial of Service (DDoS) attack. DDoS attacks can overwhelm their target’s web infrastructure and also create congestion within service provider networks that can lead to packet loss. These effects are exactly what ThousandEyes observed when Wikipedia came under attack.

During the course of the incident, ThousandEyes saw a significant drop in HTTP server availability from around the world, as well as a dramatic increase in HTTP response times. As a result, users across many regions were unable to establish an Internet connection for ongoing communication with Wikipedia servers. ThousandEyes also measured packet loss of up to 60% from our global vantage points, a condition that would have further prevented access to Wikipedia sites.

While DDoS events are an unfortunate reality of operating on the Internet, organisations should have visibility into the scope, impact and behaviour of these events and be able to validate that DDoS mitigation steps are effective.

Related stories
Top stories
Story image
Delinea’s Joseph Carson recognised with OnCon Icon Award
Delinea chief security scientist and advisory CISO Joseph Carson has been recognised as a Top 50 Information Security Professional in the 2022 OnCon Icon Awards.
Story image
Artificial Intelligence
Eight top DevSecOps trends to support IT innovation in 2022
The use of DevSecOps practices is growing, as it is increasingly seen as the best way to produce high-quality and secure code. So what are the current trends?
Story image
Artificial Intelligence
Vectra AI named as AWS security competency partner
Threat detection and response company Vectra AI has announced that it has become an Amazon Web Services Security Competency Partner.
Story image
Businesses unprepared to defend against ransomware attacks
Ransomware attacks continue to impact organisations worldwide with high costs, but businesses are still largely unprepared.
Story image
Hundreds arrested, millions seized in global INTERPOL investigation
A two-month-long investigation by INTERPOL this year involved 76 countries and clamped down on organised crime groups behind telecommunications and social engineering scams.
Story image
Network Security
Netskope announces zero trust network access updates
Customers can now apply zero trust principles across a range of hybrid work security needs, including SaaS, IaaS, private applications, and endpoint devices.
Story image
Gartner's top recommendations for security leaders
"Leaders now recognise that major disruption is only one crisis away. We can’t control it, but we can evolve our thinking, philosophy, program and architecture.”
Story image
Identity and Access Management
Ping Identity launches corporate venture capital fund
Ping Identity has launched a corporate venture capital fund to foster innovative offerings for the identity security market.
Story image
Internet of Things
ManageEngine wins big in IDC MarketScape assessment
ManageEngine's Endpoint Central service has been recognised as a leader by IDC MarketScape in several categories including Internet of Things device deployments and UEM software for SMEs.
Story image
Email threats spike 101%, remains a top attack vector
"Each year we see innovation in the threat landscape, but each year email remains a major threat to organisations."
Story image
Sternum joins NXP, collaborates on IoT security and observability
Sternum has announced it has joined the software partner community of NXP Semiconductors, a manufacturer of and large marketplace for embedded controllers.
Story image
Preparing for the digital decade with the right workforce strategies
For a decade that started under the pall of the pandemic, the 2020s is poised to end with a bang with the digital economy swelling to a high across the world.
Story image
Flashpoint unveils security offering for school boards
Flashpoint has released its K-12 risk management and security offering to provide school boards and education security practitioners with tools to recognise, prevent and manage cyber and physical threats.
Story image
Varonis strengthens security capabilities for AWS and S3
Varonis has strengthened and expanded its cloud and security capabilities, with a critical aim of improving safety and boosting data visibility in Amazon Simple Storage Service (S3).
Story image
Oracle Cloud
Commvault, Oracle to deliver Metallic Data Management as a Service
"We are excited to partner with Commvault and enable our customers to restore and recover their most mission-critical cloud data."
Story image
Without trust, your security team is dead in the water
The rise of cyberattacks has increased the need for sound security that works across any type of business, but with any change, buy-in is essential. Airwallex explains why.
Story image
IT and security team collaboration crucial to data security
Many IT and security decision makers are not collaborating as effectively as possible to address growing cyber threats.
Story image
New study reveals 51% of employees using unauthorised apps
The research shows that 92% of employees and managers in large enterprises want full control over applications, but they don't have it.
Story image
Tech and data’s role in the changing face of compliance
Accenture's study found that 93% of respondents agree or strongly agree new technologies such as AI and cloud make compliance easier.
Story image
Tech job moves
Tech job moves - Bitdefender, Cohesity, Fortinet & MODIFI
We round up all job appointments from June 27-30, 2022, in one place to keep you updated with the latest from across the tech industries.
Story image
Secure access service edge / SASE
Cloudflare adds new capabilities to zero trust SASE platform
New features for Cloudflare One include email security protection, data loss prevention tools, cloud access security broker, and private network discovery.
Story image
Digital Transformation
What CISOs think about cyber security, visibility and cloud
Seeking to uncover the minds of CISOs and CIOs across Asia Pacific, my company recently asked Frost & Sullivan to take a snapshot of cloud adoption behaviour in the region.
Story image
HP Inc
Firmware attacks significant threat in age of hybrid work
Changing workforce dynamics are creating new challenges for IT teams around firmware security, according to new research.
Story image
Amazon Web Services / AWS
Zscaler, AWS accelerate onramp to the cloud with zero trust
Zscaler has announced an extension to its relationship with Amazon Web Services, as well as innovations built on Zscaler's Zero Trust architecture.
Find out how you and your business can prevent being caught out by everything from ransomware to cryptojacking.
Link image
Story image
WatchGuard Technologies
Ransomware volume doubled 2021 total by end of Q1 2022
Ransomware detections in the first quarter of this year doubled the total volume reported for 2021, according to a new report. 
Story image
Motorola Solutions
Motorola Solutions deploys communication system to 5th Japanese airport
Motorola Solutions with its partner, Nippon Airport Radio Services deployed the mission critical communication system to Kansai.
Story image
Threat actors ramp up their social engineering attacks
As people get better at identifying potential threats in their inbox, threat actors must evolve their methods. Their new M.O? Social engineering.
Story image
Digital Transformation
Cybersecurity priorities for digital leaders navigating digital transformation
In recent years, Asia-Pacific has especially been a hotspot for cyberattacks, and as we continue into 2022, it’s evident that the problem is becoming more significant.
Story image
Cyber Criminal
Identity and access: the fight is on
Blue team defenders are used to protecting our data, applications, and users with access controls and other security mechanisms, which is why attacks like this are especially challenging when they target identity and access control systems.
Story image
Cloud Security
Palo Alto Networks bolsters cloud native security offerings
Latest Prisma Cloud platform updates help organisations continuously monitor and secure web applications with maximum flexibility.
Story image
Zero trust security adoption rises 27% in just two years
A survey of WAN managers has revealed that multi-factor authentication and single sign-on are the top zero trust features implemented.
Story image
Aqua Security, CIS create software supply chain security guide
Aqua Securityand the Center for Internet Security have together released the industry’s first formal guidelines for software supply chain security.
Story image
Stock security features inadequate in face of rising risk
"Organisations must proactively find ways of identifying unseen vulnerabilities and should take a diligent, holistic approach to cybersecurity."
Story image
Forescout reveals top vulnerabilities impacting OT vendors
Forescout’s Vedere Labs has disclosed OT: ICEFALL, naming 56 vulnerabilities affecting devices from 10 operational technology vendors.
Story image
Internet of Things
Domino's Pizza: A blueprint for secure enterprise IoT deployment
Increasingly, organisations are embracing smart technologies to underpin innovations that can enhance safety and productivity in every part of our lives, from industrial systems, utilities, and building management to various forms of business enablement.
Story image
FIDO Alliance releases guidelines for optimising UX with FIDO Security Keys
The new guidelines aim to accelerate multi-factor authentication deployment and adoption with FIDO security keys.
Story image
Industry-first comprehensive risk-based API security enhances protection
Application Programming Interfaces (APIs) have become a crucial part of operating web and mobile application businesses and are causing significant economic growth in the digital sector.
Story image
Significant security concerns resulting from open source software ubiquity
"The risk is real, and the industry must work closely together in order to move away from poor open source or software supply chain security practices."
Story image
Artificial Intelligence
Juniper study reveals top AI trends in APAC region
Juniper's research shows an increase in enterprise artificial intelligence adoption over the last 12 months is yielding tangible benefits to organisations.
Story image
Trend Micro
5G network projects driven by improving security and privacy
Trend Micro's new study reveals the prospect of improved security and privacy capabilities are the main motivations behind private 5G wireless network projects.
Story image
Kaspersky opens three new centers to boost data management
Cybersecurity company Kaspersky has opened three new Transparency Centers, one in Japan, the second in Singapore and the third in the United States.
Story image
SentinelOne integrates with Torq to empower security teams
"With Torq, security teams can extend the power of SentinelOne to systems across the organisation to benefit from a proactive security posture.”
Story image
Tech job moves
Tech job moves - ActiveCampaign, Arcserve, LogRhythm & Qlik
We round up all job appointments from June 17-22, 2022, in one place to keep you updated with the latest from across the tech industries.