Tuesday, April 18th, 2017

Time	Event
12:00p	How The New York Times Handled Unprecedented Election-Night Traffic Spike When he woke up the morning of October 21, 2016, Nick Rockwell did the same thing he had done first thing every morning since The New York Times hired him as CTO: he opened The Times’ app on his phone. Nothing loaded. The app was down along with BBC, CNN, Fox News, The Guardian, and a long list of other web services, taken out by the largest DDoS attack in history of the internet. An army of infected IP cameras, DVRs, modems, and other connected devices – the Mirai botnet – had flooded servers of the DNS registrar Dyn in 17 data centers, halting a huge number of sites and mobile apps that depended on it for letting their users’ computers know how to find them online. The outage had started only about five minutes before Rockwell saw the blank screen on his phone. His team kicked off a standard process that was in place for such outages, failing over to The Times’ internal DNS hosted in two of its four data centers in the US. The mobile app and the main site were back online about 45 minutes after they had gone down. While going through the fairly routine recovery process, however, something was really worrying Rockwell. The thing was, he didn’t know whether the attack was directed at many targets or at the Times specifically. If it was the latter, the effect could be catastrophic; its internal DNS wouldn’t hold against a major DDoS for more than five seconds. “It would’ve been incredibly easy to DDoS our infrastructure,” he said in a phone interview with Data Center Knowledge. See also: How to Survive a Cloud Meltdown His team had been a few months deep into fixing the vulnerability, but they weren’t finished. “We were OK [in the end], but we were vulnerable during that time.” The process to fix it started as they were preparing for the 2016 presidential election. Election night is the biggest event for every major news outlet, and Rockwell was determined to avoid the 2012 election night fiasco, when the site went down, unable to handle the spike in traffic. One of the steps the team decided to take in preparation for November 2016 was to fully integrate a CDN (Content Delivery Network). CDN operators, such as Akamai, CloudFlare, or CDN services by cloud providers Amazon, Microsoft, and Google, store their clients’ most popular content in data centers close to where many of their end users are located – so-called edge data centers — from where “last-mile” internet service providers deliver that content to its final destinations. A CDN essentially becomes a highly distributed extension of your network, adding compute, storage, and bandwidth capacity in many metros around the world. See also: How Edge Data Center Providers are Changing the Internet’s Geography That a CDN had not been integrated into the organization’s infrastructure came as a big surprise to Rockwell, who joined in 2015, after 10 months as CTO at another big publisher, Condé Nast. While at Condé Nast, he switched the publisher from a major CDN provider to a lesser-known CDN by a company called Fastly. He has since become an unapologetically big fan of the San Francisco-based startup, which now also delivers content to people reading The New York Times online. Being highly distributed by design puts CDNs in good position to help their customers handle big traffic spikes, be it legitimate traffic generated by a big news event or a malicious DDoS attack. (Rockwell said he did wonder, as the Dyn attack was unfolding, whether it was a rehearsal for election night.) Fastly ensured that on the night Donald Trump beat Hillary Clinton, the Times rolled without incident through a traffic spike of unprecedented size for the publisher: an 8,371 percent increase in the number of people visiting the site simultaneously, according to the CTO. The CDN has also mostly absorbed the much higher levels of day-to-day traffic The Times has seen since the election as it covers the Trump administration. The six-year-old startup, which this year crossed the $100 million annualized revenue run-rate threshold, designed its platform to give users a detailed picture of the way their traffic flows through its CDN and lots of control. Artur Bergman, Fastly’s founder and CEO, said the platform enables a user to treat the edge of their network the same way they treat their own data centers or cloud infrastructure. In your own data center you have full control of your tools for improving your network’s security and performance (things like firewalls and load balancers), Bergman explained in an interview with Data Center Knowledge. While you maintain that level of control in the public cloud, you don’t necessarily have it at the edge, he said. Traditionally, CDNs have offered customers little visibility into their infrastructure, so even differentiating between a legitimate traffic spike and a DDoS attack has been hard to do quickly. Fastly gives users log access in real-time so they can see exactly what is happening to their edge nodes and make critical decisions quickly. The startup today unveiled an edge cloud platform, designed to enable developers to deploy code in edge data centers instantly, without having to worry about scaling their edge infrastructure as their applications grow. It also announced a collaboration with Google Cloud Platform, pairing its platform with the giant’s enterprise cloud infrastructure services around the world. GCP is one of two cloud providers The New York Times is using. The other one is Amazon Web Services. Today, the publisher’s infrastructure consists of three leased data centers in Newark, Boston, and Seattle, and one facility it owns and operates on its own, located in the New York Times building in Times Square, Rockwell said. The company uses a virtual private cloud by AWS and some of its public cloud services in addition to running some applications in the Google Cloud. This setup is not staying for long, however. Rockwell’s team is working to shut down the three leased data centers, moving most of its workloads onto GCP and AWS, with Fastly managing content delivery at the edge. Google’s cloud is also going to play a much bigger role than it does today. The plan is to run apps that depend on Oracle databases in AWS, while everything else, save for a few exceptions (primarily packaged enterprise IT apps), will run in app containers on GCP, orchestrated by Kubernetes. As he works to sort out what he in a conference presentation referred to as the “jumbled mess” that is The Times’ current infrastructure, Rockwell no longer worries about DDoS attacks. Luckily for his team, there was no major DDoS attack on The Times between the day he came on board and the day Fastly started delivering the publisher’s content to its readers. Whether there was one after Fastly was implemented is irrelevant to him. “It’s no longer something I have to think about.” (Comment on this)
5:23p	The New York Times to Replace Data Centers with Google Cloud, AWS Add The New York Times to the quickly growing list of companies replacing their own data centers with public cloud services. As it continues to modernize its infrastructure, the publisher is planning to shut down three of the four data centers hosting its content and internal applications in the near future, migrating most of the workloads to Google Cloud Platform and Amazon Web Services, Nick Rockwell, the company’s CTO, said in a phone interview with Data Center Knowledge. The Times is a rare relative success story in the digital age among traditional big media organizations, all of whom have been hit hard by the internet’s rise and the virtual takeover of the advertising market by Google and Facebook. The publisher has managed to temper steep declines in print advertising revenue with a growing digital subscription business (the last presidential election and The Times’ subsequent coverage of the Trump administration have boosted that growth substantially) and by expanding into new markets, such as its recent acquisition of the product review site Wirecutter. This success story, however, has been powered by an outdated infrastructure backend, which Rockwell recently described as a “jumbled mess.” He joined near the end of 2015 after 10 months as CTO at Condé Nast, and since then his team has been working to bring that jumbled mess up to today’s standards. And that usually means shifting as many applications as you possibly can to the cloud. The Times already uses a virtual private cloud in AWS and a variety of Amazon’s public cloud services in addition to running some apps in the Google cloud. In addition, however, it has cages of equipment in leased data centers in Newark, Boston, and Seattle, as well as its own internal data center at The New York Times building in Times Square. See also: How The New York Times Handled Unprecedented Election-Night Traffic Spike Rockwell’s plan is to shut down the three leased sites, keeping only the internal facility in New York, which primarily hosts infrastructure for video editing, network equipment, and a few older applications that are hard to move to the cloud. All applications that depend on Oracle databases will be deployed on AWS, while most of everything else will run in containers on GCP, orchestrated by Kubernetes. “Plus, some other apps that we prefer to run on [virtual machine] instances will probably remain in AWS, mainly packaged enterprise IT apps,” he wrote in an email. A critical piece of the publisher’s modernized infrastructure is a CDN (Content Delivery Network) operated by the San Francisco-based startup called Fastly. It caches clients’ content on SSD drives in edge data centers located in major metros around the world and provides an unusually high level of visibility into the way their traffic flows through its network – a key differentiator from the big traditional CDNs. Using Fastly further reduces the amount of infrastructure The Times needs to deploy, in the cloud or otherwise. The service that provides the primary API for mobile push notifications for its app, for example, sends out 25 million to 30 million notifications per minute when news breaks. AWS cloud infrastructure to support this service had to be scaled roughly 40 times typical load, costing the company about $25,000 per month, Rockwell said. His team is working to switch that service to GCP in combination with Fastly (the startup is a GCP partner). He expects the new setup to bring the price tag for delivering the service down to about $5,000 per month, the cost of Fastly’s service in this particular instance being “immaterial.” See also: Can Google Lure More Enterprises Inside Its Data Centers (Comment on this)
6:57p	Opinion: De-Electrification of the American Economy Justin Fox (Bloomberg View) — For more than a century after the advent of commercial electrical power in the late 1800s, electricity use in the U.S. rose and rose and rose. Sure, there were pauses during recessions, but the general trajectory was up. Until 2007. The initial drop in electricity use in 2008 and 2009 could be attributed partly to the economic downturn. But the economy grew again in 2010, and every year since. Electricity use in the US, meanwhile, is still below its 2007 level, and seemingly flatlining. The change is even more dramatic if you measure per-capita electricity use, which has fallen for six years in a row. We’re now back to the levels of the mid-1990s, and seemingly headed lower. This is a really big deal! For one thing, it’s yet another explanation — along with tighter federal emissions rules, the natural gas fracking boom, and the rise of solar and wind power — for why the past few years have been so tough on coal miners. It means that even a big pro-coal policy shift from Washington may not result in higher demand for thermal coal. For another, it seems to settle a turn-of-the-millennium debate about the electricity demands of the digital economy. See also: Digital Realty: Trump’s Climate Order Won’t Change Firm’s Clean Energy Goals Equinix to Press on With Clean Energy Goals after Trump’s Climate Order Businessman and technology analyst Mark P. Mills, now a senior fellow at the right-leaning Manhattan Institute, kicked things off in 1999 with a report stating that computers and the internet were already responsible for 13 percent of U.S. electricity demand and would be consuming 30 percent to 50 percent within two decades. In a subsequent op-ed for Forbes, charmingly titled “Dig More Coal — the PCs are Coming,” he and fellow Manhattan Instituter Peter W. Huber argued that: “Yes, today’s microprocessors are much more efficient than their forerunners at turning electricity into computations. But total demand for digital power is rising far faster than bit efficiencies are. We are using more chips — and bigger ones — and crunching more numbers. The bottom line: Taken all together, chips are running hotter, fans are whirring faster, and the power consumption of our disk drives and screens is rising. For the old thermoelectrical power complex, widely thought to be in senescent decline, the implications are staggering.” A group of scientists at Lawrence Berkeley National Laboratory who studied energy use were dubious of these claims, and published a series of reports calling them into question. One 2003 paper concluded that direct power use by computers and other office and network equipment accounted for just 2 percent of electricity consumption in 1999 — 3 percent if you counted the energy used in manufacturing them.Since then, the digital takeover of the economy has continued apace. But it hasn’t translated into an explosion in electricity demand. The “old thermoelectric power complex” was decidedly not on the cusp of a big boom in 1999. Instead, per-capita electricity use more or less stopped growing after then. Now it is falling. Part of the reason is that a grim new economic era dawned in 2000 or 2001 that has been characterized by slow growth, declining labor-force participation and general malaise — all of which tend to depress energy demand. But if you measure electricity use per dollar of real gross domestic product, the decline is just as pronounced, and it began much earlier than the fall in per-capita demand. In an article published in the Electricity Journal in 2015, former Lawrence Berkeley energy researcher Jonathan G. Koomey, now a consultant and a lecturer at Stanford, and Virginia Tech historian of science Richard F. Hirsh offered five hypotheses for why electricity demand had decoupled from economic growth (which I’ve paraphrased here): State and federal efficiency standards for buildings and appliances have enabled us to get by with less electricity. Increased use of information and communications technologies have also allowed people to conduct business and communicate more efficiently. Higher prices for electricity in some areas have depressed its use. Structural changes in the economy have reduced demand. Electricity use is being underestimated because of the lack of reliable data on how much energy is being produced by rooftop solar panels. The Energy Information Administration actually started estimating power generation from small-scale solar installations at the end of 2015, after Koomey and Hirsh’s paper came out, and found that it accounted for only about 1 percent of U.S. electricity. That estimate could be off, and there’s surely room for more study, but mismeasurement of solar generation doesn’t seem to be the main explanation here.Which leaves, mostly, the possibility that life in the U.S. is changing in ways that allow us to get by with less electricity. This still isn’t necessarily good news — those “structural changes in the economy” include a shift away from manufacturing toward sectors that may not provide the kinds of jobs or competitive advantages that factories do. When you look at electricity use by sector, in fact, it’s the decline in industrial use since 2001 that stands out. Still, some of that decline is surely due to efficiency gains. The corporate focus on costs has increasingly come to include energy costs, and parts of the corporate world have also reorganized themselves in ways that make saving energy more of a priority. Consider the shift to cloud computing. From 2000 to 2005, electricity use by data centers in the U.S. increased 90 percent. From 2005 to 2010, the gain was 24 percent. As of 2014, data centers accounted for 1.8 percent of U.S. electricity use, according to a 2016 Lawrence Berkeley study, but their electricity demand growth had slowed to a crawl (4 percent from 2010 to 2014). What happened? The nation outsourced its computing needs to cloud providers, for whom cutting the massive electricity costs of their data centers became a competitive imperative. So they innovated, with more-efficient cooling systems and new ways of scaling back electricity use when servers are less busy. Read more: Here’s How Much Energy All US Data Centers Consume In much of the world, of course, electricity demand is still growing. In China, per-capita electricity use has more than quadrupled since 1999. Still, most other developed countries have experienced a plateauing or decline in electricity use similar to that in the U.S. over the past decade. And while the phenomenon has been most pronounced in countries such as the U.K. where the economy has been especially weak, it’s also apparent in Australia, which hasn’t experienced a recession since 1991.So is electricity use in the developed world fated to decline for years to come? Well, not exactly fated. Check out that bottom line in the last chart. Transportation now accounts for just 0.3 percent of retail electricity use in the U.S. If the shift to electric vehicles ever picks up real momentum, that’s going to start growing, and fast. Dig more coal (or drill for more natural gas, or build more nuclear reactors, or put up more windmills and solar panels) — the Teslas are coming. This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners. (Comment on this)
8:58p	Top AWS Engineer Calls Hurd’s Cloud Data Center Bluff Mark Hurd, one of Oracle’s two co-CEOs, recently told an audience in Boston that the company he leads does not have to spend tens of billions of dollars on data centers annually to catch up to the three largest cloud providers. Ten billion dollars is roughly what Amazon, Microsoft, and Google have each been investing in infrastructure annually to support their cloud services. Oracle spent about $1.7 billion last year. According to Fortune, Hurd’s message was essentially that if you have more powerful hardware, you need fewer data centers: “If I have two-times faster computers, I don’t need as many data centers. If I can speed up the database, maybe I need one fourth as may data centers. I can go on and on about how tech drives this.” Spin detected. First, Oracle’s reputation as a force in data center hardware innovation has long since faded. It gained that reputation when it acquired Sun Microsystems, but that deal took place seven years ago. Oracle’s server market share is nowhere near that of the top five suppliers. In fact, its overall hardware revenue has been falling, and the company has been making deep staffing cuts to its hardware units. Second, if Oracle knows something about cloud hardware design top cloud providers don’t, why would it hire the people who built those cloud platforms to build its own? Last October, around the time the company launched the first availability region of its new cloud platform, Deepak Patil, Oracle’s VP of development, boasted in an interview with Data Center Knowledge that he was “surrounded by several hundred people who came from Amazon, Microsoft, Google,” whom Oracle had recruited to build the platform. Patil himself joined Oracle last year after 15 years at Microsoft, the last 10 of which he spent in senior engineering roles for data center and cloud infrastructure. Read more: Oracle’s Cloud, Built by Former AWS, Microsoft Engineers, Comes Online James Hamilton, VP and distinguished engineer at Amazon, one of the top technical minds behind the sprawling AWS data center empire, called Hurd’s bluff in a blog post Tuesday: “I don’t believe that Oracle has, or will ever get, servers 2x faster than the big three cloud providers. I also would argue that ‘speeding up the database’ isn’t something Oracle is uniquely positioned to offer. All major cloud providers have deep database investments but, ignoring that, extraordinary database performance won’t change most of the factors that force successful cloud providers to offer a large multi-national data center footprint to serve the world.” The thing is, designing hardware for global hyper-scale clouds is not the same as designing hardware for enterprise data centers. Infrastructure engineers at Google learned this the hard way, and so did the teams at Facebook, Microsoft, and Amazon. It appears that the people building Oracle’s cloud infrastructure are also aware of this, since many of them are the same people. The cloud giants also seem to have learned that $10 billion a year is about what the ticket to the top cloud provider club costs nowadays. But Hurd needed an at least plausible-sounding answer to an uncomfortable question from a reporter in front of an audience. Is Oracle prepared to spend big on cloud infrastructure to really compete? Hurd appears to be reluctant to make that commitment, at least in public. (Comment on this)

<< Previous Day

2017/04/18 [Calendar]

Next Day >>