The scene outside the Supreme Court after the justices narrowly upheld the Affordable Care Act looked chaotic, yet the scene on the back end of SCOTUSblog wasn’t -- due in part to some serious planning.
SCOTUSblog is a website dedicated to news and analysis of the Supreme Court of the United States, run as a separate business by the lawyers at Washington, D.C.-based law firm Goldstein and Russell. It averages about 30,000 hits a day, but in the months leading up to the court’s ruling on the Patient Protection and Affordable Care Act, it became clear that something would have to be done to support a huge amount of traffic.
The blog staff knew that they were in for traffic problems when page views spiked during oral argument in March. Over a three-day period, the site received more than a million hits, creating a slow experience for users that was punctuated by crashes during peak hours.
“We were just really, really struggling to serve that audience,” said Max Mallory, deputy manager of the blog.
Mallory, a self-described liberal arts-type who learned IT on the fly after becoming deputy manager of the blog, said that the staff took stock of what they had and decided there was no way for them to rework it on their own. To accommodate the blog traffic they expected when they reported on the court’s decision, they would need to get outside help.
SCOTUSblog planned for huge traffic boost
Options on what to do ranged from completely redesigning the entire site to optimizing what they already had and adding more servers.
“There was tons of stuff being thrown around,” Mallory said.
Monday, June 18 was the earliest the court could have made its decision and served as the first testing day for the site’s changes. They decided to redirect traffic from the homepage to the live blog page, something they normally do on breaking news days. At one point, 40,000 simultaneous users were on the live blog, a fraction of what they expected on the big day, but it still revealed difficulties on the back end.
By Thursday, they had implemented a new plan -- split the traffic between three servers. The main blog page would be hosted on Media Temple, the service they had been using all along. That page would redirect to a landing page that housed just the live blog, which would be hosted by WP Engine. Once those readers clicked to activate the live blog, that traffic would be hosted by third-party live blogging service CoverItLive.
In anticipation of a decision that still hadn’t come that day, traffic again spiked and the site stayed afloat, but still moved slowly. The WP Engine server handled the live blog page, but the Media Temple server was swamped by redirect requests.
“Friday morning I knew there was no way based on that performance we were going to be able to handle it,” Mallory said.
So Mallory reached out to Datagram, a server provider that handles hosting for some large blogs, and asked them to put him in touch with “the best optimizer of WordPress sites.” Datagram gave him the name of Andy LoCascio and his company, Sound Strategies. By the end of the day, LoCascio was in charge of rebuilding everything from the ground up.
After bringing LoCascio on board, the team learned all their work over the previous two months was essentially a waste.
“Literally everything that [could be] wrong was wrong,” Mallory said.
LoCascio’s team worked all day Friday and Saturday, adding a high-powered NGINX deployment on top of the Media Temple server, rewriting all Apache and MySQL configurations, fixing plug-ins and reworking caching. By Sunday, everything was finished.
Most court watchers expected the decision to come down on Monday. The blog surpassed its all-time traffic record by 2 p.m. and had more than 100,000 viewers on the live blog. Everything went well, but the big day had yet to come.
Finally, the media learned Thursday was going to be the day and the team was prepared to sit and wait. But on Tuesday evening they experienced a distributed denial of service (DDoS) attack, which left them scrambling to find a way to protect themselves from a nefarious attempt to crash the site.
They decided to eliminate the chain of servers at different companies and consolidate resources. The night before decision day, they set up four satellite servers off the main Media Temple server, each of which would host a cached version of the site that would be updated on a fixed, periodic schedule.
Two more DDoS attacks came the morning of the decision, but neither worked. Then, the news they and their audience had been waiting for broke.
“Right at 10:03 a.m. Thursday, we were getting more than 1,000 requests every second,” Mallory said.
In the end, SCOTUSblog received 5.3 million page views with no crashes or lag time. Load time never climbed above one second and CPU usage never ventured above 1%, a vindication of the new design. The site previously operated around 60% to 80% CPU usage with a hundredth of the traffic.
Traffic has since subsided and is expected to fade as the court heads for its summer recess. Mallory said the system set up for the health care decision will be shut off for now, but added that he and his colleagues will be prepared for the next major Supreme Court decision.