AWS vs Azure: Return of the JEDI?

You may have come here because of the “clever” title, but I promise you this is a serious analysis of the ongoing saga around the U.S. Department of Defense’s (DoD) massive cloud computing contract, JEDI (Joint Enterprise Defense Infrastructure).

Earlier this month, this saga was reignited by a new complaint that Amazon filed directly to the DoD, which led to dueling blog posts between Microsoft and Amazon’s public relations departments that are anything but cordial.

This movie will not be over any time soon. And it’s worth analyzing why, from both a technical and a political angle.

Current State of Play

The quality of reporting on this issue has been mixed. I don’t blame the reporters. To really explain what’s going on, you need a solid foundation of both how cloud computing works from a technical and business perspective, and how federal contracting works in the context of power and influence in Washington. And this is not just any run of the mill federal contracts, but a massive, 10-year investment by the U.S. military. I have found reporting by the Federal News Network, a niche media outlet, the most useful so far.

The current state of play in one set of bullet points without legal jargons:

• The DoD gave the whole contract to Microsoft Azure.
• Amazon filed a lawsuit alleging that Donald Trump interfered with the contracting to get back at Jeff Bezos (he owns the Washington Post).
• Amazon also thinks the entire six-part technical evaluation was flawed and wants a total re-do.
• The Judge found at least one of the six categories -- Price Scenario 6 -- faulty, pauses the whole thing for 120 days (until August 17), to give all the parties time to sort this out.
• DoD volunteered to revise and reopen the bidding of Price Scenario 6.
• Amazon does not like it, filed another complaint (not a lawsuit) directly to the DoD on the new, limited rebidding.
• Meanwhile, the DoD’s Inspector General’s office (a self-investigative watchdog unit, similar to an independent auditing committee of a company) tried to investigate the whole thing and issued a big 300-plus page report
• Two main takeaways from the report: 1. DoD officials accidentally shared some Azure confidential information with the AWS team; 2. The question regarding possible political interference from the Trump White House is inconclusive (read: we don’t know enough to know)

With that context in mind, let’s dive deeper.

Assess Data Storage Pricing

The overarching technical question is whether revising Price Scenario 6 is enough. This category has to do with data storage. Based on the public portions of the court filings, Judge Patricia E. Campbell-Smith found that Azure’s solution did not meet the standards of the contract for a “cloud data warehouse” that needs to be “online” and “highly accessible” to JEDI users “without human intervention.” Azure thinks AWS’s pricing was too high and lost the bid fair and square. AWS thinks Azure’s bid was technically deficient and won under ambiguous DoD guidelines.

Which argument is more plausible?

The crux of this category seems to be a straightforward delivery of a highly available data warehouse. And evaluating high availability is both a storage durability issue and a networking issue; you can’t have one without the other.

Having data “online” and “accessible” (or “available”) “without human intervention” (aka automatically) requires some implementation of data replication plus a consensus algorithm (e.g. Paxos or Raft), so the system automatically makes copies (at least 3) of the data and synced all copies of that data to make sure they are the same (aka consistency). This implementation will likely happen both within a single data center region and across different geographical regions to guarantee “accessibility” with reasonable performance. E-commerce retailers do this so people can buy stuff. Social networks do this so people can “like” stuff. I expect much higher requirements from the world’s largest military.

All this technical work happening behind the scenes to have stored data be “online” automatically means a lot of data transfer cost via networks. Cloud industry observers are well aware of the trend that while storage cost is going down, the data transfer cost is not. And no organization of any meaningful scale would store a single copy of their data on AWS S3 or Azure Blob Storage and be done with it. Network data transfer cost is the bigger chunk to make “highly accessible” data a reality.

Lest you think I’m just mincing words here, I strongly believe that having precise definitions and shared understandings of the implications of those definitions are fundamental to any fair evaluation. If the definitions and scope of Price Scenario 6 are only about storage, it’s quite possible that Azure’s lower-priced bid does not capture the whole picture, while AWS’s bid appears expensive but is more comprehensive.

To be clear, this is not a dunk on Azure’s alleged technical inferiority. Even though I have criticized Azure’s reliability before, especially during all the COVID-19-induced usage surge (a line of argument that AWS is apparently using now), building a government cloud for the military is a unique technical challenge. Regardless of which company gets picked to build this, judging them from how their public cloud performs is not entirely fair.

If Pricing Scenario 6 is in fact limited to data storage with imprecise definitions and expectations, it is reasonable for the re-bidding process to go beyond this single category to at least include whichever category deals with networking. As far as I know, there is no public information as to what the other five categories are. Having said this, I’m not sure if Amazon’s request for a total re-do of all six categories is reasonable from a technical perspective. Did the DoD make some mistakes? Certainly appears so, based on the Judge’s ruling and the IG report. But did it mess up the entire process during the two-plus years of evaluation? Unlikely. Given the Pentagon’s current response to Amazon’s lawsuit, it clearly prefers the most limited rebidding process possible.

So what is Amazon trying to achieve with its lawsuits and complaints? I think there is a larger strategy at play that has less to do with technology and more to do with the U.S. political calendar.

Bigger Strategy: Get To November

I hate to make everything about politics, elections, and Donald Trump, but it’s unavoidable in this case. Amazon’s bigger strategy is to delay any work or action on the JEDI contract by all means possible until November when the U.S. presidential election happens. It’s betting on a Joe Biden victory. If Biden does become president, Washington will go through a complete leadership reshuffle at all levels of the government, and Amazon may very well get the total JEDI redo it is hoping for.

Amazon’s lawsuit has caused a pause until August 17, the endpoint of the current 120-day remand period. There are checkpoints along the way and the remand could be extended. This part in one of the Federal News Network reports provides a clear look ahead:

“Judge Campbell-Smith ordered government [aka DoD], AWS and Microsoft attorneys to submit a status report by June 16 to update the court on how the process is playing out. Once the reconsideration process is finished, another update is due within five days, along with each party’s views on whether the lawsuit needs to continue. But the order makes clear that the 120-day remand period could be extended.”

If the remand period is not extended beyond August 17, that scenario obviously won’t get Amazon to Election Day. Thus, Amazon filed its latest complaint directly to the DoD two weeks ago, along with waging an increasingly bitter public relations war with Microsoft to add more fuel to the court of public opinion, which always has some influence on the court of law.

The current lawsuit sits in the U.S. Court of Federal Claims. This court has at least two layers of more powerful courts above it: the U.S. Court of Appeals and the U.S. Supreme Court. If Amazon receives an unfavorable ruling from the Court of Federal Claims, an appeal to a higher level court is almost guaranteed. Amazon will use a combination of lawsuits and appeals, direct complaints, and public relations campaigns to drag this out to November. Microsoft will not be sitting idle either; its recent blog post responding to Amazon’s latest DoD complaint is full of emotional appeals to patriotism and jabs at Amazon as a sore loser, authored by a former Marine.

But will a Biden administration be so nice to Amazon that a “Return of the JEDI” will happen? Nothing is guaranteed, but what is certain is that Amazon will have a more receptive audience. Trump’s hatred towards Bezos personally and Amazon broadly is well-documented. Though to be fair, most Democrats don’t like Amazon either. Amazon’s upper leadership is stacked with former Obama-Biden administration officials, the most high-profiled being Jay Carney, who is Amazon’s SVP of Global Corporate Affairs. He was Biden’s communications director when Biden was Vice President and later Obama’s White House Press Secretary. (Disclaimer: I used to work for Jay in the White House as one of his many assistants. He was a really nice boss.)

If Biden becomes president, Amazon will use those relationships to influence the JEDI outcome. And I’m not here to single out Amazon. Microsoft would do the same. So would Oracle. So would every company who is or wants to do business with the Federal government. (When I wrote “Why Zoom Chose Oracle” a few weeks ago, I explained why Zoom's government business prospect is one of the reasons why it chose Oracle as its newest cloud provider.) All of of this is part of the “DC influence game” that gives Washington the swampy reputation it frankly deserves.

Of course, there is no guarantee that Biden will win. At this moment, his chances of beating Trump is 50/50 at best. But if what’s on the line is a second shot at a $10 billion USD contract that will most definitely open the door to many more millions of contracts with other government agencies, the expected outcome of this coin toss is totally worth the effort. All the lawyer fees and possible short-term reputational damages are peanuts compared to the potential winnings.

If you like what you've read, please SUBSCRIBE to the Interconnected email list. New posts will be delivered to your inbox (twice per week). Follow and interact with me on: Twitter, LinkedIn.