The most fascinating tech company story during this ongoing COVID-19 pandemic is Zoom. It’s meteoric rise in usage, coupled by a similar surge in its stock price, is met with increasing scrutiny of its security practices, a drop in stock price, and a whole lot of FUD (Fear, Uncertainty, Doubt) and piling on.
The rollercoaster ride will continue for quite a while longer. But the reality with Zoom, like all realities, exists in between the extremes, along a spectrum, and among a basket of tradeoffs and nuances that usually get lost in this confusing world. Yet, it’s important and worthwhile to find where that middle ground is, so I’ll give it a try.
FUD is an age-old marketing tactic. The term dates back to the early 20th century, but was popularized by its wide use in the technology industry, selling hardware and software to large enterprises. IBM was an early and prolific user. Microsoft did its fair share of it as well. It’s now commonly used by companies, big and small, looking to gain an edge in perception in front of customers in a competitive situation.
FUD used to be about technical uncertainty -- big incumbents squeezing out startups by pushing disinformation, so customers would go for or stay with the “safe choice”, while potentially losing out on new innovations and better products.
In our increasingly interconnected economy and complex international relations, FUD has now taken on a new form. Its use is no longer about technology alone, but technology shrouded in the fog of geopolitics and nationalism. And that brings us to Zoom.
There has been a lot of media attention paid to a Citizen Lab’s analysis of Zoom’s poor implementation of end-to-end encryption (E2EE). Among its findings, the encryption/decryption keys of some Zoom calls that happen in North America are routed to servers located in Beijing, and Zoom employs more than 700 engineers in China for R&D and development purposes, the latter of which is consistently disclosed as a “Risk Factor” in Zoom’s regular filings with the SEC, so it can hardly be considered a “finding”. All this information would’ve been fair game, but Citizen Lab took the unfortunate, cringe-worthy step of stoking FUD with headings like “A US Company with a Chinese Heart?”. And this sentiment has been picked up by multiple media outlets, without further analysis or context, and of course in the notorious Twitterverse.
Furthermore, Citizen Lab’s discovery of the Beijing servers did not include the fact that those servers are AWS servers. Zoom’s response blog post painstakingly noted that all its data centers in China are run by either Telstra, an Australian company, or AWS, an American company. Very little of these details made it into the subsequent media headlines. That’s sadly how FUD takes off and morphs into a narrative of its own.
To be clear, I understand that non-Chinese tech companies must form some kind of joint-ventures with local Chinese entities in order to operate. The AWS in China is not exactly the same as the AWS in the US, but it’s also not the same as a homegrown Chinese cloud provider; it exists in the middle, a nuance that is all but lost.
Concerns around Chinese espionage, both corporate and national intelligence, are no doubt legitimate and important. But generalizing every server request that happens inside China to be a cyber-spying field day is more fiction than fact, and ignores the messy complexity of how data exchange works in China -- between private companies and government, between state-owned enterprises and government, and even between different government departments. Tech companies, especially ones listed as public companies in the U.S. or Hong Kong, like Alibaba and Tencent (and Zoom), often resist giving the government user data to protect their commercial interests; they don’t just roll over like many people assume. (The Financial Times did a deep dive of this “messiness” recently that’s worth reading.)
And if employing engineers in China automatically makes your product a security threat, then we ought to place VMWare, IBM, Google, and Microsoft on this “naughty list” too, all of which employ R&D teams up to the thousands in multiple Chinese cities and make products occupying much more critical layers of the infrastructure stack than videoconferencing.
I’m not suggesting that Citizen Lab is intentionally pushing FUD to help Zoom’s competitors. But it will be used that way anyway. And I believe Citizen Lab’s overall work is very informative and valuable; its recent analysis of China’s social media censorship of all coronavirus related information was rigorous and objective. Frankly, it’s because of its good track record and reputation that made the FUD in its Zoom analysis both unfortunate and puzzling.
Pointing out the FUD by no means exonerates Zoom’s lack of security, privacy, and data collection oversight. Its false claim of having implemented E2EE is perhaps its worst mistake yet.
This mistake, however, is also the source of an important under-explored tradeoff.
A big reason why Zoom was a rocketship even before COVID-19 was because of its “just work” user experience, helping its product standout among much better known tools like Skype, WebEx, and Google Hangout. What endeared it to its users are also seemingly frivolous but office worker-friendly features like virtual background, beautification filters, and Snapchat-style filters. Zoom is very much a poster child of the trend of “consumerization of enterprise technology”, which I’ve discussed in the Dropbox context before.
All these features have performance characteristics and implications. There is generally at least some performance overhead when encryption is in place for other technical scenarios. It’s not a question of if, but how much. In the report by The Intercept, which initially exposed Zoom’s lack of proper E2EE implementation and dubious marketing claim, the cryptographer and Johns Hopkins computer science professor, Matthew Green pointed out:
...that group video conferencing is difficult to encrypt end to end. That’s because the service provider needs to detect who is talking to act like a switchboard, which allows it to only send a high-resolution videostream from the person who is talking at the moment, or who a user selects to the rest of the group, and to send low-resolution videostreams of other participants. This type of optimization is much easier if the service provider can see everything because it’s unencrypted.
Professor Green goes on to say:
If [Zoom is] all end-to-end encrypted, you need to add some extra mechanisms to make sure you can do that kind of ‘who’s talking’ switch, and you can do it in a way that doesn’t leak a lot of information. You have to push that logic out to the endpoints...This isn’t impossible...as demonstrated by Apple’s FaceTime, which allows group video conferencing that’s end-to-end encrypted.
This insight says to me that the fact that Zoom has been able to maintain its quality of service plus its many features, without any notable outage, while getting crushed by demand is at least in part because it did not implement E2EE by the books. (Maybe that’s also why no one is picking FaceTime as their work-from-home video conferencing tool of choice.)
Thus far, there has not been any thorough performance analysis done on Zoom with a perfectly implemented E2EE for every endpoint of every videoconference. (Readers: please hold me accountable and send me any you’ve found!) But there is a clear tension and set of tradeoffs between having end-to-end encryption and maintaining the level of user experience that Zoom has been delivering, and will have to continue delivering to stay competitive against much bigger but less performant players, like Microsoft, Google, and Cisco.
Are we letting the perfect be the enemy of the good? That’s at least a fair question to pose and think about.
What makes Zoom Zoom -- a low friction, easy to use way to communicate “face to face” -- may no longer be true if E2EE is perfectly implemented for every single type of call. These may or may not be tradeoffs worth making; it really comes down to the use case. Surely a meeting among government officials or corporate executives should have a high level of security and encryption, and these types of customers would pay for it. But is it really necessary for my hip-hop dance instructor, who is relying on hosting Zoom classes with a virtual empty studio background as his only source of income right now? Or the elementary or middle school teachers, most of whom are not highly technical users, trying to educate and keep their young students engaged while at home?
Zoom is most definitely at fault for not being honest and transparent about E2EE and already suffering the consequences, with lost customers, terrible PR, and a class action lawsuit filed by a shareholder. It also can’t rewind the clock and return to its not-so-distant past as just another B2B enterprise company; ordinary consumers around the world have found Zoom, and they won’t suddenly un-find it.
So taking a step back, there has to be a middle ground when it comes to performance tradeoff, where encryption requirements are contextualized in the use cases and corresponding user experience. Otherwise Zoom, business customers, and consumers may all be worse off.
Open Source to Maturity
There’s no question that Zoom, as a product and a company, will have to grow up and mature fast, fighting through both real technical challenges and plenty of FUD that’s already out there.
One path that may accelerate that growth and maturity is to open source its codebase.
From a reputational angle, open sourcing its codebase is the most effective way to establish trust and transparency. You can claim to have removed the China-based servers from the whitelist and the mistake will never happen again, but letting interested parties see and test the code and routing is much more convincing. Sunlight is always the best disinfectant.
From a technical level, open source has proven to be the most effective way to develop robust and secure software. Linux is the obvious example. (It was also the subject of much FUD from Microsoft years ago.)
In this Forbes profile of Eric Yuan, Zoom’s founder and CEO, published on April 3 and written while all this sequence of events was unfolding, Yuan hinted that he would consider open sourcing Zoom “in the next several years”. Fast forward to April 8, during Yuan’s first weekly “Ask Eric Anything” webinar on Zoom security, he now plans to open source the customer-side key-generation portion of Zoom’s encryption mechanism as it's being developed fairly soon, which would be a welcoming step if it happens.
Meanwhile another open source alternative to Zoom, Jitsi, is picking up steam, as both a standalone solution and the core to 8x8’s videoconferencing product, which has experienced a 50-times usage growth during COVID-19.
Of course, to open source something is not as simple as just tossing a codebase onto a public GitHub repository. You need proper documentation, clear processes for developers to contribute or file issues and bugs, as well as proper licensing. It’s real work; a topic I’ve worked on and written about a lot previously.
Nevertheless, the sooner Zoom can get ready to open source important components of its codebase, the better its product, technology, and reputation will be.
For what it’s worth, Zoom has demonstrated its ability to act quickly to: fix and change its product, take on new measures, and apologize over and over (and over) for its mistakes without appearing defensive. Crisis communication is never easy. I’ve been in similar shoes in the government and political context. I don’t envy anyone working in Zoom’s communications and engineering department right now.
Yuan has shown solid leadership and the needed vision, steadiness, and decisiveness to convince ex-Facebook Chief Security Officer, Alex Stamos, to help out, which is an important step in the right direction. Late last year, Yuan said during an enterprise technology event hosted by GGV Capital that taking a company public is not much more than high school graduation -- there’s a long road ahead. He grasps the fact his creation is still young, immature, and untested in many ways. Unfortunately, Zoom won’t get four years worth of college time to meet those challenges.
Chinese Version Below
在这场冠状疫情的大势中，一个最吸引人的科技公司故事就是Zoom。从使用率的飞速上涨，到股价飙升，到大家对其安全措施的漏洞开始严格注意，到股价下跌，还有在舆论中大量的FUD（恐惧、不确定性、怀疑；Fear, Uncertainty, Doubt）。
一篇Citizen Lab（多伦多大学的一个学术小组）对Zoom端到端加密（E2EE）落实程度不够好的分析引起了媒体的广泛关注。调查中的主要发现是：一些源于北美的Zoom视频会议的加密/解密密钥被路由到位于北京的服务器，而且Zoom在中国雇佣了700多名工程师做研发工作（后者一直在Zoom向美国证券交易委员会提交的常规文件中公开披露过多次，类为“风险因素”，所以也算不上是什么“发现”）。所有这些信息本身都没什么问题，可以好好讨论，但Citizen Lab采取一种故意渲染的方式，用“一家有中国心的美国公司”这种耸动的标题煽动对Zoom的FUD，已经被多家媒体报道直接接受，而不加任何分析和解释，当然也包括推特。
我并不是说Citizen Lab有意推动FUD来帮助Zoom的竞争对手。但渲染的方式肯定会被Zoom对手这么用。我其实一直觉得Citizen Lab的整体工作是信息量很高，很有价值的。它最近对中国社交媒体审查删除有关冠状病毒负面信息的分析是严谨和客观的。正是因为它良好的业绩和声誉，才使得其Zoom分析中的FUD既不幸又令人费解。