One post tagged with "mercurial"

Why Did Facebook (Meta) Say "No" to Git? A Story of Scaling, Community, and Giant Monorepos​

In the world of software development, Git is ubiquitous. It's the default tool for millions of developers and projects, almost a given, "as common as water," as our source's author points out. It's perceived as the only viable solution for managing code. So imagine the surprise of discovering that Facebook (now Meta), one of the world's largest tech companies, does not use Git as its primary version control system for its immense monorepos.

This is a fascinating story that highlights engineering challenges on a colossal scale, the limits of popular tools, and the crucial importance of human factors in technological decisions. Let's delve into the reasons why Meta chose a different path.

The Astonishing Absence of Git at Meta​

For many, the idea that Facebook doesn't run on Git is counter-intuitive. The author, whose personal experience with version control systems began with SVN before the explosion of Git, confesses his own surprise: "Throughout my life a git was common as water it was so common in fact that I assumed it was the only viable tool for creating and managing code changes". He recounts how the Facebook engineers he met were "deeply trained on material patterns and Facebook stack diffs workflow" rather than on Git.

Historically, even Google, whose engineering "predates git by over 5 years," uses its own internal system. But for Facebook, it was a more active and recent decision.

The Myth of Git's Complexity (and why that wasn't the main reason)​

Before addressing the real reasons, it's worth noting that the perceived "difficulty" of Git was not the driving force behind this decision. The author himself wonders: "I've never understood this kind of commenting get is so confusing how is G confusing like what about git is confusing". He often attributes this confusion to a lack of fundamental learning, suggesting that "most of you have just never taking taken the two hours of time it takes to learn get well enough to not be confused by any of it."

No, the reason for Facebook's shift was far deeper and more technical.

The Scaling Nightmare in 2012: When Git Reached Its Limits​

The real breaking point occurred around 2012. By then, Facebook's codebase was already "many times larger than even the Linux kernel" (which had 17 million lines and 44,000 files). With exponential growth, Git began to show significant signs of weakness for operations on such a gigantic monorepo.

The key bottleneck? The process of "statting" (checking the status) of all files. "G examines every file and naturally becomes slower and slower as the number of files increase." Basic Git operations, far from being "crippling slow," were slow enough to warrant a thorough investigation. Simulations were "horrifying," showing that simple Git commands could take "over 45 minutes to complete" as the codebase continued to grow. This was untenable for thousands of engineers.

The Call for Help and the Surprising Response from Git Maintainers​

Faced with these challenges, the Facebook team did what many tech companies would have done: they contacted the Git maintainers. Their goal was to collaborate to extend Git and better support large monorepos.

However, the response was unexpected and, according to the author, "wasn't cooperative." The Git maintainers "pushed back on improving performance and instead recommended that Facebook shared the uh Shard their monor repo" (divide their monorepo into multiple repositories).

This suggestion, though technically possible, was a "non-starter" for Facebook. They had invested heavily in a monorepo workflow, and the complexity of such fragmentation would have been enormous. Even more surprisingly, Facebook expected their offer of "free open source labor by a major tech company is well received," an opportunity to improve a widely used open-source project. The lack of cooperation was a decisive factor.

Mercurial: The Unexpected Alternative and Its Clean Architecture​

Faced with Git's limitations and the lack of support for massive monorepos, Facebook explored alternatives. In 2012, options were "scarce." Perforce was dismissed due to perceived architectural flaws. This is where Mercurial entered the scene.

Mercurial had performance "similar to git," but possessed a much cleaner architecture. While Git was a "complex web of bash and C code," Mercurial was "engineered in Python using object-oriented code patterns and was designed to be extensible." This extensibility was crucial.

The team decided to attend a Mercurial hackathon in Amsterdam. What they discovered was not just a flexible system, but also "a community of maintainers who were impressively welcoming to aggressive changes by the Facebook team." This was the perfect contrast to their previous experience.

The Internal Migration: A Masterclass in Change Management​

Convincing the entire engineering organization to migrate from Git to Mercurial was an "intimidating" task. Engineers can be "extremely sensitive about tooling changes." Yet, what followed "sounds like a masterclass in internal Dev tools migrations."

The team methodically:

1. Socialized the idea: Communicated the necessity and benefits.
2. Documented workflows: Ensured everyone knew how to use the new tool.
3. Listened to concerns: Allowed developers to express their doubts.
4. Made the definitive switch: Cut the cord with Git once the groundwork was laid.

The success of this massive migration is also attributed, with a hint of irony, to the fact that few Facebook engineers knew Git in depth. As the author notes, "it's not even a big deal" to change tools if engineers aren't attached to specific Git subtleties.

The Legacy of Facebook's Decision: Stack Diffs and an Improved Mercurial​

Facebook's decision was not without consequences for the open-source ecosystem:

• Improved Mercurial: Facebook "contributed performance improvements to Mercurial making it the best option for large monor repos."
• "Stack Diffs": Building on Mercurial's concepts, Facebook created an innovative code review workflow called "stack diffs" (or stacked diffs). This "unlocking novel code review parall parallelization" and revolutionized their development process. Former Facebook engineers exported this workflow to other companies, creating a "small but vocal Cult of Stack diff Enthusiast," even inspiring the author to create tools like Graphite.

The Human Factor and the Constant Evolution of Technology​

Ultimately, the story of Facebook and Git is a poignant reminder that "so many of History's key technical decisions are human driven not technology driven." The receptiveness of a community, the adaptability of a team, and the ability to collaborate can outweigh perceived technical advantages.

It's also crucial to note that the landscape has evolved. "A decade later GI has made significant improvements to support monor repos... today get now with some knowledge of how to do it operates well with really really large repos now." Git has progressed, and it's possible that it could now handle Facebook's needs.

Facebook's story is one of a company that had to adapt to explosive growth. Faced with the performance limitations of a dominant tool, and a community that was not ready to support its specific needs at the time, they made a pragmatic choice. It was not a rejection of Git in itself, but a response to a unique scaling problem, resolved with an innovative solution, and a testament to the power of human decisions in large-scale engineering.