What was the biggest change in Yarn 2?

The creator and maintainers of Yarn identified an issue we have in the JavaScript community and found a brilliant solution to address it. The solution is called Plug'n'Play (PnP). Surprisingly, the PnP feature exists in Yarn 1. The feature was unveiled back in September 2018, almost a year before Yarn 2's announcement. However, it was turned off by default due to incompatibility issues with well known NPM modules. As you can imagine almost no one used this feature for this reason. Then when the Yarn team released Yarn 2, PnP was on by default and can't be turned off unless you use some highly experimental workaround. I guess they were hoping this will push up the adoption rate.

But why did the maintainers want everyone to use PnP so bad?

Yarn team discovered that the current way of maintaining JavaScript dependencies is extremely inefficient and there is a lot of room for improvement. For example, 70% of the Yarn install time is used to generate node_modules. Sadly, without drastic changes, there is no way for package managers to optimise anything. This means the problems are not going to get better over time without intervention.

If you are interested here are a few other ways why the node_modules design is wasteful and impractical see Yarn: The Node Module Problem.

So, how does PnP fix these issues?

PnP generates a .pnp.cjs file that points to packages in local disk location. Instead of Node looking for a package in a folder, Yarn will tell Node where to find the packages instead. Since now Yarn only needs to generate a single file instead of creating a node_modules folder of modules during install. It is much faster and has way fewer I/O operations, meaning a massive performance boost!

By committing .npn.cjs file it is even possible to avoid running yarn install (hence the name Plug'n'Play). Node apps will also launch much faster without Node iterating over files to find what it needs. Just remember back when Windows XP supported USB devices without the user installing any drivers, this was meant to be a game-changer.

If Yarn 2's PnP feature is so awesome then why so much resistance?

As mentioned above the main issue is around the breaking changes and requires existing packages to be updated by their respective maintainers to be compatible with PnP. This caused further discussions around if Yarn 2 should be released as a separate package manager altogether under a different name e.g. Berry. It is obvious now looking back, the suggestion wasn't adopted.

In the early stages, people who tried Yarn 2 reported a lot of errors when they tried to upgrade. A lot of tools and libraries people rely on stopped working or experienced issues after the upgrade. Yarn 2 offers workarounds such as adding pnpMode: loose into .yarnrc.yml to disable PnP. But even this was a highly experimental feature at the time.

Unfortunately, not everyone is on board with this change. Facebook and Twitter both came out and said they plan on staying with Yarn 1. There must be a lot of other smaller companies which also didn't want to spend ages upgrade to Yarn 2 and risk their product stability and product roadmap.

Summary

The Yarn maintainers wanted to push the brilliant PnP feature, which has been in Yarn 1 for ages, for wider adoption. To achieve that goal, they wanted to force developers to start building and using PnP (reminds me of how Apple forces people to move to newer I/O ports).

The fact it breaks compatibility with a lot of projects caused concerns within the community. Well-known names in the JavaScript community opened stated on Twitter that they or their team wouldn't be upgrading to Yarn 2. This probably further hindered the adoption rate. But the root cause is the lack of easy migration between the two versions since a lot of the issues lies with the NPM package maintainers who needs to spend time supporting Plug'n'Play. People just didn't want to run the risk of upgrading and having that one library not working for them.

If you are familiar with Python 2 and 3 drama over the past ten plus years then you'll find the story pretty similar here. Python 3 breaking changes were introduced without real ways for developers to bridge the gap. So for years, Python devs struggled with the two major versions. Unable to move to the new version completed and unable to use newer libraries that only support v3. From my own experience, it seems like that's been getting better.

State of Yarn 2 in 2021

In all the companies I've worked at over the past 3 years, only one company looked into adopting Yarn 2. As far as I'm aware, other than a presentation showcasing its benefits, that didn't go anywhere. I almost can't find any discussions online regarding Yarn 2. I don't think Yarn 2 is dead, but compared to Yarn 1 the adoption rate is much slower.