Why open source projects (sadly) favor new users, and what you can do about it

If you‘ve been in software development for a while, you‘ve likely noticed a common trend – open source libraries and frameworks that generate a lot of excitement and rapid adoption when they launch, only to be abruptly abandoned by the community a year or two later as the next shiny new thing emerges.

As developers, we‘re naturally drawn to tools that make it quick and easy to get something working. We‘ll happily trade configuration for convention if it means we can have a slick demo app up and running in an afternoon instead of a week. And this makes perfect sense! Time is our most precious asset and if a new technology can save us some, we owe it to ourselves to check it out.

The problem is that this natural gravitation toward saving time in the short-term is inadvertently causing harm in the long-term, and not just to the individual projects we abandon. By consistently favoring technologies that optimize for rapid initial development at the expense of long-term maintainability, I believe we‘re harming the overall health and sustainability of the open source ecosystem.

The churn of JavaScript frameworks

Nowhere is this dynamic more apparent than in the fast-paced world of JavaScript development. A 2018 analysis by NPM, Inc. found that a staggering 97% of npm packages are dependent on other packages that are less than a year old.^1 In other words, virtually the entire JavaScript ecosystem is built upon tools that have only recently gained popularity, and will likely be replaced within the next year by something even newer.

This rapid turnover is evident when you look at the shifting popularity of JavaScript frameworks over the past decade. Consider the rise and fall of these once-dominant tools:

Framework	Release	Peak Popularity	Decline
Backbone.js	Oct 2010	Early 2013	Mid 2014
AngularJS	Oct 2010	Late 2014	Early 2017^2
Ember.js	Dec 2011	Mid 2015	Mid 2016^3
Meteor	Jan 2012	Early 2015	Early 2016^4

Each of these frameworks saw explosive growth upon their initial release, capturing the attention of the JavaScript community and gaining widespread adoption. But in each case, their time in the spotlight was short-lived. As soon as the next big thing arrived, developers started to jump ship, leaving behind codebases and communities that had been built on what was now perceived as a "legacy" technology.

Prioritizing initial ease of use

To understand how we got here, let‘s take a look at the key factors that drive adoption of open source libraries and frameworks:

Hype and buzz – The more people talking about and sharing a new technology, the more other people will be curious to try it. Public interest compounds itself.

Perceived value – Glowing reviews and impressive demos featuring a technology boost perception of its usefulness and drive further adoption.

Ease of getting started – The less friction there is for a developer to begin using a technology productively, the more likely they are to try it out and stick with it.

What‘s notably absent from this list is anything about the long-term experience of using a technology in a large, real-world application. This isn‘t because developers don‘t care about long-term maintainability and scalability. Of course we do! It simply reflects the reality that it takes a lot more time and effort to build a large system and form an educated opinion about how well a technology holds up over time…time that most people evaluating something new naturally don‘t have.

As a result, the qualities that lead to rapid adoption of an open source library or framework are disproportionately aligned with the experience of getting started as a new user:

• Concise, approachable documentation with a low barrier to entry
• Minimal upfront configuration
• Sensible defaults and conventions (aka "auto-magic")
• A fast "time to hello world"

Conversely, some qualities that tend to matter much more in large, long-lived applications are:

• Explicit, transparent behavior that‘s easy to reason about
• Rich configuration options to customize for an application‘s specific needs
• Strong encapsulation and modularity
• Comprehensive error handling and logging
• Performance and resource management at scale

This isn‘t to say that these two sets of priorities are necessarily in opposition. The very best, most well-engineered projects find a way to serve both new users and long-term users well. But accomplishing that is extremely difficult, and in many cases the incentives are stacked against even trying.

Dan Abramov, co-author of Redux and Create React App, describes the dilemma facing open source maintainers:

Early on, a project‘s success is directly tied to how much attention and excitement it can generate. There‘s pressure to make a big splash with an initial release and "wow" people with either new features or incredible simplicity. Long-term maintainability or scalability don‘t drive adoption.^5

The brutal truth is that for a new open source library or framework, it‘s a lot easier and more effective to focus on new user adoption than it is to go the extra mile to also address the needs of long-term users.

Consequences of short-term thinking

While catering to the "hello world" experience is effective for driving adoption in the short-term, it often comes at the cost of neglecting deeper issues that don‘t become apparent until an application grows in size and complexity.

Some concrete examples of how this commonly manifests:

• Tight coupling and poor separation of concerns – In an effort to make things "just work", many tools combine multiple responsibilities into a single, tangled abstraction. This feels convenient at first, but it quickly becomes a maintainability nightmare as an app‘s requirements evolve.

• Magic strings and implicit dependencies – Favoring convention over configuration is a popular way to minimize the amount of boilerplate code needed to get started with a tool. But taken too far, it can leave developers at the mercy of a framework‘s hidden behavior and "magic" naming patterns. Good luck trying to trace an issue through a system built on stringly-typed references!

• Lack of modularity and extensibility – Keeping an API surface small makes a tool initially easier to learn and understand. But if the underlying architecture isn‘t designed with clear, loosely-coupled boundaries, it becomes very difficult to customize the tool for more advanced use cases that weren‘t considered up front.

• Overuse of mocking and shallow testing – Writing genuine, end-to-end tests for a large system is hard. It‘s much easier to slap together some quick and dirty unit tests that mock out all of the interesting behavior. Tools that promote and even celebrate heavy mocking can lead developers down the primrose path to a false sense of security.

The prevalence of these anti-patterns in popular open source tools has real consequences for teams that adopt them without looking past the shiny "hello world" veneer. In my career as a consultant, I‘ve seen these scenarios play out time and time again:

• A company rebuilds their aging PHP monolith on a JavaScript stack, choosing tools primarily based on GitHub stars and Hacker News hype. A year later, the new system has become an unmaintainable patchwork of band-aids and hacks as the team struggles against the limitations of their tightly-coupled, "easy-to-get-started" framework.

• A well-intentioned developer advocates for their team to adopt the latest trending state management library. While building out the first few features, everything seems so elegant and concise compared to the legacy Flux architecture. But as business requirements evolve and new team members join, the lack of encapsulation and clear boundaries in the data model becomes a constant source of bugs and frustration.

• A startup builds an MVP for their product on a brand new, high-performance web framework. The initial development goes incredibly quickly, and the ease of deploying to a serverless platform seems like a dream compared to managing infrastructure for their old system. However, as the application grows in complexity, mysterious errors and performance problems start to emerge. The lack of visibility into the framework‘s internals and dearth of long-term production users makes troubleshooting and optimization a nightmare.

It‘s worth noting that none of these problems with immature tools would be particularly troubling if teams moved slowly and built systems incrementally. The real danger comes from our industry‘s obsession with speed and desire to constantly adopt the latest and greatest technology.

In a few years we're all going to laugh at the idea of rewriting our production apps every 18 months just to keep up with JavaScript framework churn.

— Tom Dale (@tomdale) April 4, 2018

The road to sustainable open source

If you‘re an open source maintainer, you may read all of this and think "well if you can‘t beat ‘em, join ‘em." After all, if the most reliable way to achieve success is to cater to new users, why fight it?

I would humbly suggest that even if it‘s in your project‘s short-term interest, it‘s in the long-term interest of the entire open source community to try to break this cycle. The more we can do to promote technologies that are not just easy to adopt but built for the long haul, the healthier and more sustainable our ecosystem will be.

Some thoughts on what maintainers can do:

• Be transparent about your project‘s priorities and design tradeoffs. Make it clear when you‘re making a short-term optimization that may have long-term consequences. Education is the first step towards changing expectations.

• Engage with your power users and learn from teams using your project to build large applications. Incorporate their feedback into your roadmap. Actively solicit input on scalability and maintainability concerns, not just feature requests.

• Write and promote content that emphasizes building for production, not just getting started. Highlight the architecture and design patterns that make your project well-suited for large, long-lived systems. Provide concrete examples and best practices whenever possible.

• Celebrate and amplify community members who build impressive, non-trivial projects using your library or framework. Make them the heroes of your ecosystem, not just the people with the prettiest demo apps.

• Resist the temptation to over-optimize for the "hello world" experience at the expense of deeper concerns. Every time you consider adding a bit of auto-magic or tightly-coupled convenience, think about the long-term maintainability implications. A small bit of extra friction for new users may pay huge dividends for long-term users.

As developers evaluating open source tools, we have even more power to influence the dynamics of the ecosystem with our choices:

• Look past the hype and FOMO around new tools. Wait until a project has proven that it‘s stable, reliable, and well-suited for building large applications before betting your product on it. Be wary of glowing reviews from people who have only built toy apps.

• Evaluate technologies holistically, not just on how easily you can get started with them. Examine (or ask maintainers about) the architecture, testing strategy, performance characteristics, and other attributes that will matter to you long after the initial shine has worn off.

• Before adopting a tool, look for examples of it being used successfully in large, production applications. Seek out war stories from teams maintaining non-trivial systems with it over a long period of time. If you can‘t find any, that may be a red flag.

• When choosing between two options, heavily favor the one that seems to be playing the long game. Imagine the kind of system you want to be working on 3 years from now, not just what will get you to a demo fastest. You (and your successors) will reap the rewards of your good judgment down the line.

• Vocally support and evangelize projects that embody long-term thinking. Publicly share your success stories of using tools that have held up well for you over time. Help bust the myth that popularity is the same as quality.

My sincere hope is that by being more deliberate in how we grow and consume open source, we can gradually shift the incentives that have led us to the cycle of churn we‘re in today. There will always be a place for tools that are optimized for getting started quickly – they play a vital role in making software development more accessible. But we have to stop glorifying adoption velocity above all else.

The health of our open source ecosystem depends on tools that are optimized for the long run. Tools that are boring, in the best possible way. Let‘s celebrate and support the maintainers who are prioritizing sustainability over shiny new features. Let‘s champion the codebases that are stable and reliable, not just the ones that are new and trendy. Together, we can focus the incredible energy of the open source community on solutions that will stand the test of time.