Jake Pusateri
When building a modern web app, one of the most important considerations after ‘Does it work?’ is ‘Does it load quickly?’. The importance of performance has been studied numerous times, and increased performance leads to happier users, higher conversion, and a better web. For a more in-depth analysis, take a look at this resource by Google: Why Performance Matters.
To increase performance, specifically by reducing load time, the answer is usually reducing the amount of JavaScript sent to the user. The popularity of tools like npm and webpack makes sites easy to build, but also make it very easy to get a bundle size measured in the megabytes. In this series, I’m going to talk about several techniques for reducing bundle size - without sacrificing features.
Strategy #1 - Eliminate Duplicates
A common source of needless bloat is including the same package multiple times in your bundle. This happens because JavaScript files are the true unit of code import in node’s resolution algorithm. Your code may say import _ from ‘lodash’, but node and correspondingly webpack use a resolution algorithm to turn that into ./node_modules/lodash/index.js. When paired with npm’s ability to install the same package multiple times on version conflict by nesting node_modules folders, you can end up in a situation with very similar packages appearing twice in your bundle, using twice the space they should, and taking twice as long to load. This issue is particularly nefarious because your app will otherwise load and work fine, just slower than it should be loading.
Detecting duplicates
To detect and manage duplicates in your bundle, a tool is needed. Because webpack is able to write a detailed programmatic description of a completed build, there are multiple tools to choose from. Personally, I use webpack-stats-duplicates. If you generate a stats.json from your build already, the following command will allow you to check for duplicates.
npx webpack-stats-duplicates stats.json
If no duplicates are detected, the command will exit 0 with no output. If duplicates are detected, the command will exit non-0 with a list of the duplicates found.
Duplicate module warning found in webpack configuration 0
./node_modules/warning
./node_modules/react-router/node_modules/warning
The above shows an example output of webpack-stats-duplicates. The tool is saying the module warning is found twice - once at the root node_modules and once underneath the react-router package.
I suggest putting this command as a step in your build system right after the webpack build, thus making the build fail on unexpected duplicates. If you use gulp, there’s a gulp plugin. If you use another build system, the command line version is available, or there is a programatic API.
Fixing duplicates
A tool breaking the build every time a duplicate is found is not helpful if you can’t fix the duplicates, so let’s review the strategies available. Fixing duplicates is fairly context-dependent on the duplicates, but a couple of strategies usually fix the issue.
Incompatible ranges
The most common reason that the same package ends up in your bundle multiple times is that not every package in your tree has overlapping semver ranges, so no one version can satisfy them all. For example, you may be requesting lodash: ^4.0.0 in your package, but a package you import called react-slowpoke is requesting lodash: ^3.0.0. To avoid duplicates, all modules in the tree must agree on a version of lodash. To find out which ranges are incompatible, you should start by running npm ls $package on whatever package is duplicated to see the npm tree of which packages request a given package.
jake@shmocking-sherver:~/seriestrackr$ npm ls warning
[email protected] /home/jake/seriestrackr
├─┬ [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected]
│ ├─┬ [email protected]
│ │ └── [email protected] deduped
│ └── [email protected]
└─┬ [email protected]
├─┬ [email protected]
│ └── [email protected] deduped
├─┬ [email protected]
│ └── [email protected]
└── [email protected]
Once you know the package importing the package in question, you can look in its package.json to see the semver string that is being used. In this case the npm ls shows enough to know that warning@3 and warning@4 are both required at different places in the tree. The action required is to upgrade the furthest behind packages to match the newer import, warning@4. react-transition-group, material-ui, and history would need to upgrade their dependency, and you would need to install their updates to remove warning@3 from the tree. Since these libraries seem well-maintained, it looks like they all have updates available that resolve this issue. If they did not, filing an issue on the project asking to upgrade warning to the most recent version would be appropriate. Some upgrades are simpler than others and you may even be able to submit a PR to help out the maintainer yourself. The warning@3 to warning@4 major version bump looks to be due to a license change and internal refactoring and probably does not break the public API.
Improper installation
Another common cause of duplicates is the npm installation process. Sometimes you’ll have packages that have compatible or even identical semver strings that refuse to dedupe according to npm ls. If you’re feeling lucky, you can remove your package-lock.json file and your node_modules folder and npm install again. Running the resolution process from scratch is pretty reliable and usually fixes the issue. If you’d like the package-lock.json file to be preserved, another option is to npm install the package in the root, then uninstall it. This tends to work since duplicates that happen like this are usually nested dependencies not being installed at the same time. Installing to the root and then removing the root installation forces the dedupe to occur properly.
Whitelisting
If you’re working under time pressure or none of the above are practical, there is a way to whitelist certain duplicates without disabling the webpack-stats-duplicates tool entirely. Adding a .wsdrc file with the key ‘whitelist’ as an array of allowed duplicates will prevent those particular duplicates from breaking your build. For tiny packages like warning, this can be the right answer if fixing the duplicate is impractical at the time.
{
“whitelist”: [
"./node_modules/warning",
"./node_modules/react-router/node_modules/warning"
]
}
Use the whitelist file sparingly, as once silenced, a duplicate error is easy to forget. Make sure to not let whitelist entries live around for too long, working to resolve the duplicate at the source.
If you’re interested in making your bundle size small and keeping it lean, you’ll probably be interested in Watchdog. Watchdog is a tool I’m working on to encode best practices around dependency management, responsible importing, and preventing regression of your hard-won performance gains.