zodern

Optimizing CI for Meteor

Many Meteor projects run tests and deploy from CI. If you are not careful, building a Meteor app in CI can take a lot of time. Fortunately, it is not difficult to make it much faster, and this article will show how to achieve it.

Many of the improvements come from caching. Most CI services handle caching in a similar way. You provide a list of paths, and a cache key. Some services treat the cache as immutable for a key, so the key must change to update the cache.

Here are the caching docs for some popular CI services

The example keys are in the format for Github actions. CircleCI has a very similar format.

Installing Meteor #

One of the first steps in CI is installing Meteor. Meteor is installed at ~/.meteor, where it stores a sqlite database with a list of all packages and their versions, and a local copy of the Meteor Tool and any packages used by your app. Installing Meteor itself isn't much slower than using the cache, but the cache also saves time from downloading a list of packages to update the sqlite database and downloading the packages used by your app.

By default Meteor installs the newest version of Meteor. Since you only need the version used by your app, you can use the installer for that version by adding it to the installer url. For example, if your app used Meteor 1.8.1, you can run the installer from

https://install.meteor.com/?release=1.8.1

The ~/.meteor folder can become very large, which makes updating and restoring the cache slow. It is recommended to have a version in your cache key (the example cache key starts with v1- for the version). To reset the cache you can occasionally increase the version, such as after updating the Meteor version used by your app.

Another option I tried for one project is to run meteor-cleaner with the --keep-scanned option to remove any packages and Meteor versions no longer used by the app.

When meteor is cached, you need to add the Meteor command to the path. This can be done by modifying the Path variable. If Meteor isn't cached, it, of course, should be installed. Here is one way to do both:

export PATH=$HOME/.meteor:$PATH
command -v meteor >/dev/null 2>&1 || { curl https://install.meteor.com/ | sh; }

Some projects will have the CI cache the /usr/local/bin/meteor file and restore it instead of modifying PATH.

The cache key includes a hash of the app's .meteor/versions file. This file contains the complete list of packages that should be installed, so is a good indicator of when the cache needs to be updated.

NPM #

You probably use npm ci in CI for faster installs and to catch when the package-lock.json file is out of date. npm ci deletes the contents of the node_modules since it is faster to reinstall from scratch, so there is no reason to cache that folder. The ~/.npm folder caches the downloaded npm packages and other data that helps npm to install the app's dependencies faster.

Some npm packages, such as cypress and puppeteer, download large files when installed. If you don't need them in CI, these packages allow you to set specific environment variables to skip downloading the files.

Meteor Build Cache #

Meteor heavily relies on these disk caches in .meteor/local to provide good build performance. You can find more details on what these folders are used for in Exploring the .meteor/local Folder.

One folder missing is .meteor/local/bundler-cache/linker. For most apps this saves a few seconds, but due to how quickly the linker cache grows in size it can take much longer than that for the CI to update or restore it.

To check that the cache is working correctly, you can set the METEOR_PROFILE=100 env var. Except for production builds or when there were a large amount of changes since the cache was last updated, no single step in the build process should take over 30 seconds.

The cache key should include the git commit. Most commits probably change code in your Meteor app, so you want the CI to always update the cache.

Minify #

If you are deploying from CI or creating a production build for other reasons, the slowest part of the build process is probably minifying the client. I created zodern:standard-minifier-js which uses caches to save time. For the minify cache to be used in CI, you should have the CI cache the paths listed in the previous section.

If your app doesn't have a client, you can use the zodern:remove-client-js minifier. This minifier removes all client js from production builds, removing the need to minify any code.

Meteor Packages #

If your app has local Meteor packages with npm dependencies, the npm dependencies should be cached. Otherwise, Meteor will install the dependencies and rebuild the package. Instead of caching the package's whole .npm folder, the paths above are more precise. Sometimes Meteor leaves behind garbage directories with a copy of the package's npm dependencies, and caching those and restoring them would take more time. When using the example cache key, make sure you replace <cache path>.

For most apps, these changes will speed up building for tests from a few minutes to ~45 seconds + the time to restore the caches, and save a few minutes when deploying.

Have a comment? Please Send me an email

Want More? Join my monthly newsletter for new blog posts and news about Meteor and my projects.

Have a Meteor app in production? I run Monti APM for production monitoring to help improve performance and reduce errors.