File watching in Meteor
One area of Meteor that has always interested me is file watching. This is one of the main tasks handled by the Meteor tool (the part of Meteor that provides the
meteor command and builds and runs the app) and it has to balance multiple conflicting qualities:
- It should detect changes quickly so it can start the rebuild sooner and reduce how long you have to wait after making a change to see it
- It should reliably detect changes. Rebuilds are much faster than the initial build, and having to manually restart Meteor after a file change isn't fun.
- It should be efficient
There are multiple ways to watch a file:
- Polling. At a regular interval, it stats the file and checks if it has been modified. This is a reliable way to detect file changes, but it uses a lot of cpu. You can either check very frequently to detect changes quickly but be very inefficient, or check less frequently, increasing the efficiently but taking much longer to detect changes
- Native file watchers. These are very efficient and detect changes quickly; however, they are not always reliable. There are many situations where these aren't available, such as in network drives and anyplace that uses network drives (wsl for the main disks, docker volumes on some operating systems, etc). Also, they work inconsistently and sometimes weirdly (many watching libraries try to fix this). And you need to be careful about not creating too many.
History of file watching in Meteor #
Jan 2012 Back then, Meteor was called Skybreak, and was at version 0.0.41. If you look at the repo, there are many aspects that are recognizable to Meteor developers today. Before this point, file watching was very basic. However, during this month, many large changes were done such as renaming to Meteor, supporting IE 6 and 7, and changing how file watching is done (part 1, part 2). During the build, it creates a list of files and folders it uses. After building, it uses native file watchers for everything. This causes some issues with creating too many file watchers (especially initially, when it watched each file and folder in the .meteor directory).
One interesting thing is during the build it simply created a list of paths to watch. After the build, it would then check the paths to create the initial state it would watch for changes to. If a file was modified during the build, it wouldn't be detected.
June 2012 Meteor switches to using polling for files, checking them every 500ms, in order to better support vim. Supporting vim seems to be a common problem for bundlers - one popular bundler fixed this same issue in 2019. Switching to polling would have also fixed the previous issues with creating too many watchers.
This introduces an interesting scenario where it might not detect a file change on OSX. Meteor watches the file's mtime (the time when the file was last modified). On OSX, at least at that time, the mtime only had a second precision. If the file was modified twice during the same second Meteor started watching the file (once before the watcher started, and once after), it would not detect the change since the file's mtime would still be the same. This and other issues are addressed a year later by having Meteor read the file content's a second after it starts watching it to check if it changed.
Aug 2013 Watch Sets are added. These are still used today, and fix race conditions that could cause changes to files that happen during a build to not be detected. As Meteor reads files and folders, it adds them to the watch set at the same time. This way, what Meteor uses during the build, and what it uses to compare when looking for changes is the same. In addition, they can be written to disk and read later and reliably used to check if anything changed (this is used to know when local packages should be rebuilt).
Along with this, native file watching was completely removed in favor of only polling. This caused some issues with using a large amount of cpu, but it also worked more reliably.
Dec 2014 In an effort to fix the high cpu usage from using polling, all polling was replaced with native file watchers, using the same file watching library as the Atom text editor,
pathwatcher. This of course had issues when native file watching wasn't available, which was common on the setups people used to run Meteor on Windows (this was before Meteor supported Windows). Meteor tried something that is very tempting (I've thought about trying it too in Meteor before learning that Meteor had already tried), to modify a file to test if native file watchers work. Unfortunately, that creates false positive when Meteor is running an app that is mounted in a virtual machine - native file watchers detect changes done in the virtual machine, but not done on the host machine.
The final solution was to poll files for changes every 5 seconds in addition to using native file watchers. This tries to balance using less cpu than before when it checked every 500ms, while still being able to quickly detect file changes with native file watchers. This wouldn't be as pleasant when native file watchers aren't available, but there was an env var that could be used to go back to the old behavior. The downside though is this approach can create a very large number of native file watchers.
Sept 2016 There was a brief attempt to use chokidar, which is a watching library used by many developer tools. It ended up not working well with Meteor, so Meteor switched back to pathwatcher.
Oct 2016 Optimistic functions were introduced. These have helped to greatly improve Meteor's rebuild time. They cache the results of file operations (stats, file content and hashes, etc) in memory, and use file watchers to know when to invalidate the cached results.
June 2017 There were reports that Meteor was sometimes crashing on Linux. This was eventually tracked down to be caused by creating too many native file watchers. Meteor realized that usually when developing you repeatedly modify the same set of files. Meteor came up with the strategy of first polling all files once every 5 seconds, like it was previously. The first time a file is modified, it starts polling it every 500ms, and also creates a native file watcher which would detect changes even quicker. This is how Meteor still largely works.
Even though I really dislike that delay of up to 5 seconds the first time modifying a file, this has a really nice balance between not using too many resources, working reliably, and detecting changes within a reasonable time frame. My favorite part is this just works for anyone without requiring any config, which was one of the main goals of Meteor (I'm not sure if this still is a main goal; Meteor seems to be moving in the opposite direction). Many developer tools use only native file watchers by default which don't always work, or have a much less efficient polling implementation than Meteor.
Jan 2019 It was discovered that setting up the watchers was adding a few second delay at the end of every rebuild before you could use the app. There were a few optimizations to drastically improve this (sometimes, Meteor would check the same file or folder over 100 times while setting up the watchers). These were included in Meteor 1.8.1. This was probably the point that Meteor was the most efficient at watching.
Oct 2019 As a side effect of a change to improve rebuild performance, Meteor 1.8.2 starts watching many more files than it did before, causing high cpu usage from polling all of the files. A year later we improved this, but we still haven't reduced the number of files being watched to where it was before.
June 2021 Before this point, if a file that could have been used in your app was modified, Meteor would rebuild the app. This also affects the type of rebuild - if a file could be used on the server (except for files in
imports), Meteor would do a slower full rebuild even if the file isn't actually used on the server.
During this month, we finally addressed this. For every file that could be used, Meteor tracks exactly which ones were, and for which architectures the file was used. When a file is modified, Meteor will only do a full rebuild if the file is actually used on the server. If a file isn't used at all, Meteor will do nothing when it is modified.
Sept 2021 On Windows, Meteor has had issues with sometimes getting stuck, especially when creating a new app. It ended up being caused by the
pathwatcher library. As a temporary fix, we started disabling native file watchers for any Meteor command that doesn't really need file watching (creating an app, updating/adding/removing packages, etc.).
Oct 2021 We finally fixed the up to 5 second delay to detect the first time you modify a file. The reason we have the delay is Meteor watches too many files to create a native file watcher for each one. Fortunately, some operating systems (Windows and macOS) have recursive file watchers. This allows you to create a single file watcher for a directory, and get notified for any changes to that directory or any subdirectories. Meteor creates a single recursive watcher for the app, and for each local package used by the app. With this, Meteor is able to almost immediately start a rebuild as soon as you modify a file.
There are three downsides we know of:
- Meteor gets notified for every change, including all of the changes done by Meteor and build plugins while building the app. So far we haven't found any performance issues from this, but it could be possible.
- Neither Windows or macOS watch subdirectories that are symlinks. For now they are watched with polling, though that still has the up to 5 second delay. Someday I want to look into a way to handle these, but it probably won't be soon.
- Linux doesn't support recursive watchers. For now we still use pathwatcher and use the old strategy of only creating native file watchers after a file has been modified for the first time. On linux, the most efficient way is to create a native watcher for each directory. I am waiting for one of the libraries for native file watching to add a way to exclude some directories on linux (such as
.meteor/local) so we don't have issues with too many native file watchers.
I am really happy with the current state of file watching in Meteor now. Most of the time it detects changes instantly, with well tested fallbacks for when native file watchers don't work that do a good job of balancing efficiency and detecting changes quickly, all without you having to think about it or configure anything.
Have a comment? Please Send me an email
- Previous: Optimizing CI for Meteor
Want More? Join my monthly newsletter for new blog posts and news about Meteor and my projects.
Have a Meteor app in production? I run Monti APM for production monitoring to help improve performance and reduce errors.