My trusty s3-publish module fell over when I last attempted to update my blog.
The error informed me that the maximum call stack size had been exceeded. In retrospect, it’s not hard to see why.
Escaping Callback Hell
At the time I wrote s3-publish, I was extremely enamored with the async and neo-async modules. So much so that I began to see every problem as a nail to be hammered with a callback-based queue. Eventually, my static blog was comprised of enough files to exhaust the maximum callback stack. Yikes.
Clearly, my approach had to be completely rethought. As luck would have it, I have since become very fond of the async/await syntax which is a much better fit. Adding a dash of p-limit allowed a specific number of promises to be executed in parallel and I was off to the races.
Redesigning the API
As part of the rethought approach, I decided to expose a proper API this time. Version 1x of s3-publish was first and foremost a CLI and it’s implementation did not give much thought to exporting the functionality in useful bundles.
To that end, I also decided to give yarn workspaces a big hug and make the project a monorepo aided by lerna.
s3-publish is now a meta package that bundles the CLI and all dependencies with webpack.
- @s3-publish/core introduces the concepts of a
Provider(that providesFileandOperationobjects) and aPlanner(that plans operations) - @s3-publish/provider-fs implements a provider capable of reading/writing to the local file system
- @s3-publish/provider-s3 implements a provider capable of reading/writing to Amazon S3
- @s3-publish/cli exports a runnable script and configuration utilities
- @s3-publish/loggers implements log message handling for the CLI
Rethinking Configuration
Unfortunately, support for the old configuration schema had to be dropped in favor of something more versatile. See the ConfigFile interface documentation for details.
The default configuration file has been renamed to .s3p.config.js and the new schemaVersion is 2.
The --config option can be used to specify an alternate config file name if you’d prefer to keep using s3p.config.js.
If any users have trouble migrating their s3p.config.js files to .s3p.config.js files, please create an issue on GitHub and I’ll do my best to help.
Improvements
- Now supports local directory targets and S3 origins (meaning files can be transferred FS→S3, S3→FS, S3→S3, FS→FS)
- Direct S3→S3 transfers via COPY request
- Greater control over all types of request parameters via delegate objects
- Optionally load ignore patters from a .ignore file (supports gitignore syntax)
- Ability to ignore target files
- Support for custom providers and loggers
Project Sprawl
Along the way, s3-publish has spawned a few sister projects. These are separate in that they are not part of the monorepo, but they were created to address a specific need in s3-publish:
- argv-walk - walk an argv array with a handler function using minimist conventions
- twisters - display multiple simultaneous animated spinners
- typedoc-plugin-jekyll - create a .nojekyll file in the docs output
- typedoc-plugin-lerna-packages [FORK] - organize docs by lerna package
- typedoc-plugin-resolve-by-name - resolve interface references by name
Wrap Up
I’m quite proud of how this rewrite turned out and I hope it’s useful to folks. If I broke something you rely on, feel free to create an issue on GitHub.