My trusty s3-publish module fell over when I last attempted to update my blog.
The error informed me that the maximum call stack size had been exceeded. In retrospect, it’s not hard to see why.
Escaping Callback Hell
At the time I wrote
s3-publish, I was extremely enamored with the async and neo-async modules. So much so that I began to see every problem as a nail to be hammered with a callback-based queue. Eventually, my static blog was comprised of enough files to exhaust the maximum callback stack. Yikes.
Clearly, my approach had to be completely rethought. As luck would have it, I have since become very fond of the
async/await syntax which is a much better fit. Adding a dash of p-limit allowed a specific number of promises to be executed in parallel and I was off to the races.
Redesigning the API
As part of the rethought approach, I decided to expose a proper API this time. Version 1x of
s3-publish was first and foremost a CLI and it’s implementation did not give much thought to exporting the functionality in useful bundles.
s3-publish is now a meta package that bundles the CLI and all dependencies with webpack.
- @s3-publish/core introduces the concepts of a
Operationobjects) and a
Planner(that plans operations)
- @s3-publish/provider-fs implements a provider capable of reading/writing to the local file system
- @s3-publish/provider-s3 implements a provider capable of reading/writing to Amazon S3
- @s3-publish/cli exports a runnable script and configuration utilities
- @s3-publish/loggers implements log message handling for the CLI
Unfortunately, support for the old configuration schema had to be dropped in favor of something more versatile. See the ConfigFile interface documentation for details.
The default configuration file has been renamed to .s3p.config.js and the new
schemaVersion is 2.
--config option can be used to specify an alternate config file name if you’d prefer to keep using s3p.config.js.
If any users have trouble migrating their s3p.config.js files to .s3p.config.js files, please create an issue on GitHub and I’ll do my best to help.
- Now supports local directory targets and S3 origins (meaning files can be transferred FS→S3, S3→FS, S3→S3, FS→FS)
- Direct S3→S3 transfers via COPY request
- Greater control over all types of request parameters via delegate objects
- Optionally load ignore patters from a .ignore file (supports gitignore syntax)
- Ability to ignore target files
- Support for custom providers and loggers
Along the way,
s3-publish has spawned a few sister projects. These are separate in that they are not part of the monorepo, but they were created to address a specific need in
- argv-walk - walk an argv array with a handler function using minimist conventions
- twisters - display multiple simultaneous animated spinners
- typedoc-plugin-jekyll - create a .nojekyll file in the docs output
- typedoc-plugin-lerna-packages [FORK] - organize docs by lerna package
- typedoc-plugin-resolve-by-name - resolve interface references by name
I’m quite proud of how this rewrite turned out and I hope it’s useful to folks. If I broke something you rely on, feel free to create an issue on GitHub.