Background jobs, while being used in all web applications (processing webhooks, interacting with 3rd party APIs, or powering core features), did not benefit from the developer experience improvements that arose in all other layers of the Node.js API stack: quick and reliable databases with Supabase or easy Serverless deployment with Vercel.
Today, even for simple use cases, working with background jobs in Node.js necessarily requires some infrastructure knowledge—either by deploying and scaling an open source solution (ex: BullMQ) or using an IaaS such as AWS SQS with Lambdas, which comes with complexity and limited features (no support for dead letter queues, dynamic concurrency, or throttling).
At a large scale, you will need to solve how to handle rolling restarts, how to auto-scale your workers, how to safely deploy without interrupting long-running jobs, how to safely encrypt jobs’ data, and how to version them. Once deployed, your background job’s code lives in a separate part of your codebase, with its own mental model (queues and workers). Finally, most solutions provide technical dashboards which are not always helpful in debugging production issues, so you end up having to build custom dashboards.
Most companies we talked to try to handle those different aspects, building custom similar solutions and using developers’ time that could have been used on user-facing features.
Bryan and I are technical founders with 10+ years of experience working at start-ups of all stages (e.g. Algolia, home of HN Search!), from tech lead to CTO roles. Like many developers, we got asked many times to work on background job stacks and invest time into tailoring and scaling them for product needs.
I even dedicated most of my time at Algolia to building a custom background jobs pipeline to power the Algolia Shopify integration: ingesting partial webhooks from Shopify, enriching them given customers configuration, in FIFO order per shop, with the Shopify rate limited API, for thousands of shops and the equivalents of 3 millions of jobs per day. Given the complex and unique product requirements of the Algolia Shopify Ingestion Pipeline, the only solution (at the time and context) was to build a custom background jobs stack combining Redis and Kubernetes.
When consulting with some startups, we witnessed some developers choosing to keep some slow API routes calling 3rd party APIs synchronously instead of investing time in setting up background jobs. When looking back to the recent increase of productive zero infrastructure solutions in the Node.js ecosystem, we were surprised that the experience with background jobs remained unchanged. We decided to build Defer, so working with background jobs, CRONs, and workflows would match the current standard of Node.js developer experience.
Inspired by Next.js, Remix, and Netlify design, background jobs in Defer become background functions that live in your application’s code, with direct access to all configuration options: retry, concurrency, and more (https://docs.defer.run/features/retries-concurrency/) , and no specific mental model to learn. Your background functions get continuously deployed from GitHub with support for branch-based environments, allowing you to test new background jobs in no time, before safely moving to production.
Defer works for all kinds of Node.js projects, not only serverless ones. It does not require you to learn any new architectures or adapt your system design—you just turn your code into background functions using coding patterns you already know, ex: map-reduce, or recursion. Defer brings features such as configurable retries (advanced backoff options), throttling, and concurrency at the background job level, which other solutions either require you to implement yourself or are simply not available. Finally, the Defer Dashboard is the only background jobs Dashboard to allow developers to quickly find executions based on business/product metadata, ex: “Show all executions for `user_id=123`) to quickly debug product issues.
Defer’s infrastructure, written in Go, is composed of 3 main components: a Build pipeline, a Scheduler, and a Runner. The Build pipeline enables us to build any Node.js project without requiring any configuration file (https://docs.defer.run/platform/builds/). The Scheduler relies on Postgres for persistent storage of your jobs (no risk of losing some)—all jobs’ data is encrypted—and on Redis, as an atomic counter to handle features such as concurrency and throttling (https://docs.defer.run/platform/executions/). Our infrastructure runs on AWS EC2 - leveraging auto-scaling groups, using the containerd API directly from Go.
We run a progressive deployment approach to enable uninterrupted long-running jobs (some of our customers’ jobs run for more than 5h) while releasing updates multiple times a day. Once your application is up and running, the Defer dashboard gives you all the essential information to operate background jobs: activity histograms, performances, and Slack alerting upon failures. The executions list comes with rich filters, allowing you to quickly find all the executions linked to a specific customer or other business metadata.
In short, we ensure that you get all the essential features, with the best developer experience, and with a fully managed infrastructure and observability tools so you can focus on building your product.
All of this would be meaningless without a free plan for small and side projects and usage-based pricing, so that’s what we offer: https://www.defer.run/pricing. If you want to give Defer a try, you can get started with a simple GitHub login, without any credit card information required, and our docs are at https://docs.defer.run.
We would love to get to read about your experience with doing background jobs in Node.js and feedback on what we’ve built. We look forward to your comments!
by Andrey
Azimov