I remember getting a simpler form of a priority ordered job processing system for a take home interview question. I declined to do it. Just wasn't interested in creating usable compilable code for someone else outside of a paid job.
In creating my new Product for task management, I required a way to do job execution to schedule account oriented operations outside of API interactions. When tasks get created they need to be searchable, so a job is submitted to execute that asynchronously. The jobs would have to occur at least once and others would be repeatable. Users will want to remove reminders, so the jobs needed to be cancel-able. Some jobs will have higher priority than jobs.
- One bonus was supporting backlog execution such that it did not get executed on schedule then the job would be executed as many times as needed until it caught up with its schedule.
The interesting part of the architecture is that it is not a service, but a library to enable services (any application really) to perform jobs. APIs and other services can submit jobs. If I wanted to scale up later such that a service was dedicated to job processing then the library enables it from the get go. Many consumers can take tasks and execute them from the queues they are subscribed to. Consumers cooperate, on the queues they are subscribed to, to keep jobs moving through the system which I think is unprecedented. 😁🥳 Consumers should be allowed to fail while other consumers can pick up the job. I have seen no open source systems or libraries that do what I wanted to do.
The most important part of this was the facilitation of asynchronous job
execution using existing applications with a goal of minimizing
resource usage. A library fit this easily by enabling decentralized job
execution via message passing between applications on a persistent store.
I have no dedicated job processing services, nor do I need them right now, and this is acceptable at a small scale. As I scale up, I will want more capacity, so spinning up another service is easy. If the jobs being processed take too much resources then I can further scale by putting dedicated task processors for specific queues. If I need to scale down, then remove job processor instances. Scale up and scale down isn't something I've seen from any job processing system.
Resumption is supported by natively supporting job states via contextual data. This enables checkpointing, so that a job does not need to keep executing the same things over and over.
This library is built in NodeJs using Redis as the data storage. Why Redis? Because Redis is a great high performance database with easy backups. The amount of data structures it supports enables great applications and libraries to be built like this library. 😀 The age old thinking that Redis is just a cache needs to be aged out because it relegates it to caching layer, creates more work supporting a second database such as MySQL and more cognitive overhead.
Is it Open Source? No. No plans to do that.
Future for this library? Execution of many many jobs. 😁