Ran into a couple of articles today about async work, job queues, and worker pools! Love this stuff. :)

  • Atlanta, Job Queues, Batch-wise Operations: In which they go with a single, fast consumer written in Go and this gives them lots of runway until the next evolution of their process. They go to tasks of batched items from single-item tasks long before they’ll have to think about adding workers. (Small, simple architecture)
  • Transactionally Staged Job Drains in Postgres: Talks about how to enqueue a job for work later during a transaction when transactions can fail and be rolled back. (On rollback, any entities the job needs to proceed may have not been committed to the database nor will they ever be.)