- Handling billions of invocations – best practices from AWS Lambda: Request processing for aws lambda has evolved over time to be able to scale with a growing number of requests per second. Techniques for management async usage are included and become more and more sophisticated:
- single shared queue for all tenants
- multi-queue for tenants random placement
- multi-queue consistent hashing
- multi-queue random power of 2
- multi-queue shuffle sharding (choose n queues for a tenant, pick smallest backlog queue for new messages)
- Data Aggregation: S3 Table Buckets with DuckDB & CRON Extension, Using Arrow Server to Serve Aggregated Data: DuckDB has a cron plugin that is used to refresh a materialized view periodically based on data it’s able to pull in from S3 Tables
- 10 Years of Engineering Ladders: One of the earliest engineering ladders I can remember reading back in the day …
- shadcn/ui Build your component library: UI components that can be used in most popular frontend frameworks (eg react, vue)
- Why is observability so expensive?: Matt Klein creator of envoy talks about our default inclination to collect all the telemetry. Do we have to do this? He’s working on tech that would allow a system to run relatively lean in terms of metrics, logs, traces collection but that can be dialed up as needed. Collecting perf data from a system can be extremely expensive and the vast majority of it is never looked at.
- Another observability 3.0 appears on the horizon: Counter point to Matt’s arguments about data collection for systems and what is given up when we monitor less. (There is value in thinking about aggregation at metric write time to the tsdb but we still collect some things …)
