It's a big step forward. I've talked about Data Collection, Monitoring and Visualization before, so this has always been on my mind. Just glad to get something down now.
My Idea has been a TSDB/LogDB on every machine that can send to a historical store. One should be able to query and stream observable data from any machine from an authorized point of access to diagnose issues. With that streaming capability, my idea has been that you could create rules that trigger violations and enough/combination of violations would trigger incidents that would notify people. Visualization would be via a UI that allows explorations, dynamic dashboards and saving snapshots of data either locally or remotely. Templating is a big part of it all.
Obviously something that did all of this is a sell-able product, but do I want to run a SaaS to provide that? No. Datadog is probably good enough for most people since they are typically reactionary i.e. out of mind out of sight or don't think about it don't see. I wouldn't have too much fun running it as a business.
The TSDB is built in RUST lang and on top of RocksDB, of course. I make use of finite state transducers too because lets be honest it's hard otherwise to provide fast tag search without it. Inverted index, roaring bitmaps, quantile compression, etc. Lots of advanced stuff that most people will never use.
I support maximum resolution up to 100 nanoseconds. I don't support anything higher resolution than that as it is important to avoid u64 as much as possible. I don't want to blow up space requirements where every record has unnecessary u64 or even u32 when u8 or u16 can save factor of 2 - 8 times space. Relative time and resolution setting slash numeric datatype sizes. Each record is pretty much u16 + value (u64 or f64) instead of u64 + (u64. or f64). If you have millions of data points, the space savings are enormous think 10mb vs 16mb = 37.5% constant savings. Compression on top of that saves even more space.
There's obviously more technical details to the metric storage, but I'll stop here.
I fully intend to add log support and log search to it. Nothing crazy just saving logs and making use of whatever query language is there with the library used. i thought about using an FST to store the logs, but while querying would be fast, it would be annoying trying to create ways to expose what an FST provides. While it would be fastttttttt to query, it is not ideal. I have a library in mind already as I've used it before like rocksdb.
Network communication library uses ZeroMQ. Technically, its a private library that provides async RPC, Pub/Sub Event support on top of ZMQ. Serialization is all MessagePack. RMP-Serde baby! I've talked about this private network communication library before. I use it in all of my applications. I'm sure I'll open "sores" it some day.
This TSDB scales from small to big. It is only one step though of a multi tier architecture.
I've thought about making this public but nah. There are too many libraries private included.