Tracking issue for kv_unstable (structured logging)

Tracking issue for the implementation of #296. See also #149

Initial implementation (Build out initial key values API #324)
Integration with frameworks and loggers:
- slog
  - Zero-allocation conversion between slog::Value and log::kv::Value. We should be able to do zero-allocation conversion from log to slog, but may need to allocate from slog to log.
- tracing (log: Support structured values tokio-rs/tracing#341)
  - Zero-allocation conversion between tracing::Value and log::kv::Value in the event! macro (Make it easier to create a Value from owned or erased external types #339)
  - Capture std::error::Errors as Values
- fern
- async-log
- std-logger (Print key values Thomasdezeeuw/std-logger#14)
  - Some way to tell whether a Source has any key-value pairs (Add a count method to Source #329)
- femme
- env_logger (Add support for Key-Value data in log records. rust-cli/env_logger#137)
- antlog (an experimental wrapper to prove out requirements)
  - A way to get specific values (Add a Source::get method for finding a single value #335)
  - A way to coerce a value to a common type (Support coercing structured Values into primitive types #379)
- Please suggest others to track!
Explore macro support (key_value capable log macros #343)
- log!("msg {}", interpolated, { key: value }) (kv macro support #353) - We ended up having to revert this in log, but are experimenting externally
- log_kv_macro
- log_macros
Integration with serialization frameworks:
- serde
  - Conversion between serde::Serialize and std::fmt::Debug (Add key and value methods to DebugMap rfcs#2696). See serde_fmt.
- sval (Add structured support to kv::Value using sval #342)
Micro benchmarks
Documentation and examples
Stabilization review
Open questions
- Should Source::count return a Result<usize> instead of a plain usize?

Add support for Key-Value data in log records.

@KodrAus

@KodrAus thanks for all the work you've put into this (and linking to this issue from a forum post).

I was curious: is there a release planned that includes the beta version of kv? I actually wanted to use the beta for async-log, but given there wasn't any release yet it wasn't possible 😅

@yoshuawuyts

Hi @yoshuawuyts! 👋

Right now any release of log we publish will include this API under the kv_unstable feature gate, but we're not ready to commit to backwards compatibility just yet. So if you depend on log from git then you can access the API:

[dependencies.log]
git = "https://github.com/rust-lang-nursery/log.git"
features = ["kv_unstable"]

I think what we need are a few draft PRs in some of these log frameworks that tease out any breaking changes we should make, and then will be able to publish a release that replaces the kv_unstable feature gate with a kv one.

Right now any release of log we publish will include this API under the kv_unstable feature gate,

Oh oops, maybe I was being a bit vague. The last release on crates.io was done in October 2018, which means currently no releases ship with kv_unstable on crates.io. Which in turn (I believe) means we can't ship any versions to crates.io.

In order to help test this feature, it would be nice if there was a release on crates.io with the ["kv_unstable"] feature available. For experimental technology such as async-log I think we can get away with publishing a version that directly depends on this feature, and keep updating it as it's polished.

So my question is: could a version be of log be published to crates.io that includes the experimental flag?

P.S. Thanks for taking time out of your day to reply to my questions!

In order to help test this feature, it would be nice if there was a release on crates.io with the ["kv_unstable"] feature available. For experimental technology such as async-log I think we can get away with publishing a version that directly depends on this feature, and keep updating it as it's polished.

Ah this sounds fair enough. I think we can get pretty far without needing to break anything, but it's probably going to happen at some point. But right now it is harder for folks to get started with these new APIs, so once #339 lands I'll put together a release so you can depend on it from crates.io and browse the docs on docs.rs.

@KodrAus

@KodrAus in the top-level post there's an entry about "Explore macro support". Do you have any plans to experiment with this?

We've integrated key-value logging in lrlna/femme#1; all that's missing now is a way to create key-value pairs and send them to a logger. Even a small code example would be helpful, but ideally there'd be a small log crate that could work with this.

Also: is it perhaps useful to open a separate issue to track/discuss logging macros?

edit 2019-07-21:
Ah, I think I figured out how to log a kv-pair!

let record = log::Record::builder().key_values(kv_pairs).build();
log::logger().log(&record);

Ah yes that Record::key_values method is the way you can attach key-value pairs to a record in the absence of macro support. It needs to be a &dyn Source.

Opening a separate issue to design macro support sounds like a good idea! We’ve got a couple of constraints but also plenty of opportunity to improve things.

Macro issue created: #343

Created a crate for shared types in key-value logs (to mark pairs of logs) -- https://github.com/yoshuawuyts/log-types. Not sure yet how useful this is, but I think it might be a fun reference of how to create source/key/value structs, and probably a useful experiment (:

Having some standards around well-known key-value pairs is something I've also been exploring in a little wrapper. It seems like it'll be useful for a more composable logging ecosystem.

@KodrAus

@KodrAus Yeah for sure! -- I wonder which others to add; I only really thought of spans::{start,end} and http::{request,response}. Though I guess timestamps, host, ip might be common too?

Oh also, I saw femme is not listed yet; could we perhaps add it to the tracking issue?

Merge #199

I'm hesitant to add to the list of todos because I really want this feature, but it might be worth adding fern as another crate to integrate with. I don't use it (I use slog), but it seems like it has quite a lot of usage (600,000+ all time downloads on crates.io).

@KodrAus

@KodrAus I took a look at value-bag (from the perspective of using it in the log crate). Overall I think it looks good 👍

For the capture_* and from_* methods I have the same comment as I made here: #584 (comment) (so let's continue the discussion about this there).

We could get the size down from 3 to 2 words (on 64 bit) by using tagged pointers, but I don't know if that is worth all the unsafe code.

One final note is around the Fill trait, which isn't used by the log crate, but it was a little surprising to me that the Fill::fill method seems to be called multiple times on every access (?). That wasn't clear to me by (quickly) scanning the Fill docs. Maybe a note in the docs would be sufficient.

We could get the size down from 3 to 2 words (on 64 bit) by using tagged pointers, but I don't know if that is worth all the unsafe code.

Thanks for this work. Looking forward to it being released proper.

Some systems generate a significant amount of log data in debug mode, so it might be worth the effort, but personally I wouldn't want a change at this point to hold up release of this feature. It seems more important that structured logging become a stable feature sooner rather than later.

Add support for Key-Value data in log records.

log is used by over 50000 crates, which means you'll inevitably pull it in as a transitive dependency. I'm therefore concerned about value-bag being added as a dependency here. Will this remain an optional, non-default feature?

@KodrAus

log is used by over 50000 crates, which means you'll inevitably pull it in as a transitive dependency. I'm therefore concerned about value-bag being added as a dependency here. Will this remain an optional, non-default feature?

I'm afraid not, it's the core of the Value type (about 50% of this feature ;) )

Maybe we can move it to the rust-lang org? /cc @KodrAus

feat: Add support for Key-Value data in log records.

feat: Add support for Key-Value data in log records

@chloekek

@chloekek I think that's a reasonable concern. In fact, in #613 I've made that dependency fully optional for structured logging, so if you just want to log simple primitives, like numbers, strings, booleans, or formattable values, you won't pull it in.

For some background on what that value-bag library is (as well as a serialization framework called sval), they all started out as the implementation directly here in log, but grew in complexity to a point where it seemed best to split them out. I didn't originally try move them to a repository in the rust-lang organization I think because I didn't want it to carry any sense of being an "official" solution to dynamic values. So I created an organization: https://github.com/sval-rs and it's been living and continuing to evolve there.

I would happily move that repository anywhere or add any collaborators of log to that organization 🙂

feat: Add support for Key-Value data in log records

Navigation Menu

Repository navigation

Tracking issue for kv_unstable (structured logging) #328

40 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking issue for kv_unstable (structured logging) #328

Description

Activity

yoshuawuyts commented on Jul 2, 2019

KodrAus commented on Jul 4, 2019

yoshuawuyts commented on Jul 4, 2019

KodrAus commented on Jul 6, 2019

yoshuawuyts commented on Jul 21, 2019

KodrAus commented on Jul 21, 2019

yoshuawuyts commented on Jul 22, 2019

yoshuawuyts commented on Sep 1, 2019

KodrAus commented on Sep 1, 2019

yoshuawuyts commented on Sep 1, 2019

atroche commented on Sep 17, 2019

40 remaining items

Thomasdezeeuw commented on Sep 4, 2023

scripturial commented on Sep 15, 2023

zopsicle commented on Jan 9, 2024

Thomasdezeeuw commented on Jan 9, 2024

KodrAus commented on Jan 31, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions