Commit graph

27 commits

Author SHA1 Message Date
Lambda
99924e5779 Add admin commands to get and reset tracing filters 2025-02-17 17:41:14 +00:00
Lambda
5eab758bd2 Make tracing filter reload handles mutable
Makes the following diff nicer.
2025-02-17 17:40:55 +00:00
Charles Hall
e74c8687f5
drop dependency on once-cell 2024-11-07 11:36:13 -08:00
Charles Hall
b03c2a15b3
add observability infrastructure for cli subcmds 2024-10-25 11:27:11 -07:00
Charles Hall
6ab87f97dd
include traceresponse header if possible
This can help with debugging.
2024-09-26 19:01:15 -07:00
Charles Hall
b0d1cc1b63
bump otel to v0.24.0
Someone contributed opentelemetry-prometheus support for v0.24 and this
version also doesn't put stupid requirements on the tokio version. This
version of the OTel ecosystem also fixes an apparent bug with some hacks
I plan on doing in the future...
2024-09-23 14:22:55 -07:00
Lambda
14afa1357e tracing: allow configuring service name
This is essential when consuming tracing data from multiple servers.
2024-08-24 19:31:39 +00:00
Lambda
2c19abc535 Add OnDemandHashMap and TokenSet 2024-07-22 18:07:00 +00:00
Lambda
f89e1c7dfc
Allow tracing filters to be changed at runtime
ReloadHandle is taken from conduwuit commit
8a5599adf9eafe9111f3d1597f8fb333b8b76849, authored by Benjamin.

Co-authored-by: Benjamin Lee <benjamin@computer.surgery>
2024-06-27 16:04:23 -07:00
Lambda
e0e7d8fd91
Make observability more configurable
The following are now configurable:

- the OpenTelemetry endpoint,
- the tracing-flame filename, and
- whether the log output should include timestamps (useful to disable if
  it goes straight into journald).
2024-06-27 16:04:23 -07:00
Lambda
df571818f1
Make tracing filters configurable per backend 2024-06-27 16:04:23 -07:00
Lambda
98d49554ce
Move observability config to separate config section
This renames:

allow_prometheus -> observability.metrics.enable
allow_jaeger -> observability.traces.enable
tracing_flame -> observability.flame.enable
log -> observability.logs.filter
log_colors -> observability.logs.colors
log_format -> observability.logs.format

New config values in these sections will follow.
2024-06-27 16:04:23 -07:00
Benjamin Lee
a909e2079b
config options for log format and color usage
We want to be able to disable colors for complement logs (since they
are likely to be opened in a text editor). There's no pressing need for
alternative log formats, but I'm interested in whether the 'pretty'
format will be easier for debugging.

I chose to add 'log_*' options rather than making a separate 'log'
section for now. There's been some discussion about trying to separate
the tracing/logging stuff into more structured sections, but that can
happen later.
2024-06-13 21:21:14 -07:00
Charles Hall
c46eaed0e0
parse configured EnvFilter once
This allows the error handling to be done upfront instead of for each
use. In particular, the `toml` error now points to the span of text in
the config file where the misconfigured EnvFilter value is. This is much
better than the previous error that did not indicate what was actually
causing it to happen.
2024-06-12 19:56:59 -07:00
Charles Hall
0c2094a56f
record FoundIn with metrics instead of traces
This is much more efficient in terms of network use and data storage,
and also easier to visualize.
2024-06-06 20:56:36 -07:00
Charles Hall
9364d44ce2
use strum instead 2024-06-05 17:41:39 -07:00
Charles Hall
792300d220
derive Copy on FoundIn 2024-06-05 17:41:39 -07:00
Charles Hall
784b86a1f6
make stringified names mirror variant names 2024-06-05 17:41:39 -07:00
Charles Hall
38382128ac
rename FoundIn::{value -> as_str} 2024-06-05 17:41:39 -07:00
Charles Hall
22dd7f1a54
move FoundIn to observability.rs 2024-06-05 17:41:36 -07:00
Charles Hall
04ecf4972e
record histogram of http requests 2024-06-03 18:18:44 -07:00
Charles Hall
a0b92c82e8
set up opentelemetry for metrics
Also adds an `allow_prometheus` option (disabled by default) to expose
a `/metrics` endpoint that returns Prometheus data.
2024-06-03 18:18:44 -07:00
Charles Hall
94fda7c875
improve usage of Resource
* Extract into reusable function (we'll need this later)
* Merge the values we want to override over the defaults, instead of
  dropping the defaults completely
* Don't unnecessarily allocate (the `vec![]` usage is gone)
2024-06-03 18:18:44 -07:00
Charles Hall
507de063f5
don't hardcode EnvFilter for tracing-flame 2024-05-27 10:06:26 -07:00
Lambda
7a154f7416
observability: allow combining multiple tracing layers
Previously, jaeger, tracing_flame and stderr output were mutually
exclusive. There's no good reason for this.
2024-05-27 10:03:44 -07:00
Lambda
263edcc8a1
observability: don't drop tracing_flame drop guard until exit 2024-05-27 10:02:42 -07:00
Charles Hall
c17ab5328d
factor observability things into its own module
And also create a new error for the observability initialization.
2024-05-27 10:01:00 -07:00