TSDB Birds Eye View Growing pains are common and databases are no exception. A year and a half ago, the Loki team started talking about how to approach order of magnitude improvements in cardinality, query throughput, and reliability. As we put the finishing touches on our new index layer, let’s take a look at how we’re trying to stay ahead of the curve. Loki’s new index is built atop a modified version of TSDB.

Loki is multitenant by default, meaning that it can ingest, store, and query for different users (teams, organizations, etc) in the same process. This has some attractive benefits, namely that it’s much more economical to run a single database with multiple tenants than multiple databases with single tenants. Loki does not have any sophisticated authentication on it’s own, so tenancy is determined by a special X-Scope-OrgID http header attached to all requests.

I get asked this all the time, so here’s a list of things to take care of before running Loki in a production environment: Collect it’s own logs Make sure Loki is ingesting it’s own logs. This can be done just like you would for any other application (i.e. via promtail) and ensures that when something goes wrong, you’ll have logs available to debug. For an even more reliable setup, log them to a different Loki cluster, but this may not be feasible depending on your scale/investment.

What is Loki? Loki is an open source log aggregator with a narrow focus: to help you understand your applications. It’s built for the needs of software operators and has been continuously refined in the course of scaling Grafana Labs. It’s the tool we built for ourselves, seamlessly integrating with open source tools. It is cloud first, Kubernetes first, Prometheus first. In other words, Loki is a time series database for strings.