How and where logs are stored on a server

Published on 2026-09-12 by frfrand

Dónde busca cada cosa, qué cambió con journald, por qué un log sin rotación llena el disco, y los comandos que resuelven la mayoría de las búsquedas.

When something fails, the answer is almost always already written somewhere. The problem is knowing where.

Two systems coexisting

On a modern Linux there are two places where logs go, and it's worth understanding the difference.

journald (from systemd) stores them in an indexed binary format, with metadata: which service each line comes from, which user, which process, with what priority. It is queried with journalctl.

Text files in /var/log still exist. Many services (nginx, for example) write their own files directly, and in many distributions rsyslog copies part of the journal to files for compatibility.

That duplication explains a frequent confusion: the same message can appear on both sides.

The commands worth memorizing

For a service:

journalctl -u miapp -f # live journalctl -u miapp --since "1 hour ago" journalctl -u miapp -n 200 # the last 200 lines

For system errors:

journalctl -p err -S today

The -p option filters by priority, with the classic syslog scale: 0 emergency, 3 error, 4 warning, 6 informational. Filtering by err leaves out the noise and shows what matters.

And for text files:

tail -f /var/log/nginx/error.log grep -i "timeout" /var/log/nginx/error.log | tail -50

A shortcut that saves time

If nginx returns 502, the problem is almost never in nginx: it means your application did not respond. The error is in your application's log, not in nginx's.

A 504 is different: your application did respond, but it took longer than nginx expected.

Knowing that difference saves many searches in the wrong file.

Rotation, and why the disk fills up

A log grows indefinitely if no one cuts it. Rotation takes care of that: it renames the file, starts a new one, compresses the old ones, and deletes those that exceed retention.

For text files, logrotate does it, with its configuration in /etc/logrotate.d/. Each service leaves its file there.

For the journal, the limit is configured in /etc/systemd/journald.conf:

SystemMaxUse=500M

Without that limit, the journal can grow until it occupies a considerable percentage of the disk.

And the command you will want the day the disk fills up:

journalctl --disk-usage journalctl --vacuum-time=7d # deletes anything older than 7 days

Two details that bite

A log rotated while a process has it open. If rotation renames the file and the process keeps writing to the old descriptor, the space is not freed and new lines go to a nameless file. That is why rotation configuration includes notifying the process to reopen its files. When you see that the disk does not free up after rotating, it is usually this.

Container application logs do not go to the journal by default: the container engine captures them. If you don't configure rotation there, those files grow out of control. It is a very common cause of full disks on servers with Docker.

What is worth logging

More is not better. A log with everything is as useless as an empty one, because no one reads it.

Three habits that make logs useful:

• One identifier per request, appearing in all related lines. It allows reconstructing what happened in a specific case.
• Structured format (JSON) if you are going to process them with tools. Plain text if you are going to read them with your eyes.
• Never sensitive data. Passwords, tokens, and personal data do not go to the log. It is one of the most common forms of leakage, and it almost always happens through a debugging session that was left on.

In ArduMaker, the logs of each service and each deployment are viewed from the panel, with their history, without having to enter via SSH to look for them.