If that sounds familiar, your logs are not the problem. Your logging strategy is.
Every Linux system generates logs. The problem isn't that there isn't enough data, it's that there isn't enough useful data. If you don't handle your linux log management well, your observability layer could become a problem. On the other side, a good logging technique decreases the time it takes to fix problems from hours to minutes. This post talks about the 10 linux logging best practices that will help you tell the difference between a system you can troubleshoot with confidence and one that makes too much noise at untime.
These criteria apply to all Linux distributions, whether you're using a RHEL server, a CentOS stack, a distributed Ubuntu environment, or any other Linux distribution. You may run a real terminal example for each rule right away. No theory without evidence.
Logging controls what gets recorded. File access and ownership are separate concerns.
If you need to restrict who can read your log files, that is a job for
chmod and
chown.
Rules
- A Linux system: RHEL, CentOS, or Ubuntu (all examples tested on these)
- sudo or root access — some log paths and journalctl filters require elevated privilege
- systemd running — Rules 08 and 09 use
journalctlandlogrotate - Basic familiarity with the terminal — if you're new, start with
Basic Linux Commands first
logger [OPTIONS] MESSAGE # write a message to syslog
tail -f FILE # follow a log file in real time
grep PATTERN FILE # search log output for a pattern
awk '{print $0}' FILE # parse and process log fields
Linux Logging Best Practices Start With Writing for the Reader, Not Yourself
When you write a log entry, picture a colleague — exhausted, on-call, mid-incident — who has
nothing but your log output and a deadline. Every log entry you write is a message to that person.
Make it count. The single biggest shift in your linux logging best practices is writing for the
reader, not the author.
INFO Processing finished
INFO Payment confirmed userId=u_9921 orderId=ORD-4477 result=SUCCESS latencyMs=84
The second version tells you who did it, what happened, and how long it took — without opening a single database query.
More Output Is Not More Visibility — Log with Purpose Only
"Just add more logs" is advice that sounds helpful and destroys your system. Every log line costs
you disk I/O, storage space, and — most critically — the time it takes a human to read through it.
Flooding your output with loop-level noise buries the entries that actually matter.
for item in "${items[@]}"; do
echo "INFO Processing item: $item"
process "$item"
done
for item in "${items[@]}"; do
process "$item" && (( success++ )) || (( failed++ ))
done
echo "INFO Batch complete: success=$success failed=$failed"
INFO Batch complete: success=9987 failed=13
Heavy synchronous logging can reduce application throughput by 30–50%.
Always summarise loop output and use asynchronous log writers in production environments.
See Linux System Monitoring for performance tooling.
Understand What Each Linux Log Level Means — and Never Mix Them Up
Log levels are a contract, not a suggestion. The moment you log a wrong-password attempt as
ERROR, your on-call alert fires for something that happens a hundred times a day.
That's how alert fatigue starts. Here is the catch-block mistake that most teams make:
log.error("Login failed", e) # wrong — this fires your pager every time
log.info("Login attempt: bad credentials", userId) # expected path
catch Exception e:
log.error("Auth system fault", userId, e) # true system failure
Ask yourself: does this require a human to wake up and act right now?
If yes → ERROR. If it might become a problem later → WARN.
If it's just tracking what happened → INFO.
Never Write Sensitive Data Into a Log File — Mask It Always
This is the rule with the most severe consequences when broken. A log file with a plaintext
password, API token, or card number sitting in /var/log/ is a data breach waiting
to happen. It is also the easiest rule to follow — mask before it hits disk, every time.
The following must never appear in any log file under any circumstance:
- Passwords and passphrases
- API tokens, session keys, OAuth secrets
- Credit / debit card numbers (PAN)
- National ID numbers, passport numbers, SSN
- User home addresses and geolocation data
Every Useful Log Entry Needs These Six Fields — No Exceptions
A log entry without context is a riddle. Here are the six fields that make every entry
self-contained — readable and actionable with no additional lookups.
Structured JSON Logs Outperform Plain Text in Production
Plain text logs are written for humans to skim. Structured JSON logs are written for machines
to index, search, and alert on — and still perfectly readable by humans. In any linux log
management setup that uses ELK, EFK, Loki, or Datadog, unstructured text is a dead end.
"timestamp": "2025-10-02T14:22:15Z",
"level": "INFO",
"service": "order-center",
"traceId": "c7d2eb9f",
"message": "Payment confirmed",
"userId": "u_9921",
"orderId": "ORD-4477",
"amount": 149.00,
"latencyMs": 84
}
JSON structured logs are machine-readable and work directly with
ELK,
EFK, Loki, and Grafana without any custom parsing rules. Make JSON the default format for every production environment
from day one. See the Linux System Monitoring Cheat Sheet for toolchain options.
Keep Loops Out of Your Log Calls — and Guard Expensive Debug Statements
Two habits that silently degrade your logs: logging inside loops (covered in Rule 02), and
building expensive strings for debug messages that get evaluated even when debug is disabled.
In other words — the string gets built whether or not the log line ever gets written.
log.debug("User: " + user.name + " ID: " + user.id)
log.debug("User: {} ID: {}", user.name, user.id)
# For expensive calls, add a level guard first
if log.is_debug_enabled():
log.debug("Detail: {}", build_expensive_context(obj))
Teams disable DEBUG in production but forget that string concatenation in log calls still
executes. On a high-traffic system processing 50,000 req/s, that wasted CPU adds up.
Switch every debug log to placeholder syntax and guard expensive calls — it costs one line
and saves measurable overhead.
Use journalctl to Search and Filter System Logs Like a Pro
tail -f /var/log/syslog is a blunt instrument. journalctl is a scalpel.
It queries the systemd journal with filtering by unit, priority, time, and output format.
Here are the five commands you will use in every real incident.
Traditional flat files in /var/log/ are written by rsyslog or syslog-ng.
journald (queried with journalctl) is the systemd binary journal —
faster to filter and supports structured output. On modern RHEL 8 and Ubuntu 22.04, both run
side by side. Use journalctl for systemd services; use /var/log/
for application logs that write their own files.
a. View logs for a specific systemd service unit
Apr 03 09:11:22 server1 nginx[1024]: Started nginx.service
b. Filter by priority — errors and above only
Apr 03 08:55:12 server1 sshd[4421]: error: Could not load host key /etc/ssh/ssh_host_ed25519_key
c. Show logs within a specific time window
d. Follow live log output in real time
e. Export as JSON for machine parsing
"__REALTIME_TIMESTAMP" : "1696253535000000",
"_SYSTEMD_UNIT" : "nginx.service",
"PRIORITY" : "6",
"MESSAGE" : "Started nginx.service"
}
Stack -u, -p, and -S in a single command for laser-precise
incident investigation:
journalctl -u nginx -p err -S "2025-10-02 14:00"
See Linux System Monitoring Cheat Sheet for more filtering patterns.
Set Up Log Rotation — Disk Space Is Not Infinite
A logging strategy without rotation is a ticking disk-full incident. Left unchecked, a single
busy service can fill /var/log/ in days. logrotate handles this
automatically — here is a production-ready configuration.
daily # rotate once per day
rotate 30 # keep 30 days of history
size 100M # rotate if file exceeds 100MB
maxsize 100M # hard cap per file
compress # gzip rotated files
delaycompress # compress on second rotation cycle
missingok # no error if log file is missing
notifempty # skip rotation if file is empty
dateext # append date to rotated filename
dateformat -%Y-%m-%d # e.g. app-2025-10-02.log
copytruncate # rotate without restarting the process
}
Drop one config file per application into /etc/logrotate.d/. The main
logrotate daemon picks them all up automatically via cron. No need to edit
the global /etc/logrotate.conf for per-app rules.
Use cron
if you need custom rotation schedules beyond daily/weekly/monthly.
Build Alerts That Fire on Signal — Not on Noise
An alert that fires too often trains your team to ignore it. An alert that never fires means
something is silently broken. The goal is alerts that fire only when a human genuinely needs
to act. Here are the four patterns that cover most production scenarios.
Count errors per minute directly from a log file using grep and awk:
| awk '{print substr($1,1,16)}' \
| sort | uniq -c | sort -rn | head -10
8 2025-10-02T14:21
2 2025-10-02T14:20
In other words — 41 errors in one minute compared to 2 the minute before. That is your spike. That is your alert trigger.
For teams managing more than a handful of servers, a centralised platform (ELK stack:
Elasticsearch + Logstash + Kibana, or EFK with Fluentd) gives you all four alerting patterns
out of the box with no shell scripting. See
Best Linux Monitoring Tools
for a curated comparison.
A Mistake I See Often — and One That Cost a Team 4 Hours
A payment service went down during a peak traffic window. The logs showed only this:
ERROR Process failed
ERROR Process failed
No userId. No orderId. No error code. No trace ID. Four engineers spent four hours manually
cross-referencing database records to find that a third-party payment API was timing out.
That's not a logging volume problem — that's a logging content problem. Rules 01
and 05 in this guide would have cut that incident to under 10 minutes.
Every ERROR entry must carry enough context to reproduce the scenario without any additional
data source. If your ERROR log requires a database query to understand — it is not a log entry,
it is an incomplete sentence. Go back to Rule 05 and add the six fields.
questions. Apply Rules 01 and 05 first (they have the highest immediate impact), set up
log rotation from Rule 09 before you forget, and graduate to structured JSON from Rule 06
when you're ready to hook into a monitoring platform. The next logical step from here is
mastering Linux system monitoring
— because good logs and good metrics are two sides of the same observability coin.
Log files in /var/log/ are only as safe as their permissions.
If a sensitive application log is world-readable, any user on the system can read it.
Lock down log file ownership and permissions with
chmod and
chown after setting up
any new application logging. A typical safe baseline: chmod 640 and
chown root:adm.
People Also Ask
/var/log/ contains flat text files written by traditional syslog daemons
(rsyslog, syslog-ng). journald is the systemd binary journal, queried with
journalctl. On modern systems both run simultaneously. Use journalctl
for systemd service logs; use /var/log/ for application-managed log files.
The binary journal is faster to filter and supports structured JSON export natively.
Use INFO for key business events you want to track under normal operation.
Use WARN for conditions that are not yet broken but need monitoring.
In production, disable DEBUG and TRACE entirely — they
generate excessive output and carry a measurable performance cost under load.
Use journalctl -p err to filter the systemd journal to priority 3 (error) and
above. For flat log files, use grep "ERROR" /var/log/myapp/app.log.
To also catch critical and emergency entries, use journalctl -p crit.
Structured logging writes log entries as machine-parseable data (typically JSON) instead of
free-form text strings. Each field — timestamp, level, traceId, userId — is a named key,
not part of a sentence. This makes logs directly searchable by field, compatible with ELK
and EFK stacks, and filterable without regex. For any system generating more than a few
hundred log lines per minute, structured logging is the only practical approach.
Set up logrotate with a configuration file in /etc/logrotate.d/.
Define a maximum file size (e.g. size 100M), a retention period
(e.g. rotate 30), and enable compression (compress).
Test your configuration without applying it using logrotate --debug.
See Rule 09 in this guide for a complete working example.
Yes — synchronous high-volume logging can reduce application throughput by 30–50% under
load. The main culprits are disk I/O, string construction inside log calls, and thread
contention in multi-threaded environments. To mitigate: use asynchronous log writers,
avoid logging inside loops, use placeholder syntax instead of string concatenation, and
keep DEBUG and TRACE disabled in production.
From your first terminal command to advanced sysadmin skills — every guide here is written in plain English with real examples you can run right now.
