LINUX IS NOT SECURE! A practical server-hardening manual
A problem-and-solution hardening runbook for Ubuntu Server, Debian and Rocky Linux: patching, SSH keys, firewalls, Fail2ban, AppArmor or SELinux, audit logs, file-integrity checks and proof that the result actually works.
Scope
A problem-and-solution hardening runbook for Ubuntu Server, Debian and Rocky Linux: patching, SSH keys, firewalls, Fail2ban, AppArmor or SELinux, audit logs, file-integrity checks and proof that the result actually works.
Was I hacked?
Perhaps. A Linux prompt is not a certificate of innocence, and a penguin is not a security control.
Treat the server as a possible incident — not merely a hardening project — if you find any of the following:
- a listening port or service that nobody can explain;
- a new user, SSH key,
sudoersentry, cron job or systemd unit without an approved change; - successful SSH logins from unfamiliar addresses;
- security tools, logs or package repositories that were disabled or altered;
- unexpected outbound connections, miners, web shells or processes running from
/tmp,/dev/shmor a web-upload directory; - package verification or file-integrity checks reporting unexplained changes to system binaries;
- an unsupported operating-system release exposed to the internet.
If that is your situation, do not harden over the top of the evidence and declare victory. Isolate the host at the cloud firewall, hypervisor or switch; preserve volatile evidence and logs; rotate credentials and tokens from a known-clean device; investigate the entry point; and usually rebuild from trusted media. See the unknown-script incident-response guide for the first-hour sequence. Hardening a compromised machine is rather like fitting a better lock after discovering somebody already lives in the wardrobe.
If there is no sign of compromise, continue. This guide turns a newly installed or lightly managed server into a defensible baseline for a small production environment.
Linux is not secure — it is securable
Linux gives you strong security mechanisms. It does not know which ports your application needs, which administrators should log in, whether last Tuesday's kernel may reboot today, or why somebody exposed PostgreSQL to the entire planet.
A default installation is a general-purpose compromise. A hardened server is a documented decision:
internet
|
[cloud/network firewall]
|
[host firewall: only required ports]
|
[patched service + least privilege]
|
[AppArmor or SELinux confinement]
|
[audit logs + integrity monitoring + backups]
No single package supplies that stack. Fail2ban is useful, but it is a doorman counting failed attempts; it is not a moat, a patching policy or an incident-response team.
Scope, versions and assumptions
The commands below target three widely deployed server families current at review on 4 September 2026:
- Ubuntu Server 26.04 LTS; the same approach applies to supported 24.04 LTS hosts;
- Debian 13 “trixie”;
- Rocky Linux 9 or 10, representing the RHEL-compatible family. AlmaLinux administrators can use the same
dnf,firewalldand SELinux sections, but should confirm package names in their own repositories.
The worked example is a public web server requiring SSH, HTTP and HTTPS. Replace every capitalised placeholder before running a command:
SERVER_IP 203.0.113.20
ADMIN_USER opsadmin
ADMIN_IPV4_CIDR 198.51.100.24/32
ADMIN_IPV6_CIDR 2001:db8:1200::24/128
The documentation addresses the operating-system host. Nginx, Apache, PHP, WordPress, PostgreSQL, containers and mail servers each require their own hardening. A well-locked plant room does not make every machine inside it correctly assembled.
Before touching SSH or the firewall
Problem: a perfectly valid hardening command can cut off the only working administrative session.
Solution: create recovery and evidence before changing access.
- Take a provider snapshot or verified backup.
- Confirm that the provider's serial console, KVM or rescue mode works.
- Keep the current SSH session open.
- Start
tmuxso a network wobble does not abandon a half-written configuration. - Open a second terminal for every access test.
Install tmux if necessary:
# Ubuntu or Debian
sudo apt update
sudo apt install --yes tmux
# Rocky Linux
sudo dnf install --assumeyes tmux
tmux new -s hardening
Record the starting state:
date --iso-8601=seconds
cat /etc/os-release
uname -a
ip -brief address
ip route
sudo ss -lntup
sudo systemctl --type=service --state=running
sudo systemctl --failed
Save the output with the change ticket. You cannot prove an improvement if you have forgotten what existed before breakfast.
Problem 1: the server is already behind on security fixes
An internet-facing server with an elegant SSH banner and an old kernel is an elegantly labelled target.
Ubuntu 26.04 LTS
Install current updates and the tools used in this guide:
sudo apt update
sudo apt full-upgrade --yes
sudo apt install --yes \
unattended-upgrades needrestart ufw fail2ban \
apparmor-utils auditd audispd-plugins \
aide aide-common chrony libpam-pwquality
Enable daily package-list refresh and unattended upgrades without surprise automatic reboots:
sudo tee /etc/apt/apt.conf.d/52notmyfault-hardening >/dev/null <<'EOF'
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Automatic-Reboot "false";
EOF
sudo unattended-upgrade --dry-run --debug
systemctl list-timers 'apt-*'
Expected result: the dry run completes without repository errors and an apt-daily-upgrade.timer has a next-run time. Ubuntu enables security updates by default on normal Server installations, but checking the timer is cheaper than believing the installer remembers your intentions.
Debian 13
First confirm that trixie-security exists in your APT sources:
grep -R --line-number -E 'trixie-security|security.debian.org' \
/etc/apt/sources.list /etc/apt/sources.list.d 2>/dev/null
If no security suite appears, fix the repository configuration before proceeding. For the classic one-line format, Debian's documented entry is:
deb https://security.debian.org/debian-security trixie-security main contrib non-free non-free-firmware
Then update and install the baseline:
sudo apt update
sudo apt full-upgrade --yes
sudo apt install --yes \
sudo unattended-upgrades needrestart nftables fail2ban \
apparmor apparmor-utils apparmor-profiles auditd audispd-plugins \
aide aide-common chrony libpam-pwquality
Use the same /etc/apt/apt.conf.d/52notmyfault-hardening drop-in shown for Ubuntu, then test it:
sudo unattended-upgrade --dry-run --debug
systemctl list-timers 'apt-*'
Rocky Linux 9 or 10
Apply updates, enable the Extra Packages for Enterprise Linux repository for Fail2ban, and install the baseline:
sudo dnf upgrade --refresh --assumeyes
sudo dnf install --assumeyes epel-release
sudo dnf install --assumeyes \
firewalld fail2ban fail2ban-firewalld \
policycoreutils-python-utils audit aide chrony \
dnf-automatic libpwquality
Configure automatic security updates:
sudo cp -a /etc/dnf/automatic.conf \
/etc/dnf/automatic.conf.before-notmyfault
sudo sed -ri 's/^upgrade_type[[:space:]]*=.*/upgrade_type = security/' \
/etc/dnf/automatic.conf
sudo systemctl enable --now dnf-automatic-install.timer
systemctl status dnf-automatic-install.timer --no-pager
Rocky rebuilds Enterprise Linux packages, so errata metadata may not always behave exactly like a subscribed RHEL host. Confirm that the timer runs and review its journal rather than treating the word security as a magical incantation:
sudo systemctl start dnf-automatic-install.service
sudo journalctl -u dnf-automatic-install.service --since today --no-pager
All three: find out whether the update needs a reboot
# Ubuntu and Debian
if test -f /var/run/reboot-required; then
cat /var/run/reboot-required
cat /var/run/reboot-required.pkgs 2>/dev/null || true
fi
# Rocky, where the needs-restarting command is available
sudo dnf needs-restarting -r
Schedule the reboot. “Patched on disk, vulnerable kernel still running” is a disappointingly popular configuration.
Problem 2: administrators log in directly as root or with passwords
Risk: internet bots can attempt passwords indefinitely, shared root access destroys accountability, and one stolen password becomes a complete server.
Solution: create a named administrator, test an SSH key, and only then disable remote root and password authentication.
Create the named administrator
Ubuntu or Debian:
sudo adduser opsadmin
sudo usermod --append --groups sudo opsadmin
sudo install -d -m 0700 -o opsadmin -g opsadmin /home/opsadmin/.ssh
sudo install -m 0600 -o opsadmin -g opsadmin /dev/null \
/home/opsadmin/.ssh/authorized_keys
sudoedit /home/opsadmin/.ssh/authorized_keys
Rocky Linux:
sudo useradd --create-home opsadmin
sudo passwd opsadmin
sudo usermod --append --groups wheel opsadmin
sudo install -d -m 0700 -o opsadmin -g opsadmin /home/opsadmin/.ssh
sudo install -m 0600 -o opsadmin -g opsadmin /dev/null \
/home/opsadmin/.ssh/authorized_keys
sudoedit /home/opsadmin/.ssh/authorized_keys
sudo restorecon -Rv /home/opsadmin/.ssh
Paste the administrator's public key — one line beginning with ssh-ed25519, sk-ssh-ed25519@openssh.com or another approved modern key type. Never copy the private key to the server.
From a second terminal, force a key-only test:
ssh -o PreferredAuthentications=publickey \
-o PasswordAuthentication=no \
opsadmin@203.0.113.20
sudo -v
Do not continue until both commands succeed.
Apply an SSH server baseline
Create a drop-in rather than rewriting the distributor's entire file:
sudo tee /etc/ssh/sshd_config.d/40-notmyfault-hardening.conf >/dev/null <<'EOF'
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitEmptyPasswords no
MaxAuthTries 4
LoginGraceTime 30
X11Forwarding no
AllowAgentForwarding no
PermitTunnel no
ClientAliveInterval 300
ClientAliveCountMax 2
EOF
If your organisation uses PAM-based multi-factor authentication, do not disable keyboard-interactive authentication blindly. Configure and test the intended AuthenticationMethods, commonly publickey,keyboard-interactive, with your identity provider's instructions.
Do not paste fashionable cipher lists from a 2018 blog. Supported distributions maintain secure OpenSSH defaults, while a frozen custom list ages badly and breaks clients. Remove a specific legacy algorithm only when your inventory proves it remains enabled and your compatibility testing approves the change.
Now validate the syntax and inspect the effective settings:
sudo sshd -t
sudo sshd -T | grep -E \
'^(permitrootlogin|pubkeyauthentication|passwordauthentication|kbdinteractiveauthentication|maxauthtries|x11forwarding|allowagentforwarding|permittunnel) '
Expected result includes:
permitrootlogin no
pubkeyauthentication yes
passwordauthentication no
kbdinteractiveauthentication no
If the effective value differs, an earlier configuration or a Match block is winning. Fix the conflict; do not merely admire the drop-in you wished OpenSSH had used.
Reload without closing the original session:
# Ubuntu and Debian
sudo systemctl reload ssh
# Rocky Linux
sudo systemctl reload sshd
Open a third new session as opsadmin. Test that root and password logins fail:
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password \
opsadmin@203.0.113.20
ssh root@203.0.113.20
A failure is the expected result. Your existing session remains the emergency rope until testing is complete.
Optional: restrict which accounts may use SSH
On a simple host you may add:
AllowUsers opsadmin deploy
to the drop-in. List every automation and break-glass account first. AllowUsers opsadmin is excellent until the backup system called backup discovers that philosophy at 02:00.
If the host is not a bastion and nobody needs tunnelling, add DisableForwarding yes. Test backups, configuration management and deployment jobs afterwards.
Problem 3: services listen on every interface because nobody checked
Start with the truth:
sudo ss -lntup
For the worked web server, the intentional public listeners are usually:
22/tcp SSH, preferably restricted to the administrator's address or VPN
80/tcp HTTP, normally for redirect or ACME validation
443/tcp HTTPS
A database should ordinarily bind to localhost or a private application network. A host firewall is a second control, not permission to leave Redis, PostgreSQL or an admin panel listening on 0.0.0.0.
Ubuntu: configure UFW without losing SSH
sudo ufw default deny incoming
sudo ufw default allow outgoing
# Best: fixed administrator address or VPN range
sudo ufw allow proto tcp from 198.51.100.24/32 to any port 22 \
comment 'SSH from admin address'
# If the admin address is dynamic, keep SSH public but rate-limited
# sudo ufw limit 22/tcp comment 'rate-limit public SSH'
sudo ufw allow 80/tcp comment 'HTTP'
sudo ufw allow 443/tcp comment 'HTTPS'
sudo ufw --dry-run enable
sudo ufw enable
sudo ufw status verbose
If the server has public IPv6, add the approved IPv6 administrator range as well or choose the public rate-limited SSH rule. Confirm /etc/default/ufw has IPV6=yes; otherwise you have secured half the internet and left the other half a small side entrance.
Debian: use native nftables
This example is for a conventional host, not a Docker or Kubernetes node. Container engines manipulate Netfilter and require a design that accounts for their own chains. Do not use flush ruleset on such a system without understanding the result.
Create a candidate file locally on the server:
sudo cp -a /etc/nftables.conf /etc/nftables.conf.before-notmyfault
sudo tee /etc/nftables.conf >/dev/null <<'EOF'
#!/usr/sbin/nft -f
flush ruleset
table inet host_filter {
chain input {
type filter hook input priority filter; policy drop;
ct state invalid drop
ct state established,related accept
iifname "lo" accept
ip protocol icmp accept
ip6 nexthdr ipv6-icmp accept
ip saddr 198.51.100.24/32 tcp dport 22 accept
ip6 saddr 2001:db8:1200::24/128 tcp dport 22 accept
tcp dport { 80, 443 } accept
limit rate 5/second counter log prefix "nft-input-drop: "
}
chain forward {
type filter hook forward priority filter; policy drop;
}
chain output {
type filter hook output priority filter; policy accept;
}
}
EOF
sudo nft --check --file /etc/nftables.conf
Replace both example address ranges. If you do not administer over IPv6, remove the example IPv6 SSH rule rather than allowing somebody else's documentation range, which will be useless but impressively tidy.
Apply only after the syntax check succeeds and while the recovery console is available:
sudo nft --file /etc/nftables.conf
sudo systemctl enable --now nftables
sudo nft list ruleset
Test a new SSH session immediately. To recover from a mistake in the still-open original session:
sudo cp -a /etc/nftables.conf.before-notmyfault /etc/nftables.conf
sudo nft --file /etc/nftables.conf
Rocky: use firewalld and preserve SELinux
Identify the active zone before changing it:
sudo systemctl enable --now firewalld
sudo firewall-cmd --get-active-zones
sudo firewall-cmd --zone=public --list-all
The following assumes the public interface belongs to the public zone. Add the restricted SSH rule before removing the globally allowed SSH service:
sudo firewall-cmd --permanent --zone=public \
--add-rich-rule='rule family="ipv4" source address="198.51.100.24/32" service name="ssh" accept'
sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --permanent --zone=public --remove-service=ssh
sudo firewall-cmd --check-config
sudo firewall-cmd --reload
sudo firewall-cmd --zone=public --list-all
For IPv6 administration, add an equivalent family="ipv6" rich rule with the genuine administrator prefix before reloading. If your address changes, leave the ssh service enabled and rely on keys plus Fail2ban until a VPN or stable management path exists.
Verify from outside, not from the server itself
From another network or a VPS you control, scan only your own address:
nmap -Pn -sT -p- 203.0.113.20
nmap -Pn -sV -p 22,80,443 203.0.113.20
Expected result: only the deliberately exposed ports respond. A provider security group, NAT gateway or upstream firewall may make the external result differ from the host firewall, which is exactly why both views matter. The external Nmap guide explains how to distinguish an open port, a version finding and an actual vulnerability.
Problem 4: SSH logs fill with password-spraying noise
With password authentication disabled, most password bots cannot enter. They can still waste resources and obscure useful logs. Fail2ban can temporarily block addresses that repeatedly trigger authentication failures.
Ubuntu and Debian Fail2ban configuration
Never edit /etc/fail2ban/jail.conf; package upgrades replace it. Create a local drop-in:
sudo tee /etc/fail2ban/jail.d/10-sshd.local >/dev/null <<'EOF'
[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 4
bantime.increment = true
bantime.maxtime = 24h
backend = systemd
usedns = no
[sshd]
enabled = true
port = ssh
mode = normal
EOF
sudo fail2ban-client -t
sudo systemctl enable --now fail2ban
sudo fail2ban-client status
sudo fail2ban-client status sshd
On Debian's nftables firewall, Fail2ban's packaged default should select a compatible banning action. Check it rather than guessing:
sudo fail2ban-client get sshd actions
sudo nft list ruleset | grep -i fail2ban
Rocky Fail2ban configuration
Use the firewalld action supplied by the EPEL package:
sudo tee /etc/fail2ban/jail.d/10-sshd.local >/dev/null <<'EOF'
[DEFAULT]
bantime = 1h
findtime = 10m
maxretry = 4
bantime.increment = true
bantime.maxtime = 24h
backend = systemd
usedns = no
banaction = firewallcmd-rich-rules
[sshd]
enabled = true
port = ssh
mode = normal
EOF
sudo fail2ban-client -t
sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd
If the service fails, inspect the actual error:
sudo journalctl -u fail2ban -b --no-pager
sudo fail2ban-client -d | less
Do not test by deliberately banning the only address from which you can administer the server. Use a separate test source and keep console access. To remove a known accidental ban:
sudo fail2ban-client set sshd unbanip 198.51.100.24
Do not put broad office, VPN or cloud ranges into ignoreip without thought. A compromised device inside an allowlist becomes an attacker with diplomatic immunity.
Problem 5: the operating system's mandatory access control was disabled “temporarily” in 2023
Traditional Unix permissions answer “which user owns this?” AppArmor and SELinux also restrict what a compromised process may touch. They can turn a vulnerable web process with a shell into a vulnerable web process that still cannot read everything on the host.
Ubuntu: verify AppArmor
Ubuntu installs and loads AppArmor by default:
sudo aa-status
systemctl status apparmor --no-pager
sudo journalctl -k --since today | grep -F 'apparmor="DENIED"' || true
Expected result: the module is loaded and relevant profiles are in enforce mode. Do not place every experimental profile into enforce mode at once. Introduce one service profile, exercise the application, inspect denials, and then enforce it:
sudo aa-complain /usr/sbin/example-service
# exercise normal application functions, then review the proposed rules
sudo aa-logprof
sudo aa-enforce /usr/sbin/example-service
Replace the example with a real profile that exists in /etc/apparmor.d/. If a service breaks, diagnose the denial; disabling AppArmor globally is not diagnosis.
Debian: confirm AppArmor really loaded
Debian 13 normally enables AppArmor, but minimal and upgraded systems deserve verification:
cat /sys/module/apparmor/parameters/enabled
sudo aa-status
sudo journalctl -k --since today | grep -F 'apparmor="DENIED"' || true
Y and loaded profiles are the expected result. If the kernel module is not enabled, follow Debian's current AppArmor boot instructions during a maintenance window. Do not add kernel parameters and reboot a remote machine without a tested rescue path.
Rocky: keep SELinux Enforcing
getenforce
sestatus
sudo ausearch -m AVC,USER_AVC -ts recent
Expected result: Enforcing. If it is Permissive, investigate current denials, correct file labels or policy, then test enforcement:
sudo restorecon -Rv /var/www
sudo setenforce 1
getenforce
Persist the intended state in /etc/selinux/config only after the application works:
SELINUX=enforcing
SELINUXTYPE=targeted
If SELinux is Disabled, re-enabling it requires relabelling and a reboot. Follow the Rocky/RHEL procedure with console access. Do not solve a web permission problem with setenforce 0; use ausearch, restorecon, documented booleans and narrowly scoped labels.
For example, a web application that must make outbound network connections may require the documented boolean:
getsebool httpd_can_network_connect
sudo setsebool -P httpd_can_network_connect on
Enable it only if the application genuinely needs that capability. “The website works” and “the website may connect anywhere” are different requirements.
Problem 6: permissive kernel defaults survive because nobody owns them
These settings reduce common information leaks and unsafe redirect behaviour without inventing a bespoke kernel. They are a baseline, not a substitute for application isolation.
Create the same file on all three distributions:
sudo tee /etc/sysctl.d/60-notmyfault-hardening.conf >/dev/null <<'EOF'
# Reduce kernel information exposed to unprivileged users
kernel.kptr_restrict = 2
kernel.dmesg_restrict = 1
# Protect common link and FIFO attacks in writable directories
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.protected_fifos = 2
fs.protected_regular = 2
# Do not keep set-user-ID core dumps
fs.suid_dumpable = 0
# Basic network hardening for an ordinary non-router host
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
EOF
sudo sysctl --system
sudo sysctl kernel.kptr_restrict kernel.dmesg_restrict \
fs.protected_hardlinks fs.protected_symlinks \
net.ipv4.conf.all.accept_redirects net.ipv4.tcp_syncookies
We deliberately do not force reverse-path filtering, disable IPv6, or turn off IP forwarding in a universal copy-and-paste block. Those changes can break multihoming, asymmetric routing, VPNs, containers and routers. Set them only after classifying the host's network role.
Problem 7: privileged changes happen, but the evidence evaporates
Make the system journal persistent
On all three distributions:
sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/20-persistent.conf >/dev/null <<'EOF'
[Journal]
Storage=persistent
Compress=yes
SystemMaxUse=1G
MaxRetentionSec=30d
EOF
sudo systemctl restart systemd-journald
journalctl --disk-usage
One gigabyte and 30 days are examples; size them for the host. Local logs help with mistakes and short incidents. A root-level attacker may alter them, so production systems should also forward security logs to a separate, access-controlled destination.
Add useful audit rules
The auditd package records kernel audit events. Add watches for identity, privilege and SSH configuration:
sudo systemctl enable auditd
sudo service auditd start
sudo tee /etc/audit/rules.d/50-notmyfault.rules >/dev/null <<'EOF'
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/sudoers -p wa -k privilege_scope
-w /etc/sudoers.d/ -p wa -k privilege_scope
-w /etc/ssh/sshd_config -p wa -k ssh_configuration
-w /etc/ssh/sshd_config.d/ -p wa -k ssh_configuration
EOF
sudo augenrules --check
sudo augenrules --load
sudo auditctl -l
Test with an authorised, harmless metadata change:
sudo touch /etc/ssh/sshd_config.d/.audit-test
sudo rm /etc/ssh/sshd_config.d/.audit-test
sudo ausearch -k ssh_configuration -ts recent -i
Expected result: audit records identify the user, executable and affected path. Watches are not alerts by themselves; send or query them in your monitoring system.
Useful daily questions:
sudo ausearch -k identity -ts today -i
sudo ausearch -k privilege_scope -ts today -i
sudo ausearch -k ssh_configuration -ts today -i
sudo aureport --auth --summary
Problem 8: system files change and the only baseline is somebody's memory
AIDE records file metadata and cryptographic hashes. Build the baseline after patching and approved configuration changes, then store a copy outside the server. A database held only beside the files it measures can be replaced by the same attacker.
Ubuntu and Debian
sudo aideinit
sudo aide.wrapper --check
sudo ls -lh /var/lib/aide/
Package scripts and filenames vary slightly between supported releases. Read the final aideinit output and confirm which database became active before scheduling checks:
sudo systemctl list-timers | grep -i aide || true
sudo grep -R --line-number aide /etc/cron.* /etc/systemd/system \
/usr/lib/systemd/system 2>/dev/null | head -50
Rocky Linux
sudo aide --init
sudo cp -a /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz
sudo aide --check
Schedule a daily check with a systemd timer or your monitoring platform, and alert on unexpected changes. Do not automatically accept a new database after every update: review known package changes first, then update the baseline. Otherwise the burglar merely receives a freshly printed inventory.
Problem 9: forgotten services and weak local privilege controls remain
Disable only what you understand
List enabled and running units:
sudo systemctl list-unit-files --type=service --state=enabled
sudo systemctl --type=service --state=running
sudo ss -lntup
For each unexplained listener, identify its package and owner before acting:
sudo systemctl status SERVICE_NAME --no-pager
sudo systemctl cat SERVICE_NAME
# Ubuntu or Debian
dpkg -S /path/to/executable
# Rocky Linux
rpm -qf /path/to/executable
Then, if the service is genuinely unnecessary:
sudo systemctl disable --now SERVICE_NAME
Do not blindly paste a list that disables rpcbind, cups, avahi-daemon or Bluetooth everywhere. They are unnecessary on many servers and necessary on some. The secure action is to remove an unneeded capability with an owner and rollback plan, not to win internet points for the shortest systemctl output.
Improve sudo accountability
Create a small sudo policy drop-in:
sudo tee /etc/sudoers.d/40-notmyfault-hardening >/dev/null <<'EOF'
Defaults use_pty
Defaults timestamp_timeout=5
EOF
sudo chmod 0440 /etc/sudoers.d/40-notmyfault-hardening
sudo visudo -cf /etc/sudoers
Do not grant broad NOPASSWD: ALL merely to quiet automation. Give service accounts explicit commands, use configuration-management privilege escalation, and keep human administration attributable.
Find dangerous permissions without “fixing” the filesystem automatically
sudo find / -xdev -type f -perm -0002 -print
sudo find / -xdev -type d -perm -0002 ! -perm -1000 -print
sudo find / -xdev -type f -perm /6000 -printf '%m %u:%g %p\n'
The first two queries find world-writable files and directories lacking the sticky bit. The third inventories set-user-ID and set-group-ID executables. Review package ownership and application requirements. Never pipe these results into an automatic chmod; /tmp and privileged binaries exist for reasons.
Problem 10: time, backups and recovery are treated as somebody else's department
Authentication logs, certificates and distributed event timelines require correct time:
sudo systemctl enable --now chronyd 2>/dev/null || \
sudo systemctl enable --now chrony
chronyc tracking
chronyc sources -v
timedatectl status
Expected result: the clock is synchronised and at least one source is reachable.
Backups are not created by this article because destinations, retention and data ownership differ. The minimum production design is:
- automated backups of application data and configuration;
- an encrypted copy outside the server and outside the same cloud account where practical;
- immutable or append-only retention against ransomware and administrator mistakes;
- restore tests with measured recovery time;
- no long-lived backup credentials stored where a compromised web process can read them.
Hardening lowers the chance of failure. Restore testing limits the length of the funeral.
Verification: prove each layer instead of trusting the commands
Run the local checks:
cat /etc/os-release
sudo systemctl --failed
sudo ss -lntup
sudo sshd -t
sudo sshd -T | grep -E \
'^(permitrootlogin|passwordauthentication|kbdinteractiveauthentication|pubkeyauthentication) '
sudo fail2ban-client status sshd
sudo auditctl -l
sudo ausearch -k ssh_configuration -ts today -i
sudo systemctl is-active chronyd 2>/dev/null || \
sudo systemctl is-active chrony
# Ubuntu or Debian
sudo ufw status verbose 2>/dev/null || sudo nft list ruleset
sudo aa-status 2>/dev/null || true
# Rocky
sudo firewall-cmd --state 2>/dev/null || true
getenforce 2>/dev/null || true
From an external system you own:
nmap -Pn -sT -p- 203.0.113.20
ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password \
opsadmin@203.0.113.20
ssh root@203.0.113.20
Then test the application, monitoring, backup and deployment paths. The expected end state for the worked web server is:
| Control | Expected evidence |
|---|---|
| Updates | automatic update timer active; no repository errors |
| SSH | named user with key; root and password login rejected |
| Network | only intended ports visible externally |
| Brute-force control | Fail2ban sshd jail active and using the real logs |
| Confinement | AppArmor loaded or SELinux Enforcing |
| Audit | test change appears under ssh_configuration key |
| Integrity | AIDE baseline exists off-host and a check completes |
| Time | chrony reports synchronisation |
| Recovery | a recent backup has been restored in a test |
For a second opinion, run a host audit tool, but read findings rather than worshipping the score:
# Ubuntu or Debian
sudo apt install --yes lynis
sudo lynis audit system
# Rocky (from EPEL)
sudo dnf install --assumeyes lynis
sudo lynis audit system
Lynis recommendations are prompts for engineering decisions. A higher number does not prove that the mail server still accepts mail, the application still deploys, or the backup still restores.
What this baseline deliberately does not automate
Some popular “hardening scripts” apply hundreds of controls without understanding the server. This guide does not automatically:
- disable IPv6;
- mount
/tmpwithnoexec; - blacklist uncommon kernel modules;
- force a particular password lifetime;
- replace OpenSSH's maintained algorithm defaults;
- block all outbound network traffic;
- apply every CIS Level 2 control;
- reboot production after unattended updates.
Each can be appropriate. Each can also break containers, package installers, monitoring agents, legacy integration, high-availability networking or emergency access. Classify the host, test in staging, document exceptions and roll out with configuration management.
If verification finds something genuinely suspicious
Hardening stops and incident response begins when evidence suggests a live intrusion.
- Restrict the host at an upstream firewall without powering it off unless safety requires it.
- Preserve cloud audit logs, authentication logs,
journalctl, process and network state. - Record suspicious accounts, keys, processes, hashes, addresses and timestamps.
- Rotate passwords, SSH keys, API keys, session tokens and backup credentials from a clean device.
- Search peer systems and central logs for the same indicators.
- Determine the exposed service or stolen credential that provided entry.
- Rebuild from trusted media when root compromise is possible; do not rely on deleting the obvious file.
- restore only reviewed data and configuration, then apply this baseline before returning the host to service.
The goal is not to preserve the server's feelings. The goal is to restore a trustworthy service.
The maintenance rhythm
A secure build decays. Give it a calendar:
Daily
sudo systemctl --failed
sudo journalctl -p 0..3 --since '24 hours ago' --no-pager
sudo fail2ban-client status sshd
Weekly
sudo ss -lntup
sudo ausearch -k identity -ts week-ago -i
sudo ausearch -k privilege_scope -ts week-ago -i
Review AIDE results, backup jobs, new accounts, firewall changes and pending reboots.
Monthly
- scan the public address from outside;
- restore a sample backup;
- review administrators and SSH keys;
- remove expired exceptions;
- patch staging, reboot it, test it, then schedule production;
- compare the host to its declared configuration.
At more than a handful of servers, convert the approved baseline to Ansible or another configuration-management system. Manual hardening teaches what the controls do; automation prevents server number 47 from becoming the charmingly unique one.
Final judgement
Linux is not “unsafe”, and it is certainly not automatically safe. Its advantage is that the controls are visible, scriptable and well understood. Security arrives when somebody chooses a supported release, reduces exposure, removes weak authentication, keeps mandatory access control enabled, records important changes, checks integrity and proves recovery.
Install Fail2ban, by all means. Then keep going.
References
- Ubuntu release cycle and LTS support
- Ubuntu Server firewall documentation
- Ubuntu security-update documentation
- Ubuntu Server AppArmor documentation
- Debian 13 “trixie” release information
- Debian Handbook: Security
- Debian Wiki: periodic and unattended updates
- Debian Wiki: using AppArmor
- Rocky Linux firewalld guide
- Red Hat Enterprise Linux: security hardening
- Red Hat Enterprise Linux: installing security updates
- OpenSSH
sshd_configmanual - Fail2ban upstream jail configuration
Reviewed 4 September 2026. Commands must be tested against the exact operating-system release, network role and application before production rollout.