Processes and Services

Filesystem and Permissions treated a Linux box as a pile of files sitting still: owners, permission bits, a tree hanging off /. That is the system at rest. Nothing on that disk does anything until something runs it.

This lesson is the same machine in motion. A program leaves the disk, lands in memory, and starts consuming CPU time. While it runs it is a process, and the list of processes is the single most useful thing to read on a machine that has just been accessed. It says what software is actually running, what is running as root, and what is listening for connections from outside. Files describe what could happen. Processes are what is happening.

A process is a program that is running

An executable file on disk is inert. /usr/sbin/sshd is just bytes with the execute bit set until the kernel loads it into memory, hands it a slice of CPU time, and lets it run. That running instance is a process. The distinction is worth holding onto: one file can become many processes. Start the same program three times and there are three processes, each with its own memory, each unaware of the others.

The kernel gives every process a number the moment it starts: the process ID, or PID. The PID is unique among the processes alive right now, and it is how every tool in this lesson points at a specific process. Nothing else identifies one as precisely. Two processes can run the same program, the same command line, under the same user, and they still never share a PID.

Processes form a tree. Every process except one was started by another process, called its parent. A command typed at a prompt runs inside a shell, the program that reads typed commands and starts them, so the shell is the parent of whatever it launches. Close the shell and, by default, its children go with it. The one exception, sitting at the root of the tree, is PID 1: the first process the kernel starts at boot, the ancestor of everything else.

ps: a snapshot of the process table

The kernel keeps a live list of every process, the process table, and ps prints it.

$ps aux

Read the columns left to right. USER is the account the process runs as, and it is the first thing a practitioner checks: code running as root is a bigger prize and a bigger risk than the same code running as an unprivileged user like alice. PID is the process ID. %CPU and %MEM are load. STAT is the process state in short codes, R for running or runnable, S for sleeping and waiting on something, Z for a zombie that has finished but has not been cleaned up by its parent yet. The last column, COMMAND, is the full command line with arguments, which is why it rewards a close read. It often spells out a config file path.

top: the live view

ps is a photograph. By the time its output finishes scrolling, the numbers are already stale. top is the video. It redraws every few seconds, sorted by CPU use by default, so the process hammering the machine floats to the top.

$top

The header earns a glance before the process list. load average is three numbers: the average count of processes wanting the CPU over the last 1, 5, and 15 minutes. On a 4-core machine a load average near 4 means fully busy, and well above 4 means work is arriving faster than the CPU can clear it. Many systems also carry htop, which shows the same data with color and per-core bars and arrow-key scrolling. It is friendlier, but it is not always installed, so top is the one worth knowing cold.

Signals: how one process is told to stop

Killing a process is not done by reaching into memory and deleting it. One process asks the kernel to deliver a signal to another. A signal is a small, predefined message from a fixed catalog, each one carrying a specific meaning, and a program can be written to react to most of them. A few matter constantly:

• SIGTERM (15) is the polite request: stop, but finish the current work first, flush data, release locks. This is what kill sends by default.
• SIGINT (2) is what the shell delivers to the foreground process when Ctrl+C is pressed at the keyboard. Interrupt. Most programs treat it as quit.
• SIGKILL (9) cannot be caught, blocked, or handled. The kernel destroys the process on the spot, with no chance for it to save state or clean up.
• SIGHUP (1) originally meant the terminal hung up. Daemons often repurpose it to mean reload their configuration without restarting.

The command is kill, and despite the name it only sends a signal.

$kill 4021

Reaching for kill -9 first is a common mistake. SIGKILL gives the process no chance to close files, remove its lock files, or finish a half-written database record, so it can leave the system in a mess that a plain SIGTERM would have avoided. The correct order is SIGTERM first, wait a few seconds, and escalate to SIGKILL only if the process ignores the polite request.

Foreground and background jobs

By default a command runs in the foreground: the shell starts it and waits, and the terminal is tied up until it exits. Appending & runs the command in the background instead. The shell starts the process, prints its PID, and returns the prompt immediately, leaving the work running.

$sleep 300 &

A background job still belongs to the shell that launched it, and that has a consequence worth carrying into the next idea. Close the terminal and the shell sends SIGHUP to its children, which by default die with it. A job that needs to outlive the terminal has to be started with nohup, detached with disown, or handed to a system built to keep processes running no matter who is logged in. That system is usually systemd.

Daemons and services

Some processes are never meant to stop. An SSH server waits for logins for as long as the machine is up. A web server answers requests around the clock. A database sits ready for queries at all hours. A process like this, running in the background with no controlling terminal, started at boot or on demand and expected to run indefinitely, is a daemon. The name is old Unix, and the convention is a trailing d: sshd is the SSH daemon, crond runs scheduled jobs, systemd supervises the rest.

The word service is the management layer's name for the same idea: a daemon that the system knows how to start, stop, and restart on command. In everyday use the two words get swapped freely.

systemd and systemctl

PID 1, the process at the root of the tree, is the init system: the first thing the kernel starts and the ancestor of every other process. On most current Linux distributions that process is systemd. Its job is to bring the machine up and keep the daemons alive, starting them at boot in the right order, restarting them if they crash, and stopping them cleanly on shutdown.

systemd describes each service with a unit file, a plain text file that records which binary to run, which user to run it as, and what to do if it exits. A service unit file ends in .service and lives in /lib/systemd/system for the ones shipped by packages, or /etc/systemd/system for local overrides and custom services. Reading those files is how a practitioner learns what a box is configured to run, and under whose account.

The command that talks to systemd is systemctl.

$systemctl status ssh

Loaded shows the unit file path and whether the service is enabled, meaning set to start automatically at boot. Active: active (running) is the state right now. Main PID: 412 ties the service straight back to a row in the ps output above, closing the loop between the two views. systemctl start, stop, restart, enable, and disable control services, where enable and disable govern boot behavior and start and stop govern the running state this minute.

The port a service listens on

A daemon that accepts connections has to be reachable, and Networking Foundations set up how. A host has one IP address, but a single host runs many services, so a port number picks out which service a connection is meant for. A service waiting for connections is said to be listening on its port. sshd listens on port 22, a web server on 80 or 443, PostgreSQL on 5432.

The command that shows every listening socket, and which process owns it, is ss.

$ss -tulpn

The Local Address matters as much as the port. 0.0.0.0:80 means the web server accepts connections on every network interface, reachable from other machines on the network. 127.0.0.1:5432 means the database is bound to the loopback address, the host talking only to itself, so it answers the local machine and nowhere else. Same kind of running process, very different exposure, and the pid= field ties each one back to a specific row in ps and a specific service in systemctl. Those three commands are three windows onto the same processes.

Every command in this lesson answered its question by printing text: a table from ps, a status block from systemctl, a list of sockets from ss. Finding the one line that matters, the single root-owned process or the one service bound to 0.0.0.0, means cutting that text down to it. Text Processing is the next lesson, and it covers the tools for exactly that: the three streams every command reads and writes, the pipes that chain one command into the next, and grep, cut, awk, and sed, which turn a wall of output into the finding hiding inside it.