Read the original article here.

---

Understanding the Kernel: The Core of the Operating System

In the journey of building a computer system from the ground up, understanding the core components is paramount. While the hardware provides the raw computational power, the operating system breathes life into it, making it usable for running applications. At the heart of every modern operating system lies the kernel. This document explores what the kernel is, why it's essential, its major responsibilities, and different architectural approaches, providing a foundational understanding crucial for anyone delving into low-level system design.

What is the Kernel?

A kernel is a computer program at the core of a computer's operating system that always has complete control over everything in the system. It is the portion of the operating system code that is always resident in memory and facilitates interactions between hardware and software components.

Think of the kernel as the central nervous system of your computer. It's the first major software loaded after the initial boot process (handled by the bootloader), and it remains in memory throughout the system's operation. Its primary function is to manage the system's resources and provide services to applications, acting as a crucial layer between the hardware and the user-level software.

In the context of "building from scratch," creating a kernel is one of the most significant and complex challenges. It requires a deep understanding of the underlying hardware architecture – how memory is accessed, how the CPU executes instructions, how devices signal their needs, and how to manage these resources efficiently and securely for multiple potentially competing tasks.

Kernel Space vs. User Space: The Privilege Divide

One of the most fundamental concepts in modern operating system design, heavily tied to hardware capabilities, is the division between kernel space and user space.

Kernel Space: The protected area of memory where the kernel's critical code and data reside. Code executing in kernel space (often said to be in "supervisor mode" or "privileged mode") has unrestricted access to the system's hardware and all areas of memory.

User Space: The area of memory where application programs (like browsers, word processors, games) run. Code executing in user space (often in "protected mode" or "user mode") has limited access to hardware and can only access memory within its own allocated address space.

This separation is not merely an organizational structure; it's a critical security and stability mechanism enforced by the CPU hardware. When the CPU is in a privileged mode (like supervisor mode), it allows instructions that can directly manipulate hardware, manage memory mapping units (MMUs), and disable interrupts – operations that are dangerous if performed by untrusted code. When the CPU is in a less privileged mode (like user mode), attempts to execute such instructions or access protected memory regions will trigger a hardware exception (often called a "trap"), which the kernel intercepts. The kernel can then terminate the offending program, thus preventing a single faulty application from crashing the entire system or accessing data it shouldn't.

Imagine building a multi-user system. Without this separation, one user's program could easily read or modify another user's data, or even deliberately or accidentally overwrite the operating system's core code, leading to a system crash or security breach. Kernel space provides a secure sandbox for the OS itself, shielding it from the potentially untrusted and complex code running in user space.

The Kernel-Application Interface: System Calls

Since user-space applications are restricted from directly accessing hardware or protected kernel data, they need a defined, controlled way to request services from the kernel. This is achieved through system calls.

A system call is the mechanism by which a process requests a service from an operating system's kernel that it does not normally have permission to run in user space. It provides the interface between a user process and the operating system's privileged functions.

Examples of services requiring system calls include:

• Accessing files (opening, reading, writing, closing).
• Creating or terminating processes.
• Networking operations (sending/receiving data).
• Accessing specific hardware devices.
• Requesting more memory.

When a user program needs a kernel service, it makes a system call. This isn't a regular function call. Instead, it involves a deliberate transition from user mode to kernel mode. The specific mechanism varies by hardware architecture, but common methods include:

1. Software Interrupts: The program executes a special instruction that generates an interrupt. The CPU hardware handles this interrupt, switching the CPU into supervisor mode and transferring control to a predefined kernel interrupt handler. The kernel examines the interrupt details (often stored in registers) to determine which system call was requested and its parameters.
2. Call Gates: Some architectures provide special instructions or data structures (call gates) that allow a controlled transfer of execution to a predefined entry point in kernel space, while automatically switching the CPU to a higher privilege level. This requires hardware support.
3. Special System Call Instructions: Newer architectures might have dedicated instructions (like syscall or sysenter on x86) specifically designed for fast and efficient transitions into the kernel for system calls.
4. Memory-Based Queues (Less Common for Direct Calls): For scenarios with many non-blocking requests, applications might place requests in a shared memory region that the kernel periodically checks.

Regardless of the mechanism, the key is that the user program doesn't directly jump into arbitrary kernel code. The kernel defines specific, validated entry points for system calls. Before fulfilling the request, the kernel performs checks (e.g., does the user have permission to open this file? Is the requested memory valid?). If the request is valid and safe, the kernel performs the operation in its privileged context and then returns control (and potentially a result) back to the user program, switching the CPU back to user mode.

Typically, application programmers don't interact with system calls directly. Operating systems provide standard libraries (like the C standard library or Windows API) that wrap these low-level system calls, providing a higher-level, more convenient function interface (e.g., printf eventually uses a system call to write to the console, malloc might use system calls to request memory).

Key Responsibilities of the Kernel

The kernel acts as the primary manager of the computer's resources. Its main responsibilities include:

1. Resource Management (General)

The kernel is the ultimate arbiter of system resources. It ensures that multiple processes can share the hardware effectively and prevents them from interfering with each other in harmful ways. This involves:

Resource Management: The kernel's task of allocating, scheduling, and protecting system resources such as CPU time, memory, and I/O devices among competing processes.

• Arbitration: Deciding which process gets to use a resource when multiple processes want it (e.g., scheduling CPU time).
• Optimization: Managing resources to maximize system throughput or responsiveness (e.g., caching frequently accessed data, optimizing disk access patterns).
• Defining Execution Domain (Address Space): As discussed with kernel/user space, the kernel defines and enforces the memory boundaries for each process, preventing them from accessing each other's data or the kernel's data.
• Protection Mechanism: Implementing and enforcing the rules (often via hardware support) that govern access to resources within an execution domain.

2. Process Management

A computer runs multiple programs simultaneously (or appears to). The kernel is responsible for managing the lifecycle and execution of these programs, known as processes.

Process: An instance of a computer program that is being executed. Each process has its own memory space, resources, and execution state.

• Process Creation and Termination: The kernel handles the loading of programs into memory, allocating resources, and setting up their initial state. It also manages their termination and cleanup.
• Scheduling: Deciding which process should run on the CPU at any given moment and for how long. The kernel's scheduler aims to share CPU time fairly and efficiently among all runnable processes, often giving the illusion of parallel execution even on a single-core processor (though modern systems are multi-core).
• Context Switching: The mechanism by which the kernel saves the state (CPU registers, memory pointers, etc.) of the currently running process and restores the state of another process, allowing the CPU to switch execution between them. This is a fundamental operation for multitasking.
• Process Cooperation: Providing mechanisms for processes to interact with each other, share data, and synchronize their activities.

Inter-Process Communication (IPC): Mechanisms provided by the kernel that allow different processes to exchange data and information. Examples include pipes, sockets, message queues, and shared memory.

Synchronization: Mechanisms (like semaphores, mutexes, monitors) provided or supported by the kernel to help processes coordinate their actions and avoid race conditions when accessing shared resources.

3. Memory Management

Memory (RAM) is a crucial but often limited resource shared by all running programs and the kernel itself. The kernel's memory management subsystem is complex and vital for stability and performance.

Memory Management: The kernel's task of allocating, managing, and protecting the computer's memory resources for the kernel and all running processes.

• Allocation and Deallocation: Granting memory blocks to processes when they need them and reclaiming memory when it's no longer in use.

• Virtual Memory: A technique that allows programs to use a larger address space than the physical RAM available. The kernel achieves this by using secondary storage (like a hard drive) as an extension of RAM.

  Virtual Memory: An operating system technique that provides an application with the illusion of a large, contiguous block of memory, even if the physical memory is fragmented or limited. The kernel maps these virtual addresses used by the program to actual physical addresses.

• Paging and Segmentation: Common methods for implementing virtual memory. Paging divides memory into fixed-size blocks (pages), while segmentation divides it into variable-size logical blocks. The kernel manages page tables or segment tables to translate virtual addresses used by processes into physical addresses.

  Paging: A memory management scheme where the operating system retrieves data from secondary storage into primary storage in same-sized blocks called pages. Allows non-contiguous physical memory to appear contiguous to a process.

  Segmentation: A memory management scheme where memory is divided into logical blocks called segments, which can vary in size.

• Demand Paging: A strategy where the kernel only loads pages into physical memory when they are actually needed by the running process (a "page fault" occurs when a process tries to access a page not currently in RAM). This allows processes to start faster and use more virtual memory than physical memory exists.

• Memory Protection: Enforcing the separation between kernel space and user space, and also between the memory spaces of different user processes, preventing unauthorized access. This is often achieved using the MMU hardware based on the kernel's page/segment tables and CPU modes.

4. I/O and Device Management

To be useful, a computer must interact with the outside world and persistent storage. The kernel is responsible for managing all connected hardware devices, from keyboards and mice to disk drives, network cards, and displays.

Input/Output (I/O): The communication between an information processing system, such as a computer, and the outside world or another computer system.

Device Driver: A computer program that operates or controls a particular type of device that is attached to a computer. Drivers provide an abstraction layer, translating generic OS requests into device-specific commands.

• Abstraction: Providing a consistent, high-level interface for applications to interact with devices, hiding the complex, low-level details of specific hardware. Applications don't need to know how a hard drive works, only how to request reading a block of data.
• Driver Management: Loading, unloading, and managing device drivers, which are often implemented as modular components.
• Handling Interrupts: Devices signal the kernel when they need attention (e.g., data is ready from the network card, a key was pressed) via interrupts. The kernel receives these hardware signals and routes them to the appropriate device driver or handler.
• Device Discovery: Identifying and configuring hardware devices present in the system, especially in "plug and play" systems where devices can be added or removed while the system is running. This often involves scanning system buses (like PCI, USB).

Managing devices can be complex due to varying hardware interfaces and performance considerations. Direct device access by user programs is typically forbidden for stability and security reasons, making the kernel and its drivers the sole intermediaries.

Kernel Design Approaches

The responsibilities listed above can be implemented within the kernel in various ways, leading to different architectural designs. The choice of design has significant implications for the kernel's size, performance, modularity, and reliability.

A key concept distinguishing kernel designs is the separation of mechanism and policy:

Mechanism: The support that allows the implementation of many different policies (e.g., the ability to route user login attempts to a specific program).

Policy: A particular "mode of operation" or rule implemented using a mechanism (e.g., the requirement that a user must provide a password verified against a database).

A kernel provides mechanisms, while higher-level parts of the OS or user-space components might implement policies. The degree to which mechanisms and policies are separated influences the kernel design.

Here are the main kernel design approaches:

1. Monolithic Kernels

Monolithic Kernel: An operating system architecture where the entire operating system, including core functions (like process management, memory management, I/O, file systems, networking) and device drivers, runs in a single address space in privileged (kernel) mode.

In a monolithic design, all kernel services are tightly integrated into one large program. This approach is common in traditional Unix systems and their modern derivatives like Linux and FreeBSD.

Characteristics:

• All kernel code executes in kernel space.
• System calls are typically implemented as function calls within the kernel's code base (though invoked via the hardware mode-switching mechanism).
• Device drivers and file systems are compiled into or dynamically loaded into the kernel's address space.

Advantages:

• Performance: Because all kernel services are in the same address space, function calls between different parts of the kernel (e.g., file system calling a disk driver) are fast, without the overhead of context switches or message passing.
• Direct Hardware Access: Provides rich and powerful access to hardware.
• Simpler Implementation (initially): Can be perceived as easier to get a basic version running compared to the complexities of a microkernel's inter-process communication.

Disadvantages:

• Size and Complexity: Can become very large and difficult to maintain as more features and drivers are added (millions of lines of code in modern examples).
• Vulnerability: A bug in one part (e.g., a device driver) can corrupt data structures in another, potentially unrelated part, or even crash the entire system, as everything shares the same memory space and privilege level.
• Debugging Difficulty: Debugging kernel code often requires specialized tools and can necessitate rebooting the system frequently.

Modern Evolution: Modular Monolithic Kernels

Many modern monolithic kernels support loadable kernel modules (LKMs).

Loadable Kernel Module (LKM): An object file that contains code to extend the running kernel, usually adding support for new hardware (device drivers) or file systems, without needing to reboot the system. LKMs run in kernel space, alongside the rest of the kernel.

This adds a degree of modularity to the monolithic design. Drivers or file systems can be loaded and unloaded on demand. However, LKMs still execute in kernel space and have full privileges, so a buggy module can still destabilize or crash the system.

Examples: Linux, FreeBSD, OpenBSD, NetBSD, Solaris, AIX, HP-UX.

2. Microkernels

Microkernel (μK): An operating system architecture where only a minimal set of core functions (such as inter-process communication (IPC), basic memory management, and basic scheduling) run in privileged (kernel) mode. Most other operating system services (like file systems, device drivers, network stacks, higher-level scheduling) are implemented as separate processes running in user space, called servers.

Microkernels were developed as a reaction to the perceived complexity and reliability issues of monolithic kernels. The philosophy is to move as much functionality as possible out of the privileged kernel space and into less-privileged user space.

Characteristics:

• Kernel core is minimal, handling only essential tasks like IPC, basic scheduling primitives, and low-level memory management.
• OS services (file system server, network server, device drivers as servers) run as user-space processes.
• Communication between user-space servers and the microkernel, or between servers themselves, relies heavily on message passing via the microkernel.

Advantages:

• Modularity: OS services are separate, independent components (servers), making the system easier to design, implement, and maintain.
• Reliability/Resilience: If a user-space server (like a device driver or file system) crashes, it typically does not crash the entire system. The microkernel can often restart the affected server.
• Flexibility: Different servers can implement different policies or APIs, and parts of the OS can be updated or replaced dynamically without rebooting. New services can be developed and debugged as normal user-space processes.
• Security: A smaller codebase running in privileged mode potentially reduces the attack surface.

Disadvantages:

• Performance Overhead: Interactions between user-space servers and the kernel (or other servers) require context switches and message passing, which is significantly slower than direct function calls within a monolithic kernel. This has historically been the primary criticism of microkernels.
• Complexity of Design: Designing an efficient and robust message-passing system and structuring the OS functionality across multiple servers is challenging.
• Debugging Message Flows: Debugging issues involving multiple communicating servers can be complex.

Examples: MINIX 3, QNX, GNU Hurd, L4 family.

3. Hybrid Kernels

Hybrid Kernel: An operating system architecture that attempts to combine aspects of both monolithic and microkernel designs. They typically run some services (often performance-critical ones like networking or file systems) in kernel space for speed, while potentially running others (like device drivers) as user-space processes or keeping the kernel core relatively small compared to a pure monolithic design.

Hybrid kernels represent a pragmatic compromise aiming to achieve the performance benefits of monolithic kernels while incorporating some of the modularity and reliability advantages of microkernels.

Characteristics:

• More functionality runs in kernel space than a microkernel, but often less than a typical monolithic kernel.
• May support modular components like LKMs (similar to modern monolithic kernels).
• May still use message passing for communication between some components, but potentially less frequently than a microkernel.

Advantages:

• Performance Balance: Can achieve better performance than pure microkernels for frequently used services by placing them in kernel space.
• Flexibility: Offers a mix of tight integration and modularity.

Disadvantages:

• Can inherit complexity from both architectures.
• The line between "kernel space" and "user space" functionality can be less clear than in pure designs.

Examples: Microsoft Windows NT family (including Windows 11), macOS (XNU kernel), Symbian.

4. More Exotic Designs

While monolithic, microkernel, and hybrid designs cover most common operating systems, other architectures exist or are being researched:

• Nanokernel: An even smaller microkernel where virtually all services, including very basic ones like interrupt handling or timers, are delegated to drivers or servers. The core nanokernel only handles the lowest-level tasks like CPU mode switching and minimal interrupt dispatch. This pushes minimalism even further than microkernels.
• Exokernel: An experimental approach where the kernel's role is limited only to protection and multiplexing of raw hardware resources (like CPU time, memory blocks, disk blocks). It provides no hardware abstractions. User-space applications run with a library operating system (LibOS) that builds conventional OS abstractions on top of the kernel's raw resource access. The idea is to let applications have fine-grained control over hardware allocation and usage for maximum performance, tailored to the specific application. Highly flexible but requires significant effort from application developers (or reliance on robust LibOSes). The Xen hypervisor shares some philosophical similarities regarding explicit resource allocation.
• Multikernel: Treats a multi-core processor as a network of independent nodes rather than a shared-memory multiprocessor. Each core runs a small kernel that communicates with kernels on other cores via message passing. Designed for scalability on many-core architectures, avoiding contention on shared data structures that can bottleneck traditional kernel designs. Example: Barrelfish.

History of Kernel Development

The concept of a "kernel" evolved gradually.

• Early Systems (1950s-early 1960s): Many computers ran programs directly on "bare metal." Programs included necessary code for I/O and device control. Small resident monitors or loaders were precursors to kernels, staying in memory between program runs.
• Supervisor Programs (IBM OS/360, 1960s): Formalized the concept of a privileged program (the "supervisor") that manages resources and controls execution of other programs, providing services via supervisor calls (early system calls).
• Time-Sharing Systems (Late 1960s): The need to share a single computer among multiple users simultaneously led to significant kernel developments, including sophisticated CPU scheduling, memory protection, and virtual memory. Projects like Multics explored advanced protection mechanisms.
• The RC 4000 (1969): Introduced a system design philosophy based on a small "nucleus" for building different operating systems, laying the groundwork for the microkernel idea.
• Unix (Early 1970s): Had a profound impact. Its core kernel was relatively small initially, providing process management, memory protection, scheduling, and a simple, powerful abstraction: "everything is a file." This design promoted the use of small utility programs and pipes, defining a powerful command-line paradigm. The kernel itself was one of these programs, albeit a privileged one. Over time, Unix kernels grew significantly.
• Microkernel Research (1980s-1990s): Projects like Mach and MINIX explored the microkernel architecture to improve modularity and reliability, reacting to the growing complexity of monolithic Unix kernels. This led to the famous "Tanenbaum–Torvalds debate" about the merits of monolithic vs. microkernel designs.
• Modern Kernels (Late 1990s - Present): Most widely used commercial operating systems (Windows NT, macOS) adopted hybrid kernel designs. Linux, a prominent open-source system, evolved as a modular monolithic kernel. Embedded systems and real-time operating systems often utilize microkernels or specialized small kernels where predictability and reliability are paramount.

Building a kernel from scratch means walking in the footsteps of these pioneers, making fundamental design decisions about how to manage the machine's resources, provide necessary abstractions, and protect the system's integrity, all while grappling with the constraints and capabilities of the target hardware.

---