Hello, and welcome back to CS615 System Administration. This is week 4, segment 2. In our last video, we we just spent some time discussing the different types of software, and we did draw a distinction between the operating system -- the OS -- and several other types of software. So why don't we start out thinking about how we get one of those things, one of these operating systems, onto our disk? So in this video, we'll do just that: install an OS. But of course, remembering that we want to keep "scalability" as a core pillar in our mind, we don't just worry about getting an OS installed this one time on this one box, but rather how to ensure we can do this in a automated fashion across hundreds or thousands of systems. And before we can do that, we should probably come up with a plan of --- what files should go where. You remember the 'hier(7)' manual page from our last video -- it tells us the filesystem layout and describes the hierarchy under which we place the various files. In there, we had noticed a difference between, for example, the /bin and /usr/bin directories: everything under /usr is not needed at system bootstrap time, so can be placed onto a different partition that's mounted as part of the boot process. We also noted the presence of the "/var" directory, which contains, well, _variable_ data. But why would we care about having _variable_ data under a separate directory or partition? --- To answer that, let's look at the definitions of different types of data: Now _variable_ data - as the name suggests, is data that is expected to modified during routine operations. So that includes log files, obviously, as well as the data in the users' home directories, for example, as we expect them to write to their files with some frequency. In contrast to variable data, we then also have - static data -- data that is _not_ expected to change during the normal course of operations. This includes pretty much all your operating system files, your libraries, and applications. Sure, you may need to update those once in a while, but by and large, those do not change. And perhaps you already see where this is going: if you know which data changes, and which data does not, then you can set up separate partitions and mount the filesystems accordingly: variable data read-write with asynchronous I/O to improve performance and perhaps no-exec or at least no-suid; static data read-only. This separation then makes it more difficult for an attacker to compromise the system even if they get access, but it can also help you organize your filesystems across multiple hosts. And for that, you may want to further divide your data into - shareable data -- data that, if you run multiple hosts, remains the same across all of them; and - non-shareable data -- data that is necessarily unique to each individual host. Again, you can hopefully see why this might be useful: if you know which data sets are identical across hundreds or thousands of hosts, then you can think about how you can share it, or deploy it, or manage it in a central fashion, while the bits and pieces that are non-shareable need to necessarily be managed on a host-by-host basis. Here, let's look at some examples of these different types of data: - We create a matrix, - since variable/static - and shareable/non-shareable may overlap. So as an example of data that is static -- that is, we don't expect it to change during the normal system runtime -- as well as shareable -- that is, the same for multiple systems might be the data found under - /usr -- the common libraries and applications for a given operating system as well as the data under /opt, a popular directory or mount point for add-on software not included in your OS or not managed by your package manager. These files should not change frequently, but would be identical across your entire fleet of systems of the given operating system. Shareable data that we do expect to change might be - that under a separate data partition we specifically set aside for a project, for example, such as /var/data, or the home directories of your users. As we mentioned earlier in the semester, you might use the network filesystem to make shareable data available to multiple systems, and NFS mounted home directories are perhaps the most common use case here. Now in the column for non-shareable data, we'd list - the system-specific files, such as the unique boot blocks or the system configuration files under /etc as examples of static, non-shareable data, and - the various runtime files created by processes on this particular host as examples of variable, non-shareable data. Now of course these distinctions are dependent on your particular setup, but I hope you can see that it might be useful to plan for how you are going to manage your data in this fashion, and, more applicable to this video segment's topic, that it'd be rather useful to have decided on this layout _before_ you begin the OS installation, such that all the right partitions are created and the files installed appropriately. That is, we got our first taste of one of the more important aspects of operating system installation: not so much the actual deployment of software or packages onto the disk, but the _advance planning_ of what to put where. But alright, so let's assume we did plan out our filesystem layout and partitioning scheme and we now want to go ahead and install an operating system... --- You may be familiar with a screen like this -- a graphical installer offering you a few buttons to click to install the OS. Now this is certainly useful, and it takes quite a bit to get to this point: we must have booted from some alternate media, loaded a temporary OS into memory, initialized the graphics hardware and detected the correct display and all that so that we can interactively ask the user how they want to install the operating system. But in this class we said we want to keep an eye on simplicity and scalability, and I think it's obvious that we can't easily install a few hundred systems if for each one we have to manually click on some boxes to start the installation. And of course nobody outside the individual consumer would use such a graphical installer. Instead, in the industry we use deployment engines that are often times custom built and that allow for the completely automated installation of an operating system based on a system profile. But to be able to build such a system, we need to understand what the actual steps are that take place during OS installation. --- So let's go ahead and perform a _manual_ installation, but not using a graphical display, but instead using the actual commands that the graphical installer would invoke. So for this, we are booting a virtual machine off a NetBSD install CD, which drops us into a menu-based installer. This is still interactive and doesn't show us the commands, so we simply drop out of the menu. This lands us in a root shell on the ramdisk OS booted off the CD, and we can now run all the commands we need to install the operating system ourselves. --- So first, let's identify the harddisk available to install the OS on. There, wd0 with the geometry as shown here by the kernel. From our previous videos, we know we need to set up the partition table, so let's calculate the total sectors left if we start our OS partition at offset 63 and then use fdisk(8) to set partition 0 of type NetBSD at offset 63 of the given total size. Next we copy the actual bootcode into the boot sector... ...and mark partition 0 as 'active'. Then we create the BSD disklabel. Since our EDITOR environment variable is unset, we get dropped into 'ed', the standard unix editor. But don't worry, 'ed' is not that bad. It's just a line-based editor, so we have to specify the actions with line addresses. Anyway, here's what the disk label looks like right now, and that looks just fine. We have a single large partition for use by the OS, and in this case we have no need to change that. So we simply write the label to disk and quit. Now we're ready to create the filesystem on the partition, similar to what we've seen in a previous video, if you recall. --- Ok, so now that we have a filesystem created on our destination disk, we can mount it. We use '-o async' so that writes happen faster when we extract the data. Next, let's look at the data sets we want to extract. We see all the different tarballs here and can choose which ones we want to install. I'm going to leave out the X window system, since we're installing a server and don't need a graphical user interface. So we'll extract all of these into /mnt, where our disk is mounted, together with a generic kernel. Next we copy the bootstrap code into place... ...and install the primary boot loader for our filesystem with a 5 second countdown. Then, we create all the device nodes under /dev... and we're mostly done. "Mostly", because our system is now _installed_, but not _configured_. To do that, let's chroot into it, so we can treat it as a regular system and don't have to remember pathnames under /mnt etc. See, here we are, everything looking fairly normal. We need to adjust a few bits in /etc/rc.conf, the file that controls the system startup. On other unix systems, that might be equivalent to init.d or system.d configuration. We enable DHCP, NTP, and set a hostname. Then we update /etc/fstab to tell the system to mount the root filesystem. And _now_ we're done. So we can exit our chroot, unmount the disk, and reboot into our newly installed OS. --- When the system comes back up, we find our stage 1 bootloader here, with the 5 second delay we had specified our kernel boots up DHCP does its thing and here we are, up and running. --- Ok, so why did we go through all this work of running all these commands ourselves when there's a perfectly good installer provided by the OS? Well, as mentioned earlier, those installers don't lend themselves to automation, and what's more, in this class we always seek to actually _understand_ what the system does, how things work. And in the process, we can then extrapolate roughly just how OS installation needs to happen regardless of the particular Unix flavor in question. So let's generalize and run through the steps to install the system based on what we just observed: Obviously, we'll start with - the system being powered up. As we already covered before, there's a few things that happen, but at some point we'll be looking for a boot loader. If we're installing a new OS, then obviously we don't _have_ a boot loader yet, so we need to - boot off alternate media. In our example, and in the case of your custom one-time end-user scenario, that would be a CD, but for a scalable approach in a data center, you'd probably want to have your system netboot. Now this includes a bit of added complexity, such as - the system needing to obtain its network configuration - and retrieving the ramdisk to boot into over the network, but we'll handwave over this for the time being. Once your ramdisk has booted, you then need to - identify the hard drive to install the OS onto - partition and label the disk - create the filesystem - install the bootblocks and make the partition bootable - and retrieve the OS data itself. In some cases this will be extracting the data from the local CD, as we showed, or, in the data center, retrieving the data over the network from a local mirror via HTTPS, for example. We - then extract the data onto the disk - perhaps add whatever additional software we might need and then - perform a little bit of minimal system configuration, just as we've shown. After all that, we're good to go and can - reboot the system. Now these steps are more or less the same for most Unix systems that you would install, and they would only differ in the specific commands you'd use to perform them. And having understood these steps, you can then automate the whole process and build a deployment engine, something just about every system administrator I know has built at least twice in their career. But perhaps you'll also notice that the basics of this process are really not complicated. Rather --- the more difficult parts are happening elsewhere. That is, in order for you to be able to build an automated deployment system that can install an OS unattended across many systems, you need - some sort of hardware inventory - a method to determine which OS should go on which hardware - as well as what software to add that's not part of the base OS. This may be accomplished by defining a profile or identities for your different workloads or images, for example. Next, you need to perform at least - _some_ initial system configuration - and register the now installed system in your inventory, before you - restart it But many of these steps here require quite a bit of infrastructure and organization around them. And once again, it's not always clear where the boundaries lie between how you bring up a new system - and how you configure a running system. That is, there's some overlap with the larger topic of Configuration Management as well as service orchestration, topics we'll cover in more detail a bit later in the semester. And finally, you may also have noticed that we kept talking about an OS installation as if it was necessarily onto a -- physical or virtual -- server. But it needn't be: nowadays, it's perhaps more common --- and certainly more scalable, to build individual containers or OS images. That is, your AWS Machine Images are example of where an OS is installed not onto a running system, but instead into what becomes a machine image that can be _instantiated_. Now there's obviously a bit of a difference in creating a container for a process and instantiating a machine image, but perhaps you can see that the up-front work we discussed -- deciding what data can be shared, what is static etc. -- and the general steps we outlined, help us better understand that process as well. --- So let's summarize what we observed: - For starters, even though it may seem intimidating the first time you do it, the steps involved in installing an operating system are not very complicated. What's more, they're the same -- or similar, at least -- across the board, which is why it's useful to go through the exercise of manually installing a system. - Secondly, we saw that planning out our filesystem layout can help us streamline the OS installation process, improving its scalability and making it simpler. - Logically, then, the bulk of the work lies not in executing the same set of commands, but rather to identify the surrounding properties, factors, and system aspects that need to be accounted for. - All of that leaks into, as we said, with other areas of work in System Administration, which we will get to later in the semester, but which you may want to look into already to see how, for example, configuration management overlaps here. So a few exercises for you to improve your understanding and internalize what we covered so far would be - to repeat the process shown here for a few different operating systems. Download the install images and install them into a VM -- follow their provided installer and note which steps they are executing, then see whether or not you can repeat the process yourself by running the commands manually. Finally, think about how you could automate the process of installing these different OS -- there are some tools out there that can help you. Research and find out what they are and how they work. - Then, think about how this process differs from creating a machine image instead of installing an OS. How would you handle the host-specific bits in such an environment? What infrastructure services does the cloud need to provide to make this possible? As you can tell, you could spend a lot of time on this topic alone, and rest assured that system administrators everywhere _have_ done so. In the process, we've all found out that there's a distinction between what makes up the operating system as provided by your vendor or open source project and what ends up on your system. You will have to add more software, some of it open source, some proprietary, and some developed in-house. How we try to manage all these different components will be the topic of our next video, when we discuss package management. Until then - thanks for watching. Cheers!