Hello, and welcome back to CS615 System Administration! This is week 3, segment 3, and after in our last video we explored what it means to be a filesystem, we'll now take a look at a slightly more sophisticated implementation: the standard UNIX filesystem, or UFS. The UNIX Filesystem is the historical filesystem implemented in Version 7 of Unix, and subsequently superseded by the Berkeley Fast File System, also known as the BSD Fast File System or FFS. Now of course there are many more types of filesystems, and the implementations vary significantly between traditional filesystems, journaling filesystems, and things like ZFS, as we've already hinted at. However, the fundamental concepts remain the same, and so it's useful for us to consider how the UNIX filesystem is structured. --- Now recall from our last video, that in our filesystem we quickly identified a need to allocate a storage unit for the data we wish to store, and that we then --- equally as quickly realized that we want to be able to associate metadata with a file, but that in doing so it would be useful to separate the metadata from the file data and storing it on different areas on the disk. We then constructed a crude filesystem format to accommodate this model, but of course a real filesystem will look somewhat different. So let's take a look: --- As we recall from our earlier video, a physical -- or virtual -- disk can be divided into logical partitions that may be described in, for example, the master boot record. --- Such a logical partition may be further divided into filesystem partitions. On BSD derived systems, these partitions are described using a so-called 'disklabel', which we've also seen before. --- The OS specific partitions on which we may then decide to create a filesystem are comprised of _cylinder groups_, a concept that is applied even if the hardware backing the storage device were, say, an SSD, and thus really doesn't have any physical cylinders. Either way, we can now visualize our physical disk as looking --- like this. Now since each partition comprises these cylinder groups, we have a need to describe them in a filesystem specific manner and to store some meta information about the filesystem itself. This data is stored in the so-called _superblock_, which we generally find near the beginning of the partition. If the partition is a boot partition -- such as our primary OS partition -- then it may also contain boot blocks at the beginning of the disk as well. Now each cylinder group itself --- contains the actual data blocks -- the parts where we store the actual bytes that make up our files -- as well as a list of inodes and blocks set aside for the meta data associated with the inodes, the inode blocks. Since the entire structure of the filesystem is written in the superblock, it'd be pretty disastrous if you lost this one block, so the file system replicates the superblock in cylinder groups, thereby allowing recovery of the filesystem from a a corrupted superblock. --- Finally, the actual data that we discuss when we talk about files and directories are stored in different groups of data blocks: the inode data blocks and the file data blocks. Here, we see how the meta data about a file -- all the information that we came up with in the last video -- is stored separate from the actual bytes of the files. As you can tell, we continue to strictly follow the evergreen mantra of CS where one layer of abstraction reveals further layers as we dive deeper. But at least we end up with a reasonably easy to understand diagram, such as this one. Now let's see if we can map our understanding from this diagram back to what we actually observe on the disk: --- Here we have our usual NetBSD instance with our extra volume attached. To inspect the properties for a filesystem, we can use the 'dumpfs' utility, so let's take a look at the manual page. This tool allows us to see details about the superblock, cylinder group, and the inodes, for example. So let's give it a try! Oh, huh, look at that. No superblock. No surprise, we haven't created a filesystem on this disk yet. No filesystem, no superblock, no meta information about the filesystem - makes sense. Let's start by creating a disklabel. We accept the default of just using a single partition spanning the entire disk here, since this is our second disk and doesn't require a boot block or swap or anything. Ok, so dumpfs still doesn't know anything about this disk. So let's create a filesystem on it already. For that, we use the 'newfs' tool. The 'newfs' tool is used to create a new filesystem, and as you can tell, there are several useful options here. For example, we can specify a filesystem _block size_ -- that is, the smallest storage unit for our files. Note that this is a filesystem specific property; the storage device itself will, as we discussed in previous videos, have its own _physical_ block size -- commonly 512 bytes. Another option we can specify here is the total number of inodes to create, as well as several others. Ok, so let's create a filesystem on our disk. Here we go. We created a filesystem with the defaults: 16K blocksize across 6 cylinder groups with 21504 inodes each, and a superblock stored at these locations across the partition. Now that we have a filesystem, we can mount it. ...and create our files on it. cat1 cat2 ...aaaand snowcats So now let's look at 'dumpfs' again. With the filesystem created, we now get a lot of information displayed here, including about the cylinder groups. The dumpfs utility can _also_ show us information about the inodes in use: Here we see the information for the files we currently have on this disk, including their permissions, file size, their creation time and ownership etc. We can compare with the output from 'ls' and observe the timestamp via 'date'. inode number two, by the way, isn't missing -- it's the inode of the root directory on this disk. There it is, with size, owner and group as displayed in the dumpfs output. If we run 'dumpfs' with the '-v' flag as well, then we also get information about _all_ the inodes, including those that are not yet used. That is, we can see that our filesystem has indeed allocated a fixed size of inodes _at filesystem creation time_, and thus we are limiting how many files we can store on this disk the second we create the filesystem. In this case, that's 21503 inodes per cylinder group, meaning we're maxing this disk out at... just about 1.3 million files. --- Ok, time for another break. Let's summarize: We've seen that our - filesystem divides the disk into cylinder groups. The information about the filesystem and cylinder groups is then stored in the superblock, which - to avoid it being a single point of failure, is replicated into the different cylinder groups, thereby allowing recovery and reconstruction of the filesystem even if one of the cylinder groups is corrupted from the alternate superblock. We also saw that our filesystem has the concept of - a block size, the basic, optimal I/O unit that the filesystem will fetch or write data in, and may be different from the physical block size of the disk. Next, and in alignment with what we had come up with ourselves in our last video, we - want to separate where we write the actual file data from where we write the meta data. This allows for more efficient storage and less wasted space, but also increases I/O efficiency, since often we don't really need to access the actual data of a file: Think about how when you run 'ls -l' you are retrieving and displaying all the metadata associated with a file, but not the contents. By separating the inode blocks from the data blocks, we can rapidly fetch all the metadata without having to seek to the data blocks back and forth. Talking about inodes, we observed that - the total number of inodes -- that is, the total number of files the filesystem can hold -- is fixed at filesystem creation time. And that, together with the filesystem block size, are just two of the - various things we can tune at filesystem creation time to accommodate the anticipated use of the disk. That is, if we know that this disk we're putting in place is intended to be used to store primarily a relative small number of really large files, then you might choose a different inode density and blocksize than if you wanted to store huge numbers of tiny files. In the first case, you'd use a larger blocksize to improve I/O and select a smaller number of inodes and fragments; in the latter, a smaller blocksize and larger number of inodes. The 'fs' manual page in section 5 has a lot more details here, and it'd be a good exercise for you to try to create filesystems using different parameters and see if you can exercise and observe performance differences based on the different use cases. Alright, so with all this information and the exercises you've seen, we now have a pretty good understanding of the basics of the UNIX filesystem. And although other types of filesystems utilize different techniques to manage the storage space, a lot of what we discussed here remains directly applicable. So with that, - we can finally move on to briefly summarizing the different file types we're dealing with as well as some of their attributes as well as talking about where we mount our filesystems and what conventions exist for our filesystem hierarchy. Until next time - thanks for watching. Cheers!