Hello, and welcome to CS615 System Administration! This is week 1, segment 2, and after in our last video we talked about all the boring administrative stuff around this online class, we'll now get to make things a bit more interesting. You all have enrolled in this class entitled "Aspects of System Administration", so I assume everybody has some sort of idea what a SysAdmin does. Fun fact: you're all wrong. Well, you're also all correct, I suppose. For, you see, the job of a system administrator is not very well defined. Sure, you get 'root' in the Matrix, like our friend Tank here and... wait, that movie came out 22 years ago. Some of you weren't even born yet! Oh boy, I'm going to have to update my pop culture references, it seems. Anyway, let us talk about the job of a system administrator... --- Now of course we're biased. As a System Administrator, we view ourselves as the hero of our story, and all the other participants in our tech world as buffoons, but it's also true that _we_ are often seen by others... well, not necessarily in a positive light. We'll return to this sort of tribalism towards the end of this video, but let's think of an example of what a system administrator does. I mean, other than flipping off everybody else and single-handedly rescuing mankind. --- Here's an example: "You gotta touch the banana for Wifi Access", says SysAdmin. I like this. It's an example of the weird situations SysAdmins find themselves in, and how they might solve a problem: In order to grant people time-limited Wifi access, a receptionist would have to manage a spreadsheet and print out the codes. Instead, this person rigged a Raspberry Pi to a banana and used the voltage drop detected when you touch the banana as a signal to generate a new code. It's an example of automating a manual task with a clever and -- frequently -- unexpected, creative solution. A very SysAdmin thing to do... But let's try to be more specific. What _exactly_ does a System Administrator do? --- Sure, SysAdmins work with computers. With all sorts of computers. With workstations, with obsolete, ancient hardware, with laptops, --- with tiny little computers like this Raspberry Pi, which we just saw in action, --- but of course also with larger computers. What you see here is an HP Blade system used in data centers to allow for multiple physical servers enclosed in a single chassis. These boxes are then filled with server blades that are space and energy efficient and are connected using high-speed network and storage networks, often with hot-swap capabilities and out-of-band management capabilities, perhaps optimized for virtual servers allowing for a high compute density. --- But of course you can scale that up. Here, you see a set of SGI Altix systems, which are often times used to build some of the world's most powerful supercomputers, with each system containing up to 2048 dual-core microprocessors, and yes, they are large enough for you to step into these racks if they weren't full of computers and cables. But of course nowadays people don't only run their tasks on individual servers or CPUs, but instead on --- containers. So many containers. Containers everywhere! So naturally, SysAdmins have to be on top of containers and how to manage workloads using Docker and Kubernetes and whatnot. You'll see plenty of online memes here around containers and "shipping" code, so naturally we take it all the way to 11 and --- think about how to manage such infrastructures. That is, we're in the business of building ship shipping ships to ship containers onto our servers in the datacenters in the cloud. There's probably a recursion joke hidden somewhere in there as well. --- But yes, SysAdmins often work in data centers, and some of them look just like this. Well, to be honest, this isn't a real data center; it's the view of a local machine room at a company I once worked at. You see a typical setup with locked cages, racks along aisles with raised floors, boxes and shelves of equipment, and cables going just about everywhere. Kind of a mess. So along comes a sysadmin, --- grabs their cable ties... a second favorite tool of all sysadmins -- the mighty label maker... ...and, quite likely, an unhealthy amount of caffeine or some other perfectly legal drug... ...and after just a few sleepless night --- we've cleaned up the mess. That is, SysAdmins bring order to chaos. There's no shortage of OCD in our field, but we prefer to consider this level of meticulousness a virtue, a dedication to doing the right thing, knowing that this investment of time and effort will later pay off many times over when we need to determine which cable connects which host to which switchport, for example. There's an entire subreddit dedicated to the art of clean data center cages, racks, and overall cable porn. By the way, not all of these cables are necessarily manually run by a SysAdmin: at a certain scale, you may be able to order your racks pre-cabled already, although others consider that cheating. --- Ok, so a clean data center aisle might look more like this. As you can tell, blue blinkenlights are very popular, as is a tendency to keeping things identical. So with larger and larger computing requirements, you end up filling aisle after aisle of racks filled with servers, and then you have to start thinking about cooling for all these systems. So SysAdmins may be involved in the design of data centers and their cooling technologies. --- What you see here is not a farm in the middle of nowhere in Nebraska, but rather... a datacenter in the middle of nowhere in Nebraska. One of Yahoo's datacenters, to be precise. The buildings have a kind of funny shape, because the data centers use a patented approach for cooling: the Yahoo "Chicken Coop" design. Nope, not a joke. The engineers at Yahoo -- including System Administrators -- designed their data centers - inspired by how chicken coops distribute air flow. As a result, the requirements for cooling systems using air conditioners or other electrical systems went down drastically, and the majority of the cooling is done simply by, in a way, opening the windows. --- And those data centers have all sorts of access controls, including security cameras and biometrics readers. All of which nowadays are hooked up to the internet, run a web server, and suffer from remote code execution vulnerabilities that some poor SysAdmin has to try to patch or otherwise mitigate. Network design with segregated zones for such equipment then also may fall into the job description of a SysAdmin, or we may simply be users of these controls as we monitor their output. --- But of course to install such equipment, every SysAdmin everywhere has a giant box or a drawer full of random cables, because for some reason no manufacturer in the world wants to use the same cable as any other manufacturer, not even when you are using industry standards. Like... USB. - Which, in order to drive you insane comes in 30 varieties, and in the history of the internet not once has anybody plugged in a USB cable the right way on the first try. So connecting things is definitely a SysAdmin thing, which reminds me of the various projectors used in classrooms at Stevens, which was a big problem, since different faculty would use different devices and would never have the right adaptors. So a few semesters back, I was pleased to discover - these contraptions in every classroom, which seemed to me like a very SysAdmin-y solution: a selection of the most common adaptors chained to the projector cable. --- But of course cables are often not needed any longer either, and everything goes wireless. Which... brings with it an entire class of _other_ problems, because now you have to support people working in environments you don't control and which may in fact just be an invitation for compromise, because you can be guaranteed that somewhere in that Starbucks some clever CS major has set up a rogue wifi access point and is now sniffing all your traffic, but if the user can't get to your website, it's still your fault. And on the other hand, if you’re the SysAdmin for Starbucks here, you have the problem of supporting infrastructure for random people while making sure they don’t use your systems for things you’ll be held liable for. Good times, good times... --- So you go back to your desk and... wait, what is this now? Oh, right, a punch card. That's how we used to program. A long time ago, you wrote your code, arranged your punch cards in order, then walked them over to the _operator_, who you'd hope would feed them into the computer, but who in reality may have tripped, dropped all your cards, and you now have to put them back into the right order. Well, at least we don't have to do _that_ anymore. Although, now we have to deal with "undefined is not a function" and similar nonsense. But we _do_ write a lot of code as System Administrators, and we do a lot of typing in general, so to do our wrists a favor, --- we should use some ergonomic keyboards, like this Kinesis keyboard shown here. These keyboards are amazing. They remind you that you probably suck at touch-typing. I certainly do. When I was a System Administrator here at Stevens, one of the CS professors used one of these keyboards, and I quickly learned that I didn't actually remember my passwords very well, that they were all in my muscle memory and it took me a good minute to log in on his keyboard. Of course it didn't help that he had changed his keyboard layout to DVORAK instead of QWERTY... I soon stashed a spare keyboard in his office so if I had to work on his system, I'd be able to type normally. Anyway, so SysAdmins do a lot of typing, and of course we do the vast majority of our work on the command-line. --- That is, we operate on the console, day in, day out. System Administrator know -- or should know, anyway -- what all these messages flying by here mean, as each one is not generated by the operating system for fun, but to provide some meaningful information. What you see here is a NetBSD amd64 virtual machine booting up, with kernel messages in green and messages generated once init takes over in white, getting a DHCP lease, starting sshd, postfix, cron etc. and finally offering the login prompt. Most of the time, we don't even see those messages, since we're doing the majority of our work remotely over the internet. --- Right, the internet. That's certainly something System Administrators have a _lot_ to do with. We _use_ the internet, and we _build_ the internet, a network of networks, as illustrated in this partial map. But of course to do that, we need other equipment beyond just computers, so --- System Administrators also deal a lot with network equipment, like these ethernet switches shown here. Switches allow for network connections on layer 2 of the OSI stack, which we're all rather familiar with, but if you want to build a more complex network and connect to other networks, you'll also need --- some routers, like this one. You can tell that this has a similar form factor and design as the blade server systems we saw a minute ago, and of course that's no surprise: in order to facilitate installation in the standardized racks inside the data centers, this equipment follows the same standards, and can also often be extended or upgraded in a similar manner by swapping out individual blades hooking into the backplane within this chassis, depending on your routing needs. Why do we deal with all this networking equipment? Often times it's because as a System Administrator we're in charge of --- a web server running some service using HTTP as the universal protocol, although of course we certainly hope that it's also using TLS to secure the connections. But this isn't quite what a normal infrastructure looks like, is it? --- For starters, we usually have more than just one web server, and we probably have to store some data in a redundant database. --- Then we add a load balancer in front of the web servers and probably decide that we don't allow just _any_ traffic, perhaps by using a firewall to create a security perimeter around the web servers. --- And of course as we grow our infrastructure, we probably need to add some sort of message queue system, a large storage array, and perhaps we add Zookeeper to juggle some services... --- ...and of course we need to allow some of our engineers and developers access, so we let their laptops, workstations, and mobile devices access these internal resources, but... damn, those silly humans have a tendency to want to go home some time, or maybe there's a pandemic going on and all of a sudden people aren't in the office in more, so you need to allow folks from _outside_ your perimeter access. --- What's more, you probably have to integrate with a bunch of third party services, because nowadays everything lives in the cloud, and come on, if you're not using GitHub, Jira, and Slack, can you really call yourself a hip startup ready to make the world a better place? Of course you can't, --- so you punch some holes into the firewalls to let these connections in, and sooner or later your boss tells you that they decided that it'd be much better to take your entire infrastructure and --- move _that_ into the cloud, why not, so suddenly you are running on somebody else's computers, because honestly, that's really all "the cloud" is: somebody else's computers. But yeah, so System Administrators are involved in building, supporting, and maintaining just about every piece of this puzzle, and I tell you what: Nobody will notice. Well, until, --- things go wrong, anyway. Which they will, sooner or later. And this, when shit's really on fire, that's when people will suddenly remember you and call you up to come and fix the mess, and so --- you grab your trusty leatherman multi-tool and get to work. Seriously, just about every SysAdmin I know has one of these, because you never know what you find in the machine room, what things you have to unscrew, dislodge, or dismantle and then put back together. In fact, there's a surprising number of physical tools that system administrators end up using in their line of work. Like, for example, --- a circular saw. Which doesn't necessarily come to mind immediately when you think of System Administration, but after a few runs to the hardware store for increasingly stronger drill bits to get the holes into the surprisingly hard concrete floors to screw in the rack, I once was only able to stabilize a number of very heavy backup batteries in that rack by cutting some boards to size to support them, using just such a saw as shown here. And perhaps this gives you an idea of how varied the SysAdmin's path can be; not everybody will have to handle heavy machinery, but the ways in which the systems are connected may lead you down that path. --- Inevitably, we are dealing with systems that require electrical power, and so you may also get to learn a bit about what happens when the power goes out, and why it might be a good idea to have diesel generator (and to remember that for that to work you need to have diesel fuel) as well as to understand just how much power your systems draw, which systems have a higher priority to bring back online etc. etc. --- Alright, so long story a bit shorter: every SysAdmin has their own toolbox, from which they pull what they need to fix whatever's broken today. A lot of times that involves duct tape and WD40, or their software programming equivalents. You may have heard people refer to Perl as the duct tape of the internet, and boy, folks weren't even kidding. If you ever look under the hood of the internet, you will be surprised at how things are held together, and in your career as a System Administrator you will get a chance to learn quite a bit about the layers of glue here. --- Ultimately, though, the single most important tool in the SysAdmin's toolchest is your brain. You'll have to come up with new solutions to old problems, apply old solutions to new problems, and anything in between. Whether you end up supporting people's workstations, run internet load-bearing infrastructure in the cloud, design and operate data centers, or manage everything in between, this profession requires a lot of ingenuity and curiosity. --- Ok, so after this whirlwind tour through Jan's random collection of images, are we any closer to answering the question we asked earlier: what _exactly_ does a System Administrator do? We've pretty much seen that there - can't be a uniform job description - that there's no uniform career path - and that sysadmins often times are just considered as the people who "make things run". That is, we frequently - work behind the scenes, rarely being seen unless disaster strikes. And so, the System Administrator may fill many roles, and sometimes instead be referred to as - IT Support, Operator, Network Administrator, System Programmer, System Manager, Service Engineer, Site Reliability Engineer or any other variation thereof. --- In more recent years, the two main other job titles besides "System Administrator" widely used were "DevOps" and "SRE", and the differences as shown here really are not as specific: Rather, it depends on whatever the organization defines the role to be. What's "DevOps" in one organization is "SysAdmin" in another, and an SRE may perform the same or completely different tasks depending on the company. In fact, the title used is often a function of organization maturity and scale. It's no surprise that the term "Site Reliability Engineering" originated in part at Google, where it was described as "what happens when a software engineer is tasked with what used to be called operations". --- That is, the various areas that may comprise the day-to-day duties of somebody working in this field might be illustrated as shown here: In a small environment, you likely only have a small number of staff performing these tasks -- perhaps only a single person -- and they are responsible for _everything_ ranging from daily operational tasks to the planning of the services and infrastructure they support. In a larger environment, however, these duties are divided amongst multiple people, and in even larger organizations amongst multiple, specialized _teams_ of experts. You can come back to this graphic as you progress in your SysAdmin career, if you like, but unfortunately this _still_ doesn't really give us a definition of "System Administration". So let's try something else. --- Let's ask the dictionary. What's a "system", anyway? Hmmm, let's see... Here, how about this? "a group of independent but interrelated elements comprising a whole". I like that. What about "administrator"? "one who directs, manages, or dispenses" and "system administrator", then... "a person in charge of managing and maintaining a computer system of telecommunication system (as for a business or institution)" --- So this definition, then, seems pretty good to me, as it hits a few important aspects: --- First, there is a clear implication of the job comprising multiple aspects, with the management of computer and network systems being only the _primary_, but not the sole responsibility of the system administrator. --- Secondly, the job of a system administrator is clearly to manage these resources _on behalf of another_, meaning there is a larger organization or system involved beyond just the individual's goals or desires. So I'm afraid running your own home network does not make you a SysAdmin. But let us consider the meaning of the word "system" for just another second: --- In this class, we'll focus primarily on _computer-human_ systems consisting of - obviously a bunch of computers - the network connecting those computers - but also the human component, since a system without users is, quite literally, _use_less. Which, unfortunately, is something that it's easy to forget at times. Similarly, the users may act in a certain way that may or may not be in line with - the organization's goals and policies. As noted a second ago, the primary job function of a System Administrator is to manage these systems on behalf of another, such as a company or other organization, and so this strongly affects and influences almost all aspects of the systems we manage. And this, then, gets us back to --- what we showed earlier: the inter office politics and the human nature of neatly creating tribal allegiances based on some sort of "us vs them" feeling, and, guess what - that's really not helping at all. Instead, it's important the System Administrators understand how to work _with_ other people, from all the different backgrounds, because --- no matter how much we may like to focus on the technical aspects of our job, we have to remember that solving the technical problems is the easy part. Even if debugging your load balancers or trying to unwedge git or tracking down that one weird bug may take you hours, it's still a lot easier than dealing with people. Programming and everything surrounding the internet and the management of the systems that make up the internet... all that is really a _social_ thing, something that requires understanding of human beings and their motivations. Computering, at its heart, is a people problem. And so, with everything covered in this video in mind, I've come to think of System Administration as a profession --- where you primarily solve people's problems. Sure, often times this involves computering a bit harder than they did, and spending a lot of time convincing computers to do the thing you meant them to do, not the thing you told them to do, but the end goal remains to solve people problems, to manage the resources and systems you're in charge of on behalf of the organization and the public interest in mind. And that, then, is the job of the System Administrator, and what we'll focus on in this class. --- Ok, so now having a bit more of an idea of what a System Administrator may do, let's take a break. I encourage you to research a bit about the job descriptions, of the difference between SysAdmins, DevOps, and SREs, and follow the links included in these slides. Having covered a bit of the "what", we'll then use our next video segment to talk a bit more about the "how" of system administration and - discuss what I termed the core pillars of exceptional system design: scalability, security, and simplicity. We'll also - cover a few guiding principles and several meme-worthy laws of system administration and software engineering, and finally, we'll talk about - why people think that the internet looks like - this, but we SysAdmins, having peeked a bit under the hood, know that in fact it looks more like - this. Until the next time - thanks for watching. Cheers!