Hello, and welcome back to CS631 "Advanced Programming in the UNIX Environment". In this video segment, we're going to take a look at the process environment. We'll use what we learned about the process layout in memory to understand how the environment variables are stored and, if necessary, moved around. In doing so, we'll also get a quick look at what malloc(3) does. But first, let's quickly remind ourselves of what the environment looks like: --- In the shell, we can print the current environment by running env(1) utility. What we see then is a list of key-value pairs, many of which we are very familiar with from day to day usage of the Unix system. Setting things in the environment is then an easy way for a process to provide information to the user as well as for the user to provide information to any program written to look for a specific environment variable, and so, by convention, many Unix tools do honor and use certain commonly set variables -- see the environ(7) manual page for examples. From experimentation and common usage, you may have noticed that setting variables in one process does not have any effect on another process, and so we can conclude that the environment is process specific. But how does a process gain access to the environment? --- From our previous videos, we remember that the layout of a process in memory can be visualized like this. Here, we already noted that environment variables can be found near the high address, above the stack. --- Let's take a closer look. - Our "extern char **" is found in the bss segment, as it was declared without being defined in our code. Once exec has transferred control to the startup routine, the environment is then initialized, and - our environment variables themselves will be found near the high address, as promised. --- How did they get there? Recall from our previous segment what the ___start routine looks like: - in here, we set the environment, but then - we also pass it into main as well. --- That is, we have, in effect, two versions of the environment. --- So let's update our code to compare the 'extern char **environ' to the 'char **envp' passed as an argument. We'll count how many environment variables there are set, and print out the beginning and end of the array. --- Ok, here we go. - The environment contains 18 elements; the last element -- index 17 -- is the highest address. - Our 'extern char **environ' is found in the BSS segment, as discussed. - But the 'char **envp', since it is passed as an argument to 'main', is found on the stack, just like argc, and argv. - Note that 'envp' and 'environ' both look the same: both start at the same address and end at the same address. --- Ok, so far, so good. How do we access the environment or manipulate it, though? For that, we have the getenv(3), setenv(3), putenv(3), and unsetenv(3) library functions. getenv(3) retrieves a value, putenv(3) adds a new value, setenv(3) can be used to change a given value, and unsetenv(3) removes a value from the environment. Note that there's a difference between setting a variable to the empty string and removing it from the environment completely -- some tools may only test for the existence of a variable, not for any specific value, and a variable with no value still counts as being set! But now let's think a bit more about how these functions work. --- Remember, the environment is initially placed near the high address, so there is necessarily limited space, yet we are allowed to update, add, or remove values from this array. What does that look like? - As we said, envp, as an argument to 'main' must be on the stack. - The first element of this array, envp[0] points to an adddress above the stack -- CBB18 in this case. At this location, we find a 'char *', another pointer, - which points to the actual array of characters containing "FOO=bar", and which is found at "CC080". - How does that compare to our 'extern char **environ'? That variable is found in the bss segment - but the first element here also points at "CBB18", just like envp[0]. - Similarly, the last element in the array is found at the same address for both envp and environ. --- Ok, now let's see what happens when we use setenv(3) to change the value of a variable already in our environment. Let's change FOO to "a longer value". - After we call setenv(3), envp[0] still points at B8198 - as does environ[0]. - But the 'char *' at _that_ location now points to a different place. In particular, it looks like setenv(3) dynamically allocated some memory for the new string "a longer value", and so updated the pointer at B8198 to point to that. How did it do that? --- Well, to dynamically allocate a buffer, we use the malloc(3) function. This library function will try to allocate the given amount of memory and return a pointer to the beginning of the region it reserved. This memory will be uninitialized; if you want to have it be initialized to all zeros, use calloc(3). If you need to change the amount of memory you need, you can use realloc(3) to grow or shrink the region in memory. If it can make the change without having to shuffle data around, you may get back the same pointer you handed it, but it will still have updated how many bytes are reserved. And of course, as with all resources, once you are done using it, you should free(3) the pointer. This is similar to how we always close(2) a filedescriptor in the same scope in which we opened it; just like that, you should always free(3) any pointers you've allocated within the same scope, as otherwise you run the risk of a memory leak. --- Ok, before we get back to setenv(3), let's take a brief detour and observe how malloc(3) and friends play out in the wild: Here, on the left, is a small problem that first allocates some memory, the reallocates it to larger and smaller areas. In the middle, we see the output from the program. - The first call to malloc(3) - reserves some data - here on the heap. --- The next call to realloc ends up reserving a smaller region below the one we had initially reserved. --- Likewise, the reallocation of the larger region happens here. --- And the final reallocation to a smaller region again reserves yet a different region of memory. In this case, each of our reallocations yielded a new pointer, even when we shrank the amount of memory we reserved, meaning that you cannot make any assumptions about the pointer you get back. --- Ok, back to our environment: We saw that setenv(3) reserved some space for the new value via malloc(3). Next, let's see what happens when we _add_ a new variable to the environment: - After our call to putenv(3), envp[0] still points to the same address as before... - ...but environ[0] now is in a different place -- it's on the heap now! The pointer at this new address, however, still points to the same address that we had previously gotten back after we had called setenv(3). So why did environ[0] move? Our environment atop the stack is only so big - adding a new variable to the environment might not have fit at the top of the stack, and so in order to add a new variable, putenv(3) had to first copy the _entire_ env to a new place! That is, putenv(3) allocated new memory for the whole env array on the heap, then copied all the elements of environ -- all the pointers to the values -- over, and only then could append the new value. Which of course explains why our new value, environ[18], points to this address on the heap - ...and _that_ address here points to the location of "another variable", which is found in the data segment of the program as it was included in the program as a fixed string. So let's note that here envp and environ diverged: after putenv(3), environ was updated, but envp was not. Which makes some sense, since envp is really just a variable local to 'main', while 'environ' is a global variable. --- Ok, now, what's left? unsetenv(3). Let's see what happens when we call that function. We call 'unsetenv("FOO")', so the array shrinks by one element, and what used to be index 18 is now index 17, but still found at the same address as before we called unsetenv(3). However, unsetenv(3) removed the very first element of the array in this case, environ[0], so that all elements shifted down by one. What used to be environ[1] is now environ[0]. The address of environ[0] itself doesn't change, but what it contains, the pointer to "ENV=/home/jschauma/.shrc", is updated. And _that_ value still remains all the way at the top in the original environment, so envrion[0] can now point there. envp, meanwhile, remains unchanged. --- Alright, time to recap: - The process environment consists of an array of strings as KEY=VAR pairs. This is by convention, although you can of course put anything in here. - The initial environment is set up at the top of the process space by the startup routine and pointed to by the 'extern char **environ'. - The startup routine may further pass the current environment as a third argument into 'main'. - We saw how the elements of the array may be moved around as we add new variables, update or remove existing ones. Sometimes this involves moving them into malloc'd areas, but it's important to note that manipulation of the environment should only happen via the library functions and the 'extern char **environ', not the 'envp' passed into the function. - Finally, while a lot of tools rely on the environment or use it in conventional ways, as a defensive programmer you must always verify the sanity of the contents of any variables you plan on using. The user may be able to change them and if you're not careful, you can easily end up in undefined or at least unexpected behavior. - Ok, that about wraps things up. But before we go, here's a link for another exercise for you: think about how the environment is updated, and what might happen if a user were to add hundreds, thousands or tens of thousands of new environment variables. Or what might happen if a user adds a single variable that is thousands of characters long. Play around with this and see if you can identify the limitations and side-effects of this. Good luck, and have fun! Cheers!