Advanced Programming in the UNIX Environment

CS631 - APUE


Course Outline

In this course, students will learn to develop complex system-level software in the C programming language while gaining an intimate understanding of the Unix operating system (and all OS that belong to this family, such as Linux, the BSDs, and even Mac OS X) and its programming environment.

Topics covered will include the user/kernel interface, fundamental concepts of Unix, user authentication, basic and advanced I/O, fileystems, signals, process relationships, and interprocess communication. Fundamental concepts of software development and maintenance on Unix systems (development and debugging tools such as "make" and "gdb") will also be covered.

Students are expected to have a good working knowledge of the C programming language, have written non-trivial programs before, and to be able to competently use a Unix system with a command-line shell interface. All coursework will be done exclusively on a Unix system from the command-line. This is not an introduction to using Unix!

If you email me to waive the prerequisites, please provide information about how you meet the requirements listed above in bold. (This has the added benefit of showing that you've actually read at least this far on this page.)

This class overlaps significantly with CS392 ``Systems Programming'' -- if you have taken this class, please talk to me in person before trying to register for CS631.


Time, Date & Place:

Interactive/synchronous: Mondays, 18:15 - 20:45 Eastern, via Zoom
Online on-demand: anytime via YouTube video lectures

Instructor:

Jan Schaumann [jschauma@stevens.edu]

When emailing me, please use your @stevens.edu address. I will likely not even see your email if it is sent from gmail or any other non-stevens address.

Resources:

Content and discussions:

Assignments and meta-information:

Related links and documents:

Tool Tips:


Source Code

You will be writing a lot of code in this class. You may also find the need to read a lot of code not written by you as well as the manual pages accompanying the sources.

Our primary reference platform being NetBSD, you can install the full sources into your VM via e.g., this script.

To browse or fetch sources for other Open Source Unix variations, please see these links:


Textbooks:

The textbook used in this class is:

The following books are recommended purely for your own personal reference. They're not used in the class as a text, but are related and very useful books to have:

  • ``The C Programming Language'' -- important: make sure you get the 2nd Edition covering ANSI C.
    by Brian W. Kernighan and Dennis M. Ritchie.
    Prentice Hall, Inc., 1988.
    ISBN 0-13-110362-8 (paperback), 0-13-110370-9 (hardback).
  • ``The Practice of Programming''
    by Brian W. Kernighan and Rob Pike.
    Addison-Wesley, Inc., 1999.
    ISBN 0-201-61586-X.

Systems Used

All software development will be done on a NetBSD system. It is your responsibility to gain access to such a system prior to the start of the class. Please see this page for more information. Instructions on how to install NetBSD in a VirtualBox VM are provided to you here.

All grading will be done on a NetBSD 9.0 system. While you may choose to develop on your own personal host, you should make sure that your code compiles and runs flawlessly on this OS version.

You may also find it useful to get yourself an account on the Stevens linux-lab systems to be able to compare another system.


Grading:

There will most likely be:
  • course participation and class notes: 50 points
  • 2 homework assignment, worth 25 points each
  • 1 midterm project, worth 100 points
  • 1 group project, worth 200 points
  • 1 final assignment, worth 100 points
  • no make-up assignments
  • no extra credit
  • no curve
Letter grades will be given as follows:
  • 90% - 100% of total available points => A
  • 80% - 90% of total available points => B
  • 70% - 80% of total available points => C
  • 0 - 70% of total available points => F
Within each letter grade, there are +/- grades given at the discretion of the instructor. (Exception: there is no A+)

Plagiarism, Cheating and other ways to get an F

This really should not be necessary, but just to preempt any complains that I did not make myself clear:
Students are responsible for their own work. It is unethical (and in some cases illegal) to present as one's work the ideas, words or representations of another without the proper indication of the source. Therefore, it is the student's responsibility to give credit for any quotation, idea or data (such as statistical data or source code) borrowed from an outside source.

Failure to do so constitutes plagiarism, may imply copyright infringement and license violations and is viewed as cheating in this class.
Any incidents are reported to the Dean of Graduate Academics, as per the Graduate Student Code of Academic Conduct.

Homework Assignments:

Assignments will be posted as the semester progresses.

The following exercises are not graded, but are recommended to be completed before or soon after the given lecture to help you better understand the concepts dicussed.


Midterm Project:

Implement the "ls(1)" command as decribed in the manual page provided to you. See the full midterm project description for details.


Group Project:

Implement the "sws(1)" command as decribed in the manual page provided to you. See the full group project description for details.


Syllabus:

Homework assignments, slides, and other material listed below are from the previous semester. We will update the content as the semester progresses.
Date Topic Reading Links
2020-08-31 Introduction, UNIX history, UNIX Programming Basics Stevens: Chapters 1, 2
Unix history and basic features
Week 01, Segment 1: Introduction
[slides] [video lecture] [transcript]
 
Week 01, Segment 2: UNIX History
[slides] [video lecture] [transcript]
 
Week 01, Segment 3: UNIX Basics
[slides] [video lecture] [transcript]
 
simple-cat.c
simple-ls.c
simple-shell.c
simple-shell2.c
welcome.c

2020-09-07 Labor Day Holiday
2020-09-14 File I/O, File Sharing Stevens: Chapter 3 Week 02, Segment 1: File Descriptors
[slides] [video lecture] [transcript]
 
Week 02, Segment 2: open(2)/close(2)
[slides] [video lecture] [transcript]
 
Week 02, Segment 3: read(2)/write(2)/lseek(2)
[slides] [video lecture] [transcript]
 
Week 02, Segment 4: File Sharing
[slides] [video lecture] [transcript]
 
argv.c
fds.c
hole.c
lseek.c
openex.c
openmax.c
redir.c
rwex.c
sync-cat.c

HW1: bbcp(1)
2020-09-21 Files and Directories Stevens: Chapter 4 Week 03, Segment 1: stat(2) intro
[slides] [video lecture] [transcript]
 
Week 03, Segment 2: UIDs intro
[slides] [video lecture] [transcript]
 
Week 03, Segment 3: struct stat st_mode
[slides] [video lecture] [transcript]
 
Week 03, Segment 4: chmod(2) and chown(2)
[slides] [video lecture] [transcript]
 
Week 03, Segment 5: umask(2)
[slides] [video lecture] [transcript]
 
 
access.c
chmod.c
chown.c
myuids.c
setuid.c
size.c
simple-ls.c
simple-ls-stat.c
umask.c
Midterm Project Assignment
2020-09-28 Filesystems, System Data Files, Time & Date Stevens: Chapter 4, 6
File Systems and Storage Models: 4.4.1, 4.5 - 4.7
Week 04, Segment 1: The Unix Filesystem
[slides] [video lecture] [transcript]
 
Week 04, Segment 2: Links
[slides] [video lecture] [transcript]
 
Week 04, Segment 3: Directories
[slides] [video lecture] [transcript]
 
Week 04, Segment 4: Directory Size
[slides] [video lecture] [transcript]
 
Week 04, Segment 5: /etc/passwd
[slides] [video lecture] [transcript]
 
Week 04, Segment 6: getpwuid(2) and /etc/groups
[slides] [video lecture] [transcript]
 
Week 04, Segment 7: atime, mtime, ctime
[slides] [video lecture] [transcript]
 
Week 04, Segment 8: time(3) is an illusion
[slides] [video lecture] [transcript]
 
 
cd.c
getpw.c
groups.c
wait-unlink.c
sizeof.c
time.c
2020-10-05 UNIX tools: cc(1), make(1), revision control, diff(1), patch(1), gdb(1) CVS Documentation
Mini FAQ about the misc libc/gcc crt files
Debugging with GDB
Guide to Faster, Less Frustrating Debugging
gdb Tutorial
Git
Lecture Slides

compile chain examples
make examples
gdb examples

2020-10-12 Fall Recess
2020-10-13 Process Environment, Process Control Stevens: Chapters 7, 8
Smashing The Stack For Fun And Profit
Linux x86 Program Start Up
stdarg And The Case Of The Forgotten Registers
Note: Tuesday Class

Week 06, Segment 1: Memory Layout of a Process
[slides] [video lecture] [transcript]
 
Week 06, Segment 2: Program Startup
[slides] [video lecture] [transcript]
 
Week 06, Segment 3: Program Termination
[slides] [video lecture] [transcript]
 
Week 06, Segment 4: The Environment
[slides] [video lecture] [transcript]
 
Week 06, Segment 5: Process Limits and Identifiers
[slides] [video lecture] [transcript]
 
Week 06, Segment 6: Process Control
[slides] [video lecture] [transcript]
 
 
testcp.sh
const.c
entry.c
exit-handlers.c
forkflush.c
forkseek.c
hw.c
malloc.c
memory-layout0.c
memory-layout1.c
memory-layout2.c
memory-layout3.c
memory-layout4.c
memory-layout5.c
memory-layout6.c
memory-layout7.c
memory-layout8.c
sum.c
zombies.c

NetBSD crt0-common.c
2020-10-19 Process Groups, Sessions, Signals Stevens: Chapter 9, 10
POSIX Terminal Interface Description
Shichao's Notes
FreeBSD Process Management (from "The Design and Implementation of the FreeBSD Operating System")
Week 07, Segment 1: The Login Process
[slides] [video lecture] [transcript]
 
Week 07, Segment 2: Process Groups and Sessions
[slides] [video lecture] [transcript]
 
Week 07, Segment 3: Job Control
[slides] [video lecture] [transcript]
 
Week 07, Segment 4: Signals
[slides] [video lecture] [transcript]
 
Week 07, Segment 5: Reentrant and Interrupted Functions
[slides] [video lecture] [transcript]
 
 
eintr.c
pending.c
reentrant.c
signals1.c
signals2.c
signals3.c
signals4.c
sigusr.c
2020-10-26 Interprocess Communication I Stevens: Chapter 14
Shared Memory Introduction
Semaphores in Linux
Interprocess communication using POSIX message queues in Linux
Week 08, Segment 1: Interprocess Communications Intro
[slides] [video lecture] [transcript]
 
Week 08, Segment 2: System V IPC
[slides] [video lecture] [transcript]
 
Week 08, Segment 3: Pipes and FIFOs
[slides] [video lecture] [transcript]
 

BSD IPC Tutorial
Beej's Guide to Unix IPC

memory-layout.c
mqrecv.c
mqsend.c
msgrecv.c
pipe1.c
pipe2.c
popen.c
semdemo.c
shmdemo.c

HW#2: implement 'command(3)'
2020-11-02 Interprocess Communication II Stevens: Chapter 14
BSD IPC
How Linux creates sockets and counts them
How Linux allows TCP introspection
The C10K problem
Week 09, Segment 1: socketpair(2)
[slides] [video lecture] [transcript]
 
Week 09, Segment 2: socket(PF_LOCAL, SOCK_DGRAM, 0)
[slides] [video lecture] [transcript]
 
Week 09, Segment 3: socket(PF_INET, SOCK_DGRAM, 0)
[slides] [video lecture] [transcript]
 
Week 09, Segment 4: socket(PF_INET6, SOCK_STREAM, 0)
[slides] [video lecture] [transcript]
 
Week 09, Segment 5: I/O Multiplexing
[slides] [video lecture] [transcript]
 

dualstack-streamread.c
socketpair.c
udgramread.c
udgramsend.c
dgramread.c
dgramsend.c
streamread.c
streamwrite.c
two-sockets.c
two-sockets-select.c
one-socket-select.c
one-socket-select-fork.c

Group Project Assignment
2020-11-09 Dæmon Processes, HTTP / Group Project Discussions   Week 10, Segment 1: Dæmon Processes
[slides] [video lecture] [transcript]
 
Lecture Slides
2020-11-16 Shared Libraries Stevens: Chapter 13
How to read an Executable
Ian Lance Taylor's blog post series on linkers
Linkers and Loaders
How To Write Shared Libraries
The Executable and Linkable Format (ELF)
Week 11, Segment 1: The Executable and Linkable Format
[slides] [video lecture] [transcript]
 
Week 11, Segment 2: Of Linkers and Loaders
[slides] [video lecture] [transcript]
 
Week 11, Segment 3: Shared Libraries
[slides] [video lecture] [transcript]
 
crypt.c
dlopenex.c
evil.c
hello.c
ldtest1.2.c
ldtest1.c
ldtest2.c
main.c
setget.c
libgreet exercise
2020-11-23 Advanced I/O: Nonblocking I/O, Polling, and Record Locking / Encryption in a Nutshell Stevens: Chapter 14 Week 12, Segment 1: syslog(3)
[slides] [video lecture] [transcript]
 
Week 12, Segment 2: Non-blocking I/O
[slides] [video lecture] [transcript]
 
Week 12, Segment 3: Resource Locking
[slides] [video lecture] [transcript]
 
Week 12, Segment 4: Asynchronous and Memory Mapped I/O
[slides] [video lecture] [transcript]
 

Lecture Slides (previous semester, includes section on encryption basics)

slogdemo.c
nonblock.c
flock.c

2020-11-30 Restricting processes / Containers Course Notes
Thread scheduling and related interfaces in NetBSD 5.0
CPU Pinning and CPU Sets
Week 13, Segment 1: POSIX ACLs
[slides] [video lecture] [transcript]
 
Week 13, Segment 2: eUIDs, file flags, mount options, securelevels
[slides] [video lecture] [transcript]
 
Week 13, Segment 3: Restricted shells, Chroots, Jails
[slides] [video lecture] [transcript]
 
Week 13, Segment 4: Process Priorities
[slides] [video lecture] [transcript]
 
Week 13, Segment 5: Processor Affinity and CPU Sets
[slides] [video lecture] [transcript]
 
Week 13, Segment 6: Capabilities, Control Groups, Containers
[slides] [video lecture] [transcript]
 
break-chroot.c
busy-child.c
load.c
priority.c
mkchroot
2020-12-07 Review   Review Kahoot

Final Assignment: sish(1)