A UNIX operating system consists of a kernel and some system programs. There also some application programs for doing work. The kernel is the heart of the operating system. It keeps track of files on the disk, starts programs and multiplexes the processor and other hardware between them to provide multitasking, assigns memory and other resources to various processes, receives packets from and sends packets to the network, and so on. The kernel does very little by itself, but it provides tools with which all services can be built. It also prevents anyone from accessing the hardware directly, forcing everyone to use the tools it provides. This way the kernel can control who gets to do what and can provide some protection for users from each other. The tools provided by the kernel are used via system calls; see manual page section 2 for more information on these.
The system programs use the tools provided by the kernel to implement the various services required from an operating system. System programs, and all other programs, run `on top of the kernel', in what is called the user mode. The difference between system and application programs is one of intent: applications are intended for getting useful things done (or for playing, if it happens to be a game), whereas system programs are needed to get the system working. A word processor is an application; telnet is a system program. The difference is often somewhat blurry, however, and is important only to compulsive categorizers.
An operating system can also contain compilers and their corresponding libraries (GCC and the C library in particular under Linux), although not all programming languages need be part of the operating system. Documentation, and sometimes even games, can also be part of it. Traditionally, the oeprating system has been defined by the contents of the installation tape or disks; with Linux it is not as clear since the stupid thing is spread all over the FTP sites of the world.