How system calls works

Some examples of system calls include : 

• fork()
• execve()
• open()
• read()
• write() etc 

How system call works 

The following points explain the operation of a system call on x86-32 hardware: 

1. From a programmer point of view, calling a system call is equivalent to calling any normal C function.
2. Inside the Linux kernel, each system call has a number associated with it.
3. The kernel expects all the information like system call number, the arguments to the system call etc to be present in CPU registers.
4. A system call first invokes a C library wrapper function which does all this work of placing the information required to be placed in CPU registers. The system call number is placed in the register %eax.
5. After doing the above work, the same wrapper function invokes a trap instruction 'int 0x80'. This instruction switches the processor mode from user mode to the kernel mode and executes the code kept at location 0x80.  Note : Starting with Linux kernel 2.6 and glibc 2.3.2 a faster mechanism of using the 'sysenter' instruction is followed to switch into the kernel mode.
6. After the above step, now since the processor is in kernel mode, the kernel calls a general trap handler function system_call() to handle this trap.
7.                                                                                                                                                                                                                                                                         Copies all the relevant information kept in the CPU registers onto the kernel stack, Validates the system call number, Uses the system call number to find the corresponding system call service routine from a table(sys_call_table) of service routines maintained by kernel and Calls the system call service routine with the arguments passed from the user space. 
8. The system call service routine that is finally called by the trap handler first checks the validity of the arguments sent to it and then it does the actual work requested by the application.
9. Once the system call service routine is done with its processing, the control is returned back to the trap handler function which returns the flow back to the wrapper function in the user-mode.
10. Based on the return value from the trap handler, the wrapper function sets the 'errno' variable and returns -1 (in most of the cases) to indicate an error.
11. The value of the 'errno' variable is assigned by negating the return value of the trap handler (returned to wrapper function).
12. The application can use the strerror() API with 'errno' to get the error information in form of human readable string.