License : Copyright Emeric Nasi, some rights reserved
This work is licensed under a Creative Commons Attribution 4.0 International License.

If you are interested in GNU Linux security you might want to learn how to exploit buffers overflow.
Most modern Linux distributions and compilers have built-in functions against stack smashing and heap smashing. That is why it is not that easy to setup a testing environment.
Here are a few steps to do it.

I Install the OS

You should never test security exploits on your own machine. So the first thing we will do is to install a virtual one.
1 - Install VirtualBox
You can find it at http://www.virtualbox.org/wiki/Linux_Downloads, or use your favorite package manager.
2 - Download Debian 5
You can find minimal CD installation here http://www.debian.org/CD/netinst/.
This CD should not exceed 200Mo and the installed system won’t have any protection against buffer overflows.
3 - Create a new virtual machine
Choose the system Linux - Debian
Use minimal settings, you will just use it to test exploits not anything else.
Add your Debian ISO as an optical disk.
Run your virtual machine (VM) and follow default installations.
At the end of the installation, choose a desktop environment or the default environment.

II Install the needed applications

Login as root into your new Debian VM and follow the next steps.
1 - Easy access to your VM
If you use the desktop environment install the VirtualBox add-ons
If you use console only access, install open-ssh :
# apt-get install openssh-server
Then connect to you VM using ssh.
2 - Install a compiler
# apt-get install gcc
# apt-get install make
3 - Install a debugger, decompiler
# apt-get install gdb
3 - Install a text editor
I let you choose the one you prefer, vim or nano if you use console access. gedit, kate, emacs and many more if you use the desktop environment.
4 - Optionally
Install paxtest to verify that your system does not have any protections.
#apt-get install paxtest

III Disable stack randomization

On default Debian 5 system, you can write on the stack, however starting to learn buffer overflow is not easy because of stack randomization feature.
For example, using the gdb debugger you an notice that, at the same breakpoints, the stack pointer %esp is never at the same address.
(gdb) x $esp -> random address
Another way to check if randomization feature is on is to use the kernel customizing tool sysctl:
# sysctl kernel.randomize_va_space
If the value is > 0 then stack randomization is ON.

To temporarily disable this feature we will also use sysctl :
# sysctl -w kernel.randomize_va_space=0

If yo want to permanently disable this feature you need to add the line
"kernel.randomize_va_space = 0" to sysctl.conf (kernel customization boot file).

IV Test stack smashing

Imagine now you want to start with a good old stack smashing.
1 - Create the test folder
# mkdir smash_stack
# cd smash_stack
2 - Create the test program
Just an example :

3 - Compile the test program
# gcc -z execstack -fno-stack-protector victim.c -o victim

4 - Is the vulnerability exploitable?
The victim program copies the first argument into a 256 length buffer. To create a buffer overflow we call victim program with a 258 length argument.
# ./victim $( printf "%0258x" 0 )
-> Segmentation fault

All right, you are now ready to learn how to smash the stack!
You can get back your favorite tutorial/book.