Buffer Overflow — A Simple Explanation for Beginners Who Want to Learn Cyber ​​Security

In the past, when I first read the CTF write-up with the word buffer overflow, I felt like I was witnessing magic: someone could change the program so that it executes commands at will. In fact, if you learn it slowly, the concept is as simple as writing your name in a form column that turns out to be too long, then the writing spills over into the next column. The difference is, in the computer world, the "spill" can change the direction of the program.

What is Buffer Overflow?

Buffer overflow occurs when a program writes data into a buffer (an area of ​​temporary memory) that is greater than the capacity allocated to it. Think of the buffer as a 250 ml glass. If you pour 500 ml of water, the remaining water will overflow onto the table, right?

In computer memory, "overflow" means data overwrites the area next to the buffer. If the area next to it contains important instructions — for example a return address that determines where the program will run after the function completes — then an attacker can change the program's execution flow.

Why does this happen?

Most buffer overflow cases come from programming languages ​​such as C and C++. These two languages ​​do not automatically check whether we are writing inside the bounds of an array or variable. They trust the programmer completely. That's why C/C++ is very fast and flexible, but also dangerous if you're not careful.

Simple example: the program asks for name input with a maximum length of 10 characters. But the programmer forgot to add the check. If we send 50 characters, the extra 40 characters can overwrite the variables next to it, including the function return address.

Languages ​​such as Python, Java, Rust, or Go have built-in protections that make buffer overflows much less common. But because much infrastructure software — web servers, Linux kernels, databases — is still written in C, understanding buffer overflows remains important.

Stack vs Heap: Two Different Lands

Buffer overflow can occur in two areas of main memory:

• Stack overflow: Occurs in the stack, an area of memory used for local variables, function parameters, and return addresses. This is most often exploited because the return address is located here, so an attacker can change the program flow easily.
• Heap overflow: Occurs in the heap, a dynamic memory area allocated while the program is running. Even though it doesn't directly change the return address, a heap overflow can overwrite important data structures such as pointers, metadata, or virtual function tables.

For beginners, focusing on Stack Overflow first is enough. It is a classic "entrance" to the world of binary exploitation.

Example of Vulnerable Code

The following is an example of a simple C program that is vulnerable to buffer overflow:

#include <stdio.h>
#include <string.h>

void cek_password() {
    char buffer[16];
    int is_admin = 0;

    printf("Masukkan password: ");
    gets(buffer); // berbahaya, tidak pernah digunakan di produksi

    if (is_admin != 0) {
        printf("Selamat datang, admin!\n");
    } else {
        printf("Akses ditolak.\n");
    }
}

int main() {
    cek_password();
    return 0;
}

Here, the variable buffer only has a space of 16 characters. However the gets() function does not check the length of the input. If we enter 20 characters or more, the excess data will overwrite the is_admin variable. With the right input, we can change the is_admin value and log in as admin without the correct password.

Important note: the gets() function has been removed from the C11 standard because it is too dangerous. But many other functions are still often misused, such as strcpy(), strcat(), sprintf(), scanf(), and so on.

How do Attackers Take Advantage of It?

At a more sophisticated level, attackers don't just change variables. They write shellcode — a series of malicious machine instructions — into memory, then change the return address to point to that shellcode.

The process is more or less like this:

1. Find a program that accepts input from the user.
2. Send very long input containing shellcode + manipulated return address.
3. The program stores input in a buffer that is too small.
4. Data overflows and overwrites the return address.
5. When the function completes, the program "returns" to the shellcode address, not to the original function.
6. Shellcode is executed, for example opening a terminal or downloading malware.

In the modern world, mechanisms such as ASLR (Address Space Layout Randomization), NX bit (Non-Executable Stack), Stack Canary, and DEP (Data Execution Prevention) were created to make these attacks more difficult. But that doesn't mean buffer overflow is dead — a combination of techniques like ROP (Return-Oriented Programming) can still bypass those defenses.

Impact of Buffer Overflow

A buffer overflow attack can have fatal consequences, including:

• Remote Code Execution (RCE): an attacker executes arbitrary code remotely.
• Denial of Service (DoS): the program crashes because memory is incorrectly overwritten.
• Privilege Escalation: normal users get admin access rights.
• Data Corruption: important data is changed without permission.

Some of the biggest exploits in history, such as Code Red (2001) and Slammer (2003), took advantage of buffer overflows. Even modern bugs like Heartbleed are related to memory handling errors, although the mechanism is different.

How to Prevent Buffer Overflow

For developers, here are the most effective preventive measures:

1. Use safe functions: replace strcpy() with strncpy(), strcat() with strncat(), and sprintf() with snprintf(). These functions have length limitations.
2. Avoid gets(): this function is deprecated and very dangerous.
3. Input validation: always check the length and format of user input before processing.
4. Use modern languages: Rust, Go, Java, and Python have safer memory management.
5. Enable compiler protection: flags like -fstack-protector-strong, -D_FORTIFY_SOURCE=2, and -Wformat-security can help.
6. Disable execution on the stack: use NX bit or DEP.
7. Enable ASLR: so that the memory address is difficult for attackers to guess.
8. Audit and fuzzing: use tools like AFL (American Fuzzy Lop) or LibFuzzer to find crashes due to unexpected input.

Next Practice

If you're interested in diving deeper, try training on platforms like OverTheWire, PicoCTF, or MerdekaSiber. They have levels specifically designed to understand buffer overflows in stages. Starting from changing simple variables, then moving up to overwriting return addresses, to ROP chains.

Remember: understanding buffer overflows doesn't mean you have to be an attacker. Most cybersecurity professionals learn it to be able to write more secure code, find bugs before product releases, and understand how exploits work in the field.

Conclusion

Buffer overflow is a classic foundation in the world of cyber security. The concept is simple: write too much data into too little space. But the impact can be huge, from changing program flow to executing malicious code.

For developers, the lesson is clear: never trust user input, always limit data length, and use tools and languages ​​that help manage memory safely. For those new to cyber security, buffer overflow is a great entry point to understanding the relationship between code, memory, and exploits.

If you have experience learning about buffer overflow or have encountered a similar bug, write in the comments column. I love hearing your story.