Let’s continue with familiarizing ourselves with how the shell works.

In the previous article, we talked about the ls command and using it to list files in a directory. One question I received about this was: what happens to that list after it’s printed?

An important thing to keep in mind is that most of the commands we are dealing with at this point are not interactive. This means that they do their job and then quit. In the case of ls, the program’s job is to spit out a list of the current files in the directory, and then it’s done: we move on to other commands. I’d liken it to requesting information by mail: you send a letter with your request, and some information is sent back to you. If that information changes in the future, you won’t know that unless you receive another letter informing you of it.

Let’s do an experiment to test this principle. In so doing, we will also learn an important Unix concept: redirection.

Open up your terminal. We’re going to create a directory to play around in.

We make a directory with the mkdir program. This name, although abbreviated, should make sense. The mkdir program needs to know what we want to call our new directory, so we pass it an argument to give it this information. (If you don’t remember what “passing an argument” means, read the previous article again.) You can call your new directory almost anything: however, let’s try to stick to letters and numbers and dashes (including spaces or special characters has some special rules involved that we don’t want to get into just yet).

mkdir testing-ground

The above example would make a directory called “testing-ground”. Where does it make it? Well remember, in the world of the shell, you are always located somewhere. As we mentioned in the last article, most shells will tell you your current location in the prompt (see this picture for a reminder of where your location is displayed). If we run the mkdir command as shown above, the “testing-ground” directory will be created inside our current location. If we are in the location “~/Documents”, the location (or path) of our new directory will be “~/Documents/testing-ground”.

What location do I start in? Upon opening a new terminal, most shells start you in your home directory. This is a special place that belongs to your user: you own it, and you can create and delete things inside it at will. Though Microsoft Windows actually does now have the concept of a home directory, many people either don’t use it, or don’t know they are using it. This is because on a standard Windows system, users and programs don’t think about permissions. There’s nothing stopping you from deleting something important, because you have the permission to do anything you want.

On the other hand, on a Unix system like Linux or OS X, if you try to delete a file that is important to the entire system, you will be denied with a stern message. Serves you right.

The “home directory” is so common in Unix systems, that shells provide a handy abbreviation for it, so that you don’t have to type out its full name: ~. If you look at that picture again, you’ll see my location is “~/Documents”–in other words, I’m located in the Documents directory inside my home folder.

Alright, so we issued the mkdir command. Now we need to go inside of it! To do so, we will use the main method of changing our location: cd. This stands for “change directory,” and indeed that is exactly what it does. The cd program also usually needs to know where we want to go: in this case, we want to go to our newly-made directory, so we pass it an argument with the directory’s name:

cd testing-ground

We are now in the new directory. Let’s see what’s inside it, by doing our familiar ls command:

ls -l

It tells us that the directory is empty; this should make sense, since we only just created it!

Let’s put something inside of it. How are we going to do this? Well, we could copy a file from somewhere else, or we could move a file from somewhere else, or we could create a new file altogether.

Let’s make a new, empty file by doing the following:

>jordanrules

Using a > to make a new file might seem weird. Don’t worry, we’ll explain it in a moment.

Let’s ask our friendly friend ls to list the files in the directory again, and see if it sees our new file:

ls -l

Sure enough, jordanrules is now showing up!

The echo command. Let’s take a moment to play with the echo program. It’s one of the most simple programs: you tell it something, and it tells you that same thing back! Indeed, it’s aptly named, since it’s like talking when there’s an echo: your echo will repeat exactly what you said. Let’s try it:

echo hello

echo echo echo echo

echo RENEW!

Useless? Not at all! This is the most important part of this article. If you understand nothing else from this rambling, understand the following thing:

Commands can be chained together.

Let’s say that one more time, for emphasis:

Commands can be chained together.

We are going to do our first “cool thing” in the Unix shell now, by chaining the echo command, which we just played with.

echo hello >jordanrules

So now we come to what that > sign really means: it’s one way of chaining commands! What we just did, was chained the result or output of the echo command and put it into the “jordanrules” file.

Don’t believe me? Let’s open the file and check!

There are lots of ways to open a file. Let’s use the program less to do it. Type:

less jordanrules

Sure enough, we see that the file now contains the word “hello”. (Press q to get out of the less program)

Let’s do this again, with another phrase:

echo blah blah blah >jordanrules

Now check the file again:

less jordanrules

That’s interesting! Our old word, “hello,” is gone entirely, and the file now only contains “blah blah blah”. What if we didn’t want to replace the file’s contents, but just add to it?

We can use a different chaining operator for that. (The term “operator” comes from math, where operators are things like + and -.) Here’s an example:

echo cheerio >>jordanrules

Now open the file with less again:

less jordanrules

Sure enough, the file contains what it did before, plus our new word. Neat, eh?

I know what you’re thinking. “What if I wanted to put the phrase ‘jordan > you’ in the file?” Obviously, we can’t just type echo jordan > you >jordanrules because the > sign is special! The shell would think we are trying to chain the echo command into a file called “you”. But don’t worry, the solution is easy:

echo 'jordan > you' >jordanrules

See what we did there? We put the phrase in single-quotes. In the shell, anything you enclose in single quotes is treated “literally”–this means you can use special symbols etc. in your phrase, and they won’t be treated specially.

For this reason, it’s a good habit to always put phrases that you give to echo inside of single-quotes, at least for what we are using it for now.

Anyway, feel free to play around with creating different files, putting stuff in them, making new directories, and moving around with cd. We’ll talk more about other commands (and of course, more about chaining) in the next article!

This is how it should be! And as a programmer, it is my job to keep it this way. If we had to know exactly how cars worked to drive them from Point A to Point B, it would take years before we’d make it to nice restaurants in faraway big cities.

Like cars and many other things in life, however, if one wants to gain more knowledge, control, power, and–dare I say–joy out of this magic, digging deeper is essential. I won’t go into all the practical reasons why this is the case; if you are reading this, I will assume that there is more you want to accomplish with computers. Maybe you want to be able to troubleshoot problems yourself when they come up. Maybe you want to make certain tasks automated, or make something easier for yourself or others. Whatever the reason, learning how to manipulate your computer on a deeper level (henceforth referred to as “hacking,” which is what this word really means, and not what Hollywood thinks it means) is a rewarding experience.

So, where to start?

That is the question. Obviously to some extent, this depends on what you want to accomplish. But as that is impossible for me to ascertain for each individual, I will endeavor to introduce some things that should be useful to anyone.

These ramblings will assume you are using a Unix-based system, preferably Linux. This is partially because such systems give you the tools to hack out of the box, and partially because their communities are not only amiable to hackers, but are made up of them. Linux is built under the principle that anyone who wants to explore software deeper (and change it if they want) can (and should!) do so. Mac OS X is a Unix under the hood, so it’s acceptable.

The Shell

Any aspiring hacker should become acquainted with the shell. What is this shell of which I speak? You may know it as “the command line.” It’s called a shell because it provides a nice way to interface with your operating system, much in the same way an eggshell provides a nice way to interface with an egg. The shell is accessed through a program called the terminal.

What is a terminal?

Very simply, a terminal receives input from you, and sometimes tells you something back. In the early days of computers, there would only be one real “computer” at a given establishment. Multiple people interacted with this computer simultaneously through dumb terminals, which were not individual computers themselves, but (essentially) monitors and keyboards–ways to tell something to the computer, and get something back. Likewise, you can open as many terminals into your computer as you want: they don’t affect each other, they just affect your machine.

Nowadays, when you open a terminal program, it automatically starts a shell for you. But let’s talk about how to use both.

Getting friendly with your terminal

Terminals are quite basic programs from the perspective of the user (you). But if you want to use one, you should know where to find it! In Ubuntu, it’s in Applications -> Accessories -> Terminal. On OS X, it’s called Terminal.app, in Applications -> Utilities. If you can’t find it on your system, and Google isn’t helping, let me know.

Once you’ve got it open, you will want to spend a bit of time configuring it to your liking. Very important to me are the colors: if you’re going to be doing powerful hackery things, you should look the part! I prefer amber text on black background, but use whatever you like. Again, exactly how to do this is specific to your system, but your terminal program should have an easily-findable preferences menu somewhere. Also make sure the size and whatnot is to your liking. Note that you will want to make sure to use a monospace (fixed-width) font (which certainly will be the default for your terminal). Since you are only dealing with text and not graphics, it’s important that all the letters be the same width so everything lines up correctly.

Getting friendly with the bash shell

There are several different popular shells for a given system, with varying capabilities that you don’t need to worry about now. Just stick with bash, which is the default shell in OS X and almost all Linux distributions. That is, when you open your terminal, you are now looking at a bash shell.

In the world of the shell, the first thing that’s important to understand is that you are always working in a specific place in your system. That is, you are in your Documents directory, or the root directory, or the directory called /usr/local, or whatever. (Note that we call them “directories,” not “folders,” though they both do the same job of being a place where files are.) This is different from your normal computer-using experience, where you don’t often deal directly with files (except to save or open them), really.

Your current location should be displayed at the prompt, or the actual line that is waiting for your input.

The other important thing to keep in mind is not only where you are, but who you are. In unix-like systems, users have varying levels of privileges–that is to say, not everybody can do everything. This is important for security and privacy: you can’t delete another user’s files, or destroy something important to the operation of the whole system, when you are just a standard user. This is a Good Thing, and you’ll want to shun the ability to do dangerous things until you are sure what you’re doing.

Who you are is also displayed at the prompt, generally by means of the symbol (usually $ or #) that immediately precedes your cursor.

Anatomy of a Prompt. Click to Enlarge

So, what do you do now? Well, it’s a command line, so you enter commands!

What are these commands?

Generally, giving a command is as simple as telling the shell which program you want to run. For example, a very common command is one to list which files are in your current directory. To do this, you just type the name of the program that lists files: ls. Then hit enter. The results of running the program will be displayed. Try it on your terminal!

Of course, often you will want to give the program some information about what you want. With the ls program, for example, we often would prefer the files to be arranged in a nice table, instead of just haphazardly listed like in the example above. To tell a program something, you give it parameters. The parameter that does what we want for ls is -l. You put parameters after the program’s name, separating each one with a space. So typing ls -l gives us:

Output of ls -l. Click to enlarge.

By default, the ls program doesn’t list hidden files, because they are… well, hidden. But of course, sometimes you want to be able to see hidden files! So you pass the parameter -a (which stands for “all,” meaning we want to show ALL files). So we type ls -a and it Enter.

Ruh-roh! It showed all the files, but it’s back to that ugly output again. Looks like we need to combine this new parameter with the previous one. This turns out to be a very common thing to do. We could do this ls -l -a to get what we want. Try it.

But hackers are always looking for speedier/more efficient ways to do things, and so there are often shortcuts. When you have multiple parameters starting with a dash (-), you can scrunch em all together, like so: ls -la. The result is the same.

In the future, we’ll do more complex stuff with the shell, but these basic techniques of using the program name followed by optional parameters are ubiquitous.

What is scripting?

You may ask what the advantage is to typing commands and getting their results as text instead of using graphics. Obviously both have their strong points, but the command method has a major plus: you can script it.

Scripting is the process of doing the work once, and enjoying the benefits multiple times. It is making a list of commands that you can later execute just by typing the name of the script (with optional parameters… sound familiar?)

Scripts are commonly used to automate tasks. For example, you might write a script to backup your important documents to another location. If you’ve ever tried to consistently (daily, for example) do such backups just using your graphical file browser, you know how tedious this can be–eventually, you start backing up less frequently (or not at all) because it’s such a pain in the butt.

Let’s say that to you, “backing up” just means copying your Documents directory into a directory called Backups that’s on a different hard drive. That way, if your main hard drive dies, at least you have copies in another spot. That’s painless to do via the graphical interface, right?

1. Open file browser.
2. Click through folders until you get to Documents.
3. Right click on the folder, select “Copy.”
4. Click through folders on your other hard drive until you get to Backups.
5. Right click in the folder, hit Paste.

How would we do the same thing in the shell?
cp -R /path/to/Documents /path/to/Backup

Alright, you say. That might be faster, but it’s a lot of typing! Well, for one thing, the shell helps you out a lot: you can just start typing a little bit of the word, then hit Tab, and it will complete the word for you. So you can probably just type “D” and then hit Tab, and it will fill in the full word “Documents” automatically.Or, you can just write a script that has the above command in it, name it something like backup, and all you have to do is type “backup” at the command line, and you’re done!

But now let’s say you notice you’re having space issues. Keeping a full second copy of all your stuff means using twice the hard drive space. It’d be nice if you could compress the backups so that they took up less space.

In this scenario, most likely your workflow in the graphical interface has completely changed. It depends on how integrated your archiving program is with your file explorer. For the sake of hyperbole, let’s pretend it’s not integrated at all. Here are the steps for doing it graphically:

1. Open archive program.
2. Go to File -> Add to Archive.
3. Click through folders until you get to Documents.
4. Click OK.
5. Go to File -> Save As.
6. Click through folders until you get to Backups.
7. Think up a name, hit Save.

That last step is important: how are you going to name your backups? Are you going to put today’s date in the name, so that you have easy access to a given day’s backup? I’ve done exactly this before, in the above way listed (or pretty close), and it’s tedious.

Let’s see how we’d change our previous little one-line script (backup) to do the same thing, including the date in the name of the backup.

tar czf /path/to/Backup/`date +%F`.tgz /path/to/Documents

Now when you run backup, your documents are automatically archived and stored in your Backup directory with the name of today’s date. Neat, huh? Once you change the script once, you are done–you still just run backup like you did before. It figures out the date and everything for you.

But now the truly cool thing that really puts you ahead of the graphical way: you can tell your system to simply run backup for you. That’s right, now you don’t have to open the shell, don’t have to type anything–and you never worry about forgetting to do a backup. Telling the system to run your script is quite easy, but I won’t mention it here, as I’ve made my point. Obviously this doesn’t just apply to backups, but for any task that needs doing more than once–and you’d be surprised how many of those there are.

Anyway, hopefully that helps to give you a taste. I’ll gladly answer any emails with questions, and will give a more “hands-on” tutorial next time.