Archive for the ‘Unix’ Category

Unix: Securing a Web site with the proper file permissions

Sunday, August 29th, 2010

O.K. so you have Apache all set up and serving to the files all the files in your /www folder (or wherever). The question is, what would be the appropriate file permissions. This entry assumes you already know how Unix file permissions work and how to change them).

Setting the permissions is a three-part process. One part would be for the directories, and the second part for the files themselves. The third part is to deal with the exceptions.

With directories, the common wisdom seems to be to use “rwx–x–x”, or “711″ numerically speaking. This means all visitors can access documents within the directories, but can’t view or write to the directories themselves (if, for some reason, you wish to expose the contents of the directories, use “rwxr-xr-x” –or “755″).

In order to recursively change all the directories in your Web folder, you would use this command, from the root directory for your Web site:

find . -type d -exec chmod 711 {} \;

Thanks to the Movable Tripe for providing the above command.

For extra-added protection, you may want to place an index.html file of some sort in each directory. Then, if for some reason the directory permissions get changed to where outside folks can read them, when someone just enters the directory name in the browser (i.e. “http://www.site.com/DirectoryName”), then what will be returned will be the contents of the index.html file, rather than a listing of the directory.

Individual files, or Web pages should have a permission setting of “rwxr–r–” (“744″), which gives the owner full read, write and execute privileges, though others only read permissions. The command from the home directory to change all file permissions, but not directories, would be:

find . -type f -exec chmod 744 {} \;

Thanks again, Movable Triple

Finally, there are exceptions you should consider, depending on what advanced functionality you have on your site. First, there is PHP to think about: While PHP files work with the above permission sets, if you write to a ext file on the site, write permission needs to be added to those files.

Secondly, there is blog software, which in most cases requires residency in your Web directory structure, and needs write and execute permissions in certain places. Wikis also require some write permissions in selected folders.

Finally, please keep in mind that this post does not take into consideration issues of who owns the files (the file “owner”) or groups. That is a topic for another post.–Joab Jackson

Unix: A (somewhat) easy trick to understanding file permission octals

Wednesday, May 5th, 2010

In Unix, file permissions are represented in a set of nine letters (with an additional single letter beforehand as a prefix to signify if the file is a directory or not).

-rwxwr-r–

This cluster can be broken up into three groups (after the initial directory indicator), scanning left to right: A set of three permissions for the owner of the file, a set of three for the group (a group of other users that the user is part of) and a set of three for everybody (or “others“).

The letters used to represent permissions are “r,” “w,” and “x.” For each of the three groups, they are always in the same order: rwx. The “r” permits the reading of the file, The “w” permits writing to the file, that is to say the ability to make changes to the file, and the “x” permits the execution, or running the file (assuming it is an executable file or script).

If no permission is granted for that action then a dash (“-”) is inserted into its place, instead of the appropriate letter.

So for instance:

-rwxwr-r–

…may be broken down into three sections, one for the owner (“rwx”), one for the group(“rw-”) and one for the everyone (“r–”).

In this case, the owner has full read/write/execute permissions (“rwx”), while the group has read and write permissions but no execute permission (“rw-”) and others get only read permission (“r–”).

These file permissions are all bundled in one line, and revealed at the command line, with the “ls -l” command for listing the file attributes of all the files in a directory.

But that is not what I am here to talk to you about. I’m here to discuss….

Numerical Representation

File permissions can be represented not only with rwx’s but also in octals, or a set of three numbers in Base-8 (that is to say a number system that uses only 0 through 7). You can specify changes using octals in the command to change permissions, chmode.

These octal permissions will be three digits. For instance, the “-rwxwr-r–” above, octally speaking, will be “764.”

From left to right, the first digit represents the permissions for user, the second one is for the group and the third one is for others.

I will show you an easy way to derive this number.

The trick is to convert each set of rwx file permissions into a three-digit binary number. Then, all 3 sets can be converted into a single three-digit octal number.

The trick is understanding how binary numbering works. Binary is a Base-2 system. A digit is either 0 or a 1, and all numbers can be represented this way, given enough binary positions. You just keep carrying over, just like in Base-10.

For octal numbering systems, we’ll need only the first 8 numbers (in our base-10 system) in binary. They are:

0 is in binary 000
1 is in binary 001
2 is in binary 010
3 is in binary 011
4 is in binary 100
5 is in binary 101
6 is in binary 110
7 is in binary 111

If you memorize the formula for binary numbers, you can derive these quite easily. Reading right to left, each position of the three binary set represents a different number in the Base-10 numeric system. The first space represents a “1,” the second a “2,” and the third a “4″ (in essence, the values double for every move to the left).

So, to read the binary string, you can basically add up all the values that each 1 represents, as determined by the column that 1 is in. In other words, just look to see if there is a 1 in each column. If there is a 1 in a column, you add the number that that column represents into the total. So for instance, “111″ in binary would represent “7″ because it would be 4 + 2 + 1.

Now, what is interesting about the file permissions is that, thanks to the use of octals, the rwx permission clusters (for the user, group and other), line up exactly with the three digit binary representations.

To build a binary representation of a set of permissions, just look to see where permissions are granted, then, keeping the same 3 column format, place a 1 where each permission is granted. If there is a dash, put a zero as a placeholder:

So, from our example:

“rxw” = 111

“rw-” = 110

“r–” = 100

And, taking these numbers, in the same order as the original file permissions, you build the octal (using the binary-to-octal conversion above):

111 = 7
110 = 6
100 = 4

or, 764!

Source material taken from this book:


…as well as a class I’m taking on Unix. All mistakes are my own, however.–Joab Jackson

Unix: Getting started with vi

Sunday, April 4th, 2010

Created by Bill Joy in 1976, vi is a text editor for the Unix/Linux command line. At first glance, it may seem crude by today’s standards for text editors, but it is useful for working in remote command-line sessions.

To open vi at the command line, simple type vi. If you want to open a specific file with vi, type vi and then the filename.

One thing to keep in mind about vi is that it operates in three modes. You must be aware of what mode you are in at any given time, because each reacts differently to what you type in. The three modes:

Command mode is the default mode. When you first start vi, you are in command mode. You can not enter text. Here you are entering commands. Most keystrokes have a command associated with them.

Input mode is where you actually enter text. the easiest way to get into input mode from command mode is to type the letter “i.” Then you can start typing. (“a” will also work). To get out of insert mode back into command mode, hit the escape key.

Ex mode is used for file handling duties, as well as performing substitution tasks. It is kicked off by typing a “:” from the command mode.

For instance , if you want to save a file, you’d hit escape type in “:w [filename]” If you want to quit, type in “:q” If you haven’t saved your file since making any changes however, it won’t let you quit, unless you put an “!” at the end of the command, “:q!”

You can also combine the commands for writing and quitting, i.e. “:wq”

vi can be frustrating to use for beginners; it really is designed to be lightening fast for those who have memorized many of its myriad commands.

While you will have to figure out which commands are worth memorizing for yourself, here are a few that I myself have found handy:

(all of these are executed from the command line, unless otherwise noted):

) and ( : Jump ahead one sentence or jump back one sentence, respectively.

:[Number]: This will allow you to jump ahead by the number of lines you designate. For instance. “:4″ will jump the cursor ahead 4 lines. Using a negative number will jump back by the number you designate.

ctrl-f, ctrl-b, and ctrl-u, ctrl-d: Jump a screen (24 lines) forward, back, up or down, respectively.

o and O: will move from command mode to insert mode, but insert a new blank line. This is also handy for adding a new line at the end of the document.

dd: delete a line. (Note: This is also the first step of a cut and paste operation. See below).

dw: delete a word.

p and P: This means to paste, as in cut and paste. When you delete something with dd or dw, it goes into the buffer. This command retrieves what is in the buffer.

yy: The command allows for copying and pasting, without the cutting of copy. Typing yy copies the line that the cursor is on.

u undoes the last command (though there seems to be no undo for the edit mode).

/ and ? are search operators. Type them in and then the text you are searching for. / looks for the next instance after the cursor, ? looks for the first instance before the cursor.


Taken from this book:


…as well as a class I’m taking on Unix. All mistakes are my own, however.–Joab Jackson

Unix : Redirection basics, part 1

Saturday, April 3rd, 2010

Note: This entry does not discuss Unix pipes. That will be part 2.

One of the powers of the Unix command line is the ability to redirect input and output of either end of the command (for most commands, anyway).

By default, Unix assumes that the default input will come from the keyboard, and the default output would go to the display. So, to view all the files in a directory, you type “ls” at the command line and the program returns to the display of all the files in a directory.

But you can also direct the output of a program to another source, such as to a text file. You can also specify a new source of input.

This is done using the “>” and the “<” characters.

For instance, say you want to get a list of files in a directory, but instead of having them appear on the screen, you want to put them in a new file, called ListOfInfo.txt. then you’d type:

$ls > ListOfInfo.txt

And if you wanted to add more information to this file, you could append the info with “>>”, i.e.:

$ps -aux>> ListOfInfo.txt

(Otherwise, with just a single “>” Unix will just overwrite the contents of an existing file).

Just as “>” directs the output, using “<” will direct input. For instance…

$wc < ListOfInfo.txt

…will give you the word count of the ListOfInfo.txt file.

You can mix and match these commands. For instance…

$wc < ListOfInfo.txt > WCResults.txt

Keep in mind that not all Unix programs accept redirection, of either input or output. A command such as “mkdir” can’t accept input or redirect its work elsewhere.

* * *

When running a shell, Unix keeps three different streams, or files, for input/output purposes. Each gets a file descriptor number (More on that later). They are:

Standard Input: The file that captures the input, usually from the keyboard (file descriptor # 0).

Standard Output: The file that captures the output, which is usually sen to the terminal display (It has a File descriptor # 1).

Standard Error: This file captures the error messages from the shell or the running program (It has a file descriptor # 2).

With this in mind, “>” really means “1>” and “<” is shorthand for “0<”.

All this means you can redirect standard input output and error messages. For instance, say you want to capture an error message in a text file. You can’t do that with the standard redirect. A wc on a nonexistent file reirected to an output file will not send the error message to the file. Instead, you can type:

$wc phonyfile.txt 2> ErrorFile.txt

Note, you can also group these redirections for a single stream. “1>$2″ sends standard output to the standard error file, and “2>$1″ sends the error output to the standard output.


Taken from this book:


…as well as a class I’m taking on Unix. All mistakes are my own, however.–Joab Jackson

Unix: Converting files between DOS and Unix

Sunday, February 28th, 2010

Recently I found that, after a uploading file from a Windows computer to a Linux one, and opened the file from the command line, Ubuntu would notify me that it was converting it from the DOS format.

Even if it was a standard text file (.txt) filled with ASCII characters, it still needed converting.

Why? Aren’t text files the same across different operating systems? Evidently not.

Unix handles end-of-line signifiers differently than Windows/DOS does, according to Sumitabha Das’s book “Your Unix”.

Specifically, DOS uses two different sets of characters, “\r” (for Carriage Return [CR], or simply “enter”) and “\n” (for Line Feed [LF]) to signify the end of a line.

Unix only uses one, namely LF

These markers can both be seen by examining text files with Octal Dump.

Ubuntu anyway seems to handle DOS text files easily in day to day operation. Nonetheless, most variants of Unix/Linux have a set of utilities to convert files from Windows/DOS into Unix, and back again. They are called dos2unix and unix2dos, respectively.


Taken from this book:


…as well as a class I’m taking on Unix. All mistakes are my own, however.–Joab Jackson

Unix: Decoding binary files with Octal Dump

Monday, February 22nd, 2010

In many cases with Unix/Linux, if you want to view a file, using the cat command works just fine. The phrase “cat samplescript.txt”, will reveal, at the command line, the content of that file.

Cat won’t work for binary files, because binary files contain non-printing characters (Or non-ASCII characters). Run a cat on a binary program, such as sed, will only get you a screen full of gibberish, and may even destroy the terminal session itself.

(Storing programs as binary files is more efficient than storing them in ASCII, largely because binary programs use all eight bits in a byte [up to 256 possible combinations], whereas ASCII only uses seven [128 combinations] leaving the last bit to sign the byte).

What Octal Dump (od from the command line) does is display the contents of a binary file, including an execution files, as sets of octals.

As the name suggests, the octal numbering system is a numbering system in base eight. When used with the “-bc” option, he od program renders each byte of the program in octal.

For instance, rendering this command from the command line in the /bin directory of binary files:

od -bc sed

will return a row of six digit octals, preceded by a seven digit number that is the offset, or position, of the first byte in the line. Below each octal is a its conversion into ASCI characters, if the resulting decimal conversion falls between decimal 33 and 127.

As an aside, to convert from octal to decimal yourself, simply multiply each digit of the octal number by a successive power of eight, going from right to left. So, if the octal is 114, then you would calculate (1* [8^2] + 1 * [8^1] + 4 * [8 ^ 0]), which would equal (64 + 8 + 4), which would equal 76


Taken from this book:


…as well as a class I’m taking on Unix. All mistakes are my own, however.–Joab Jackson

Unix: Indexing files with inode

Tuesday, February 16th, 2010

In Unix, an inode is a data structure that holds information about a file, or set of data blocks. You can think of it as an index, or a collection of metadata about a file. It contains info such as the owner, the permissions, the date created and last modified, as well as the location of the data blocks that contain the information.It is kept on a disk in a separate location from the data blocks themselves.

“When users search for or access a file, the UNIX system searches through the inode table for the correct inode number. When the inode number is found, the command in question can access the inode and make the appropriate changes if applicable,” according to the online paper about inodes posted by IBM.

Each time a user creates a file, a corresponding inode is created. It is possible to run out of inode numbers. Typically, however, a disk will run out of space first before it runs out of inode numbers, according to one instructional site. Although typically, the number of inodes is set by the operating system, they can be set during the set up process of the file system.

By using numerical inode numbers as identifiers, the OS can have multiple file names, in different directories, point to the same file (Called hard linking). inodes are also handy during file system maintenance or recovery operations, such as fsck. fsck checks for lost inodes, or inodes with no pointers, and attempts to repair them.

One can use the “df” command to check the remaining percentage of inodes left on a system. For Ubuntu Linux, the command is “df -i.” To find the inode numbers of all the files in a directory, type “ls -i”

–Joab Jackson

Security: Encryption and Decryption Across Multiple Computers (& Users)

Sunday, July 26th, 2009

Say, you want to encrypt or decrypt your material on other computers.Or, you’d like others users to encrypt materuial to send to you, that only you could unlock. In either case, success in this endeavor could come from the use of the Public Key!!

I’m assuming that you already know how to encrypt and decrypt files in Windows and/or Linux. If not, please study the afore-linked pages.

For this post, I’ll just go over the general methods and leave the specifics to you. I’ll be using the free, open-source cross-platform software program Gnu Privacy Guard (GnuPG).

First thing is that you would need to “export” your key. This means the program generates a public key that other computers (including those run by other people) can use to encrypt their data that they want to send to you.

In GnuPG, this is done at the command-line, thusly:

#gpg -ao [nameoffileyouwantpublickeyin] [UID] –export

The “-o” option tells the program to send the output to a file, rather than just to the screen. You type in the name of a new file that will contain your public key. The -a option tells the program to copy the results into the file using ASCI, rather in a binary format, which could be hard to read.

Say, you want to read that document from another computer. With the same Linux or Windows setup, you download the public key file from where it was generated (or , if from another person, from their public key file). Then, on your computer, you would import a key, thusly:

#gpg –import [nameoffileholdingkey]

Security: Encrypting Files in Ubuntu

Sunday, July 26th, 2009

In a previous post, I explained how to encrypt files on a Windows system, using Gnu Privacy Guard (GnuPG). This entry will fill in the details for how I used GnuPG from the Linux command-line. I won’t recap the general procedures of encrypting and decrypting–it essentially works the same as in Windows.

Here is the How-to on GnuPG, if’n you get lost below.

The Ubuntu server comes with the GnuPG program installed. To check if you have a copy running, run the Advancing Packaging Tool (apt) on the command-line:

#apt-get install gnupg

If the version you have is the current one, it will let you know. If not it will (presumably) update it. (for those wishing to install by hand, go here).

O.K., with GnuPG up and running, the next step is to generate a key. Do this with the following command:

#gpg – -gen-key

Follow the instructions. The defaults are safe bets if you’re not sure which options to take. The passphrase you enter is YOUR PRIVATE KEY–do NOT share with others.

To find out what keys you already have, use this command:

#gpg – -list-keys

If there is more than one key, you can tell yours by the comments you added in. When you get a list of keys, each key will have its public key number on first line. Its the number on the right of the slash, directly after the number of bits used to encrypt (1024/2048, etc.). This is the Unique Number (UID). The next line is the secret key number.

To encrypt, you use this command:

#gpg -e [FileToEncrypt]

GPG will ask for the “recipient.” For our purposes, it is asking for the UID of the encryption key you wish to use.

To decrypt a file, you would do this:

gpg -d [Name of file to decrypt]

You will be asked for a password. The program will just copy the contents onto the screen. To stream the contents into a file, use this command:

gpg -do [Name of the resulting file] [Name of file to decrypt]

–Joab Jackson

SQL: Fetching & Parsing with SELECT

Wednesday, May 20th, 2009

The SELECT statement is used in SQL to select data from the database.

Some background first: SQL stands for structured query language. It is the language used for working with a relational database.

SQL commands can be split into two groups. One is for working with the data within the database (called the The Data Manipulation Language, or DML). The other works designing the tables, as well as defining the relationship among them (The Data Definition Language, or DDL).

All of these commands can be used from the command line prompt, once you’ve gained admittance to the Relational Database Management System (that is the MySQL software itself, or whatever RDMS you are using).

SELECT Falls under DML. It is the basic command for pulling data from the database, either directly from the command line, or as part of a Java or PHP connector.

The basic format for SELECT is:

SELECT [column(s)] FROM [table]

(Note for all these blog pages, you fill your own values in the brackets “[ ]“).

So, if you wanted to see all the info in the database “test,” you would write:

SELECT * from test;

(“*” is a wildcard operator. MySQL requires a “;” at the end of each statement, not shown here. Multiple columns can be selected to show, separated by a comma.)

If you just wanted to see one column of data, say the column of numbers called “measure” you would write:

SELECT measure FROM test;

So SELECT is pretty easy, yes? But it is also rather coarse-grained, returning a tree when what you want is a branch. You need to do some more whittlin’!

You build a more precise query from SELECT using additional qualifiers.

The main way of doing this is through the WHERE keyword, which come after the FROM statement that specifies the table. To further qualify you can also append, in various mixes, the BETWEEN, LIKE, IN, AND or OR keywords, along with selected values to filter by.

Finally, you can order the results by ORDER BY and filter them by DISTINCT, niether of which require WHERE, but could be used in conjunction with WHERE.

What follows are the details:

* * *

With WHERE you can pull only those records from a table that meet some “specific criteria,” as the W3C puts it. WHERE is part of a SELECT statement.

SELECT [column] FROM [table] WHERE [column]=[value you seek]

i.e.,

SELECT measure FROM test WHERE measure=0;

In this example, I used the “=” But other conditionals include…

<> Not equal
> Greater than
< Less than
>= Greater than or equal
<= Less than or equal

Now, for some other keywords:

* * *

BETWEEN: BETWEEN returns a range of data specified between two values.

The format for this is:

SELECT [column(s)] FROM [table] WHERE [column]
BETWEEN [lower value] AND [upper value]

The values can be numbers, date or even text! Text is ranked, ascending or descending, by alphabetical order. Here is an example:

SELECT * FROM Bike WHERE Date
BETWEEN 20090501 AND 20090515

…Will return all the data in all the columns in the table Bike that have a date between May 1 and May 15 2009.

* * *

IN: IN specifies specific entries. When you are looking for values within known entries, then IN is what you’d use.

SELECT [column] FROM [table] WHERE [column] IN ([value(s)])

Commas separate multiple values in the IN set. And, as always if the values are text, they should be in single quotes. Example:

SELECT * FROM Bike WHERE RideTime IN [30, 60]

..Returns all the rows in the Bike table where the RideTime column equals 30 or 60.

* * *

LIKE and NOT LIKE: LIKE is a pattern-matcher. You can structure a query with the elements that you know are in a value that you are looking for, with wild card elements both before and after the known-nugget of info.

The format:

SELECT [column(s)] FROM [table] WHERE [column] LIKE [pattern]

For these operations the percentage sign, %, is the wildcard operator. In a hypothetical database, “Turkey%” will return “TurkeyShoot” and “TurkeyBreast” and “%Turkey” will return “unTurkey” and “%Turkey%” will return all three i.e.

NOT LIKE is the same but returns all the results that DON’T match the criteria you select.

SELECT * FROM BirdWords WHERE Fowl LIKE ‘%Turkey%’

NOTE: Single quotes are used also for fetching numerical responses, dig?

* * *

DISTINCT: DISTINCT is actually a precursor to WHERE–It comes before WHERE in the SQL statement and, in fact, can be used without WHERE. It picks out all of the original values in a column, eliminating any duplicates.

SELECT DISTINCT [column] FROM [Table]

* * *

AND/OR An AND or OR statement may be placed after WHERE for further distinction:

SELECT [column(s)] FROM table WHERE [column]=[value1] AND [column]=[value2]

SELECT [column(s)] FROM table WHERE [column]=[value1] OR [column]=[value2]

SELECT [column(s)] FROM [table] WHERE [column]=[value1] AND ([column]=[value2] OR [column]=[value3])

Note, operators other than the “=” could be used. See above. In the final example AND and OR can be combined, with a parenthesis to clarify order of evaluation with the machine. Play around with these for greater nuances, i.e.

SELECT * from Run WHERE Time=60 OR Time=50 AND (Date=20090101 OR Date=20090201);

…Will return all those entries with a date of Jan 1 2009 or Feb 1 2009 in which the Run Time was 50 or 60.

* * *

ORDER BY The ORDER BY specifies how the results of a query should be returned. They results can be returned either in a numerically ascending (or alphabetical), or descending (reverse alphabetical) for whatever column you choose. Default is ascending.

Again, ORDER BY can be done without WHERE. It is placed at the end of the SQL statement:

SELECT [column(s)] FROM [table] ORDER BY [column] [ASC or DESC]

Example:

SELECT * FROM Run ORDER BY Time DESC

…Will return all the entries from the Run table ranked by time, longest time first.

Taken from the W3C School’s tutorial on SQL, as well as from SQLCourse, a helpful site I found on the Web.

–Joab Jackson