Applefritter - Security

Overview of Encryption II

Thu, 19 Jan 2006 22:04:49 -0800

In last weeks article, we discussed symmetrical key encryption, or encryption where the same key is used to encrypt and decrypt the content. The obvious drawback of this is that if the key is made public then it becomes useless, as anybody can read your messages. Because of this, a trusted courier or initial contact is required in order to set the key. For two people wanting to communicate without ever meeting each other, this is impractical. So how can we generate a shared key without the two parties ever meeting? One method of this is Diffie-Hellman-Merkle key exchange. Diffie-Hellman-Merkle key exchange is a method where two parties can arrive at a common numerical key without ever transmitting the key in plaintext, or in a format that can be cracked without brute force. To start with, both parties agree on two numbers. These numbers are plaintext and are assumed to be readable by anyone. "P" must be a prime number. The larger the prime the better. In our example we'll be using "83" just to make it simple. "G" is a common base number. This can be any integer. In our example we'll use 4. Let's say Bob and Alice want to establish a shared key. Bob's never met Alice, all he has is her address. He can send her a letter proposing encrypted communication, and in it send the values of "G" and "P". In our case, "4" and "83". Once these are established, Alice and Bob both choose any integer they want. We'll say that Alice chooses 2 and Bob chooses 5. These are labeled "A" and "B" respectively. Alice performs this equation on her side: G ^ A mod P or, once we substitute in values, 4^2 mod 83. This results in the value 16. Alice packages this into a letter and sends "16" to Bob. Bob performs: G ^ B mod P or, 4^5 mod 83. His result is "28". He writes Alice a letter with the number 28 enclosed. Now Bob and Alice both have a copy of the others number, but they need to find the same number. This is achieved by Alice using the following equation: (G ^ B mod P) ^ A mod P Alice doesn't have "B", as it was never sent plaintext to her. She does however have the answer to G ^ B mod P, which is 28. She substitutes in the values she has and finds the equation is now: (28)^2 mod 83 This results in the number "37". This is the final key. Bob receives Alices envelope and must execute the equation: (G ^ A mod P) ^ B mod P Alice has given him G ^ A mod P without ever revealing A. He substitutes in his values and gets: (16)^5 mod 83 This resolves out to the value "37". Both now have a shared key. Any eavesdropper cannot establish the key without knowing either A or B, neither of which have been sent in the plaintext. In this way, with sufficiently large values, we can negotiate a common key, secure except for the possibility of an eavesdropper using brute force to break their encryption.

Overview of Encryption I

Thu, 19 Jan 2006 21:17:26 -0800

Codes Codes and ciphers are used as a way to make sure that only the intended recipient is able to comprehend the meaning of a message sent via an untrusted route. While codes will translate a phrase into something completely unrelated, a cipher actually is a permutation of the original data. A code would take a phrase like this: We attack at dawn and transform it into something like this: Cheese is special. Because the translated message is completely unrelated to the original, it's only possible to find out what the original message was by having a code book. The code book is a specific set of phrases that may be used and their transformed versions. Using codes has several downsides:

Both sides must have a full copy of the code book
For each trusted sender, the recipient must keep another code book
Both sides must have previously communicated with each other
Only the information which was predicted to have to be sent can be sent.

The upside of using a code book is that without capturing a copy, anybody attempting to intercept the message is completely unable to decode it reliably. In other words, as long as the book is protected, a code cannot be translated without correlating messages to events. Substitution Ciphers A cipher operates on the actual letters in the message to allow for any message you may want to send. Without anticipating a message, you can't send it with code. You can send any message with a cipher. You are also only required to keep a copy of the ciphers key, or method of decryption, in order to read the contents. Simple ciphers, like substitution ciphers work like this: a b c d e f g h i j k l m n o p q r s t u v w x y z n o p q r s t u v w x y z a b c d e f g h i j k l m The plaintext is lined up letter by letter with a different character. "A" is transformed into "N", "B" is transformed into "O", and so on. A substitution cipher would take: We attack at dawn and transform it into something like this: Jr nggnpx ng qnja. The example above is done with the "ROT13" cipher, or by displacing the alphabet 13 characters. "ROT13" is short for "ROTate 13". Substitution ciphers have a couple of downsides.

Both sides must know the key
For each cipher used, both sides must know the key
Both sides must have previously communicated with each other
Ciphers are breakable by trying various permutations of letters matching to characters in the message.

One Time Pads A one time pad is a cipher that cannot be broken. It can only be used once, as the name implies. Basically both sides have an agreed upon random key which is used to cipher the plaintext. In order to use a one time pad, the pad (a random key) must be of a longer length than the message to be transmitted. The pad and the plaintext message are then combined using one of various mathematical techniques. To give an example, let's say that we have our plaintext message: We attack at dawn We also need a one time pad. Both parties must have a copy of this key. Perhaps they sent it by trusted courier or met personally. Most likely they have a large number of one-time pads and just use the next one in sequence for the next message. Let's say that particular key is: oxjelijqmfdsosdthsalqnsi Notice how the pad is longer than the message to encrypt. This means that there is no pattern in the encoding, provided that the key is truly random. I got this from mashing my keyboard while reading something else, then removing the non-alphabetic characters. We'll use a simple method to combined the two lines. First let's write the alphabet, and assign each letter a number. A B C D E F G H I J K L M N O P Q R S T U V W X Y Z 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 Now, we'll translate our plaintext message into numbers. W e a t t a c k a t d a w n 23 5 1 20 20 1 3 11 1 20 4 1 23 14 And translate our pad into numbers as well. o x j e l i j q m f d s o s d t h s a l q n s i 15 24 10 5 12 9 10 17 13 6 4 19 15 19 4 20 8 19 1 12 17 14 19 9 We'll combine the first number (23, standing for the letter "W") with the first number of the pad (15, standing for the letter "o"). 23 + 15 = 38 Now, 38 doesn't line up to a number on our list, so we'll subtract 26 from it to make it. 38 - 26 = 12 12 lines up to the letter L in our numbered alphabet. So the first character in our encrypted message is "L" Going through it like that, we end up with this line: LD KYFJMB NZ HTLG Notice that because we used a random line to combine with our plaintext line, characters that are the same in the plaintext are not the same in the encoded result. This means that using a one time pad to encode does not allow statistical analysis of the encoded text to break it. Now to decrypt the text we do the opposite process. Take the encrypted text and translate it to numbers. L D K Y F J M B N Z H T L G 12 4 11 25 6 10 13 2 14 26 8 20 12 7 We take our key (which we translated to numbers earlier in this article) and this time we subtract the pad values from the encrypted text. The first character is "L", which equates to 12. The first character of the pad is "O" which equates to 15. 12 - 15 = -3 If the result is negative, we add 26 to get it within the range of the alphabet. -3 + 26 = 23 Therefore the first character is "W", which equates to "23" on our alphabet numbers list. When we proceed with this, we come out with the plaintext again. 23 5 1 20 20 1 3 11 1 20 4 1 23 14 W e a t t a c k a t d a w n Since the pad could be any sequence of letters, by using another pad on the encrypted text, we could get any text out. And because the pad is truly random, there is no way to statistical way to know if the message is "We attack at dawn" or "No chimps in line". It should be noted that when a computer generates a random number, it is generally generating a pseudo-random number, which can render a one time pad vulnerable to being broken. One source of true randomness commonly used is radioactive decay. So why aren't these used more often? They're unbreakable as long as the pad is truly random and each pad is only used once to prevent analysis. For most uses of encryption, we don't require it to be unbreakable. We only require it to take longer than the lifespan of the information to break. If it takes a hundred years to get your credit card details then it's secure enough, because the card expires in four years. The Problem With Shared Keys All of the methods of encryption mentioned above have one huge problem with them. They require a trusted contact between the two parties at some point. This decreases the ease of use and increases the cost of use hugely. This is unsuitable for things like email, chat, and online shopping. Would you buy that book from Amazon.com if you had to first go to an office of theirs and establish a secret key. In our next article, we'll examine the ways that two parties can establish a key for secure encryption without needing a trusted initial connection.

Data Mining 101: Finding Subversives with Amazon Wishlists

Sat, 07 Jan 2006 18:20:38 -0800

Vast deposits of personal information sit in databases across the internet. Terms used in phone conversations have become the grounds for federal investigation. Reputable organizations like the Catholic Worker, Greenpeace, and the Vegan Community Project, have come under scrutiny by FBI "counterterrorism" agents.

"Data mining" of all that information and communication is at the heart of the furor over the recent disclosure of government snooping. "U.S. President George W. Bush and his aides have said his executive order allowing eavesdropping without warrants was limited to monitoring international phone and e-mail communications linked to people with connections to al-Qaeda. What has not been acknowledged, according to the Times, is that NSA technicians combed large amounts of phone and Internet traffic seeking patterns pointing to terrorism suspects.

"Some officials described the program as a large data mining operation, the Times said, and described it as much larger than the White House has acknowledged." (Reuters)

Combining a data mining operation with the Patriot Act's power to access information makes it all too easy for the federal government to violate the Constitution's prohibition against unreasonable search. Ars Technica has an article, The new technology at the root of the NSA wiretap scandal, that describes the ease with which widespread wiretapping can now be implemented. It quotes Philip Zimmermann, the creator of the PGP encryption software:

"A year after the CALEA [Communications Assistance for Law Enforcement Act] passed [in 1994], the FBI disclosed plans to require the phone companies to build into their infrastructure the capacity to simultaneously wiretap 1 percent of all phone calls in all major U.S. cities. This would represent more than a thousandfold increase over previous levels in the number of phones that could be wiretapped. In previous years, there were only about a thousand court-ordered wiretaps in the United States per year, at the federal, state, and local levels combined. It's hard to see how the government could even employ enough judges to sign enough wiretap orders to wiretap 1 percent of all our phone calls, much less hire enough federal agents to sit and listen to all that traffic in real time. The only plausible way of processing that amount of traffic is a massive Orwellian application of automated voice recognition technology to sift through it all, searching for interesting keywords or searching for a particular speaker's voice. If the government doesn't find the target in the first 1 percent sample, the wiretaps can be shifted over to a different 1 percent until the target is found, or until everyone's phone line has been checked for subversive traffic. The FBI said they need this capacity to plan for the future. This plan sparked such outrage that it was defeated in Congress. But the mere fact that the FBI even asked for these broad powers is revealing of their agenda."

It used to be you had to get a warrant to monitor a person or a group of people. Today, it is increasingly easy to monitor ideas. And then track them back to people. Most of us don't have access to the databases, software, or computing power of the NSA, FBI, and other government agencies. But an individual with access to the internet can still develop a fairly sophisticated profile of hundreds of thousands of U.S. citizens using free and publicly available resources. Here's an example.

There are many websites and databases that could be used for this project, but few things tell you as much about a person as the books he chooses to read. Isn't that why the Patriot Act specifically requires libraries to release information on who's reading what? For this reason, I chose to focus on the information contained in the popular Amazon wishlists.

Amazon wishlists lets anyone bookmark books for later purchase. By default these lists are public and available to anybody who searches by name. If the wishlist creator specifies a shipping address, someone else can even purchase the book on Amazon and have it shipped directly as a gift. The wishlist creator's city and state are made public on the wishlist, but the street address remains private. Amazon's popularity has created a vast database of wishlists. No index of all wishlists is available, but it remains possible to view all wishlists by people of a particular first name. A recent search for people named Mark returned 124,887 publicly viewable wishlists.

For an all inclusive search by name, you could compile a comprehensive list of first names and nicknames from the baby names databases available on the internet. Armed with this list, and by recording the search results for each first name, it is possible for you to retrieve the vast majority of public wishlists on Amazon.

For the purposes of this exercise, only a single name was chosen – a common male name that returned over 260,000 wishlists. I'm not going to divulge what name was actually used. Let's pretend it was "Edgar," in honor of former FBI director J. Edgar Hoover.

Before writing a script to download all the 260,000 "Edgar" wishlists, I confirmed that my actions would not violate Amazon's Conditions of Use. I also checked the robots.txt file which contains a list of directories Amazon requests not be traversed by scripts. User wishlists are not in this list, nor did the actions to be taken violate the conditions of use.

I started by doing a wishlist search for people named "Edgar" and got back a page linking to the wishlists of the first 25 matches. The url looked something like this:

http://www.amazon.com/gp/registry/search.html/?encoding=UTF8&type=wishlist&field-name=edgar&page=1

Two variables extracted from the above url are of particular note:

field-name=edgar
page=1

Changing "edgar" to "george", would generate the first page of matches for people named George. Change '1' to '2' and you'd get matches 26 through 50 instead of 1 through 25.

Using a simple 6-line shell script and the popular wget command line tool, I configured two computers on two different DSL connections to begin downloading all 260,000 wishlists in increments of 25,000. Each group of 25,000 wishlists took about four hours to download, for a total download time of less than one day. Each wishlist is located at an address like this:

http://www.amazon.com/gp/registry/registry.html/?encoding=UTF8&type=wishlist&id=1DBHU3OCV72ZW

1DBHU3OCV72ZW is the wishlist owner's unique Amazon identification number. I made up the one you see here. By directing wget only to download pages at urls similar to this one, and by incrementing the search page from 1 to 10,400, it is possible to download all 260,000 wishlists without user intervention. Using a pair of 5-year-old computers, two home DSL connections, 42 hours of computer time, and 5 man hours, I now had documents describing the reading preferences of 260,000 U.S. citizens.

I downloaded all the files to an external 120 GB Firewire drive in UFS format. The raw data occupied little more than 5 GB. I initially wanted to move all the files into a single directory to facilitate searching, but as the directory contents exceeded 100,000 items, the speed became glacially slow, so I kept the data divided into chunks of 25,000 wishlists.

Next comes the fun part – what books are most dangerous? So many to choose from. Here's a sample of the list I made. Feel free to make up your own list if you decide to try some data mining. Send it to the FBI. I'm sure they'll appreciate your help in fighting terrorism.

On Liberty by Stuart Mill. First sentence: "The subject of this essay is not the so-called 'liberty of the will', so unfortunately opposed to the misnamed doctrine of philosophical necessity; but civil, or social liberty: the nature and limits of the power which can be legitimately exercised by society over the individual." What more do you need?
Slaughterhouse-Five by Kurt Vonnegut. The classic anti-war novel.
Fahrenheit 451 by Ray Bradbury. Dystopian.
Brave New World by Aldous Huxley. More dystopian.
1984 by George Orwell. Most dystopian.
Critical Thinking by Alec Fisher. Can't have any of that.
Build Your Own Laser, Phaser, Ion Ray Gun and Other Working Space Age Projects by Robert Iannini. Obviously.
Apple I Replica Creation by Tom Owad. Building your own computer should be illegal. (ok, it's also here because I wrote it.)
The Catholic Worker Movement: Intellectual And Spiritual Origins by Mark & Louise Zwick.

Keywords

Michael Moore. The fringe left.
Rush Limbaugh. The fringe right.
Ralph Nader.
Greenpeace. Because frankly, we all know there's only one sort of person who would want a "Greenpeace: Standing Up for the Earth" 2006 Calendar.
Torah.
Quran & Koran. Like the Catholic Worker and Greenpeace, the American-Arab Anti-Discrimination Committee has also been the subject of FBI investigations.
Bible. Sure, a lot of books use "Bible" in the title, but I cast a wide net. What harm are a few false positives?

My Amazon seller ID is attached to these links. If I get any interesting statistics on how many copies of On Liberty, etc., are sold as a result of this article, I'll post them in a follow-up. If I get a call from the FBI, I'll let you know that, too.

To search for specific books, I used ISBN numbers, for the rest, keywords. All the search terms were saved to terms.txt, one term per line, for use with grep:

ls -1 | xargs grep -HiFof /Volumes/UFS/terms.txt > /Volumes/UFS/matches.txt

This command searches all wishlists in the current directory for the terms in terms.txt, then saves the results to matches.txt. Results are stored one per line, in the format:

filename:keyword

Now that I have a list of which keywords appear in which wishlists, I can sort them. I created a new folder "results" and within it created subfolders for each search term. The TCL script below creates links (similar to aliases or shortcuts) for each matched file, and stores the links within the new subdirectories:

#!/usr/bin/tclsh

				

				set fdgrep [open "/Volumes/UFS/matches.txt" "r"]

				

					while {![eof $fdgrep]} {

				   gets $fdgrep line

				   set mylist [split $line :]

				  if {[llength $mylist] > 1} {

				     lappend mylist [string toupper [lindex $mylist 1]]

				    if {![file exists "/Volumes/UFS/results/[lindex $mylist 2]/[lindex $mylist 0]"]} {

				      exec ln /Volumes/UFS/wishlists/[lindex $mylist 0] "/Volumes/UFS/results/[lindex $mylist 2]/[lindex $mylist 0]"

				     }

				   }

					}

Now, for example, the folder called "Greenpeace" contains every wishlist with that term. Another folder named "Rush Limbaugh" contains the wishlists of all the those interested in reading Rush.

On an aside, if you want to delete all the files beginning with the word "search" in a 25,000-file directory, the correct line is:

find . -name 'search*' -print0 | xargs -0 rm

This line deletes all the files:

find . -print0 -name 'search*' | xargs -0 rm

Good thing I had backups.

There's also a bug, in grep 2.5.1 that corrupts output when grep is run with both the -i and -o flags. Version 2.5.1-1, available through the Fink project, fixes this problem.

One curiousity revealed by this project is that there are quite a few people who show up for multiple books. Reading On Liberty and Build Your Own Laser, Phaser, Ion Ray Gun and Other Working Space Age Projects? We really should have a special list for you.

Here are the books, along with the numbers of people interested in reading each:

Book	# of people interested
On Liberty	7
Slaughterhouse-Five	82
Fahrenheit 451	63
Brave New World	1
1984	76
Critical Thinking	7
Build Your Own Laser	2
Apple I Replica Creation	4
The Catholic Worker Movement	1
Rebuilding Labor	2
Michael Moore	232
Rush Limbaugh	42
Ralph Nader	74
Greenpeace	5
Torah	42
Quran & Koran	74
Bible	3,771

The first match for "Bible," ironically, was a wishlist containing The Cannabis Grow Bible: The Definitive Guide to Growing Marijuana for Recreational and Medical Use. Right person. Wrong list. Another match was for The Linux Bible: GNU Testament. With Nader, I foolishly searched for last name alone. Thus, there are quite a few hits for The Lemonader along with the correct results.

If some results look suspiciously low, it's probably because in many cases I searched for a specific ISBN while the book is available in multiple formats. Only the first page of each user's wishlist was downloaded. Books are always added to the front of the wishlist which pushes older titles off the first page, so there is also a slight bias in favor of newer books.

It is possible for users to associate a shipping address with their wishlists, so that others can order them gifts. Though the full address is hidden, city and state remain visible. I already have first and last name. With this information, I can do a Yahoo People Search to obtain an exact street address and phone number. Viewing the wishlists that contained Apple I Replica Creation, I found that all four provided the user's city and state. Of these four, one was a common name that produced multiple hits in his town, two were unlisted (although one of them was in the Intelius database which I opted not to pay for), and the final individual was present on Yahoo People. So I sent him a signed copy and thanked him for his interest.

Thanks to Google Maps (and many similar services) a street address is all we need to get a satellite image of a person's home. Tempted as I was to provide satellite images of the homes of the search subjects, it just seemed a bit extreme even for this article. Instead, I opted only to pinpoint the centers of the towns in which they live. So at least you'll know that there's somebody in your community reading Critical Thinking or some other dangerous text.

City and state were extracted using a regular expression to create a file for each book containing the locations of its readers. Locations were stored one per line, in this format:

Sunnyvale:CA Salt Lake City:Utah Reston:Virginia South Hadley:MA Nevada City:CA Walnut Creek:CA Eagle Nest:NM Memphis:TN North Hollywood:CA Seattle:WA …

Using the free Ontok Geocoder service, I was able to quickly convert city and state to latitude and longitude coordinates. Ontok uses the public domain TIGER/Line data available from the U.S. Census Bureau to perform its conversion. It took less than an hour to convert all locations from city and state to longitude and latitude:

-122.035011, 37.369011 -111.903656, 40.696415 -77.341591, 38.968300 -72.574860, 42.259102 -121.013496, 39.262192 -122.063980, 37.906521 -105.263031, 36.555302 -90.045448, 35.148762 -118.377838, 34.173100 -122.329430, 47.605701 …

Google has released their Maps API, so a map of these locations can be embedded in this article. The API is simple. Plotting each point requires only three lines of code:

var point = new GPoint(-122.035011, 37.369011); var marker = new GMarker(point); map.addOverlay(marker);

This plots all of the locations on a satellite image of the United States that can be zoomed in to house level. Here are a few interactive samples:

Readers of 1984.

Readers of the Torah.

Temporarily removed due to high volume.

You.

The map pinpointing you (your local ISP, actually) requires a good bit of on-the-fly processing, so if the server is exceptionally busy it may not load correctly.

In the future, I may make more sophisticated maps using additional data. Maybe a map that includes all the books in the 260,000 wishlists? Simply searching for any book would present a map of the United States showing the locations of all the people interested in reading it.

All the tools used in this project are standard and free. The services, likewise, are all free. The technical skills required to implement this project are well within the abilities of anybody who has done any programming. The network connection used to download these files was a standard home DSL connection. The computer that processed the data was a 1.5 GHz PowerBook G4. The operating system is Mac OS X 10.4, though everything could have been done just as easily with Linux (and probably with Windows). Not a penny was spent in the writing of this article, just 30 hours of time.

This is what's possible with publicly available information, but imagine if one had access to Amazon's entire database - which still contains every sale dating back to 1999 by the way. Under Section 251 of the Patriot Act, the FBI can require Amazon to turn over its records, without probable cause, for an "authorized investigation . . . to protect against international terrorism or clandestine intelligence activities." Amazon is forbidden to disclose that they have turned over any records, so that you would never know that the government is keeping records of your book purchases. And obviously it is quite simple to crossreference this info with data available in other databases.

On a final note, the FBI is now hiring computer scientists to implement a project that sounds very similar to what I just did:

"Currently, the FBI is strengthening systems engineering in order to tie new systems together architecturally and ensure that standards for custom and packaged applications are enforced, and it needs engineers to accomplish this goal, the agency said.

"The FBI is also focusing on data warehousing as well as federated search technology, which allows a single search query to be deployed across a number of databases, regardless of whether those databases belong to the same protocol or platform.

"'Warehousing has been very successful, yet enterprise extraction, translation and loading processes must be fine-tuned,” the FBI said. “Data engineers are needed to model legacy databases for federated search and participate in legacy transition planning.'"(Computerworld)

This article is the first in a weekly series that will deal with security on the internet and practical steps you can take to protect your privacy. Much thanks goes to Robert Warwick for his help with this project and particularly for writing several of the scripts. Thanks also to Nancy Trump for editing, Michael Fincham for brainstorming, Dr. Bob for bandwidth, and digital.forest for hosting Applefritter. Article submissions are welcome. If you'd like to contact me, please do so via email.

A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z
1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26

o	x	j	e	l	i	j	q	m	f	d	s	o	s	d	t	h	s	a	l	q	n	s	i
15	24	10	5	12	9	10	17	13	6	4	19	15	19	4	20	8	19	1	12	17	14	19	9

A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z
1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26

o	x	j	e	l	i	j	q	m	f	d	s	o	s	d	t	h	s	a	l	q	n	s	i
15	24	10	5	12	9	10	17	13	6	4	19	15	19	4	20	8	19	1	12	17	14	19	9

A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V	W	X	Y	Z
1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26

o	x	j	e	l	i	j	q	m	f	d	s	o	s	d	t	h	s	a	l	q	n	s	i
15	24	10	5	12	9	10	17	13	6	4	19	15	19	4	20	8	19	1	12	17	14	19	9