Schneier on Security

Roger • September 8, 2012 10:45 AM

@Alasdair:
Interesting analysis. I made a few observations myself, which you might find interesting:

1. If this data is supposed to be anonymised, it has been badly botched. There are hundreds of complete names in there, some partial addresses, and thousands of complete email addresses.


2. TThis recognisable data is mainly from the USA, but from all parts. There are also some possibly Spanish and Italian addresses.
   
3. As noted, there are 1,000,001 rows, and no header row. We have to guess the meaning of the fields.


4. Most rows are formed of 4 fields, in which the first consists of 40 hexadecimal characters surrounded by single quotes; the second consists of 64 hexadecimal characters surrounded by single quotes; the third is a free-form field that appears to be what users type in as a welcome text (ranging from default values to addresses where to return it if found); and the 4th is a device type (iPhone, iPad or iPod touch.)


5. The file is not well-formed as a CSV file. The most serious issue is that 1139 lines have only 3 fields when most have 4. The malformed lines are all ones that had a comma embedded in the free-form string. For these rows, what would have otherwise been the 4th field is missing. So, Antisec is not so good at parsing CSV. (Don't worry, Antisec: it's harder than it looks.)


6. 30 lines are formed quite differently to all the rest. In one case, all the fields just contain the string 'disabled'. In the other 29, the fields are rather differently structured. Instead of the first field being the 40 hex chars, it starts with the 64 hex char field. The device type is 3rd, and the 4th field becomes what looks like a version number. Most interesting, though, is the 2nd field.


7. In those 29 odd records, the 2nd field is a short string always starting with the numeral 2. In most cases they are exactly 5 characters long; one is 6 and one is 11. They look a lot like decrypted passwords. There is significant repetition. (This would be consistent with hits from a rainbow table.)


8. There are many repeated values. For the first field (the 160 bit one) there are over 13,000 values that appear more than one, hundreds that appear 3 times, and even one that appears 53 times. A similar (bit not identical) pattern occurs for the second (256 bit) field. The third field is not surprisingly more diverse, but still has immense repetition. Including a surprisingly large number of values that appeared hundreds of times. For example, PdaTX.NET occurs 1,141 times.


9. Searching for the string "student" is interesting. It seems that quite a few of these devices are shared machines on campuses -- including religious colleges.

Roger • September 9, 2012 2:28 AM

I just ran the infamous fully-RFC 2822-compliant regex over it, and counted 582 valid email addresses. Of these, 434 are .com domains, and 48 are other TLDs (including 2 from .gov -- one from DOJ!) Of course the rest are country codes. The top 10 .com subdomains are:

1. gmail 118


2. hotmail 94


3. yahoo 75


4. aol 21


5. me 19


6. msn 9


7. live 7


8. mac 6


9. 163 6


10. naver 4


11. and 61 others.

It seems that 163.com is a popular Chinese portal, and naver.com is the most popular portal in Korea. Which brings us to the top 12 country codes:

1. ru 15


2. it 8


3. ca 7


4. cn 7


5. jp 5


6. nl 4


7. au 4


8. de 4


9. hk 4


10. hu 3


11. us 3


12. br 3


13. and 26 others.

Pretty global, then. By the way, two thirds of the Russian ones come from mail.ru, so I guess that must be a pretty sweet property to own (even though the Russian word for e-mail is "мейл" which is only similar.)

Most of the mailbox parts of these addresses are personal names; a few are business names.