Yesterday, while reviewing some logs I came across a curious entry in an Apache error log:

[Wed Apr 19 08:51:48.119666 2017] [core:error] [pid 29210] (36)File name
too long: [client 137.226.113.7:40907] AH00036: access to
/YesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForR
esearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongReques
tURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALo
okAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurpos
eWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisI
sAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPu
rposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWe
AreDoingItOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUs
erAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreSca
nningForResearchPurposePleaseHaveALookAtTheUserAgentTHXYesThisIsAReallyL
ongRequestURLbutWeAreDoingItOnPurposeWeAreScanningForResearchPurposePlea
seHaveALookAtTheUserAgentTHXYesThisIsAReallyLongRequestURLbutWeAreDoingI
tOnPurposeWeAreScanningForResearchPurposePleaseHaveALookAtTheUserAgentTH
XYesThisIsAReallyLongRequestURLbutWeAreDoingItOnPurposeWeAreScann failed
(filesystem path '[...]')

Formatted to plain English: Yes, this is a really long request URL but we are doing it on purpose. We are scanning for research purpose. Please have a look at the user agent. Thanks!

What does the user agent for this request have to say?

Here is the access log entry:

137.226.113.7 - - [19/Apr/2017:08:51:48 -0400] "GET [...] HTTP/1.1" 403
1471 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36
Scanning for research (researchscan.comsys.rwth-aachen.de)"

The website referenced in the user agent, researchscan.comsys.rwth-aachen.de, explains that this request is part of a research project at RWTH Aachen University in Germany and 137.226.113.7 is indeed a part of the university's network.

Interestingly …

On Sunday, 19 March around 10AM I received an email with the subject line IMPORTANT: ProtectMyID Surveillance Alert. I was busy at the time so I flagged the message and moved on with my day. I had received these alerts a few times before when applying for credit cards and getting a home loan. While I hadn't done either of those things lately, I still was not particularly alarmed by the subject alone and had many other things on my mind. Going about the busy day, I promptly forgot about the email.

Before starting the work day the next morning, I noticed and remembered the email in my inbox. Why do I even have ProtectMyId? Because my personal data was part of a breach at some point. Which breach? I don't even recall - there have been so many. I opened the email and logged in to find that a new credit account with a provider I had never used had been opened in my name. Surprise! I am a victim of identity theft.

What do I do now?

Having fallen behind a bit on Takeout Inspector, the 12 Years of Gmail series and some other projects, I decided to try to put something very simple together from beginning to end and actually launch it. One of my previous posts, Examining the Remnants of a Small DDoS Attack introduced me to the Python package ipwhois and the alternative WHOIS system RDAP. This eventually led me to a quick and simple project called RDAP Explorer...

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

After taking a look at the chat data in my export, I am finally ready to move on to some of the actual mail! Much of what I will look at here is pretty similar to what I was able to turn up with chat data. I tried to branch out a bit, bringing in a new package to create word clouds, and also refactored some of the Takeout Inspector code to form the beginning of a more "formal" report generating process (instead of just spitting out a single HTML file with only a certain subset of the data). Hopefully I can continue to improve this to a point allowing for easier report generation for any user. Anyway, on to the mail data!

• Top 10 Recipients
• Top 10 Senders
• Thread Durations
• Thread Sizes
• Activity by Day of the Week
• Activity by Hour of the Day
• Label Usage
• Subject Word Cloud

On Sunday (27 November 2016) a small website that I advise on was the victim of a DDoS attack that managed to knock the site offline. I received notice on Monday that the website was not working. I was able to ssh to the web server and quickly found that the database service was stopped. After a brief examination of the database logs (nothing too out of the ordinary), I started the service back up and sure enough the website came back online. As the website runs on Drupal, I logged in to take a peak at the Recent log messages and found hundreds of records of log in attempts from a lot of different IP addresses. User accounts on the website are only used by administrators to update content, so it was clear that the site was hit by a DDoS attack!

After getting things back online, I poked around the various log files to try to get an idea of what happened. The Drupal watchdog logs seemed to indicate that the attack started around 15:22 EST and overloaded the server's memory around 15:42 EST. The Apache server's access logs, however, revealed that the attack started closer …

In my last post, 12 Years of Gmail, Part 4: Chat, I included eight Plotly graphs on a single page. All the graphs worked correctly, but the page was taking almost four seconds to render any content at all and up to 6-8 seconds to load completely without cached elements. By contrast, the landing page of chrxs.net takes less than a second to load with visual content rendering almost immediately. The site is intentionally designed to be light weight and uses very few resources on a standard load. But Plotly graphs require a big (1MB+ uncompressed) JavaScript file in order to load with all the bells and whistles. What can be done to improve this slow load time, particularly when many graphs are on a single page?

Film strip before optimization (webpagetest.org)

The page load film strip above shows almost three whole seconds before any content is rendered. The obvious first step was to move the loading of Plotly's large JavaScript file from the page head (which loads before content is rendered) to the end of the page body, theoretically allowing the page's content to be partially loaded and rendered earlier. However, doing this created a bit of …

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

With the Finishing Touches in place, it's finally time to start looking at some of the data in my Google Takeout Mail export file. What better to start with than the Google Talk (or Google Chat, as I will refer to it) content stored within!

I am starting with Chat because I was surprised to find it all stored in the export file. It makes sense as chat history is accessible from the Chats link in the old Gmail interface (I couldn't find an equivalent in Inbox). My surprise led to curiosity and my curiosity led to obsession with trying to figure how the Chat data is stored and what information each messages contains. It turns out there are quite a few things that can be gleaned from these chat messages -

• Chat vs. Email Usage
• Chat vs. Email Usage (Cumulative)
• Top 10 Chatters
• Chat Clients
• Chat Times
• Chat (vs. Email) Days …

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

After spending last week Bootstrapping things and, somewhat related, working my way around Pelican, today I have tried to tie up loose ends so I can start spending more time thinking about what information I can get from all this data. While the package is far from complete, these "finishing touches" ended up being the three themes of this morning's work -

1. Implementing a settings file
2. Customising Plotly graphs
3. Generating random names

0
1
2
3
4

;settings.cfg
[mail]
anonymize = False
db_file = data/email …

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

Jumping back in to Python has been just as fun as my first experiences with it. After brushing off some of the dust, I have managed to put together a (very) small package that does a couple of basic things with a Google Takeout Mail (mbox) file:

1. Parses and standardizes the format of email addresses;
2. Imports key messages data in to an sqlite database;
3. Produces simple graphs of top recipients and senders.

0
1
2
3
4
5
6
7
8
9

import mailbox
email = mailbox.mbox('/path/to/email.mbox')

# The number of emails in the mbox file.
print len(email.keys())
114407

# The "Delivered-To" header of the first email.
print email …

This post is part of my series, 12 Years of Gmail, taking a look at the data Google has accumulated on me over the past 12 years of using various Google services and documenting the learning experience developing an open source Python project (Takeout Inspector) to analyze that data.

I have been slowly migrating off of a Gmail email address for a couple of months now - I established this domain, selected an email provider, set up SPF, DMARC, etc. and finally created myself a new email address. I updated the address in all of the obvious places, but still found myself using Gmail frequently to keep up. At some point I realized that the only way to finish the migration would be to do something with all the email I had hoarded away in Gmail.

When I made the transition to Gmail (from a mail server in my basement) back in 2004, I found some tool that pulled all my existing email in to Gmail using POP. So, I thought to myself in 2016, I'll just do that again! I fired up Thunderbird, set up Gmail POP access and started downloading. At some point, thousands of emails in, I decided …